← Behind the Craft
Loading summary
Transcript85 lines
- [00:00]
A
Let the model cook and don't try to push it in certain directions. My agent's running for nine hours straight now. It's crazy. More dev ins are being launched from other dev ins or programmatically than by humans. We're not trying to replace coding, we're trying to build abundance of good software and really multiply an engineer. You want the engineer to bring taste and bring this high level decision making. But agents should be doing everything else and the agent shouldn't be bottlenecking on the engineer's
- [00:31]
B
all right. Hey everyone. I'm really excited to host Jared building in residence at Cognition today. Jared has a lot of background building AI agents. So going to ask him about, you know, how to build a good AI agent, have him demo Devin's latest capabilities and also talk about, you know, the pretty comparative AI coding and knowledge workspace. Welcome Jared.
- [00:49]
A
Thanks for having me.
- [00:50]
B
Peter, why don't we start with some high level questions like how do you think someone should think about your building an agent to today and what is the number one mistake that one companies make when doing this stuff?
- [01:00]
A
I think first maybe I'd want to break it down into like two different types of what building an agent means and primarily I guess the first one is building a user facing agent or something that your users are interacting with, a chatbot or something like that. And the second type is probably where my mind goes of an agent that you use. So I spoke to a lot of DevOps teams recently that are building internal agents for their team to work better, which is like there's similarities and differences. I think when we're talking about an external agent and building things on top of the model, I, I think one of the biggest mistakes and one of the most important things is you just have to keep in mind model progress and you want to be building in a way that lines up with honestly the roadmap of the model labs and the people building models and you want to be building in a way that is not fighting gravity, so to speak. So for example, there was a lot of, I guess how long have LLMs kind of been around now? Like two, three years, four years, something like that. Since the first ChatGPT and things like tool calling, things like reasoning, remember chain of thought, people would kind of prompt their, prompt their agent in a very specific way to get this reasoning and now it's built into the model for things like this. There's a lot of kind of hacks we still need to do to get the model to follow instructions well and my general high Level advice is to build your own roadmap in a way that assumes these hacks are going to be less and less less important and these are not going to be the defensibility of your company. So today we've already reached a point where a lot of teams have gone from large dags of this node connects to this node, that connects to this prompt, that connects to this prompt. And it's, and it actually might be coming back as of like yesterday with this new kind of like dynamic workflow stuff that anthropic is working on. But still the high level principle of let the model, let the model cook and don't try to push it in certain directions. It's just going to make your life easier, it's going to save dev time and you're going to, you're going to need less scaffolding around agents as the models get better. And they're pretty good. Already got it.
- [03:20]
B
So basically like start with like a prompt and some like tools that the agent can use and just kind of iterate from there. Right? Is that kind of.
- [03:26]
A
Yeah. I almost think tool engineering is going to be the new, the new thing, just figuring out which tools to give the model and assume that the model is going to figure it out and be able to try things out. And the prompt in that case and the skills in that case are like a cheat sheet of what tools to use for what. But otherwise the model will just try
- [03:48]
B
everything just to get a little bit tactical here. So let's say the initial prompt, maybe it's just like tools kind of like what you're an agent for customer support or something. And then specifically your advice is not to include a bunch of stuff like think ultra hard or take the step. You know, here's 1, 2, 3 steps, how you should proceed. Like kind of just give us some principles instead of like very explicit instructions. Is that what you think?
- [04:11]
A
So I think everything's case by case here. So it's very hard to give blanket advice. But the, I think the mental model or the heuristic you should have is if it feels like you need to give these very ultra specific prompt instructions today, don't treat that as the, you're your defensibility or your differentiation in the market because the models are going to get better and you're going to need less and less of that. And even today, you see, I was working on a project the other day and on the best models today you just give a very simple one line like there's a weird bug with this login. Flow and it'll go and figure it out. And that was not true a year ago. That was not true probably six or eight months ago. And that means that if your prompt, if your whole company relies on this fancy prompt that says you need to reason and do this, the models are going to catch up.
- [05:09]
B
Yeah, dude, this is a big question, right? What actually is defensible for a company to build an agent themselves? Obviously, if you have proprietary data, that's somewhat defensible. Otherwise I feel like there are a lot of AI wrapper companies that are just prompts. Right. They're just like prompts and skills.
- [05:25]
A
Everything's a prompting skill. Everything boils down to prompts and skills. Right. I, I, it just maybe you're prompting humans, but I think, well, maybe we could use cognition as an example. Yeah, I don't think nobody uses the wrapper term like, oh, cognition is just a wrapper. Because I think it's very clear we built a like differentiation one on the harness itself. The harness is really good. But also through the go to market motion through the FDE and the, I mean, FD is the talk of term of the day. Right. But I, sorry, what's the fd?
- [06:03]
B
Ford for Deployed Engine.
- [06:05]
A
Ford Deployed Engineering. So kind of embedding engineers with customers and stuff like that. But yeah, what is defensibility? I'm somewhat less interested in that question. I think if you build something cool that is actually interesting, good entrepreneurs can figure it out.
- [06:22]
B
Yeah, defensibility is just like shipping and moving fast and reacting to all the latest models.
- [06:27]
A
Defensibility could just be moving fast. Exactly.
- [06:30]
B
Yeah. And it's funny you mentioned for deployed engineers because I feel like, I don't know, I feel like even like Anthropic and OpenAI, they have pretty small product teams. Right. But they have a shit ton of salespeople and for deployed engineers, because the market is actually not AI native builders are sitting on Twitter. The market is like, you know, Accenture or like these companies that have no idea what the hell this stuff even is.
- [06:53]
A
Right.
- [06:54]
B
So you got to train. I'm not sure what you're seeing at cognition, like is cognition. Are a lot of the customers like pretty AI native or like trying to get started trying to figure this stuff out?
- [07:02]
A
Yeah, we live in a very interesting world here because we're on Twitter. I don't know about you, but I'm on Twitter a lot and I and Twitter is the cutting edge, for what it's worth. We're like whatever percentage we are living in the future, we're using all the new models when they come out, we're using all the new techniques. And yeah, there's a lot of world that, there's a lot of the world that's using copilot and, and using tab tab autocomplete with code and, and that's, and that's not even a joke. This is, this is a very true thing and I think where cognition has had a lot of success and I've kind of sat into a lot of these customer meetings too as like, I guess my role is kind of interesting because I get to live in the startup world and also kind of bring that energy to the enterprise world and back and forth and some of the stuff the enterprise world actually has down better than the startup world in terms of security and some stuff like that. But anyway, my point is, yes, these. A lot of the world doesn't understand agent loops and is not AI pilled, so to speak. And it's very. You could look at forward deploy motions and say, oh, that's because the product is not good enough for people to self serve. And I think it's always good to have criticism against like how you can be better at just being more seamless. But I really think what's happening is we had a massive technology innovation on what agents can do and my agents running for nine hours straight now, it's crazy. And what that means is because it's exploded so quickly, there's so many people who like would normally in these other industries take years to become AI pilled, but if you can actually sit down with them and show them. And I remember this from when kind of agent agentic coding or coding agents were first becoming a thing at my last startup when we'd like sit down with every engineer and like watch me do cloud code so you could see how good it is. And I think the same thing's true. If you can you show people the magic and kind of like nobody believes how good these things are.
- [09:08]
B
Yeah. Until you show it to them.
- [09:09]
A
Right, exactly. The thing that makes the forward deploy motion at cognition unique is that it's less about just AI education and more about, hey, let's sit together and use agents to solve this actual problem you have. And like in this process we're actually taking tickets off their board and that, that's how you really prove it to them.
- [09:30]
B
Got it. Got it. This episode is brought to you by Riverside. I've used Riverside for years to record my podcast because it records in 4K resolution each person locally. So the audio and video still comes through clean even if a guest's wi fi gets shaky. But the reason I love it now more than ever is what happens after we stop recording. If I go in here, I can use these AI tools to remove pauses, remove filler words and and just clean up the recording. And I can also edit the transcript directly and it will automatically generate clips with captions ready to publish to YouTube, Spotify and all types of social media platforms all from one place. As a one person creator business, that matters a lot. Riverside is the upgrade your content workflow needs. Try it at creators.riverside.com Peter Yang and use code Peter Yan at checkout to get one month completely free. That's creators.riverside.com Peter Yang now back to our episode before we have you demo like using using Devin, let me ask you this. So you know before cognition you worked at PromptLayer, so you have a lot of knowledge about kind of observing how these agents work. And I think obviously there's kind of evals and stuff like that. I just feel like, okay, my point of view is that when companies think about building agents, they tend to overcomplicate things. They want to build an orchestrator and all this kind of stuff, or they want to build really complicated evals when you haven't even shown this stuff to a real customer yet. So how do you think about the rollout phase of these AI agents once it gets to production? It should be pretty reliable and not go off the rails. But how do you go through the phases of getting there?
- [11:08]
A
Well, there's this classic trope of the AI psychosis when someone just comes to you and they said, I've spent the last month building my personal Claude code inventory system and they have like 13 agents doing different things. And yeah, I think I agree with you. Perfect is always the enemy of complete and with a lot of these things, less is more. With a lot of these things, you don't actually need tons of scaffolding to get an MVP out. And I think the other advantage of kind of living in this AI age, on one hand everybody can build so much quicker, so more polish is expected. But on the other hand, AI users kind of understand that it's a probabilistic technology and it's not sometimes it's going to need a little bit more prodding and users I think are a little bit more forgiving with really novel stuff in AI. So I guess, yeah, my, my take on it is teams should not it is a common failure case that teams get stuck on evals and stuck on building the Perfect testable system because they're coming from this world of deterministic code and they're like, okay, just like in the old world, I needed to have tests for every single thing. I need to do that in my agent. And frankly, you're not going to be able to do that. And some of the best teams, believe it or not, are just shipping wild to prod without eval set. So I would say if. If you're a team that's building like one of your first flagship agent products or something like that, done is better than perfect. And don't get stuck on planning it out and just do it. And you'll be surprised at how good these things are and how easy it is to get to 80%. And now the last mile is the hard part and that's going to take a year and that's like 80%. With agents, you could take an hour and then the last part takes a long time. But a lot of people don't get to the 80% because they want to plan out the whole hundred percent.
- [13:09]
B
Yeah, yeah. I mean, the last mile is basically still like human. Like human craft and taste, you know, like with all the AI stuff I do, like I always spend the last 10% actually looking at it, like looking at AI's output, you know.
- [13:22]
A
Right, and that's the differentiation.
- [13:24]
B
Yeah, that is differentiation. Otherwise, just putting all slop, like all over the place, you know?
- [13:28]
A
Exactly. I have. I'm building. One of the projects I'm working on right now is kind of like an open claw style assistant, meaning I say open class style, meaning it has a heartbeat and it's proactive and building the harness. I've like spent very little time on the evals. I had Devin build evals for me and look at the evals and iterate on its own, but I'm saving the part where I'm actually sitting down and tweaking it until the end when everything else is built in the scaffolding and you're getting the responses and the responses are pretty good. And then when I want it to be really good, that's kind of like the finishing touches is my opinion. And the finishing touches, as we know, take way more time.
- [14:12]
B
And just to real quick question, like, what kind of evals are you building for? Like, just like, does the core run evals? Like, yes, no evals, you know, end to end testing kind of evals?
- [14:23]
A
Yeah, in this case. So without. I haven't released it yet. So without giving away the full spec of it, it's. It's A personal assistant, chatbot, something like that. And it proactively can notify you, but you could also chat with it. And I am the evals I have IT doing right now. I basically just told Devin, make a bunch of examples for data recall, for tool use, for response length. And basically I just had Devin build a sanity tech, or if you're. If you're using GPT a lot, build a lot of smoke tests is the word GPT 5.5 loves. But I had IT build a lot of these sanity checks. So it has basically purely as a tool that Devin could then use to check its own work and then iterate on its own and, like, see how far it could get autonomously.
- [15:21]
B
Okay, got it. Okay. So some of it is maybe just like, like browser use, computer use, kind of like just clicking through the flows,
- [15:27]
A
and, you know, it does that. But in this case, I am evaluating the agent response that it's building. So it's basically a few programmatic endpoints that result in a markdown file, and then Devin just reads the markdown file to see what the old outputs and the new outputs are and makes its own decisions.
- [15:48]
B
Got it. Okay, got it. All right, man. Well, do you want to show us your Devin setup? And maybe we can start with, I guess, what makes Devin actually unique, different from, like, you know, Codex and cloud.
- [16:01]
A
Cloud code. Totally. Happy to. Happy to. So maybe I'll give a brief. Before diving into the share screen. Maybe I'll give a brief overview. So I think there's a few things that make Devin unique. Maybe high level is the easiest way and philosophically is the easiest way to explain it first. So I think the first thing is at Cognition, our main focus is software engineering. So we. I use Devin for a lot of stuff like design and whatnot. But, like, that's not our main focus. Our main focus is how do you write rigorous, real software and how do you improve the software engineering lifecycle? And that means we have a lot of time spent on testing on verified code, on how the agent verifies its own code, on computer use and stuff like that. And I think we really stand out if you are doing hardcore software engineering and you need to write code that doesn't break. And that's why we have so much traction in kind of like Wall street banks and a lot of these Fortune 100 enterprises because their code can't break. And they're doing this real kind of not vibe coding. They're doing coding. And I think that. So that's like. I think everything kind of boils down from there and from there. There's a few assumptions we've made that I think stand out. So we've been building like Devin as a cloud agent from day one, and that means we focus a lot on asynchronous work. So you. I guess there's two types of work, and I think we all probably are starting to experience this synchronous work when you're behind your desk at work and you're doing a single stream. So maybe you're designing the landing page for something and you're going back and forth with it. Where Async work, which is you're working on eight different things and you're, you're managing this agent and then going back and managing this agent. And I, I think this cloud agent paradigm is the future. I think it's. Everyone's starting to agree that it's the future. It's just faster and, and it enables a lot of stuff in Devin that I think is pretty unique, such that when I started it, when I started at Cognition, I spent a lot of time talking with everyone on the eng, like every single engineer, like, how do you use Devin? What's your setup? What's your trick? And one of the best things I heard, and the biggest shift for me, and this is what I'll kind of show you, is having Devin set, spin up a bunch of other child Devin and be the orchestrator of other agents. And the cool thing is every child Devin is its own vm. So it has its own computer. It has its own computer use, it has a display, it's able to test its own code. And then this master agent is able to pull down those other child dev and send messages to them. So I think cloud agents are one way we stand out. Just the maturity of that product, I think the ecosystem as well. So we have Devin Desktop, which is our new kind of newly launched desktop product. We have the Devon cli, we have the Devon Cloud, and they all talk to each other and it's really, I think that's the second big way we stand out. It's a multiplayer tool, it's tool for the team as opposed to just an individual developer's kind of like switchblade and.
- [19:23]
B
Oh, it's like a. It's like multiplayer, as in like actual other human multiplayer, not just a bunch of agents, right?
- [19:28]
A
Yeah, yeah, yeah, yeah. I mean, it's agent agentic multiplayer with human multiplayer. And yeah, it's super useful because you can save kind of macros and knowledge and skills for your whole team. But you can also inter Devin can interact with other people's Devins and you can, you can see theirs and there's a lot of kind of that like shareability enterprise features there and honestly the biggest surface people use Devin on is Slack and, and automations and actually I'll add one third thing and then, and then we'll go into the demo. The third thing you should understand about like why Devin is different and it kind of goes with this async work but we really believe and it's already happened this month I think had a big switch more dev ins are being launched from other dev ins or programmatically than by humans meaning automation. So every time a datadog alert goes on you can pipe it to Devin or Slack just responding to customers. So we believe in the same way that like during the self driving car phase it wasn't not a phase but during the self the beginning of self driving cars people had a lot of GPUs and you wouldn't want the GPU to sit idle because you're losing money. In the same way if I'm a developer I'm going to sleep. I want my agents doing stuff. I don't want the human to be the bottleneck anymore. And that's kind of this whole async cloud ecosystem comes together. Does that make sense?
- [21:01]
B
I think it makes sense if you're on these all you can eat $200 plants. But if you're paying per API, do you think it still makes sense Uber is capping spending of their engineers and stuff?
- [21:13]
A
Yeah, it's a great question. The answer is yes. To be honest I two reasons. One, and this goes back to the beginning of our conversation, you should assume that we're going to be getting. I think models are going to get both more expensive and cheaper but the prices of regular tokens is probably going to go down over time. So it's depreciating. But the second reason I think is more important and this is kind of a. This is actually what really convinced me to join cognition. I really like and Scott talks about this a lot our view that we're not trying to replace coding, we're trying to build abundance of good software and really multiply an engineer. And for that reason you want the engineer to bring taste and bring this like high level decision making. But agents should be doing everything else and the agents shouldn't be bottlenecked on the engineer. So if you are bringing in ROI and bring and solving real problems it's worth higher token spend. But it's not it's not worth it if the agents are doing stupid things and are not worth it. But like, like I think there are a lot of companies, I think most companies in technology, if they could pour more money into engineering and have outsized impact, they would. Yeah.
- [22:38]
B
But yeah, I think the keyword there is good, right? Good, good engineering because you know like I can also generate like a shit ton of slop that like a bunch of apps that are not good. And is it like use of tokens? I'm not sure.
- [22:51]
A
100%. 100%. And as like as a harness that's kind of independent from the model labs. The nice thing is we don't have, we have the, we don't have the incentive to make users spend more money because we don't have our like revenue tied to our next version of GPT or this or that. We just want users to use the best model for the best thing whether it's a cheap or the cheapest too. And it's kind of all trade offs like how good is the model, how expensive is the model? And yeah it's. I think the incentives are aligned. I, but I get. But you're 100% right. You don't want to spend money on slop but if you can spend money on real engineering work, it's worth it 10 out of 10 times.
- [23:36]
B
And like you know we should get to the demo but I feel like inside companies a lot of the real engineering work is not like zero to one fee features. It's like you know, refactoring shit and like you know, trying to improve existing product like really complicated messy code base like that kind of stuff. Right. So. Yeah, yes.
- [23:54]
A
And I think, I think Devin specifically as a harness is really shines on brownfield tasks as opposed to greenfield. So like our biggest users are using us for large scale code migrations or bringing more like coverage test coverage or kind of just these, these developer grunt work you can kind of call it. And the stuff that developers like if
- [24:24]
B
it's not very sexy.
- [24:25]
A
Yeah, yeah, it's not, it's not, not the most like you're not designing the Parallax landing page and it can do that too but it can do this hard boring stuff that is a blocker for your team and there's a lot of. You'd be surprised how many big companies in the country have about 10 people who understand their code base out of thousands. And you. It's because it's in a COBOL or it's in a mainframe or it's in this old technology and you need to modernize it. Yeah.
- [24:58]
B
Cool. All right, dude. Well, should we get to that demo then?
- [25:01]
A
Yeah, let's do it. Let's do it. So this is a project I'm actually working on. I haven't released it yet, but I'll probably. Hopefully we'll see who releases this first. Right. But this is just something super simple. I wanted it for myself. You know, when you do the space on Mac and it gives you the quick look, it. It didn't. Doesn't work with markdown by default. So I just had Devin build a simple project for that. But I think it would be good to illustrate how Devin works. So this is a Devin session. You could think of it as just kind of your cloud agent, and in this case, it built it. We have an actual browser to see what's going on. So let's wake this up because the. This was. I was working on this a bit ago, but you could see what's going on in the actual VM that the computer has. I could see the changes made. And this is kind of this whole agent interface. So now you can see it's awaiting instructions. But I love this test app feature. So Devin test. I waited to click this button so we could do it together. This sometimes takes a little time, but Devin. Devin's now spinning back up the VM and it's going to actually do an integration test and click on stuff and make sure all the links work in this landing page that I'm building. So you can see it's entering test mode and it's clicking. We can kind of see what's going on, what's being clicked. But this is kind of. As you see, I think a lot of people understand what an agent harness is, but this is an agent harness in the cloud that has a browser and has a full computer and has full computer use and has a bunch of other stuff. So does that make sense?
- [26:44]
B
Why don't you pitch to the viewers here? What's so great about being the cloud versus Local? You know, like, totally. Well, first of all, what's so great about it?
- [26:52]
A
You can close your laptop.
- [26:55]
B
Yeah.
- [26:56]
A
If it's in the cloud. You know, this is kind of a. A meme these days that everyone's leaving their laptop open all day. With cloud, you. You can just set it and forget it, and it's running on a different system. But what's even better is if I'm doing something really big, like let's say I want to redesign the landing page. Can you spin up. Let's say 10 child Devin sessions. Each one should create its own redesign. And you can see it's testing it out. Over here is the live view, but. And I can also control it if I want to. But so what I'm asking this, this is kind of a trivial example, but I'm. I'm saying maybe I don't like this design of this landing page. I'm asking it to spin up 10 other VMs or 10 other child Devins. And child Devin is like, you could think of it as a full software engineer. And in this case I can't really. You can imagine I've done projects where I've had 30 different Devins at the same time. You can imagine locally. This is not so fun. My computer probably can't handle this, especially with multiple VMs. But even more so, it's just Async. I can spin it up from my phone. I can spin it up by messaging it on Slack. I've actually talked to startup teams where the CTO told me they do most of their work just kicking off Devin's on the commute to work. And then once they get to work, they pull down the repo, they click around, make sure it's good. Sometimes they just watch the video from Devin and they push it and I got it. I'm very bullish on cloud agents, I will say also, I found myself also doing a hybrid approach where I have a local agent that spins up its cloud agents and hands off sessions to the cloud and then pulls it down and kind of monitors them. So I can start to work synchronously when I want to do that.
- [28:57]
B
Okay, so basically like this whole project is in GitHub somewhere and then you basically spun up a bunch of work trees, like clones to redesign.
- [29:05]
A
Well, in this case it's not even work trees because they're on their own vm. So they're just pulling it down and they're making a new branch and making a new pr. And then the Devin can kind of talk to these sub Devin. But this is a. I think this is a Devin specific paradigm of how to do real AI pilled work. And it's by doing this kind of agent fan out. I've talked to people on our engineering team who are doing kind of like the model training and they tell me they spin up 100 Devins to look for things because agents are much better when they're doing one thing specifically and kind of really focus, just like humans are, to be honest.
- [29:48]
B
So basically the master Devin can actually go back and forth with some of these other Devins to make the landing page good. Right. It's not like they're going to go through this whole checklist, but it can go back and forth.
- [29:56]
A
Yeah, the master Devin, if I say check or we could say make sure each child Devin response with a screenshot send this message. It'll now like a human would to the agent. It'll prompt each of these Devins with a new message that'll say something like this. So it, you see it has some way to interact with them.
- [30:26]
B
Okay, this is actually, this is actually really, really good, dude. So I've, I've been using, you know, these AI coin tools and I've been chatting with them across different threads, but it's me chatting with them. I haven't really kind of adopted this pattern of doing a master child thing. I mean this is kind of a trivial example. But like in reality, like what are some good use cases for this master child thing?
- [30:47]
A
Yeah, you know, so a migration is one of the most obvious ones. Well, before I explain it, let me, let me show you one of these message sends so we can, we could go back to our agents and we can open one of these. I think it's sent to the, this one and you can see here, it impersonated, this is my like username on the Org but it sent this message over and it's working on this new redesign. But yeah, but, but yeah. Anyway, to answer your question, the, the, the real world use cases of this ancient fan out paradigm is when you're doing something that is a very large project. For example, a huge migration of a code base from let's say COBOL or From Honestly from JavaScript to TypeScript or React native to Swift. This is really useful because you want to break up the work. Well, it's useful for two reasons. One, you could parallelize things so it's just way faster to do it. But two, you could break up the work to keep the context window small so that each agent has a very specific focus. Can work on testing that thing and, and it's part of this thing that kind of. I started explaining of Devin and cognition. As a company we care about agents writing verifiable code and writing code that works. And part of that is breaking it down into chunks that are verifiable and testable. And not just these agents, these, the, all these 10 Devin agents are not just writing code and returning it, they're writing code, they're running it, they're taking screenshots, they're looking at it. It's really a full fledged teammate and
- [32:48]
B
that is already baking to just the product. I don't think do some crazy skills or prompts to. They can work with that way.
- [32:54]
A
Correct. This is a core feature and then you could get really cool when you're using Devin Desktop locally or Devon Cli locally and it's interacting with these cloud agents and it's handing off work to the cloud and parallelizing that way. I think this is, I'd say where the world is going. I'd say this is probably the advanced usage of coding agents. You don't have to do all this stuff but it's, it's helped me a lot and it's been, it's been a big learning as I moved away from all the other agent products into Devon when I started at Cognition.
- [33:31]
B
Interesting. Yeah. So I was going to make a joke that this is how you token max, but maybe actually save tokens because like each, each one only has like a smaller context window than like trying to do it all in one thread.
- [33:43]
A
Totally. I mean I'm, I'm still a pro token maxer for what it's worth. But yeah, you do save and I think we're wor. Working on. So we have our own model that we build called Suite and it's specifically designed for software engineering and it's designed to be much quicker and much cheaper. So the, the other pretty cool thing about Devin that I like a lot is I don't have to keep track of which is the best model that exists because we have a pretty cracked engineering team and they're spending a lot of time on evals, they're spending a lot of time on this sort of work and they route it to the best one. And we're starting to think a lot more about how do we do the trade off between. Between cost and time and efficiency and, and intelligence such that maybe some of these could be traded spun up with a smaller models. And this is. I mean you see there's no model selector here. So this is something that you let Devin handle.
- [34:48]
B
Oh, interesting. You know, I always get kind of suspicious sometimes when companies do this because like I, I just think they're trying to save costs. But maybe that's not true. Maybe it's.
- [34:59]
A
Well, I'll tell you today by default it's using the best models for everything. We, we have features that are currently in development and should be out soon that in this kind of. In the. For cloud agents specifically can be handed off to more light versions for simpler work to save money again because we're independent from the labs, we don't make like our goal is not for you to token max. Our goal is for you to ROI max. And so, so it's for, I think it's not for us to save money. And then also if you're using dev and like if you're using a local agent, you can choose the model yourself.
- [35:43]
B
Got it. This is helpful. So then going back to the keeping things simple thing. When you build stuff with Devin, it looks like it just gives it pretty casual commands. Do you have a bunch of agent MD and a bunch of custom skills set up or just kind of.
- [36:02]
A
Yeah. So we have a few other features I didn't actually quite show you that closely. We have some things you can add here. Like we have some knowledge, playbooks, macro skills, that sort of thing. And what, what those are, are. The goal of all those is, and I'm just getting distracted by these screenshots coming in, but the goal of all those is to like share knowledge amongst the team. So by default I think the best thing Devin does is it remembers how to set up environments. So if you have this whole local environment set up to run your app, it'll do it once and then it'll save what it learned. So it could be much quicker next time. We also have kind of ask. So ask. I can jump to this one. This was a test I just ran for you guys on openclaw Repo. It's kind of we pre index the repo, make it easier to just ask questions and this is where you can kind of do plan mode and ask things and have it write a plan that you can hand off to a real Devin session. But yeah, for skills macro, I mean I use them all. Is the short version okay, got it. I think this is like obviously a trivial example because this is not actually like this is just a landing page. It doesn't need to run anything, it just needs to make sure links work. But I think where Devin really shines is like on real engineering and here's their automations section where you can hook it up to the other work and you can unblock it on humans. And we have kind of a wiki, deep wiki, which is also something people love where it's creating, creating docs for your repo and I think if you're in a company with a really large code base, some of these companies have docs for their repo and it's just an instant win.
- [37:56]
B
Oh, it just creates docs automatically.
- [37:59]
A
Yeah, yeah. And that's part of the indexing process on a repo. So maybe let me, let me open one. Let's do. This is a little repo I made. I'm having Devin look for an apartment for me in San Francisco. So. But it's creating docs of how this process works and this is obviously a trivial example too, but you can see kind of how that would.
- [38:22]
B
All right, dude, well let me ask you a couple of questions to close our interview. So I think, yeah, I do think obviously the space is very competitive. I would love to get your take on, for example, CODIS and cloud code. They're competing against each other. They're both trying to expand into knowledge work beyond coding. Then you have this whole class of more beginner friendly like lovable replitations and so on. And then you have more hardcore tools for coding like Devin and maybe Factory and some of these other tools.
- [38:52]
A
Right.
- [38:53]
B
Like obviously you work at Devon, so maybe you're a little bit biased. But how do you think this whole thing is going to play out, dude? Do you think all these players will continue to exist or do you think this will be a consolidation or.
- [39:04]
A
Yeah, it's a good question. First of all, I think, I think the market's bigger than people realize. And this is not to say it won't be a winner take all, but it's just. I'll tell you the reason I joined Cognition was I kind of realized that there's one problem that matters today and it's building a coding agent. And all the biggest companies, all the labs are working on it. We consider ourselves an agent lab. We're working on it. And just the, it's just the power of this is such a big problem in such a big market and if you solve, solve. But if you build the, a great coding agent, it's, it's just the 80%. It's a power law when it comes to basically any other problem and basically every other problem is downstream of that. So that's why I think everybody's rushing for it and that's why I think there's so much competition. Competition predictions are hard. I'm really bullish on non technical work with agents. I, I think one on one hand it's very smart that we're focusing on software, but I, I use agents for a lot of non technical stuff too. So I don't think that's going away. I think there's a lot of great work by the labs doing it and by non labs. If I had to say something I'm more doubtful of is kind of These beginner first beginner only build version one project products that kind of like lovables and, and the wixes of the world. I think I'm, I love the idea of democratizing coding and letting more people to do it. I, I think there's probably going to be a bifurcation where we're going to have hardcore engineering, real engineering and then like play engineering. And I don't mean that in a derogatory word way either. I mean it to say single use software short. My, my friend is a founder and he's kind of started coining the term short form software. So kind of something that exists just for one time or a few times and it's not a business in the same way. And I think it's a completely different paradigm. So I think I haven't seen anything, there's not really anything built for that in the same way they're building for hardcore engineering and end up there.
- [41:35]
B
I do think if anything these tools are going to eat the market of more traditional UI focused tools like a PowerPoint or some of these more traditional tools because you just tell the agent to do the work and then the last mile, maybe the agent can help with too or you can manually tweak stuff.
- [41:52]
A
Oh yeah, I think, I think there's a lot of artifacts that are going to be second tier artifacts that we don't care about anymore, to be honest. Code, this is a little more controversial, but I don't, I argue internally. We have debates on this, to be honest, but I think code is going to be a second order artifact where you're going to be interacting with something higher level than that. PowerPoint, same thing for sure. I don't, I don't. I create all my slides with agents now, video editing. I think there's a lot of point and click. I mean if you want my radical opinion, and I'll give it quickly, is, yeah, is. I think I could see. I think humans are going to return to the world where we just have a wooden desk and we don't have this whole monitor set up and keyboard setup and mouse. I think the mouse is kind of antiquated. I think the keyboard is antiquated. I don't think we need that. I just want a button and a whisper flow and I think that's all I need.
- [42:52]
B
Yeah, okay, got it. Interesting button. You have to see what the product is, right? Don't you have to see the UI or you don't think there will be a UI anymore?
- [42:59]
A
Yeah, that's the hard part. I don't know. Yeah, there might not be a UI anymore. I think there's. I think design and aesthetics matter a lot. So there's probably still a world where you need some sort of monitor. But maybe I could see. What if it just prints it out and I could look at it there and then I don't need it. I. I think there's a lot of unique stuff to be made here, but I think this whole concept of the ergonomic keyboard and the mouse, if your job is managing a fleet of agents all day and, yeah, you don't need that. And you're like, I barely type. I type a little. Okay, I type half the day, but I'm typing less and less and I'm talking way more to agents. And that's natural. I'm a human. I like to talk. I mean, not everybody likes whisper flow because some people are more yappers than others, but I just think the desk setup is a little unnatural. I don't like this two monitor thing I have going on. I'd rather just have.
- [43:52]
B
I see, that's funny. That's funny because, like, you know, like, for example, Peter Steinberger, the open cloud guy, he has six monitors or something, right? But like, I guess it makes sense because, like, if you're a manager, you're not looking at the monitors of six of your reports all the time. You know, you just kind of give them directions and hopefully they do the work. Right.
- [44:08]
A
Yeah, right. There's a lot of different workflows, so I think that's something that maybe some workflows won't need it anymore, but I want them, I want us to return to the Mad Men style desk.
- [44:22]
B
Okay, Let me get like another, maybe a hot take and then we can wrap up. So I've been struggling with this a lot, man. Like, I feel like. So I've. I've hooked up APIs and, you know, MCPs and whatever to, you know, all these agents. So now I don't even go to the website of these apps anymore. I just like, query my agents and I feel like if I own one of these other products, like these SaaS products, it feels like I'm losing the relationship with the customer. Right. Because the customer is just talking to me through cloud or something. It's almost like, how do you design a product to be agent first versus human first? So it's like a pretty radical shift if you think about it that way.
- [44:57]
A
Yeah, yeah, 100%. I think, I think we're still trying to. I think as a society, we're Figuring that out. I, I guess my, my version of my take in that world is that I think everything can be reduced to an API endpoint or API call, meaning there's no reason an agent can't build a house. Because what does it take to build a house? You need to, you need to probably buy the property, you need to zone it, you need to buy the supplies, you need to hire the people, you need to tell the people what to do with the supplies. You need to take pictures of it after. These can all be discrete steps that an agent calls out to some API somewhere to do. And there's, yeah, there's, I can see agents building homes and building towns and, and running businesses autonomously and everything's an API endpoint. And I look rent to human I think was kind of a little bit of a like tongue in cheek it felt like. But I'm unironically believe that that's a really good idea.
- [46:06]
B
It's hard as a company though because like if I just make everything API endpoint and then you know, clock and just get all my data, the human doesn't have to talk to me. It's like what kind of like you know, do I even have a brand left or is it just a bunch of APIs I'm giving away?
- [46:19]
A
No, it's a great question. What, what is brand? But maybe the brand is something for agents. Maybe it's you're writing it in markdown and like it's you find a way to convince agents and AEO and. But yeah, I don't.
- [46:33]
B
We'll figure it out.
- [46:33]
A
Yeah, I don't have a good answer. That's a good question though.
- [46:37]
B
Cool. Jared. Well, well, I mean where can people find you and learn more about Devin?
- [46:41]
A
Yeah, you can find me. You find me on X at X Twitter at I am Jared Z. I M J R E D Z. Find Devin at Devin AI Cognition. AI is the company that builds Devin. And yeah, just start using it. You can sign up, you can try it out for free, you can use it for free and let me know if you have any questions using it. I'm always eager to help help agent built people.
- [47:06]
B
Cool. All right Jed, well I'm go off and spin up some agents now but it's great chatting with you, I love it.
- [47:11]
A
Well, thanks for having me.