Martin Casado (2:22)

So not everybody's going to be familiar with Convex. It's trying to do something people have been talking about in database land for decades. It's almost kind of a white whale. So maybe just give a quick background on what you're working on at Convex and then we'll kind of move over to the AI stuff.

Sujay Jayakar (2:35)

Yeah, sure. So Convex is a reactive database and it's a database that's built from the ground up to make application development as easy as possible. So there are a bunch of implications for that. But I think the starting point is we just casually use databases that are over 30 years old without thinking about it. And we don't do the same for programming languages, we don't do the same for our libraries. And Convex is just like you're saying, we're trying to go after that white whale of could you make application development an order of magnitude plus more efficient if you rethought some of those things from first principles? So everything, for example, is reactive by default. You don't have to handle state management at all. Everything is type safe end to end. And yeah, I mean, it kind of has all of the pieces that you need to make a modern application just integrated and configured entirely in code.

Martin Casado (3:30)

So practically what this means to, you know, us lay developers. So I use Convex on a lot of projects. So let's say I'm writing some Web app and JavaScript, pragmatically what it means is I just take my JavaScript and it gets run in Convex and then I get transactionality, I get reactivity, I get the ability to query things and I don't have to like resort to, you know, SQL and all the foibles with SQL to do that. So you actually just basically end up getting a transactional backend while still just writing JavaScript. Is that fair?

Sujay Jayakar (3:58)

Yeah, exactly. I mean there's. JavaScript is one of the most popular languages in the world and it's so intuitive and why can't everything be in it?

Martin Casado (4:06)

Sujay also was one of the primary developers of AI Town. So actually AI Town was started by Yoko, who works on our team. And then I kind of pitched in for a little bit, but then you and Ian did a lot of the work on that. So for all of you that use AI Town, Sujay is kind of the unsung hero of the backend for that. Now you wrote a book blog post recently on benchmarking AI generated code. And by the way, for those listening, if you haven't read it. Strongly recommend you read it. It's given me actually kind of the most insightful intuitions I've had since the beginning of this whole AI generation code discourse. But I think, you know, if you're open to it, it'd be great to kind of go through your key insights while going through it and then we'll try and dig into the details.

Sujay Jayakar (4:48)

Yeah, totally. I think the kind of core insight is that being able to fully build full stack apps, it's still not a slam dunk for autonomous AI agents today. It's sometimes within their capabilities, sometimes it's not. And what's really interesting where since it's on the edge of what's possible, there are things that we can do to make it work and things that we can do that make it definitely not work. So the kind of what we observed is that there's a few factors that really impact autonomous coding performance. One of them is just having really strong guardrails. We saw that systems that can give really quick feedback and have very strong boundaries on what's correct versus not help guide these models towards being able to achieve a lot more. We also observed that models are really good at writing code. You know, you mentioned SQL before. They're not so good at doing stuff like evaluating RLS rules or reasoning about why a SQL query does or doesn't work. Code is the thing that models are surprisingly good at. Right. And then the other thing we. And we'll dive into this. I think having really good library choice and choosing good abstractions is a really important task for getting good performance. On AI coding. You have the set of things that you want the model to do, and that's how we describe our tasks. But it's equally as important to pick what you don't want the model to do. And so if there's some really hard problem or something that just exists already is a good library, don't let the model reinvent it from first principles because it'll likely mess it up.

Martin Casado (6:24)

You know, as part of the blog post you actually introduced a new benchmark. So it'd be great to hear your maybe spicy take on why a new benchmark is needed. Because there are so many benchmarks already that are in use today.

Sujay Jayakar (6:37)

So we called it Full Stack Bench. And so the benchmark is if an AI agent has a front end app that implements something like we start with a chat app to do app, like a file manager app. It's kind of like common full stack app patterns. If it starts with just the front End and then as part of the test, we choose which backend it wants to use. Can the model fill out the rest of the picture? Can it take all the things that were done on the front end and map them into how it sets up the database, how it sets up APIs, how it sets up subscriptions? And making these types of benchmarks and running them is a lot of work. So to your point, why create a new one? And frankly, it's because we saw a gap, right, like when we started getting really interested in this, when for Convex we had a bunch of our users write into us and say that Convex is like a really good target for, for AI coding tools. We have some customers that are themselves like AI cogen startups and then they make apps for their users that use Convex for the backend. They said some things work super well on Convex, some things really don't work well, and some things are kind of in the middle. So to be able to understand that a little bit more rigorously, that's when we wanted to have this experimental framework. So we took a look around and we tried to see for these tasks that people are doing, are there good benchmarks out there? And the answer was no. And you know, I think this is like the type of thing that's surprising. There's you know, publicly available ones like Sweepenge recently, there's like Speedlancer right from OpenAI and these are kind of much more narrowly scoped to things that are easy to create data sets for. Like just take a public GitHub repository, look at all of the commits, look at the GitHub issue that led to it, assemble those without a lot of supervision. And then there's kind of dinky, very narrowly focused problems, right, like your code forces like competition coding, but for the tasks that people are actually doing, like if someone has cursor and they're using agent mode in there to build something, there aren't really great publicly available benchmarks. And this is an interesting theme in itself is that I think we've seen that a lot of companies consider high quality evals to be their secret sauce and aren't putting them out publicly. But for us, we saw that and then decided we would make our own. And then we also made it public.

Martin Casado (9:00)

I'd love your thoughts on the utility of these benchmarks for just an average schmo like me. I'll just give you a quick anecdote. So, you know, Claude 3.7 dropped, which is amazing. And I do a lot of just lay development at home, just for fun, right? I mean, I'm like, I'm a super casual developer. It's not my day job. It's something I kind of do to relax. And so I've been using cursor in cloud 3.5, 3.7 dropped. I used 3.7 and I found I got into a lot of trouble because it was actually maybe too clever by half. Like, it's just kind of do all of these edits and then I ended up having to unwind them. And so I went back to 3.5 for now and I'm sure I'll figure out 3.7 over time. But so when you think about these benchmarks, do you think that their value extends to the independent developer? Is it more for systems builders or is it more for the end? Like, how do you think think of the utility broadly?

Sujay Jayakar (9:49)

I think if you are building a product where that product then uses these models. So maybe not so much as if you're a developer just using Cursor or using ChatGPT. Your tasks are like the things that you directly care about, right? So it's, you know, the need to experimentally run many conditions is not there as much. But I think, you know, for so many people now are building apps where their apps are calling out to these models and then they want to provide a good user experience. Right. For their end users. There's a really good blog post of your AI app needs evals. And I think this is one of the things that's just chronically underappreciated is that I think people just assume you can slap together some random stuff and then it's going to turn into a good product. But as a web development noob back in the day, you know, like learning CSS and it's like the thing where you like tweak one thing and then something else twice totally breaks. And I think going through the rigor of writing, like, specifically what problems do you want the AI agent to solve? What do solutions look like? How are those solutions graded? And compiling a suite of those is one just a really good elicitation of what you're actually trying to do in your product. But then say, for example, if 3.7 comes out, you can then automatically see how does 3.7 perform on, let's say, 100 different things that your product needs it to do? See that it improved on some, regressed on others. And then if you are tweaking your prompts to make it work better on three. Seven, you can then also very easily verify that you're not regressing 3.5. Right.

Martin Casado (11:25)

Just seeing that I understand here. So the benchmark gives you general kind of understanding of what different platforms do better or worse, but, like, it doesn't help you on your independent job for that. Evals are. Are better and probably underappreciated, but you need to get pretty comfortable with how you do evals on these things, which has definitely been my experience, just for the listeners on this podcast. I mean, I think everybody understands at a high level what evals are, but not really how they're applied to these kind of modern AI. So maybe it's worth just talking very quickly about, like, how you view evals, like, what they are, how do they fit in this process?

Sujay Jayakar (11:56)

Yeah, totally. So it's, you know, and it. I actually found it very fun. This is my first time kind of writing a big set of evals, and, you know, in this case, like, first kind of coming up with what is the task that we're trying to do. Right. So in this case, say, writing convex code, it could be that given a kind of like a quiz question, given a prompt, say, write the backend for a full stack app that needs to be able to list messages in a channel or post a new message to a channel. So the first part of the eval is these, like, task descriptions. What are you feeding into the model? And then given that the model then will give you some output. And some of the eval design is making sure that you tell the model to format its output in a particular way or whatever. And then in the framework, we parse that output and we compare it to a reference solution, and then we grade it out.

Martin Casado (12:48)

That sounds really hard. Like, how do you. Is it like syntax? Is it semantics? Is it correct? Like, how do you even think about comparing these two things?

Sujay Jayakar (12:58)

Yeah, and so sometimes also, there isn't a gold standard solution. Right, Right. So if you just get this random output, how do you compare it with.

Martin Casado (13:07)

Yeah, right.

Sujay Jayakar (13:08)

Yeah. And I think this is a big part of the experimental design. Right. Because it could be that, like, you know, say, for example, if we have a task that's just like, implement the backend for a chat app, then the output for that could look like anything. Right. And it could legitimately look like anything. I've seen this, like, kind of general trade off between generality of the task description and gradability. So, you know, instead of just saying, write me the backend for a chat app, if you say, write the backend for a chat app where it has a get request, that slash list Messages that takes in this, you know, JSON object and returns this in this sorted order. It might be more descriptive than you'd ideally, like if you're just prompting some random agent, but it makes.

Martin Casado (13:51)

You can grade it now.

Sujay Jayakar (13:52)

Yeah, yeah. For a different eval set for us is that we have the reference solution and we have the model generated solution and then we actually spin up convex backends for both, push the code.

Martin Casado (14:07)

Oh, interesting.

Sujay Jayakar (14:09)

And then we have some human written unit tests that compare the results across both. But then we also have some like, backend introspection APIs to say, like, what are the schemas for both? Those should 100% match. What is the API description with all of the types and like those should 100% match. So being able to kind of do that more automatically and be more gradable is another piece of our kind of experimental design.

Martin Casado (14:30)

Are there things that the agents aren't super good at that, you know, you had to build specific tests, or is it, you know, just most gradable things are the same?

Sujay Jayakar (14:40)

Yeah, there's a kind of very interesting, at least in our experience, relationship there between difficulty and variance. You know, like we've seen that when.

Martin Casado (14:49)

And by variance you mean variance between models or within the same model.

Sujay Jayakar (14:52)

Even within the same model. Like consistency in its ability to come up with a result given the same input.

Martin Casado (14:58)

Sorry, I don't mean to interrupt, but given the same input. Really?

Sujay Jayakar (15:01)

Yeah, totally. Because when we have these coding agents that are spending an hour in one of these.

Martin Casado (15:06)

Yeah, that's just a lot of context and state and.

Sujay Jayakar (15:09)

Yeah, yeah. So, you know, for our most difficult task, we like have it implement this files app where there's different project namespaces, there's this hierarchical authorization model where groups can be nested in each other, but not in cycles, and then groups can be attached to folders. Like, all this stuff is designed to be pretty complicated. And in addition to that, there's also a lot of code. Right. It starts with 4,000 lines of code for the front end and then adds a few thousand lines as it implements the back end. Seen just a lot of variance on some. Sometimes it'll manage to keep the right things in context as the cursor agent kind of navigates through implementing it. I don't think it's public what their context management looks like, but I think they'll just evict a very important piece of the input prompt or some file they worked on along the way, and then it might not ever find that again. Until a human has to get involved saying, hey, you're just totally stuck.

Martin Casado (15:59)

So you find for the more complex problems, the variance goes up and then. So how do you think about reigning in that variance when it comes to, you know, let's say I'm a developer and like I actually care about things. I mean, is there a solution?

Sujay Jayakar (16:15)

Yeah, and this is like one of the, I think big takeaways from the full stack bench work is that having very strong guardrails and guardrails that can have. That can be applied quickly.

Martin Casado (16:25)

What do you mean by guardrails by the way? I mean, I guess we all intuitively understand what you mean by guardrails, but specifically, is it like type safety? Is it good semantics?

Sujay Jayakar (16:34)

Yeah, I think type safety is just like one of the best examples. Right. Because it's, you know, especially with TypeScript, like there's so many invariants JavaScript, but.

Martin Casado (16:43)

You have none at all.

Sujay Jayakar (16:44)

Yeah. I mean it's also funny too. I remember when TypeScript first came out, like all the PL nerds were hating on it writing, myself included. Right. It's not sound. It's just like trying to embed every single pattern and it varying from the JavaScript ecosystem into types. And then after a while I feel like there's a sense of admiration if you just give up on your principles. If you just go like give out decidability entirely, you go through the five.

Martin Casado (17:07)

Stages of grief and you kind of end up at acceptance. Yeah.

Sujay Jayakar (17:10)

Taking away anything from this.

Martin Casado (17:12)

Yeah.

Sujay Jayakar (17:12)

I think changing one's task and changing the tools that an agent uses to pick things that are more type safe, where type safety can be like cursor agent will, as soon as a model generates code, use the type information from the language server to feed it right back in as context. So it'll just immediately fix stuff without any real iteration loops. So I think that's one of the big takeaways of that. If you want to decrease that variance as it's exploring, kind of having type safety can keep it on the straight and narrow.

Martin Casado (17:43)

You know, what about like runtime guardrails, like languages that are mostly referentially transparent or you know, like things beyond type safety. Do these matter or.

Sujay Jayakar (17:53)

Yeah, it's a good question. I mean I think the ease of writing tests, I mean this is another thing that models seem to be very good at. So if the kind of semantics of a language generally encourage things patterns that are more testable, that's another way of managing the model's trajectory. Right. As it solves or the agent's trajectory. And I mean I think my initial reaction there though is I think that shows up more for reasoning than the actual authoring of the code. Like, you know, I think this is one of the things you observed with the full stack bench experiments was that a lot of the time would be that, you know, the model would think it's done and then we would start to grade it and then we'd say like, hey, this isn't working. Here's a screenshot of the error in the developer tools and then being able to go from that error and debug it and narrow it down to what the problem was relies on a lot of reasoning. Right. And you know, we saw the model just pretty consistently get stuck. Stuck on things like react hook rules. Right. Which are kind of like hard to reason about, you know, stuff like RLS rules for in SQL where there's not a very clear procedural execution semantics. Those are the things that I think show up for reasoning and places where it's easy for the model to just blow off the happy path and get totally stuck.

Martin Casado (19:16)

So maybe are you comfortable talking about your experience with the different types of models we've seen?

Sujay Jayakar (19:21)

Because we have another set of benchmarks which test model knowledge. I think this is one of those really interesting problems for dev tools right now. And you know, a bunch of people have talked about it where like with knowledge cutoff but also just like the amount of data and pre training that is anchored on just existing systems, it's very hard to build new abstractions. Right.

Martin Casado (19:44)

Even in context.

Sujay Jayakar (19:46)

And so that's the thing, I think that's what ends up saving us. But without any prompt engineering. Right. It's the defaults for what these models are good at is not going to evolve over time and it's going to be close. It feels like it's kind of like a uniform sample, right. Of what's out there on the Internet. And one of the first things we did was try to understand this phenomenon for different models and then like you're saying patch it up in context. And that's. I think the saving grace here is that these models do also seem to be very steerable and their ability to learn stuff from just, you know, very small numbers of examples and guidelines and context has been very cool. You know, I mean this is a thing where we see with Claude 3. 5 versus 3. 7 we notice that when it comes to at least knowledge about convex, it mostly stayed the same. It was a little bit worse. And that's probably just because our prompts were tuned for 3.5, but we didn't see a significant improvement.

Martin Casado (20:38)

Well, this includes knowledge, but also any sort of like in context, learning is the same.

Sujay Jayakar (20:42)

The artifact of this test is what we tell people to put in their cursor rules.

Martin Casado (20:47)

Oh, I see.

Sujay Jayakar (20:48)

So a new model comes out, we see what it knows and what it doesn't know, and then we fold that insight back into the cursor rules and then see if it improves it. And yeah, it's been very interesting. You know, for example, with using gemini, like the 2.0 flash versus I think we run it in CI with O3 mini. We've done it with 01. Just don't want to pay for it. Yeah, right. And talk about not paying for it. Tried it with GPT 4.5 recently. I was like, I was going to do one run, never touch it again.

Martin Casado (21:24)

How much was it? How much did it cost to do it with 4.5?

Sujay Jayakar (21:26)

Well, we like, I mean, it's I think 600,000 input tokens and I think roughly. I have to go double check, but I think, you know, same rough order of magnitude for output tokens. Well, it's not by that much. No, but still, like what it was like what, like $50, I think.

Martin Casado (21:41)

Yeah, okay. Yeah, yeah, okay, that's fine. Yeah, yeah, yeah.

Sujay Jayakar (21:44)

Could have done a lot of that. $50, Martin.

Martin Casado (21:46)

Yeah, yeah, no, I've been actually looking at pro. You know, like, the pricing of these models is pretty remarkable, right. If you do like whatever, GPT4 oh, it's $2.50. If you do mini, it's 47 cents or something like this. And so, you know, they're. And do you find there's a big trade off between the more expensive models and the cheaper models? Like, if I use the cheaper model, to me, like they're actually materially worse than the more expensive models. And so I think. And I haven't found out how to bridge that graph, unfortunately. And as a hobbyist, maybe this is just like my problem. I don't want to spend a lot of money on my stupid little, you know, programs I work on is I always want to use the smaller models, but they just don't find them to be as good. And so maybe a specific question to you, A. Do you find that this dichotomy is real? And the second one, is there anything that I can do to use the cheaper models or am I screwed?

Sujay Jayakar (22:28)

This is like for say 40 versus 4o mini, mini or.

Martin Casado (22:31)

Oh, it's exactly that. Yeah, exactly.

Sujay Jayakar (22:33)

Versus haiku. Yeah, we've had very poor results with.

Martin Casado (22:36)

That with the small models.

Sujay Jayakar (22:37)

Yeah, I know it's interesting to look at OpenAI's docs and we haven't tried this, but OpenAI's docs for their regular fine tuning API are actually pretty discouraging of using it for improving model performance. If you're using 4o and 4o isn't cutting it, they like, you know, like reading between the lines a bit. They like don't want to set up expectations that it'll get a lot better if you provide 4.0a lot of examples for fine tuning. What they do say is that if you have a very specific task and 4.0is good enough, you can then fine tune 4.0 mini with your label data and then get it up to that level of performance at a much cheaper price. But I think it's like a hobbyist that sounds like a whole too much of a pain. The Gemini stuff is pretty good, right? And that's what we see in our benchmarks and it's cheap for price performance.

Martin Casado (23:28)

I found Gemini to be the best actually. But a lot of it is like the actual price component, which I mean, I'm guessing from Google standpoint this is one area they can flex because they can vertically integrate, right? I mean all the way down to the hardware. So that's probably what they're doing is just they can just offer it cheaper because of that. Okay. So I just actually want to go through kind of some of the high level talking points because this has been great. So the first one is, you know, there's a gap in benchmark when it comes to, you know, like this kind of big back nd. I mean, how would you characterize it? Backend or full stack? Like full stack type workloads. Stateful. Is it stateful in transactions or is that the primary thing?

Sujay Jayakar (24:00)

Yeah, I think it's, you know, integrating all of these different systems. You know, I think there's already a ton of work out there for front end stuff and being able to work across multiple systems. So going from the front end to the API layer to the database, being able to make intelligent decisions across all of them, those are all necessary components to actually build a real app. And it's just seeing, I mean even looking at stuff like three bench, right. It's often just very focused on something kind of narrow.

Martin Casado (24:28)

So big multi component systems is what this benchmark is good and it does deal with heavily stable stuff, heavily transactional stuff, like real stuff.

Sujay Jayakar (24:34)

Yeah.

Martin Casado (24:35)

Two, you found quite a bit of variance even within the same model. And then the way to Reduce that variance was guardrails. One that we've called out particularly was just type. Safety is actually pretty meaningful.

Sujay Jayakar (24:46)

Yep.

Martin Casado (24:47)

3. From the perspective of an independent developer, benchmarks are great, but really you have to rely heavily on evals, and that's kind of where the secret is. I mean, it's very interesting to me because there's so few evals that are public right now. So, I mean, and I don't think most people know how to do a good eval set. I mean, do you have any thoughts on the state of evals being in.

Sujay Jayakar (25:08)

The position where we were trying to get started. Right. And figuring it out? I think it is like a, you know, I think folks have, you know, probably correctly identified that evals are really valuable and that, you know, I remember the Vercel CTO talked about this with V0, where, you know, they, I think started by trying to protect their system prompt really carefully. And then eventually we're like, that's actually not that valuable. We will let people get our system prompt. It's the evals that actually let you build a real app. And I think that gap between an organization that has a great UL set for their domain versus one that doesn't is just really big. And then folks have identified that and then not shared them. Right. And I kind of hope for more of an open source types moment. Right. Where even if an app is open source in the code or the system prompt, that's not the hard part for development in a lot of ways. And I think it'd be very cool for more than just like benchmarking models against each other. But if there are. Yeah. Sets of evals for particular applications getting built that can be put out there and kind of worked on together.

Martin Casado (26:16)

Yeah. So I have to ask this because you're such a strong systems thinker and it's relevant to evals, which. Is there any hope for incremental evolution in this space? My experience is you have a new model, everything changes. Right. And I was like, I mean, I haven't found them all that useful for that. You have to end up rewriting a lot of your eval. So much of even evals are just dealing with the same code base in the same model. And I get. Even if I increment the minor version number of a model, everything changes. And so do you think that we've reined in this problem of like, how do you do real software engineering with a model that's generating code? If the Change of Cloud 3.7 is such a perfect example that just Changes so much stuff like is this, I guess the question is, do you think this is manageable or have you found any tricks or is it still Wild West?

Sujay Jayakar (27:02)

This is just my speculation.

Martin Casado (27:04)

Unprompted. Hot take. Yeah, please.

Sujay Jayakar (27:06)

Yeah. I think some of this is an artifact of the way that models are currently developed. Right. And they're trained and that we have these huge companies that are, you know, fighting over being premier foundation models and what goes into their pre training. But then I think more importantly what happens during post training is just a complete black box to everyone.

Martin Casado (27:30)

Right, I see there's a lot, there's a lot of benchmark hacking for sure. One thing I noticed about Cloud 3.7 to this point, it's really good at lovable bolt type stuff. So I'm like make a pop up box in HTML. It's like magic and amazing and so like the areas that they're focusing on are clearly very specific and they just, you know, it feels almost vanity ish as opposed to like solving the core problems. So if that's any sort of, you know, anecdotal evidence to your theory.

Sujay Jayakar (27:58)

Totally. The problem you were talking about specifically, right. Where if you are kind of have some coding workflows and you have patterns that are built up on three, five, I think, you know, who knows if this will be in five years or 10 years. At some point down the line I can imagine that that type of task would be codified into some type of benchmark and then, you know, easy for them to run in their reinforcement learning loops and easy for them to. So easy for them to train against and then that could be something that, because you know, we've seen how steerable all of these models are with all the different types of fine tuning and prompting and I, you know, I do believe that if things are a little bit more open that there could be some more consistency in this way and that we wouldn't all be scrambling when a new model comes out to run it on our own benchmarks and be like, oh, like this worked or this didn't work or here's what I have to change in my prompts.

Martin Casado (28:50)

I don't know if you've put up a leaderboard for the models themselves with you know, the full stack bench that you're working on. But I think that would be great just because in my experience it's the most useful benchmark if you're building a real system. Like it's just not a vanity thing. Hey, have you done that? I should know this. And then if not, do you plan on doing that just so that the model creators consider this benchmark?

Sujay Jayakar (29:16)

Yeah, not yet. We mostly focus on just Cursor Agent with Claude, and so it's been done with Claude 3.5, Claude 3.7. I tried it with the OpenAI models using cursor Agent, but I, you know, I get this, like, feeling that they're not really optimized for the OpenAI models. And so I've been trying to think of, you know, maybe using ATOR and then using that with all the different models as kind of like a more fair comparison. But, you know, it's one of those things where it's could spend all this time designing something that's trying to be as fair as possible, but it probably might be interesting just to try Tricursor with them and then have that be on the leaderboard.

Martin Casado (29:50)

All right, so listen, great work on Full Stack Bench. I really enjoyed this conversation. Any predictions on where this stuff goes or anything else that you want to comment?

Sujay Jayakar (30:01)

I think for users of these tools, I think, like, the takeaway for me is that these tools, if we're trying to solve very complicated tasks, they're, you know, it's still kind of on the boundary of what's possible and not to have them plot out these trajectories, these plans of how do they do complicated coding tasks step by step, and how do they know they're making progress and how they make sure they don't get into loops. So I think maybe the takeaway for me, in a very broad sense here is we can work a lot on writing better task prompts. You know, we can work better at when we open up a new chat view on Cursor, like, what do I type in there and how do I get better at that? But I think there's a whole nother space of options for saying, how do we change the tools we use, how do we change what libraries we use or what frameworks we use to have some of these properties of having better type safety and guardrails, and that is, you know, kind of secretly a part of the prompt. And I think spending some time optimizing that and thinking about how the model will chart its way through the course of getting from the starting point to a solution can lead to a very large amount of improvements.

Martin Casado (31:11)

Amazing. Okay, I have to ask. So do you use AI for coding yourself? I mean, literally, I think you're probably the best developer I know. I know a lot of developers. So do you use AI for coding?

Sujay Jayakar (31:21)

100%.

Martin Casado (31:22)

Does it make you faster and if.

Sujay Jayakar (31:24)

So, how much I think it makes. It's. I think it's hard to quantify, right, Because I think it makes me, you know, maybe two times faster. The kind of idea of there being just so much pent up demand for coding in the world and you know, I do all types of things I would never do otherwise. I write little custom debuggers for when I'm working on a tool and visualize data and it's honestly delightful. It's been really fun.

Martin Casado (31:46)

You've spent a lot of time now working with these models and how they generate code and you know, what sort of advice or mental model would you give to somebody that's using them on like how to approach maybe the most effectively or safely prompting or kind of what sort of workflow or how to think about it when using these models for code?

Sujay Jayakar (32:05)

Yeah, I mean I think there's like a lot of analogies here to even just like working with humans, right? I mean I think there's like when it comes to breaking things up into steps and you know, like working with some junior engineers, right. It's the first work on getting every, all of your interfaces written out and getting it so everything type checks and then git, commit, make sure that once you get to a commanding position, you will never be pushed off of it, right? And then from there you get it.

Martin Casado (32:30)

Working first, get it the basics, get it committed. So like if it screws it up, you can kind of go back to.

Sujay Jayakar (32:36)

That one and then from there totally you might. And then, you know, if you have a small little extension, right, you have a place you've committed, you know, it works. Try something small, design it to be evaluatable. Can you know whether that works or not? If it doesn't work, revert. Those are all types of things that I think as humans we have to learn, right, as programmers, but I think we're pretty intuitively good at and I don't think models are amazing at it yet and maybe they will in the future, but we haven't seen it.

Martin Casado (33:04)

Awesome. Thank you so much, cj.

Sujay Jayakar (33:06)

Thanks, Martine.

Podcast Host / Narrator (33:08)

There you have it, another episode in the books. We hope you enjoyed it and if you did, please do rate and review the podcast and share it far and wide with your friends and colleagues. And as always, keep listening for more exciting episodes.