← AI and I
AI and I
How Two Engineers Ship Like a Team of 15 With AI Agents | Kieran Klaassen, Nityesh Agarwal
00:54:02
Loading summary
Transcript139 lines
- [00:00]
Dan Shipper
You're figuring out how to do compounding engineering. There's two people on the team, but it really feels like there's 15.
- [00:07]
Kieran
Coding with AI is more than just the coding part. Utilizing it for research, for workflows. It should be used for everything.
- [00:15]
Nitesh
We haven't touched Windsurf or Cursor in the last three weeks. Both of those have agentic coding capabilities. But Plaude just takes it one step further by simplifying it by a factor of 10.
- [00:28]
Kieran
We've been really leaning into, let's AI do the work for us, and we're just managing the AI.
- [00:34]
Dan Shipper
You speak your feature into Claude code, and then it does all the research to create that long document and then just adds it into GitHub issues. That's really cool. What you did first is spent time building a prompt that effectively builds other.
- [00:49]
Kieran
Prompts, having an idea that has a lot of outcomes. This is part of the compounding effect. We had, I think, six or seven running at the same time because we were just like, new idea, let's go. New idea, let's go.
- [01:15]
Dan Shipper
Kieran Nitesh, welcome to the show.
- [01:17]
Kieran
Thank you so much for having us.
- [01:20]
Dan Shipper
Tom, I'm psyched to have you. So for people who don't know, both of you work on Quora, which is every's AI email assistant. Kieran, you're the gm. Natasha, you're an engineer. And beyond the fact that Quora is a really cool product, and I'm really excited to bring that to everybody who listens to the show or watches this show, I wanted to do an episode with the two of you because I think that you're figuring out a new way to do engineering, because, really, Quora has, you know, there's two people on the team, but it really feels like there's like, 15, because you've got. You've got agents who are pulling down PRs and working on branches, and then you're, like, pushing them up and other agents reviewing, and it's just like this kind of crazy thing that it's a new way to build software. And. And Kieran, you said something the other day that, like, really stuck with me, which is like, you're figuring out how to do compounding engineering. So with each piece of work you do, you're making it easier to do the next piece of work. And I just think that it's really important to bring what you guys are learning to everybody that watches the show, because it's like, we. We have new tools, and so we need new principles and new workflows. For using those tools. And so I'm really excited to talk to you about that.
- [02:39]
Kieran
Yeah, thanks. It's really fun to build Quora, but, like, being part of every. And, like, being in an environment where you get access to tools, like access to thinking, access to exciting new ways to. To work, really helps us rethink how we build. So, like, it's. It's really an experiment. We're building a product Quora, but at the same time, we're figuring out how we should build. And. And that's super interesting. And we're, like, right in the middle where people say, what do you think of this new model? Like. Like, how do we use this research tool? And we're just trying things out. And Natash and I, we've been, like, really feeling a shift in the last weeks, I would say, where we're like. Like, things are changing and we're not the only ones. Like, we hear other people say that as well, but not a lot of people. And what we've learned is a lot, and we want to share a little bit of what we learned. And also what we know is, like, we're just barely starting. We're scratching the surface of this, and it's a big shift that's happening right now by new models, by how people think, by mcp, by, like, just. It's a lot. And yeah, like, it's great to talk about that from different perspectives. Yeah.
- [04:00]
Dan Shipper
Yeah, I agree. And I think it is so special to be at every. Because we do, like, every day, there's someone new in the discord who's like, I built an AI agent. Do you want to use it before we launch it? And so we get access to OpenAI models before they come out, and sometimes anthropic models. And so we have this, like, early edge. And then you guys are so good at figuring out how to actually incorporate them into, like, a production process. So you said, Kieran, that something changed. So I guess I want to get a sense of what you think changed and what, like, draw the broad strokes of what the workflow is that's starting to emerge for you guys?
- [04:39]
Kieran
Yeah, for me, like, obviously, it's like, everything coming together. But I think the biggest thing is a realization in myself that coding with AI is more than just the coding part. And it's really about, like, utilizing it for research, for workflows, for everything. Like, it should be used for everything. And we're now at a point where the agents are good enough that they can actually do everything. So we need to rethink again, like, hey, cursor windsurf. Like the old school way of coding was great, like more of the vibe coding. Like that was one step and then now it's the realization, oh, actually we can just give a task and it will do it, but still the work needs to be done by like, what do we do? How do we do it? And just a realization that we should lean into that more and really go deep. And it's like Claude code. Like it's just good coding agents or agents available that actually start to work with new models, like Claude 4. Like really good at following directions and instructions. And it's all of that coming together that I realized, like, oh, we're here. Like the future is here. This thing we've been talking about, that was going to be the agentic evolution, suddenly it works and it's working in real world, non experimental playing. It's just like we're building an app and it's working, building the app.
- [06:12]
Dan Shipper
So what I'm hearing is like, it's not just about developing with AI, it's all the things that go into developing that you're using AI for and that the thing that you're using the most for. This is Claude code. Is that right? And if it's right, tell me about. For people who don't know Claude code or haven't used it. Give us a little introduction to Claude code and then tell us about exactly how you're using it.
- [06:35]
Kieran
Yeah, Claude code is basically the coding agent version from Anthropic that uses CLAUDE under the hood. And it runs in your terminal as a CLI tool, which is kind of.
- [06:50]
Dan Shipper
Do you want to share your screen and like show us?
- [06:52]
Kieran
Yeah, yeah. So Claude code is a tool that you use in your terminal. And I know for non technical people this is like, this is scary. But I've converted friends who were not technical to use Claude code and they were like, oh, this is great. But it's really simple. You just hit, you start your terminal, you say claude and an interface will pop up.
- [07:17]
Dan Shipper
And basically for people who are listening instead of watching, he's in his terminal. It's the classic black screen that feels like you're using DOS or something. And he just typed claude. And then we just got a thing that says welcome to Claude code. And there's a little text box for him to type in any command.
- [07:34]
Kieran
Yeah. And why is this different or what makes this different? This has access to the directory or the computer. So it can look through files on my computer already. It can run things on My computer, it can take screenshots of websites, it can search the web like it has tools, but way more tools than available in a normal cloth version. And that's important because engineering work, like building stuff you do need more tools than just like the basics. You need GitHub to see what you need to build or what the status is or what the CI pipeline does. Like do the test feel like having all these things available in one coding agent actually makes it possible for me to have a workflow or like a thing I do actually be done by an agent. And that's the important thing. Like really the compound word comes in by doing more than just coding because lots of like if you talk to an engineer, like most of the work is maybe coding, but Maybe it's actually 20%, maybe 80% of the work is like figuring out what to do next or understanding what people like, what their feedback is and how to interpret it. And what you can do here is you can for example, like a fun way is like to, to use it. Say let's say what did we ship in the last week? So like it, it knows stuff. So I'm asking it what we shipped and it will most likely look at the git log because that's how we track what we did ship. And yeah, so it looks through the git log, it looks at what we merge to main. And yeah, that's a fun way to use it. And for example we can use this for product marketing and it says oh these are the bug fixes, a brief skip functionality, chat panel, state, email summary, XML tags, major feature features, breathe health monitoring, time zone, auto detection. These are all things we released. And now I can say and it's.
- [09:52]
Dan Shipper
Written in a nice write up that anyone can read technical. Yeah and it's actually a lot for two people. There's what like six major features and like five important bug fixes and three infrastructure updates. Like that's a lot.
- [10:08]
Kieran
Yes, it's a lot. And like this week we've been really leaning into like let's AI do the work for us and we're just managing the AI. One other thing for example is if you have someone come to you like oh what is the status on this? Or like what are you going to ship next week? Let's see what it will do. Can you see what is in the pipeline and what will come out soon?
- [10:35]
Dan Shipper
So this is awesome. Nitesh while this is going, if you, if you want to jump in at any point, feel free to at some point. I'll lob it to you, but also like, just I'll, you know, feel free to jump in. Yeah, sure.
- [10:45]
Nitesh
Yeah.
- [10:51]
Kieran
So we'll see. I don't know if it has project access, but like you get the gist. Like if you have the information connected to the agent, it's very easy to use it and it's very important to use a tool you're familiar with. And at this point, I think CLAUDE code works the best. For me, it is the most flexible because it doesn't only solve coding issues and that's important. Lots of these coding agents are made to code, but I want to do more than coding. I want it to be like a, like a support in engineering in general. And I think the CLAUDE team really thought about that. They made it not too specific and they kept it general while actually being really good at solving things and looking at what it did, thinking about the mistakes it made and self correcting. So that is stuff coming together that's very hard that makes it possible to use now. Yeah.
- [11:52]
Dan Shipper
What's the difference between coding and cursor and agentic coding?
- [11:57]
Nitesh
CLAUDE code is such a simple departure from the cursor and Winsurf that we're used to. Like, both of those have agentic coding capabilities, but Plot just takes it one step further by simplifying it by I think like a factor of 10. So what Kieran was telling earlier about how Plaud code may feel intimidating because it is a terminal, but in reality it is like so much simpler than the Windsurf and cursor because there is nothing except a text button, text box. Here's no like command K, no shortcuts, no accept, delete, reject, remove. Like there's nothing, it's just a text box. And it works because the model, the underlying CLAUDE model is so much more capable now, so it's able to work for longer and do tool calls. So it's like a simpler ui, which makes it at the same time more powerful. Even though, even though, like the underlying model behind like cursor and CLAUDE code is the same.
- [13:08]
Dan Shipper
Yeah.
- [13:08]
Kieran
And an example of this is this morning I was pulling some metrics and was like, why didn't we get any responses to this form?
- [13:18]
Dan Shipper
And then, and for context, like, basically we have a form that we ask people how disappointed they would be if they could no longer use Quora so we can tell how well we're doing. And you noticed we have a weekly meeting where we go through all the metrics and you notice that no one had filled out that form. So you're going into cloud code. And you're asking, hey, like, why is no one filling out this form?
- [13:41]
Kieran
Yeah, I was like, there has to be something. Like, this form was not sent. And I asked like, hey, 14 days ago, something went wrong. Can you see what went and what it did? It made a checklist to dos, like fetching recent log changes to the controller, searching the code base. So it looked through what changed around that date and it found we removed a piece of code that adds people there, which is here. Like it says, hey, actually you just need to add this. And I said, okay, do it for me. Create a pull request. And it did that. And I said, oh yeah, by the way, I'm also going to create a script that will then add everyone that we missed to it, migrate it, and that was it. And the fun part was, like, it didn't cost me any energy. Like it was as easy as me writing it down in GitHub to look at later. I don't need to. I just ask it and it does it immediately, which is really nice. It's like the Inbox zero. Does it take less than five minutes do it kind of thing?
- [14:50]
Dan Shipper
Yeah, I think the thing that people may not fully realize is that that's a thing. That task could take anywhere from like 30 minutes to a couple hours without, without AI. And it's not just that it would require you to like focus on it and like put aside time to like sit down and do it. And now you just sort of like send off requests like that and then you can send off another one and another one. You have a bunch of these like sort of working in parallel. So give me like a snapshot of, of what that looks like concretely, like what your actual workflow is. What are you actually doing? How many tabs do you have open? Like, are you actually doing any hand coding yourself? Do you have like five in parallel? Are you just using cloud code? Like, give, give, give us, Give me a sense of that.
- [15:34]
Kieran
Yeah, I'll show you my screen as well. Maybe nitesh you can, you can tell what we did before. Like when we got early access to Claude, like we were excited what we did. I'll show share my screen.
- [15:46]
Nitesh
Yeah, so this is like one day before the CLAUDE livestream was scheduled. We were like, okay, tomorrow coding is going to change. We'll have a much more capable model which will be able to one shot everything that we want. We're basically going to get like a coding genie for us. So the best, most productive thing for us to do today, instead of doing Our regular scheduled programming, we should just jam for a two hour call where we make a massive list of issues that we want the future, like tomorrow's superior model to solve. And we did that. We created like 20 issues in terms of what we want to fix. What were the things that we were planning to work on and prepared the system for the new cloud model.
- [16:37]
Kieran
Yeah, and it was funny because Natash had like, he prompted ChatGPT to say, hey, tomorrow we have a. We reached AGI. Can you, can you. Yeah, can you help us come up with everything we need to do and like prepare the AGI to solve everything? We did. And then we fed that into the prompt improver of anthropic and then we use that as a prompt and we created wait.
- [17:08]
Dan Shipper
Before you move on. Before you move on. So, so for people, for people who are listening. So basically you have this sort of trello board type thing inside of GitHub Kanban board and for each thing that you've identified as what you want to do, it looks like you have a document that lays out in detail. Okay, if it's a feature or it's like a bug fix or whatever, lays out in detail what it is and how to actually do it. Can you open up one one of them? Okay, so like a feature is you want to generate, you want to have AI generate synthetic data. And it has. This document has everything from a problem statement to like a solution vision to all the requirements and all the technical requirements and like a bunch of, a bunch of stuff. But it's. And even it has, seems like it has implementation steps with day counts and stuff like that.
- [17:54]
Kieran
So, which is funny. So this is one day is like one second.
- [17:58]
Dan Shipper
Okay.
- [17:59]
Kieran
Yeah, yeah, yeah. So this, we use close code and we have this custom prompt that we generated to create these because, like it's a lot of work to create these. And even with ChatGPT, like there's a lot of steps. You need to look at all the code. You need to think about it a lot. You have to think about it. Yeah, there's a lot of thinking, so it's really hard to do well. So what we did, we created a command in cloud code. A command is kind of a custom prompt that you use a lot. And ours is like, hey, there's a feature.
- [18:31]
Dan Shipper
This is a command in Claude code or a command in cursor, because you have cursor open.
- [18:35]
Kieran
Yeah, I have cursor open because that's how I edit files. But it's Claude code. So you can see we're In Claude and, and I can use this command by hitting ccy, which is cloth code. And then I say something like a problem, I have like a bug, a problem, anything. So it's very low friction. So I have this CCI command and Nitesh and I were just jamming. We're like, oh, what if we do this? Oh, that sounds cool. And then a voice to text and it starts. So let's see how this works. And then while it's running we can go over the thing. So I want Infinite scroll in Quora where if I am at the end of a brief, it should load a next brief and it should go until every brief that's unread is read.
- [19:29]
Dan Shipper
So like, so yeah, I just want people to understand, like Kieran almost never types anything and does all voice attacks. So he was just doing voice to text into his, his, into his terminal, into Claude code with I believe an internal, as of yet unreleased internal every incubation called Monologue, which he is the number four biggest user of. But still, still under wraps. But, but you know, a little preview in here coming soon. And basically what, what it seems like it's doing is it's, it's taking that. Is it turning, is it turning that into the, that document that we were looking at earlier or is it actually going and executing it?
- [20:11]
Kieran
Yeah, so what it does is it will insert whatever I said here in the feature description and then it will follow all these steps. And these steps are research, research best practices. So one is grounding itself in the code base, so researching what exists. Then it's researching best practices. So it's searching the web, finding open source patterns. So it's like grounding it in best practices in general. Then it's will present a plan and when I say yep, sounds good, like I like that. Review human in the loop for the plan because sometimes it does it wrong, but most of the time it's right. Then I say, yep, sounds good. And then it creates the GitHub issue and it will put it in the right lane and all that.
- [20:57]
Dan Shipper
Oh, interesting. So it's like that whole Kanban we were looking at in GitHub earlier. You've created a way for you. You speak your feature into Claude code and then it does all the research to create that long document and then just adds it into, into GitHub issues. That's really cool.
- [21:16]
Kieran
Yeah, it's, it's an important step because it's like this is different from cursor coding because in cursor normally you skip this step because the tool is not really made for it. Like, the tool's made to code. Yes, you can do create, you can create markdown files and all of that, but let's lean into like an issue tracker. It exists and it works well and people use it and it already hooks into existing like patterns. Like it, like we, we can give this to a developer and they can implement it.
- [21:50]
Dan Shipper
Yeah. And one of the things that just to point out is like, you're running this and I think one of the special things that when we saw Opus 4 for the first time, we were like, holy shit. Is that it just runs forever with, without any intervention and then gives you a pretty good result, which we've had sort of agentic type things for a little while, but it's just a way different level of autonomy and quality than we've ever had before. And it's like just checking things off of this to do list in a way that I think other agent loops are just going to be a lot less thorough.
- [22:26]
Kieran
Yes, absolutely.
- [22:27]
Nitesh
Me and Kieran have like a fun thing going on where we're trying to see who can have clock code running for the maximum amount of time. Kieran is stopping the list right now.
- [22:37]
Kieran
He 25 minutes.
- [22:39]
Nitesh
For 25 minutes. I'm only at eight minutes right now.
- [22:43]
Dan Shipper
Oh, man. How. How did you get it to go so long, Kieran?
- [22:49]
Kieran
A very, very long plan includes. Yeah, it's just very, very complicated, long plan and also include a lot of tests and just make sure that it runs all the tests and fixes all the tests and Interesting. It goes pretty long.
- [23:04]
Dan Shipper
Yeah, Take me through. Wait, I want to understand. How did you make that prompt? That. That creates the prompt, like the prompt that creates the research document. So like, how did you know which elements to put in? Did you just use? Did you just do the same thing where you use the Claude prompt improver or the anthropic prompt improver to make that or. Yeah. How did you think about putting that together?
- [23:27]
Kieran
Yeah, this is part of the compounding effect. It's like having an idea that has a lot of outcomes. So this was what Nitesh sent me. He said, we just got AGI, it got delivered and we can write software. This was your initial, initial prompt, which is kind of fun. Like, like, it's very dramatic. And then chatgpt said, I'm ready. Okay, so now do this. And I was like, okay, yeah, that's fine, that's fine. But like, do you know the anthropic console prompt improver? You're like, what is that. Well, anyone that doesn't know this is the. They change it. Is this console? Yeah, it is. Oh yeah, it is. Okay. They change it a little bit. But this is great because basically you paste in a prompt or something like that and you can say, yeah, we all have thinking and you click generate and it will improve the prompt automatically. And you think like, how good can it be? It's pretty good because it's also very low friction. So like it's very easy to just take a minute to see if something comes out, if it works. If it doesn't work, delete, it doesn't matter. We were just jamming and we were like, well, we're going to come up with 30 research tasks, so like we better have a prompt. So I just copied this prompt and.
- [24:54]
Dan Shipper
That became the document. Yeah.
- [24:55]
Kieran
Okay, into here and change the arguments and then you can trigger those in Claude by doing slash. And we have these two custom prompts here.
- [25:07]
Dan Shipper
And then I think that actually gives me a much better idea of like what you mean by compounding engineering. Because what it says to me is what you did first is spent time building a prompt that effectively builds other prompts because those research documents are effectively prompts for cloud code. And so now that you have a prompt that builds prompts, every time you want to make a new feature, you have to specify less. Like you just say the little feature and then it'll go do the research to like build it out into a big document versus before. Every single time you have to do a feature you have to say at first I want you to research it and then I want you to, to like think through all these like different corner cases or the ways that, you know, I like things built or whatever. I think that's so, that's so cool. And, and what's also really interesting to point out is it's working while we're, while we've been talking. And that's just a different way to code. Like, I, we were, you know, we were on the phone together like last week or the week before and we were testing this out together and I shipped a feature that went to prod while we were talking about, which I'm not in the code base at all, so it's like kind of crazy that that actually happened. And it's, it's like, it's a kind of more social way to code. Like we're coding right now building stuff, which was not possible before. Hey there, Dan here. I wanted to take a one minute break from the episode to tell you about our Latest sponsor. All right, let's play a game. What powerhouse productivity tool is also free for individual? Nope, not that one. Try again. You may not expect this, but it's Microsoft Teams. Yep, the same teams that big enterprises swear by also has a free plan for individuals. Whether you're jamming on a side project or bootstrapping a startup or building a community, Teams has all of the features that other platforms Nickel and Dyneouf are using. You can get unlimited chat, 60 minute video meetings, file sharing and collaborative workspaces, all for free. And the real magic is that everything is integrated in one seamless collaborative workspace. That means there's no more hopping between different applications for messages, meetings and file sharing. Teams puts it all at your fingertips to save you time and money. So ditch the app overload and the subscription fatigue and use teams to experience effortless collaboration. Today, are you ready to streamline your workflow? Head to AKA msevery to use teams for free. Your productivity will thank you and so will your wallet. This episode is brought to you by Addio, the AI native CRM built for the next era of companies with Addeo Setup takes minutes. Connect your email and calendar and it instantly builds a CRM that mirrors your business with every contact enriched and organized from the start. From there, Addio's AI goes to work. It gives you real time intelligence during calls, IT prospects leads with research agents, and it automates your team's most complex workflows. Industry leaders like Union Square, Ventures, Flatfile and Modal are already building the future of customer relationships on addio. Go to addio.com every and get 15% off your first year. That's a T T I o. And now back to the show.
- [28:26]
Kieran
Yeah, absolutely. And oh, so it's while we were talking, we did the research and we created this issue which is cool. And we had I think six or seven running at the same time because we were just like, new idea, let's go, new idea, let's go. And what we also did, we went through user feedback, we read emails, we just everything we could we gathered and we were just like brainstorming and it's really fun because if you're in this brainstorming place, you can just kick off agents and see what comes up, what they come up with and take another time to then review. So what we do also is yeah, like to agree with you on like it's really fun to do this together on a call because that's where magic happens. And there is still a human review step here because we Found that we want to look at it, see if it makes sense, if anything is missing. Like, this is having taste, experience, intuition. Like the bug I solved earlier with the email not going out. Natash did the same with his close code, but it didn't give the right answer. Yeah, so. So there is like, there is still like a human touch of intuition. Like I hinted at look at the history and that actually made it think into the right direction. And Natash didn't add look at the history. And then it said, no, everything works fine. So there is still like intuition and it's like.
- [30:03]
Dan Shipper
It's still a skill. It's still a skill.
- [30:05]
Kieran
It is a skill for sure. Yeah. It's not. Yeah, it's absolutely a skill. There's no magic prompt that does everything. Like it is about using it the right way and using it to its strengths. For sure.
- [30:18]
Dan Shipper
Yeah, yeah. Nitesh, how have you found this all? Because I know, you know, Kieran is like a long time, like Rails, like expert person who's just like an incredible programmer. And I think you're a little bit earlier in your programming journey. So what has that been like to come to every start working on Quora and start working on it in this way?
- [30:40]
Nitesh
Yeah, this has been incredibly eye opening, I would say, because honestly, my experience with programming is that two years ago when ChatGPT came out, I thought, okay, now it's perfect for me to teach myself programming and build that SaaS application that I always wanted to. So I taught Myself programming using ChatGPT from the very first day. So I have gone through all the transitions. I went from ChatGPT and then when cursor came out, I shifted the workflow to cursor. And then when Windsurf got better, we shifted into Windsurf. And I was always thinking like, okay, I am at the forefront. I don't know any of my friends who are doing so much with AI, and I'm at the forefront. Then I join Emery and start working with Kieran. And Kieran is at a whole other level.
- [31:29]
Kieran
In our meetings.
- [31:30]
Nitesh
He's never writing code, he's never typing, he's always speaking into the computer. And so I was like, okay, I need to log that into the workflow. And then even when Claude code came out, Kieran actually pushed me into using it. And clearly it is now the way to program. Like, me and Kieran, both of us, like, we haven't even touched Windsurf or Cursor in the last three weeks or so. Or even if we do touch it, it's usually just because we want to read. It's basically like we're using it because we don't have VS code on our computer. It wouldn't matter if it was VS code like the older VS code or cursor and stuff, because all the AI stuff is happening with cloud code now. And it's really fun to have be in this position where the entire coding landscape just changes completely every three months and you realize nobody's at the forefront.
- [32:33]
Dan Shipper
I gotta say, I'm jealous of you learning to code right when ChatGPT came out because I learned to code from books like 20 years ago.
- [32:40]
Kieran
BHP four for dummies.
- [32:43]
Dan Shipper
Yeah. Like basic for BA. Learn basic in 24 days. Like Sam's. Teach yourself basic or whatever. Yeah.
- [32:49]
Kieran
Delphi five.
- [32:51]
Dan Shipper
Yeah. And also it's so funny for you to say, like, I thought I was, I thought I was sort of at the, at the forefront of AI coding. And then I joined every and started working with Kieran because it just reminds me of, I don't know if like there's a scene in Star wars, the prequel Episode one, where like they're, they're under the water and, and they're like being attacked by a sea monster and it looks like they're gonna die and then another bigger sea monster comes out and just like eats the. Eats the one that's killing them. And Qui Gon is like, there's always a bigger fish. There's always a bigger fish. And yeah, Kieran is the bigger fish.
- [33:30]
Kieran
But I feel, I feel the same. Like, like you say that about me, but I'm like, I'm. I have no idea what I'm doing. Like, I need to, like, I'm running behind. We need to do like a million more things. So that's just the reality of the landscape. Like, there is always more, but it's really about practice. Like, you should practice using AI. You should push yourself every day. If you don't like, you'll miss very cool stuff.
- [33:56]
Dan Shipper
Yeah. Well, I guess I'm curious personally and also for people in the audience, what are the problems with this? Right, so basically it sounds like you're moving to a form of coding where you don't touch the code. You're one level above. And so what are the problems that come up with that and how are you solving them? What are the new engineering practices that you need to incorporate in order to make sure that things go well?
- [34:23]
Nitesh
For me, the most important realization for me has been this thing that I always keep going back to, especially with shortcode. I read this in that management of that high output management which the Intel CEO wrote like 50 years ago. And the first chapter he mentioned something like how in any production process you should fix any problem at the lowest value stage. And I just can't stop thinking about that statement. Because AI and plot code can now do so many things for us, it has become really important to focus on the earliest part of things. So what I mean by that is when we see that when we are using the workflow that Kiran just showed to create a GitHub, like a very detailed GitHub issue, then it's very tempting to start another cloud code to ask it to just, hey, go now, work on this GitHub issue and fix it. But that's actually going to be a problem because there are chances that the plan that Lord was able to give in that issue, it wasn't the direction that you wanted to go. And you want to catch that before you ask Claude to go and implement the solution and then you want to fix it over there.
- [35:47]
Dan Shipper
That makes perfect sense. I really, really like that idea. The thing it reminds me of is just, it's like all this stuff is like, it's like a lever and like, the further out you get on the lever, like the more power you have, but also the more power you have to go in the, in the wrong direction. Like every little inch makes a big difference at the end. And, and so trying to catch it earlier, I think is the thing that makes sure that you're not shooting off into space or this lever metaphor is totally breaking. But like, you know what I mean? Like if you, if you point a rocket at the moon, like one inch means thousands of miles of difference. And so I guess the same thing is true with AI stuff. And I think that's actually a good lesson for me because I, I tend to want to like, rush through the planning stuff. It's like, it's just hard for me to like look at a document like that. Like the thing that's that Claude is writing and concentrate on it. How have you guys found that?
- [36:49]
Kieran
Yeah, it's, it's kind of boring to read most of the time, but you can make it more fun. Like you can say just like, minimal, minimal. This is too much. Just. But, but then the thing is, then it misses things again. So it's actually important. So for code, I like it to focus on user stories or like asking questions and answering them. So let's say like, hey, what are some questions a good PM would ask about this? Like that we should Consider and give, like, two options. Like, that's. It's more fun to read that than, like, week one we'll do this, week two, we'll do that. Like, that's. It's like PRDs are boring and you can make them a little bit more fun or give more examples, or you can shape that research. And that's normally what we do in the human review step. It's like, do we see any red flags? Do we need more stuff to be added because it will save so much time.
- [37:52]
Dan Shipper
That actually reminds me of something that we're finding in another part of the business. So Danny, who's been on this show, is the GM of Spiral. And inside of Spiral, we're building a writing agent. So you can think of it sort of like hot code, but specifically for writing tasks. And I think there's something similar about that, where sometimes you want that writing agent to shift into like an interview mode, where it, like, tries to understand more about what. Who you are and what you want, rather than just like spitting out a bunch of stuff that you then you have to read through. And it sounds like there's maybe something missing here in Claude code or these sort of coding work, coding workflows where it would be really nice instead of having to read that, like, long document, it's finding ways to ask you questions so that the thing it outputs is more likely to be right without you having to read through the whole thing.
- [38:41]
Kieran
Yeah, absolutely.
- [38:42]
Nitesh
That's an interesting idea for a custom command. Click here. We should totally try that.
- [38:46]
Kieran
Yeah, for sure. Like, this is something we should automate and make better. Like, for sure. And at the same time, it knows a lot because it has access to your code base and your style. And like, that's very powerful. So like, like you have the code base and it's actually pretty good doing it. Like, I think in addition to, like, making it very good at the beginning, I think just boring. Traditional tests and evils are very important as well, because how do you know what you did is actually working? Well, you can open a console and click through it, but, like, why just have it, test it, write a test for it? Like, just, just the bare minimum smoke tests are great where you just see does. Does it kind of work? Because otherwise it does way too much. But it's a very good way to have it iterate and fix things by itself. And we haven't tried it as much yet, but we, we use the FIGMA MCP where we say, hey, implement this from figma. And then now there is like, you can have Puppeteer take a screenshot for a mobile version and then say, compare the two. Like, we haven't really tried it out, but, like, we want to try more of that out. So there are these checks in place, tests in place that you normally do manually, and the same for prompts, like evals for prompts. So I kind of think of an eval as like writing a test for code. An eval is a test for a prompt. And what I've seen last week as well, I had Claude Code run an eval and then say, actually, it fails four out of ten times. I said, run it ten times. Does it always pass? No, four times. It doesn't. I said, oh, look at the output. Like, why didn't it call that tool? It was a cool tool call test. And it says, oh, yeah, it wasn't specific enough. And I say, okay, just keep going and change the prompt until it's passing consistently all the time. And it did it. Like, I just walked downstairs, got a coffee, walked up, and that was it. So evils are also very powerful because they will tell you if a prompt works. And similar to writing code, a test says your code works. So leaning into those more boring, traditional ways is also very powerful.
- [41:20]
Dan Shipper
That makes sense. I have a thought, because one of the things I think is really special, and I think, nitesh, you're in this boat too, so tell me if I'm wrong. But one of the things I think is really special about you, Karen, is that you just test everything. So, like, you've tested every single agent, Nitesh. Have you? Have you used a lot of the agents as well?
- [41:41]
Nitesh
That's Karen.
- [41:42]
Dan Shipper
Okay, well, I think we could still do this. I think it'd be kind of fun. I want to spend five minutes with Kieran doing a S tier through F tier ranking of agents. And so what I'm gonna do is I'm gonna share my screen and I'm gonna. I'll call out an agent, and then you tell me where it ranks. Are you game?
- [42:07]
Kieran
Yeah, let's do it.
- [42:09]
Dan Shipper
Okay, cool. Let's see. Does. I guess let's do cursor. Yeah.
- [42:19]
Kieran
So it's fun because cursor, like what cursor Is it Claude 4? Is it Max?
- [42:25]
Dan Shipper
Is it cursor on the best possible settings?
- [42:29]
Kieran
And is it the background agent? Or is it the okay, cursor traditional best possible setting clause? Like, that's the confusing part about cursor. Windsurf. Like, there are, like a million versions of it. And, like, why don't you just have the best version. And that's what I love about certain agents. They just say, look, this is the best agent. So that's why it wouldn't be the best. I would say a.
- [42:57]
Dan Shipper
A. Okay. Cursor is.
- [42:58]
Kieran
Cursor is very good with CL4.
- [43:02]
Dan Shipper
All right. Wind Surf.
- [43:06]
Kieran
C. Because they don't have Claude 4. It's ridiculous because three. Three weeks ago they would. Would be A, and now they're not.
- [43:17]
Dan Shipper
Wow. Wow. Okay.
- [43:19]
Kieran
Because I switched from Windsurf to. Or from cursor to Windsurf like, a few months back, but I switched back.
- [43:31]
Dan Shipper
Okay, so we've got Windsurf is a. Windsurf is a C. Cursor is an A. Let's see. Devon.
- [43:43]
Kieran
It's a B.
- [43:45]
Dan Shipper
Why?
- [43:49]
Kieran
It's like, it's not as integrated. It's a little bit hard to set up. And the code quality is, like. It's not as well rounded as cursor or cloth code. I don't know if they use cloth for the background, but, like. Like, it's not as usable as the. The others.
- [44:11]
Dan Shipper
Charlie.
- [44:14]
Kieran
Charlie is like, for code reviews. So we use Charlie for code reviews mostly. So I haven't really used it as an agent as much. I think Charlie as an agent is B, but it's A. As a code reviewer. Like, I. I really like the code reviews. It does. So that's interesting. Like, it. It's really good at something.
- [44:37]
Dan Shipper
And then what about Friday?
- [44:40]
Kieran
I put Friday higher than cursor, maybe between S and A. And. And it's funny because they don't even use Cloth four yet. They're still, like. They're still working on how. How they, like, really make it work. Well, it's 3.7, but, like, why I like it there. It's definitely different than cloth code, but Friday has a very opinionated way of working, and I love their opinions. And it really works well. And it just does it. Like, you give an issue, they make a plan you approve, and it does creates a pull request. And I've seen it do this stuff that. That I couldn't do with cloth code. Like, for example, implement this FIGMA design. It just one shot at a FIGMA design for the assistant. And I've seen moments where more multiple moments like that where it did things where like, wow, okay, this. I taste the future, which is really unique. And it's a small team as well, so really cool thing.
- [45:51]
Dan Shipper
Codex.
- [45:53]
Kieran
This is B for me.
- [45:58]
Dan Shipper
All right. Codex is a B copilot.
- [46:04]
Kieran
I haven't used copilot.
- [46:08]
Dan Shipper
You never use GitHub Copilot.
- [46:10]
Kieran
No, I mean I used it three years ago, but I like. No, I. Okay, let's be fair. I tried it maybe a half a year ago and after one second I stopped using it.
- [46:22]
Nitesh
It.
- [46:25]
Dan Shipper
Where do you rank it?
- [46:28]
Kieran
D was not agentic, but I mean I should try the new version for sure.
- [46:34]
Dan Shipper
We have not tried. Yeah, we haven't tried the agentic copilot, so that's not totally fair, but. Okay. Are we missing anything? I feel like we are.
- [46:44]
Kieran
Claude.
- [46:45]
Dan Shipper
Claude, obviously Claude Code, but I assume S tier baby.
- [46:51]
Kieran
We have Factory as well.
- [46:54]
Dan Shipper
Oh yeah. Where do you rank Factory?
- [46:57]
Kieran
It's interesting. Factory with certain things is like better than any others, but it's not my style. It's like Factory is for more enterprisey people that are very nerdy and want like absolute bangers of code and it's actually good like multi repo stuff like that. It's a little bit hard to use because it's on the web but also local, so. So I rate it B, maybe a little bit below codecs and Devin. Yeah, but it's like there is a use for it for sure.
- [47:29]
Dan Shipper
There's something going. There's something good there, but it's maybe not for us.
- [47:32]
Kieran
It's not my thing. Yeah.
- [47:35]
Dan Shipper
Amp Also M. What's that?
- [47:38]
Kieran
Amp A M P amp oh.
- [47:40]
Dan Shipper
Amp yeah. Yeah.
- [47:43]
Kieran
I would put it S tier under cloth coat between. Whoa.
- [47:52]
Dan Shipper
Another S tier.
- [47:55]
Kieran
Yes.
- [47:56]
Dan Shipper
All right.
- [47:56]
Kieran
Why? It's very good at just getting work done. The ergonomics are pretty good. Good tools already. Like. Like people. People use that tool that build it. They're dog fooding. Like you can feel from Claude code and amp There are developers that love agents and they're just building the best thing and they're trying new things. So. Yeah, that's it. Let's see.
- [48:26]
Nitesh
This is exactly why Kieran is the big fish.
- [48:32]
Kieran
I mean you are.
- [48:33]
Dan Shipper
Yeah. You're stringing them together. Like they're using Claude code and Friday and other stuff all at the same time. Which is. Yeah. The thing that is really cool.
- [48:41]
Kieran
Yeah. Like. Like there are. Yeah. Like how I think about, like I'm thinking about it more is like you're interviewing for a role and you find a developer, solve a certain problem. I think it's similar with coding agents. Like Friday is good at like doing UI now. So if I need UI work, I will go to Friday. If I need to do research, I go to Claude Code. And yeah, there is a. If I want a code review, I use Charlie like it's. It's fun. And agents work together. You don't need to have one agent. We have Claude code and that's because.
- [49:15]
Dan Shipper
Charlie like works in GitHub, so you can just like CC Charlie and Charlie will do the code review on the print.
- [49:21]
Kieran
Yeah, so we use GitHub and pull requests and normal developer flows so humans can hook in. So we can hire someone that's very good at specific thing and review code and then Claude code will just do the work. But it's very powerful because it is just an ecosystem that we refined over like 20 years or whatever. Like, and it works. So let's lean into that. And that's probably why Copilot will probably be fine since it's in there already. Yeah.
- [49:54]
Dan Shipper
Wait, you actually did that recently? Like we had some infrastructure things where, you know, we've handled tons and tons and tons of emails at Quora, so we had some infrastructure issues to work out. And you seems, I think you brought in someone who's like a real expert and then worked with them in a specific agentic way that you got what you needed from them, but it was less work for them.
- [50:15]
Kieran
Yeah, yeah. So like, like there was no issue yet, but we wanted more visibility in, in delivery of like the most important things. And like I'm not very good at it or like I know stuff, but like let's bring in someone. And what we did, we just had a conversation, like two hour call and I recorded everything and at the end I just fed that into Claude and say, okay, can you make two issues, research issues from this? And like 10 minutes later I said, okay, here are the issues, can you review them? And he was like, holy, what? Like this guy, like he's not an AI skeptic, but he's like, he's very good at what he does and normally what he does AI is not good at yet because like there are things AI is not as good at yet. But he was very impressed with it and he had like very good like comments on it to iterate over it. And like what we basically did, we just iterated more quickly through ideas because we had something to talk about. And then I said the next day when we, he was like, did the human review, like, let's go. I just use Claude code to implement it. And we sat down and did the code review. So it's like, it's just accelerated. What would have taken two weeks maybe is now in like a few hours, which is really cool.
- [51:37]
Dan Shipper
I love it. Well, there you have it. You've got your tier list of agents Claude Code takes the cake. We've got Amp coming up coming up in second, and GitHub Copilot, unfortunately, bringing up the rear, but with room for improvement once we try out their agentic capabilities. Anything else you guys want to say or talk about before we end today?
- [52:01]
Kieran
Everyone should use Claude code or try it out. Even if you're not technical, subscribe for their Max or Pro plan. It's only $100 per month. You have unlimited access. If you're skeptical about being technical, that it's very easy. And I've seen people. A friend of mine, he used cursor and I said, just use cloth code. It's better. Like, how much better can it be? And he said, yes, it's better. And he rebuilt everything he did with Cursor Vibe coded into cloth code. And he's like, yeah, this is great. Great. He felt that next step. And you should. Everyone should try it and really push. Push their tools.
- [52:40]
Dan Shipper
Yeah, Nitesh. Any other words of wisdom?
- [52:45]
Nitesh
Just be sure to check the AI's work at the lowest value stage. You want to catch those problems early.
- [52:53]
Dan Shipper
Yeah, that's a great one. And also use Quora Quora computer. Check it out. It's pretty awesome. We're shipping new things all the time. Thank you both for coming on. This is a true pleasure. I cannot wait to see what else you cook up over the next couple months, and we'll talk soon.
- [53:09]
Kieran
Thank you.
- [53:10]
Nitesh
Thank you so much.
- [53:19]
Podcast Host/Announcer
Oh, my gosh, folks, you absolutely, positively have to smash that, like, button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure, unadulterated knowledge bombs. About ChatGPT Every episode is a rollercoaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor. Hit, like, smash, subscribe, and strap in for the ride of your life. Life. And now, without any further ado, let me just say, Dan, I'm absolutely, hopelessly in love with you.