Loading summary
Transcript29 lines
- [00:00]
Dan Shipper
Humans are the bread in the sandwich, and the AI is in the middle.
- [00:04]
Kieran
AI is whatever you put on your sandwich. If you ship something or do something, if you want it to be your own. You cannot fully automate everything. It's like art. If you want it your own, it needs to be from you or somehow be connected. So I believe it's so important to do things you enjoy and you love, and it's very important to make it feel. Feel great. Because the bar is high, the bar will always get higher. The beginning and the end, the middles can be automated pretty well. And Trevin at some point said, oh, it's kind of like a sandwich, which was like, very funny.
- [00:55]
Dan Shipper
Kieran, welcome to the show.
- [00:57]
Kieran
Hello, Dan. Happy to be here.
- [01:00]
Dan Shipper
So, for people who don't know, you are the GM of Quora, and you are also the creator of compound Engineering, the engineering framework and plugin that everyone inside of ever uses, and everyone who's really coding in with agents is at least aware of, if not using. And so a pleasure to have you on the show. Thank you.
- [01:21]
Kieran
Yeah, it's always great.
- [01:23]
Dan Shipper
So I love getting to chat with you and getting to work with you, because every once in a while you have a thing that you do or you figure out that I'm like, holy shit, that's definitely the future. And you just figured something out, along with Trevin Chow, who also helps out on compound engineering. And I think it has massive implications for how programming works. And then I think we can also translate that to the rest of AI and its impact on work. And one of the things you've been doing, so you have this compound engineering plugin that you've rebuilt the engineering workflow for how you should work with agents. And in thinking about that and thinking about where a human is used and where a human should not be present inside of that process, I think you've found something really interesting and deep about, in general, how humans and AI are going to interact with work. So do you want to explain a little bit about compound engineering and the process that you've created, and then also explain this insight about where humans fit?
- [02:31]
Kieran
Yeah, absolutely. So compound engineering is a philosophy of doing engineering work, but we realize it applies to more than just engineering work. It's product work as well. It's design work. It could be knowledge work, it could be other things. But how I build it is. While building Quora, I had AI and I was like, how can I use AI to do better work more quickly? And the initial version of Compound Engineering really evolved around four Steps, which is planning. First you make a great plan, so it's very clear what you need to build and do. Then the work part where the agent does the work and implements it and actually writes the code, does the design work, or whatever work needs to be done. The third is review. So slop comes out or whatever you call it, something beautiful comes out. One of the two, like, something comes out, but how do you know it's good? And traditionally this is like a code review or like a PR that you review and see, like, hey, this can be improved and there's some iteration going on there. And then the most important step is the compound step, which is if anything comes up during that review or during the planning that you think like, oh, this is a good learning, probably we'll run into this again. You can compound that knowledge back into the system and we store that as knowledge inside the repository. And agents, next time when they go into planning or when they go into work or review, they can see the mistakes they made before, so they won't make it the next time. And that's really the power. Like, that's by far the most powerful thing that is in this plugin. But we start to realize more, like, first of all, the work phase is kind of done. Like, it works if you have a good plan, it does the work, and it's pretty good. And then the review makes it a little bit better.
- [04:37]
Dan Shipper
Yeah. And by that you mean, like, having an entire phase dedicated to work in this whole system doesn't necessarily make that much sense when all that really means is run the model, let the model do the thing.
- [04:49]
Podcast Host/Promoter
Yeah.
- [04:50]
Kieran
So there needs to be a step. But what I mean by done is I don't need to care or like, I don't need to think about it. I trust it. And this is not like, trust me, bro, just works. But this is like, I seen if you put in a good plan, like, it does the plan, like, it executes on the plan. LLMs are very good at just following steps, doing deep work, like working for hours, days even now, and that thing is kind of solved, and the review starts to get there too, and the planning starts to get there too. And then there's this next step is like, okay, so if all these things work, where, where do I have to do anything because, like, automate my job? Yeah, yeah. Did I automate myself out of a job? If everything works, like, where do I work? What is still the bottleneck? And there are two things we started to know. Like, Trevin, he's a very, very Great contributor to the Compote engineering plugin. Like, he is a product person and he is like, I need more on the product side, which is like before the planning phase. So he added first a brainst and an ideate step. And the ideate step is like really going wide. Like, it's like, okay, let's come up with ideas in a room full of interesting people with angles. Brainstorm is more like, I have a problem, but I don't really understand exactly what and how. So it's very much brainstorming with you around the problem. And the first thing we notice there is like, the top is very important to be super well in the loop with a human and really ask a lot of questions and really think hard. Like, the human should think hard, the LLM should support the human. But then after that, the planning phase. If you have a good brainstorm an idea, what problems you solve, it can create a very good plan. And the human needs not to be in the loop. So that's the first realization where it's like, oh, hey, here's good to be in the loop versus not to be in the loop. And you can see other like spec driven development, for example, or other ways to do things. They assume that it's always good to have people in the loop. And I disagree. I think it's very important to know when to be in the loop versus when to hand it off, because that means we can think harder at the moments where we need to think harder. And that's the first one. So the other one comes at the end. So like something comes out. How do you validate it's good? Well, it's already tested because we have browser automated testing. It clicks through. All the requirements are very clearly specified and it says, yeah, everything works, but the beauty comes in. When a human looks at it, clicks around and has a feel like, this doesn't feel good. We can polish it even more, we can make it even better, we can increase or like, we can do something that's still missing or make it more beautiful, make the design better. And this is something I've learned from doing Pomodoros, where ideally, if you do Pomodoro's, the old school way is like you start with a task and if you finish after 15 minutes, you have 10 more minutes to work on the same task. You cannot switch tasks. And sometimes in that space something beautiful happens because you will go deeper, you will go further that you would do. And I think this is the other moment which is all the way at the end when Everything is done where you can just elevate everything and make it even better. And I think that's also what we need to do because if we don't do it, it will be all slop all the same. And it's very important to make it feel great. Because the bar is high, the bar will always get higher. So this is kind of what we realized, like the beginning and the end and the middle is kind of SOL can be automated pretty well. And Drevin at some point said, oh, it's kind of like a sandwich. Which was like very funny. And then is now referring to the AI sandwich, which I think is very cool. And I think the sandwich here is like when do you need to think about what you do and really use your brain versus offload it to the LLM.
- [09:17]
Dan Shipper
We've all been there. You're sitting in an important meeting and you're trying to pay attention, you're trying to stay present, but you have this lingering underlying anxiety that you're going to forget everything, that you're going to miss the important detail. Forget the decision, forget the action item. Let something important slip through the cracks. That's why I love granola. It's an AI powered notepad that works in the background while you're in your meetings. It takes notes on everything that gets said, transcribes action items and helps get rid of that feeling. You don't have to worry about whether you're going to miss something because granola has you covered and now lets you stay present in meetings. I've been using granola for a long time, almost since they came out, and it's amazing for this, it doesn't join the meeting like some of those other clunky meeting note takers. The UI is really fast and well considered and it feels like it's sort of just transcribing all the important moments in my work life. And that gives me the confidence to get great work done. And what's even cooler is you can chat with your notes afterwards. You can run detailed research reports on how your week was, how you act as a leader, how you performed in particular difficult conversations, and how you can do better. It's really a power tool for anyone who cares about their meetings and also cares about how they show up in those meetings. It also has these things called recipes which are pre made prompts for common tasks like negotiating, coaching or summarizing. I even have a recipe that I made that's in granola that you should check out. Once you try it on one meeting, it's really, really hard to go back. The notes are always better than what you could do manually and it helps me be much more present. Instead of frantically typing all the time. Head to granola AI every for three months free with the code every e v e r y that's granola AI every for 3 months free. And now back to the episode. Humans are the are the sandwich. The bread in the sandwich and the AI is in the middle.
- [11:02]
Kieran
Yeah, the AI is whatever you put on your sandwich.
- [11:05]
Dan Shipper
Yeah, exactly. And I think that's really interesting and really cool because A it gives me a good mental model for how I should be working with coding agents. But I think that also applies to the rest of knowledge work. And I think this is such an important question now because we have all these questions about oh my God, like what are agents going to do and is everyone going to lose their job and all that kind of stuff. And I think software engineers are a little bit of the canary in the coal mine. And so far what we found internally at every is we're absolutely not like we still hire software engineers. We need software engineers. But the way that you're working and what you're doing looks a lot more like managing. If you're doing it well, you're still involved, but you're involved at the beginning, in the end as sort of this sandwich. And I think the same is going to be true of every other kind of work, whether that's copywriting or strategy or design. And I think there are deep reasons why that is the case that I think will be interesting to talk about. And I want to start with an objection that I think people will have which is like, okay, like for now agents can't do the idea in the brainstorm, but pretty soon they will. So then what, then what happens? There are now they're starting to do the beginning of the, of that process. And I think that there's something interesting here where if you look within any given local frame of a problem. So to take a non coding example, the problem might be my knee hurts and I want to solve that problem. But you can say my knee hurts is the same as this feature is broken or customers are anxious about this part of the product or whatever, any problem. If you take that frame and you say, okay, well the solution is maybe for your, if you're, if we're talking a knee hurting thing, the solution is take Advil. Any part of that process, you know, getting to the store or whatever, can be automated. Let's say doordash can go do It. But there's always, even once you've solved it in that way, there's always a larger frame within which to think about the problem. So an example is if your knee hurts, you might need to stretch your IT band or you might need to stop running on hard surfaces every day. And each one of these is sort of addressing the same problem at a different level of the stack at a different from a different frame. And humans are very good at flipping and changing frames like that. And our job is to set the frame or set the bounds within which we solve the problem. And I think it's going to be very, very hard for agents to do that well by themselves. And there are deep reasons for that. But does that. Do you get that? I know it's like a little bit hand wavy and the knee hurting thing is a little hard to understand, but does that resonate for you?
- [14:19]
Kieran
Yeah, for sure. Yeah. It's like this all comes down to building an environment where the agent will thrive. And you do that by picking the right things. And this is why it's so important to have humans with experience and humans with taste and humans that just want to click around and say this is shit or this is great and why it is shit or great. And I think it's similar to the Advil example. If you keep doing that, it's probably your friend will say, yo, that's messed up. Just go like, go fix the problem instead of denying the problem. And maybe it will work for you for a little while, but you need someone to shake you up. And in that case that's the human or that's the other person. But I do think it will also be more automated. Also the ideation you can say, okay, let's have a Persona of 100 people and run simulations of how they think and how they behave. And clearly we're going there too, where we run simulations of millions of people, see how things work, and probably you'll learn something from that and there will be more automation and maybe even that step in the front will be fully automated. But I do think in the end, if you ship something or do something, or make like a statement in the world, if you want it to be your own, which you need to say yes or no at some point, you cannot fully automate everything. Like it's maybe a little bit like art, like making art. Like if you want it your own, you need to just. It needs to be from. Be from you or somehow be connected. So I believe having those moments where you decide, this is what I just Enjoy. And that's why it's so important to do things you enjoy and you love are very important.
- [16:32]
Dan Shipper
Yeah, I agree. And I think, yeah, you can imagine it being like, okay, yeah, we're going to simulate a bazillion people and then we're going to make decisions based on what we think they would do. But that would still only cover a small set of the decisions that some someone might make.
- [16:49]
Kieran
It will never be fully. It's a moving target. Like, we always get something new. And then again, there's a layer up that we then can even make bigger
- [17:00]
Dan Shipper
impact on, especially because. Especially for a lot of these decisions. The feedback loops on these decisions, like the data is really rare. You may only get a couple moments in your career where you gather the data that helps you decide about a particular thing. And that's very hard to get into language models, especially because it's hard to get. And they need a lot of it. And so that sort of rare expertise that is encapsulated in an expert who has a personality and a worldview is hard to get. And you're right, it's also always moving. And I don't know, that makes me very excited about this stuff because I feel like we've been wandering in the woods for a long time on, okay, what is AI progress going to mean? And how are humans going to be involved and all that kind of stuff. And it just feels very much to me like the simple answer is ride the bottle. Or to mix the metaphor, be this, be this bread in the sandwich. And if you do that, you're going to be fine. It's going to be like really, really, really, really great.
- [18:05]
Kieran
Yeah, I agree. And it will be different for different people because, yeah, you need to change some things. Like you cannot keep doing what you're doing because if writing code only, you need to find your way of writing code. Like, yes, you can write code, but maybe it's about beautiful code and maybe you find also lots of value in just seeing beautiful codes. Like someone looks at the UI and says, oh, this is beautiful, this works great. Maybe you want that for code. Some people don't care about that, but they're like, oh, but the UI should feel great and just really polish it. Go extra, like wherever you feel joy. But also it's way more product focused. So as an engineer, you're going to become either more of a manager, but also more of a product person. So it's, I think, like a product manager, product engineer. Like, it's, it's more of those things as well. So there will be some changes, but lean into making beautiful stuff and whatever that means to you. That can mean beautiful code, beautiful abstractions, beautiful architecture, beautiful design, beautiful copy. I think it's very important to lean into what is beautiful to you because then you will find a way to utilize an LLM to make something that gives you energy instead of drains you all the way.
- [19:40]
Narrator/Advertiser
It may not look like it, but Naveen is a dictator. You can speak faster than you can type, so dictators choose to do so whenever possible, while those confined to keyboards deal with finger cramps and input lag Voice allows dictators to convey ideas as naturally as they sound in their heads. And in the future, as AI tools improve, we will see a rise of dictators around the world. More and more dictators are choosing monolog from every it learns, transcribes and translates across different disciplines and languages, adjusting its output format to match your context, allowing you to stay in flow. Be a dictator. An idea by every every the only subscription you need to stay at the edge of AI.
- [20:32]
Dan Shipper
Yeah, and I think there's a deep reason why language models are not going to be as good at that. There's one deep reason, which is it's just not going to be yours if you didn't decide it, if you didn't do it. But another deep reason is you can think of language models as being a superintelligence that has been kept in a box for the last year and has no idea of what's going on in the world except for whatever it gets right when it pops out of the box. And because of that, it ends up being. Its outputs end up being a little bit more generic and less personal to you and your situation. And you can see this in all the stuff that's like, okay, yeah, all the AI writing that's like it's X not Y or, you know, all that kind of stuff, it's just going to do all that. And to truly solve a problem well, or to truly make art, or to truly make a product that resonates with people, it's going to have to be really well tuned to the exact problem that you're trying to solve or the exact form that you're trying to make. And language models need a lot of help to get there. And that's why you have to be on either end of them to set the frame of the problem and then make sure the details are really right at the level of execution at the end. And I don't think, I think that they will get better at doing this. But I actually think they're much further than we think they are from being able to do it all end to end. My, my, my, my general bar for AGI is whenever it is economically profitable or makes economic sense to run an agent 247 where it never turns off and OpenClaw is like pushing in this direction but it doesn't run 24. 7. It runs on a schedule, it has a heartbeat. But it's not like you just say hey openclaw, just go and just do a bunch of stuff and just work all the time, spend tokens all the time on stuff and it's worthwhile. We're not even close to that. And yes, we sometimes have well specified tasks that we can send a model off to go for 24 hours on. But again it's not changing frames, it's not finishing the task and be like cool. Now I'm going to pick the next one and that's going to take five minutes and the next one I'm going to spend four days on it. It's like it's, we're not even close to that and I think we're going to need some fundamental changes to language model architecture to like let them learn better, for them to get to a point where they're, they're, they're running 247 and I think that will, if they are running 247 like that, they'll be a lot closer to. I'm sensitive enough to context to actually do interesting creative things, but we're not there yet.
- [23:19]
Kieran
Yeah, I agree. One other way to look at it. So I have a music background. I studied classical composition and I think one of the beautiful things about music is yes, Suno can create songs, but it will never capture a live performance or coming up with the melody. And it's something internally in the human. Like as a composer or a musician, if you perform something and you deliver this to other people that they, they feel that like it will not be like, like sure, if you're a dj, it's, it's maybe somewhere in the middle but like there is something like performing. Like you see something, you express something. And I think there is some of that element in these steps as well where you see something and you're like, oh, it feels a little bit off here because I don't know why but I wanted to change it a little bit with the step at the end and suddenly you're like kind of performing or iterating or you're making stuff, you're putting something in the world. And I feel that's special. Like, practicing a piece for, like, playing it a hundred times is not very creative as a musician. And this is kind of the middle part. But at the end, the performance is where you bring it out into the world to the people. So I think that's a special moment. And there is a little bit of a link for me with doing this polish step at the end and at the start is maybe coming up with a piece, like, if you're a composer, like, coming up with something out of nothing. And this is also a special moment. And normally everything in the middle is kind of boring. It's just work. And I feel these moments are still special. And it kind of works for making software or other things with LLMs as well.
- [25:18]
Dan Shipper
For me, I think that's totally right. I love this art angle that you have. And another way to say this is all the work exists on the spectrum from it being totally rote to it being art. And art itself has many tasks within it. Any kind of creative work has many tasks within it that are more rote or less rote. And if you're trying to map work on that spectrum, the stuff that is more rote is just going to be stuff that you're not going to have to do anymore. And that is a big opportunity to move a lot of the work that we do to the more creative, to us, probably more interesting parts of work, and to recognize that that frame is always changing or is always moving. So as. As certain things get wrote, other things become things that humans start to do. And, yes, those will get automated too, but, like, we're going to also keep keep moving down along that spectrum. And the final thing that's not automatable is, like, art made by humans who feel something. And I think that's beautiful.
- [26:31]
Kieran
Yeah, it's still scary because what if you're in the middle and you want to move, or if you want to figure out what that is to you? Because this might sound very abstract and weird to some people if you're not an artist or haven't, like, really, like, felt this in moments. Like, it sounds maybe a little bit like, oh, but like, that's not me. But I do believe everyone has this, like, think of it, like, what brings you joy, like, what. What lights a fire in you? Like, what do you get excited about? Like, I think that thing you should, like, lean into, like, whatever that is. And that can be beautiful writing or that can be very structured lists or whatever it is. Like, anything that just brings you happiness. Like, you should do more of that using LLMs in your work, because that's good.
- [27:30]
Dan Shipper
I agree. Kieran, always a pleasure.
- [27:34]
Kieran
Thank you. Yeah, let's see where this goes.
- [27:37]
Dan Shipper
See you next time.
- [27:38]
Kieran
See you. Bye.
- [27:47]
Podcast Host/Promoter
Oh, my gosh, folks, you absolutely, positively have to smash that, like, button, button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure, unadulterated knowledge bombs about chat GPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat craving for more. It's not just a show, it's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor. Hit like smash, subscribe, and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely, hopelessly in love with you.