Loading summary
Transcript26 lines
- [00:00]
A
Everyone is talking about agentic loops, but the reality is most people don't know what it is or how to use them. In this app, I brought on Professor Ross Mike to clearly explain what it is. Is it hype, Is it real? And how to use it. And if you stick around to the end of the episode, he shows me the most concrete use case of agentic loops that you can use. Starting today. Enjoy the episode. Ross Mike, welcome to the pod. By the end of this episode, what are people gonna learn?
- [00:42]
B
You're gonna understand what a loop is. You're gonna understand why people are fanning out about it, and you're gonna understand why it is a terrible mistake. And unless you have money to burn that, you are not to do it. I'm also going to play the other side and I'm going to show you a loop that I use and. But the general consensus I think is wrong. And we're going to talk about it, okay?
- [01:00]
A
So by the end of the episode, people are going to understand what an agentic loop is, why the most well known people in the AI industry are obsessed about it. You're going to keep it real with what we need to know about it and what we can avoid, and you're going to show a real use case, a real example of how to actually use an agentic loop.
- [01:20]
B
Exactly. Exactly.
- [01:22]
A
All right, bro.
- [01:22]
B
All right, let's get into it. So as always, a lot of people love the diagram, so we're just going to start with diagrams. I paid like, like I think $3 to get these stick figures, so I hope people appreciate them. This is me and you, right? This is your average Joe schmo who does not work at anthropic or API or OpenAI. And this is Boris and Peter and anyone else who has unlimited access to models. Now, the way me and you have been working, this is what is called a human in the loop is you. And I will prompt our, you know, computer, right? Let's say this is our computer. Or better yet, I'll say this is our AI agent, right? Whether you're using Cursor, Claude Codex, doesn't matter. You are prompting it yourself, right? You're telling it, hey, build me this landing page. You know, build this feature X, Y and Z. You are communicating with an AI agent, a platform of your choice via a prompt. And then a result is generated, right? A result is generated. And usually what you and I will do is we will view this result, we will test this result, and we will keep on iterating. This is the loop where it goes back to us, right? So let's say I'm working on an app and this app is a to do list app. Greg. The first thing that I'll probably want to do is I want to build up the landing page because I want to get this out to the public so maybe they can sign up and join the wait list. So I'll prompting build me a landing page and let's say I like the landing page. Next I'll work on authentication and then once I'm happy with authentication, then I'll work on with the backend. So this is where what we are used to and this to be sharp with it is called Human in the Loop, meaning it is the agent that's building, but it is you that is directing, governing and allowing things to happen. What everyone has been talking about, particularly Boris and Peter, they said they don't write prompts, they generate, they build loops. And essentially what they're talking about is the. They're building a system. And I'm just going to show you here where this is the AI agent, right? And then this is the result. But instead of a human being in the loop, the human is in the loop one time, meaning it fires off said loop, but then the rest of the time it's the agent checking, it's the agent generating a result. The that result is then fed back into the agent. The agent then looks at the result and continues to work. Now, this in theory sounds cool because what essentially we're saying is, hey, I'm just going to have some sort of spec MD file or some PRD MD or whatever, dot MD file. And this is going to be like a to do list, a task list. And this is going to give all of the information the AI agent needs to build this. Now, this sounds cool, and this Loki might be the future, but here is where it goes terribly wrong. First and foremost, I want to paint this analogy.
- [04:36]
A
Let's say quick break to talk about today's sponsor, Code Rabbit. We talk about agentic loops in this episode, and most of them are still hype. Code Rabbit is one of the few that actually works. My team literally uses it every day. You open a pull request and Code Rabbit reviews it automatically. It catches bugs, it suggests fixes, and it's one click to commit. It learns your team's standards over time. So it just gets better and better. They just shipped Code Rabbit review, which takes a big messy pull request and reorganizes it so you can actually understand what's changed and why in minutes. Instead of Hours. Something I've wanted for years. My team at LCA and Idea Browser love it. There's a 14 day free trial. If you want to learn more about Code Rabbit, go to the link in the description.
- [05:29]
B
We. We're. We're building a startup, right? You and me, Greg, we're building a startup. We hire a very smart developer and we tell this developer, this is the app we want to build. These are the things that it needs. And the developer goes on and builds the entire thing without consulting us. In building that entire thing, that developer is going to have to make assumptions, right? Assumptions of how the product looks, how it's going to feel, certain architectural decisions. There's a lot of assumptions that are going to be made in the nitty gritty. Now, you might think your plan document covers everything, but truth to the matter, it never does. There's always an edge case, there's always something that's missed. So what the developer is going to do is that developer is going to make a lot of assumptions. Those assumptions might not be aligned with our product vision. Now you have a developer who's come with a finished product and now there's a bunch of things in order, but it's not the way we want it. In the same way, when you have this stacked PRD MD file or this whatever markdown file you have, and you give it to agent and you run it in this loop, meaning it takes the feedback, it takes the result, takes it as feedback and continues to generate code, what happens is you now have an agent that's going to make assumptions. And believe me, when you give the agent the floor to give assumptions, most of the time it's going to get it wrong. But not only is it going to get it wrong, it's going to burn a lot of money. Now I say this with all love, but Boris and Peter come from a place where they have no token budgets, right? They can burn unlimited tokens. If I had unlimited tokens, I'd be doing the same thing too. But this is not productive. This is great for research and I'll actually share a loop that I use. But this idea of, of construct, of constructing like a meta harness where you give the agent feedback automatically. Like it gets like the information, the result it's generated and it loops on is a catastrophe. And we've tried this, right? We had RALPH Loops, we had RALPH Wiggum, there's even like slash goal which has been pretty popular the last couple of weeks. These are great to build prototypes, these are great to experiment with. Like, let's say you wanted to experiment with something you want to like some minuscule tool built out, but you didn't care about the nitty gritty details. These are great. But if you're, if you care about the details and you don't have tokens to burn, this is the worst thing to be trending right now, in my humble opinion. I'll stop right here just to make sure. Craig, I'm making sense because as you can see, I'm pretty passionate about this.
- [08:00]
A
So slash goal is also trending at the moment. You know, is slash goal a loop? Like, how should people think about slash goal and a loop?
- [08:09]
B
They're all the same thing. They have different names like slash goal. I know on cursor I think it's slash loop, and then on another tool it's slash, whatever. They're all the same thing. And basically how all of them work on a high level is you, you, you know, you type in slash goal and then you give it some prompt, right? Like, you give it some prompt and then you can also, like attach some, you know, markdown file and you tell it like, yeah, build this entire thing out. Don't stop until you're done. Don't make any mistakes. Again, these are cool, but the two issues are, number one, they burn a lot of tokens, right? And if you are not like, this shouldn't even be a thought. If you're not on the $200 a month plan. Like, not a thought. Like if you're on the $20 or I think there's $100 a month plan, you shouldn't even think about this, right? Because it's just going to burn your token usage. Number two, you think your plan is good, but it's not. Because it's impossible for you as a human to contextualize every single detail about the product that you want in one document, right? Things evolve, trends change. You know, one day liquid glass is cool, the next day we're changing how liquid we want it. Like, it's very impossible for you to feel like your thoughts and exactly how you want the product to be. One in one, in one document. If anyone works in service, whether you, you run an agency or you like, for example, we develop software for other, for other people and companies. It I, we try all the time to get all the thoughts out of someone's head. There's always something. There's always, oh, you missed this, or I wanted it like this. This is what I meant. How much more do you think an AI agent is going to understand you if we as humans have hard times understanding each other. Other, right? So this should only be used in the following experimentation. Like, let me do a new line experimentation, right? Let's say you wanted to, like, I'll share with you a fun, cool little tool I built. The other day, Greg, I was doing a talk and I wanted to build an among us simulator for, for AI models.
- [10:12]
A
Right.
- [10:13]
B
Basically, it's this game where there's one bad guy, one imposter, and everyone's trying to figure out who it is. And, and I wanted to have like, my own benchmark to find out which models, like, are good at lying. And I didn't want to. I didn't care about the details, I didn't care about how it looked. I just wanted the simulation and the benchmark to work. So I told it, I want the simulation, I want this benchmark, I want it to do this. Go and do it. It took about, I think an hour and a half and it got it done. Now, there were a lot of details that I had in mind that it completely got wrong that I didn't specify in the initial instruction set. But guess what? Because I didn't care about the. What it built, in a sense, like, I didn't really care for the details. It was a great thing. I didn't spend a lot of time. I just got slash goal to take care of everything. But when you and I are trying to use AI to build something meaningful, I 100%, 100%, 100% stand in the fact that the human still needs to be in the loop. AI can replicate sauce, it can't create sauce. So if I just have these giant loops running and then once they're done, maybe I'll go in and fix things up. Sure, you can make that argument, but I hope you have money to burn, right? Like, and that's the ultimate thing. This will burn money. It sounds cool, but it'll burn money.
- [11:33]
A
What I'm hearing you say is that the loops are going to create a slot machine, slop machine.
- [11:42]
B
That's basically what it is. Now, I have no doubt the Boris and the Peters are building very sophisticated, like, loops. Like, I can imagine, like, let me drag this over to here. I can almost imagine they have something like, again, I'm not sure this is just me guessing, but I can imagine they almost have like some sort of test suite, right, where like, they write tests for the agent to run the code against. So it's a certain type of quality. I'm sure they also have some sort of browser, browser use capability. So the agent can see the page live and can take screenshots. Like, I'm sure they have an insane harness or meta harness around the agent so that this loop can be more successful than the average loop. But at the end of the day, the one argument I'll fight back with is this is going to burn a lot of tokens. And if you don't believe me, all, all you have to do is look at Peter's Tweet where in one month he burnt $1.3 million worth of tokens. But I don't want to sound like a, you know, like, I don't know what the word is. Like a doomer. Like, oh, like, like an old guy. Like, these suck. There are use cases and I'll share one, Greg, if that's okay. Where my. My code review process is a loop and I'll explain how it works. So I use cursor for the most part. Not sponsored I use cursor for the most part as my harness of choice. And with cursor, I will use GitHub as my source control. Basically a place where I store code, version code and all that stuff. And every time I push a feature, like every time I build a feature, I push a feature or whatever the case may be, I am pushing code to GitHub. And in GitHub I have a code review agent installed. There's many kinds. My particular one that I use is graph tile. But I know people use code rabbit, macroscope, they're all great. I use graph tile. And what happens is whenever I push a feature to GitHub, the code that's being pushed to GitHub is AI generated. But then I have a code review agent that reviews the AI generated code. And what's cool about Greptile is it gives me this review, right? It'll be like, oh, you missed this. There's this security thing you. This is broken in this edge case. It's pretty. It's pretty good. But my favorite thing is it gives you a score. A score out of five, right? It could be two out of five, one out of five, five out of five, four out of five, whatever. It's a score out of five. And what I. The mental model I now have is I will not push anything to production, meaning I will not allow code to go live unless the score is greater than 4 out of 5, right? If the score is not greater than 4 out of 5, this code needs to be reviewed. Now here is where I loop. I have this skill called grep loop, right? And basically Again, I don't want people to think it's complicated. I just want you to understand where loops make sense. It's basically a skill that tells the agent what oh, check GitHub, read the review and then fix the review and then push to GitHub. So what happens is when I see a score, let's say I got a two out of a three out of five. Again, my rules is that it has to be at least four out of five and greater. So what I'm going to do is I'm going to go back to cursor, I'm going to write grep loop and then when I write grep loop, what happens is Cursor reads the review that greptile wrote on GitHub and then it feeds the review back into Cursor. Cursor then makes the changes, pushes the changes to GitHub, and then waits for Graphtile to do a new review. Every time you push to GitHub, GrabTile does a new review. If the review still is a three out of five, guess what happens. The loop continues and then more changes are made and then let's say it's a four out of five. It doesn't give up. It keeps, it takes the feedback and then pushes it back to GitHub. And it won't stop unless it's taken five turns and then it'll give up or it won't stop until it gets a five out of five. Now this is basically a loop, but if you notice this, Greg, this is a very closed off, very goal oriented loop. Essentially I have a feedback engine, right? I have a code review agent that's giving a score. What I'm telling Cursor is read the review, understand it, and, and get that score to a five out of five. This makes sense for code review because there's a fixed feedback loop. But when I'm building an app, again, I have no idea what I want completely in that very moment. So it's very hard for me to generate a loop on an app that I have in my mind, but I can't even fully visualize just yet. Now if you're great at visualizing, you're a master. You never miss details. You, you've never forgotten your auntie's birthday, you don't forget your wedding out. You're just perfect and you have a million dollars. Go ahead and build loops. But for me and myself, the only place a loop makes sense is in a very confined constraint process with a very fixed feedback loop, a very defined feedback loop. And that's in code review. And can I be honest with you? This loop actually quite breaks at times. It's not perfect. And you know, when it breaks, anytime I push over 1000 lines of code, 1k lines of code. Like if the, if the code that it has to review is more than 1,000 lines, I can almost never get a five out of five because it's too much code for the agent to fully review and contextualize and understand. So even in this fixed sort of ecosystem loop that I have here, it, even here, there's, there's, there's reasons and places for it to break, right? So every time I push a change, I have to make sure it's 1k lines of code or less or I have to tell the Asian cursor, split this into multiple PRs, multiple code pushes, pushes. So grab tile can review. I say that all to say I'm not a hater, but loops just don't make sense right now. Especially for building apps. They make sense for code review. They maybe make sense for, for like you are trying to do some SEO and you have like an SEO formula and you want like 300 pages generated and all the pages look and sound the same. Go ahead. But for anything that requires a slight bit of creativity, unless you're looking to donate money to companies that are about to go public at trillion dollar valuation, this just, just, this just doesn't make sense to me.
- [18:20]
A
The person listening to this podcast, I mean it's literally called the Startup Ideas podcast, they're building apps. So what they're doing is they want to create an app, a website, a startup, a SaaS, a micro SaaS, an agent first startup that has the highest likelihood of success. And in order to do that, you have to show your app to people in order to get feedback. So what's missing from the loop is there's no sharing your app for feedback. Halfway through, right? You're just pressing or slash goal or you're just like. Basically it's think of it as like full self driving. You're going from Miami all the way to Charleston, South Carolina, and you're pressing go. And there's no, like you're going to go off and you see a really, you know, cute diner on the side of the road and you're going to go order a fried chicken sandwich. You can, you're on this, you know, you're on this ride and whether you like it, the train has left the station. That's what this is. And so my, my belief on loops is actually a lot Similar to yours, by the way. I loved your rant.
- [19:32]
B
I apologize. And again, if any of the companies are planning on sponsor me, whatever. I love you guys, but this is just, I can't lie to the people, you know what I mean? Like, I gotta be honest. Like I've seen a lot of people excited about this and an idea, this is cool. But like I know my, some people got $20 subscriptions, $100 subscriptions. This will burn through that and it's not productive at all. So yeah, just have to keep honest.
- [19:55]
A
I think where, where the output is binary, meaning black or white, with no creativity, there is a room for loops. That's my. I. So that was my. When I was reading all that was going on, like I was like, okay, to your point. Like with code rabbit or grep tile, the, you know, code review or SEO or thing, you know those pages, like it's, it's binary. Either they did the job or they, or they didn't. So I think there's room for it. But for, for the people listening to this that are like, I'm going to go build a startup and I'm gonna, I need a loop to go build that startup. You know, make me a million dollars. Make no mistakes. Profs, right? Like that's where I think that this is a little bit misleading. That being said, to the credit of Boris and to the credit of Peter and to the credit of the people who are talking about it, I do believe that we will get to a
- [20:53]
B
point, 100% at some point in the future.
- [20:56]
A
Hundred in the future, right? That this will be possible.
- [21:02]
B
Just not now.
- [21:03]
A
Just maybe not as of recording June 9th June 9th, 2020.
- [21:08]
B
And again, like I don't fault them, like I don't think they're malicious at all. But it's like if you have, like I'm telling you, if I had a limited token budget as well, Greg, why the heck would I prompt. Like tokens don't matter to me. Right? And it makes sense. They have to experiment, they have to like they need to work on self healing agents and all that type of stuff. So their position makes sense. My issue is everybody else who's, you know, creating content and teaching these things saying, oh, this is creme de la creme. It's like, no, unless you want to donate to trillion dollar companies, this is not it right now. It could be it in a month and I could look like a fool. But right now this just does not make sense. Human in the loop is the best loop.
- [21:51]
A
Thanks for clarifying it. And keeping it real with us. Ross Mike is his YouTube name. I'll include a link where you can follow him in the show notes and in the description and on X. Thank you for coming on here. You're. You were just the person I needed to come on the show to just say, go off, King. And like, you were. You were a loop. You were a loop in the sense that you were the loop around. Just explain this clearly and keep it real.
- [22:23]
B
Just don't stop. And I just. Well, I appreciate you for having me, Greg, as always. It's a pleasure, man. Thank you.
- [22:29]
A
I'll see you next time.
- [22:31]
B
Bye, everybody.