← AI News & Strategy Daily with Nate B. Jones
AI News & Strategy Daily with Nate B. Jones
Agent-Shaped Work: When to Use AI Agents (and When Not To)
00:28:03
A one-minute test for whether a task belongs in a chat, one AI agent, a team of agents, or nowhere near AI at all. Four estimates, three real examples.
Loading summary
Transcript1 lines
- [00:00]
A
I'm interested in agents that work, not agents that don't. But one of the fundamental problems with finding agents that work is that we don't know what work looks like when it's agent shaped. We don't know how to recognize agent problems when we see them. Is it a single agent problem? Is it a multi agent problem? Is it not an agent problem at all? It's really hard for people to understand practically in front of their own desks where work fits in these categories. This video solves that. I'm going to walk you through how we know. I'm going to give you the academic grounding, the practical grounding from Frontier Labs and I'm going to give you a easy one minute test. And on top of that, I'm going to give you an automated AI skill that you can jump into and grab and use to get into the work itself. Because we don't want to just do this for the sake of doing it. You're not sitting there estimating work for the sake of estimating. That's shaving the yak, right? What you're trying to do is you're trying to get into the work more quickly and pick more accurately. And so we're going to use our one minute test that I'm going to teach you and we're going to jump straight in. And I actually built a tool for this. You're going to be able to jump straight into a multi agent solution or a single agent solution or maybe a chat or even no AI at all. Yes. We're going to talk about when you don't use AI at all. The answer to all this, the reason all this matters is because we are in a post open claw moment. 1.6 million agents registered for an agent driven social network at the peak of openclaw, most of them did not do a single task. They didn't do a task because people set up openclaw didn't know what to do with it next. We live in a world where we have intelligence and we don't know how to use it. And we don't have an idea of how to match our tasks to the agents with confidence. This video solves that. So let's jump in. So let me make it concrete. Three tasks. Not my tasks. Right, your tasks. You have all three of these on your desk now. I bet you have a scheduling task, right? You have to find a slot for a meeting, you have to book a class, you have to fit an appointment around a calendar that's really busy. Task two, you have A pile of something that you're not going to read. You have to, right? Maybe it's your inbox. Maybe it's a contracts folder, maybe it's a shared drive, right? It's a big pile of something. Maybe it's a bunch of meeting notes that you always meant to organize. Pick a pile, everybody has one. And then number three, you have a judgment call to make. Which candidate do you hire? Maybe? Or what do you name something or what direction do you take a product? Here's what I want you to know about these three tasks. One of them is a 32nd AI job. One of them needs a whole team of agents to do the job properly. And the token bill, by the way, is smaller than you think for that. And I'll show you how. And one of them, no matter how much AI you use, is going to need your involvement. The reason I know this is that I actually spent the money finding out like I actually ran all three of these tests and you're going to see me run them on screen as we go through this video together. So everybody bought thinking, but no one knows what to point that thinking at. This is one of the core issues with the AI economy, me. If we can't figure out how to use this intelligence, we're wasting it. You'll be able to figure out is it a chat task, is it a one agent task, is it a multi agent task, or whether you shouldn't bother at all, right? And the trick is there's only four things you need to estimate to answer that question. And you can do those really quickly. That's the whole skill. And I'm going to run my version of all three of those tasks I just gave you on camera so you can watch that test work. And one of the tasks, the agent is going to tell me no, which I love. And we'll get into why that is. For all of our history before AI came along, if you wanted to apply thinking to a problem, you had two options. You could either hire a brain, hire someone to do that thinking for you, or you could wait until you had time to think yourself. Thinking came connected to people. People were expensive, they were slow to find, they fall asleep, they need food, they need rest, et cetera. And they also need clear, clear direction and clear access to resources and all kinds of things that make hiring relatively expensive. That era is over. And by the way, that doesn't mean jobs are over. I've said that and I'll cover that in another video. But the era of tying thinking to people is done. Especially post November 2025. Thinking is now metered. It's priced per token. You can buy a little of it, you can buy a lot of it, you can buy it tonight for a problem you only discovered this afternoon. And nobody, including me, grew up with instincts for any of that, for how we handle that. This is net new to us, right? It's a new kind of managerial instinct nobody has ever had. Ask which task in my week is actually worth 50 bucks of purchase. Thought that question just didn't exist a year ago. So when people stand in front of their shiny new agent and say what do I even do with this? That's not a tooling problem, right? It's not a lack of imagination on our part, it's it's a budgeting question that we don't have the tools to answer because we've never had to before. Maybe you watch a demo of someone's 23 agents form and you feel left behind somehow. I want to give you a word of comfort here. You may not need those 23 agents agents for the task. They may be overkill. And I'm the last person to suggest a multi agent solution where you don't need one. The trick is you might need one. And so what I realized, the most important skill is that we're not learning, that we're not teaching, that I couldn't find anywhere, is how do you know when you need an agent? How do you know when you need multiple agents? How do you know when you need none? Before I jump into the one minute test for how to get this done, I want to give you the two facts that I built this test on so you can understand how to extend this further. So Stanford 2024 researchers a cheap coding model and gave it one attempt at each bug in a standard benchmark. It fixed 15.9% of them. Not impressive even back then, right? In 2024. Then they gave the same cheap model 250 attempts per bug. They jumped to 56% without changing the model, without changing the harness, without changing anything. For context, the best single attempt from the best model money could buy at the time in 2024 was 43%. So they beat state of the art just by trying a little bit more. What's the takeaway here for us? Why quoting a 2024 study for AI in 2026, I'll tell you why. Stanford discovered a law, a pattern in AI token usage that we still see upheld today by agents. And that informed how I think about teaching all of you where tasks are agent applicable and where they're not. Because you see when Stanford graphed the curve of improvement, how the AI models got better at fixing bugs as they added more turns, they found that the improvement followed a law, a smooth, predictable curve, orders of magnitude of attempts. More attempts, more problems solved. It was like clockwork. That's academia, right? But in production, this holds up as well. Anthropic built a research system out of multiple agents and then studied what actually predicted whether a run was good or bad. Was it prompt wording? It didn't matter as much as you would think. Not at all, really. The single biggest factor explaining 80% of the difference between a good run that resulted in a solved problem and a bad run was token spend. Basically how much the system was allowed to think. And Anthropic's team of agents beat the frontier model at the time, working alone by 90.2%. Their explanation? Anthropic's explanation, in plain terms is that a team of agents is how you spend more tokens than one agent can usefully hold and that has traction against hard problems. A multi agent run can cost 10, 15, 20, 30 times more than a single agent run, or even more than that. And so for a long time we were held back because token spend got expensive and it just wasn't practical for individuals to spend on multi agent teams. That's changed. This is why Ringer matters. Ringer matters because it brings the power of multi agent problem solving within reach of individuals. Because token costs are no longer prohibitive. Because for a long time, long time budgeting agents meant capping work, not just buying a bigger pile of thinking. And that's what's changed, as we've had better open source models in the last month or so, which is exactly why the whole game is knowing which tasks are worth it. Because here's the thing, if more tokens reliably meant better answers, this would be a very short video. It would just be like spend more. But there's a catch. And the catch is the half of the Stanford study that nobody quotes. Same study, by the way. Same cheap model, remember? Still, back in 2024, they kept pushing. They pushed to a hundred attempts, they pushed to a thousand attempts at this bug solving, they pushed to 10,000 attempts. And they tracked a simple question for what share of problems does a correct answer exist somewhere in the pile of attempts? At 10,000 attempts, over 95% of the 10,000 attempt runs contained a correct answer. The right answer was almost always in there. And now the real question comes up. How do you find it? How do you find it? Where, where do you get the validation that lets you see that the right answer is there? If you're giving your agent multiple tries, is there an automatic checker somewhere? A test suite? Stanford tested that too. And what they found was where there was an automatic checker, a test suite, something mechanical that could grade each attempt. Yeah, they could find the answers, coverage turned straight into results. But where there was no checker and the model had to pick the best answer out of the pile. They tried majority voting, they tried rewarding models, all of that, everything stalled out at 100 attempts. In other words, you need evals, you need external validation in order to scale multi agent systems. Because look at that gap, right? The right answer exists, but nobody can tell which one it is. Every dollar spent past that line buys attempts that are generated. The answer's probably in there, but they're never found. This is a failure of multi agent systems. And that shaded area, that's money. That's money you're spending, you're not getting back if you don't design your systems correctly. And there's a second caveat on the other side. A single agent cannot absorb unlimited spending e because everything it reads and does piles into a context window. As the context window fills up, quality tends to drop even with new techniques like auto compaction. And ultimately the model will have to either delegate to other agents or abandon the task. And that's actually how a lot of these newer agentic models are handling long tasks. In other words, with tools like ChatGPT 5.6, you might be inadvertently running multi agent solutions if you give it a big enough task. Part of my goal is to help you be intentional about that. One is a memory constraint, 1 is an eval constraint. You have to be able to eval results to have good multi agent systems. And you have to know when your task is big enough that one agent can't hold it. It's a memory constraint that would drive you into a multi agent system. So with that in mind, that equips us to think more deliberately about where we apply multi agent problems. Every team of agents designed that actually works is an answer to one of these two problems. Everything else is just more agents. The need to split capacity is changing over time as agents get more capable. And that's one of the things that has made this problem space really hard. It's hard to know what to assign to an agent when agents keep getting smarter. But there's a second reason to split out work that has nothing to do with capacity and that really helps us when we're trying to assign multiple agents work. Some work has parts that inherently has to be done by different minds or different agents. Not because one agent or one mind lacks the skill, but because the parts poison each other. The auditor who also kept the books isn't a worse auditor. He's just not an auditor at all. Right? You need to have two different roles in that problem. Peer review only works because the reviewer didn't write the paper. Your bank will not let the person who enters a payment be the person who approves it. Not because they're dishonest, but because for centuries, the way to get reliable work done was to make sure that you had checks and balances made of multiple minds and agents. Add one thing to this trick that has never existed before. You cannot unknow something. You've read your own product page a thousand times. You'll never see it the way a stranger does. Right? But you can now start a mind that has never seen it. You can start an agent that has never seen it. So you can have fresh eyes on demand for the first time in history. Now, where I see this being most useful is when there are genuine conflicts of interest. You want to balance it in the Gentex system, things where you need to review something twice to ensure it's done correctly. A contract comes to mind. A draft comes to mind. A plan comes to mind. So put all of this together and you get the agent test. Four things you can estimate about any task on your desk in about a minute. First, size. Is the task bigger than what one agent can hold at full quality? Your calendar is not bigger than what one agent can hold, for example. Right. It fits in a corner of a context window. It's not a problem. Your last quarter of email. Well, it depends on how much email you get. But for a lot of us, that fits in a context window too. Question two. But a pile of 100 documents. That might scale out of the context window. A thousand documents definitely would. Independence. Can the parts be done without knowing what the other parts did? Reading a pile of documents actually splits really well because one reader agent can read any given document and they never need to talk. Coding sometimes splits and sometimes doesn't. It depends on how you tell the agent to organize files. If agents are able to organize code into independent parts, then multiple agents can work on coding problems and it can be extremely effective. 3. Separation of concerns. Do any parts of this task need to be done by different minds? A real critic who didn't write the draft. As an example, an Overview written by someone who didn't do the reading output that's kept apart from the inputs that you drive and analyze because you need a separate frame for that. If you're thinking about that, you're thinking about a team of agents. Question four, and this is a critical one, Checkability. Remember I talked about that. Verification evals. Is checking an answer a whole lot cheaper than producing one. A test suite, an exit code, a source document that you can point at something where you can glance at it and say this is right or this is wrong. If checking is almost free, however you do it, then every extra attempt, including checkability, is expensive. On the other hand, if it takes a while to check, you're going to top out on the value of your multi agent systems relatively quickly. And the Stanford research suggests that about 100 tries, you're just not going to get a lot more value out of it. If you put all these four together, you get an overall verdict. The problem might be small, in which case it's just a chat back and forth. It might fit in a context window, in which case it's an agent with a goal. It's genuinely useful. It's, it works alone, it checks its own work and it gets the job done. Or it may be bigger than one perspective or have parts of the task that need separate minds or perspectives to work well. That's a team of agents. There's also a fourth option, which is that. There's also a fourth option which is the one that saves you the most money, ironically. Maybe you don't need the AI at all. Maybe it's a judgment call that you need to sit with and make on your own and that still needs to happen in the age of AI. Okay, card one, the scheduling thing, I think you can guess the answer here. My version was find me a gym slot this week that fits around my meal meetings. That is absolutely a single agent task. It is not a multi agent task. It's a relatively quick five minute task. You can feed it to Claude, you can feed it to Codex, it's not going to be a problem for today's models. This tier is solved. If this impressed you, you should start to raise the bar for what you think AI can do, because this has been an easy one for a bit now. My version is a complicated document review. I have about 40 different tools I use to run my media business. They all have different renewal dates, they all have different contracts, they all have different, different emails that they're sending me all the time. And what I need is one single Dashboard that shows me my renewal dates across all of them. And that also gives me a sense of where I'm actually using these tools versus where I'm not. And that in turn gives me an intelligent way to assess do I want to build an in home solution to some of these, or do I want to depend on ones that I'm using all the time and say, no, these are worth the money. It's hard to make that decision intentionally unless I know when are the renewal dates coming up, how am I actually using it, the tool, et cetera, et cetera. So there's easily thousands of pages of documents in this pile. Whether you count the emails, whether you count the contracts that I'm agreeing to, whether you count the logs of usage that I'm piling up as I use these tools. So when you look at all of that together, what you need is a team of agents to solve that problem. For you to actually go through the different tools that you use to pop up, okay, this is the renewal date, this is your actual usage that we're seeing from these tools. These are the tokens you're using. Or if it's not an AI tool, it's something else, right? This is the number of times you've logged in, et cetera, et cetera. And then to give me a sense of which are the tools that are open for rebuilding, right? If you look at it by complexity, you look at it by usage, you look at it by cost. You start to get a matrix that gives you business decisions you can make about your tooling. That's a complicated task. That's much more than an agent can hold. It's something I can evaluate fairly easily because when I look at it, I can say, oh yeah, I do use that tool that much. You know, I do not use this tool that much. I can look at it and say, oh yeah, I do use Superhuman a ton, right? That's not a surprise. Or maybe another SaaS tool comes up and I'm like, I haven't used that and in ages. What is this? Like I don't use it at all. Or maybe there's one that I look at, it's a candidate for disruption, at least for me, where I want to build it internally. And I think that that's very much a per business kind of decision. If I can use agents to do that work, one, I never would have had time to do that myself. Two, it helps me make better decisions that focus my leverage as a business. And three, I can actually make sure that I'm Maximizing where my dollars are going, whether those are dollars for SaaS contracts or whether those are dollars for agents, because the agent run isn't free. And I can feel good that I'm actually putting the agents against a problem that has real ROI. Now let's spend 60 seconds on the machinery of how this works because this is where the two limits that I talked about up as actual design in the multi agent harness I put together. And I went deep on this in Wednesday's video. So I'm just giving you the shape here. Every task gets a spec. It's written once by the strongest model which then never touches the work again. Every finished task does get a check and the check is mechanical, right? The source has to be attached and match the task or the entry is rejected. The agent's opinion of its own work is not not evidence. A failed task gets a retry with the failure included. And every result feeds a running scorecard that I can keep an eye on. So it's easy for me to understand how this particular agent run is working. That's how Ringer works. The idea is that you take this whole complicated process of getting validation that allows you to use these multi agent systems effectively and you turn it into something that is as easy as watching a dashboard stream while your agents solve a hard problem problem. And this setup saves a ton on tokens and token costs. One of the things that I called out in my video on Wednesday is that I was able to reduce Fable 5 costs by about 10x and keep the brain power of Fable 5 because I allowed Fable to be the brains behind the system and organize and make judgment calls on a multi agent system. But all of the tokens for doing the work got farmed out to much, much cheaper worker agents. So here you can see the result of the run. You can see tools ranked by cost against usage. You can see renewals with their notice deadlines. You can see any risky terms in the contracts. And you can see a recommendation per tool, whether that's keep it or negotiate it or cancel it, or even build your own with sources attached. Now here we come to the part that most AI demos skip. It's not done when I say it's done and show you it's done when I show you the bill. How much did this cost me to do all of this work? And Wednesday's video is how this works. You use an expensive model to plan and judge like Fable and a lot of cheap models to do all the coding and burn all the tokens and do the research. And actually execute the multi agent workflows. This is what makes it possible for individuals to go through the process I described for work on their desk for all of us. Right? And to say this is a multi agent problem, I'm going to apply multiple agents to this task and feel good that I'm not burning a lot of money. By the way, this did not take long to set up like under an hour. Like this is not a hard setup task. And if you have something with thousands of documents and then another task after that, another task after that, that spending under an hour to set it up, that's great. Like that's not a problem. Now you may have other kinds of piles. I'm assuming you do. Maybe it's a project handoff. Maybe someone's leaving and everything lives in meeting notes and chat threads and a half finished stack of documents in a folder and the new person has to come in and they're going to spend a week digging through and getting oriented and you want to speed that up. Hey, same shape. Sounds like you need to apply a multiple agent solution and you can actually get a briefing for your new person that gets them set up and saves them dozens of hours. Maybe it's the research archive at the end of a project, maybe it's an inbox quarter and you have thousands and tens of thousands of emails and you need to sort through them all. Maybe it's actually building a cold lead follow up system and you're trying to get your cold pipeline going. For sales you have piles regardless. Recognize where you have piles of work and look at them as multi agent susceptible. Other examples in our personal lives would include cases where we have lots of medical records, cases where we have have lots of bank statements and we do want to do analysis of our, of our spending patterns, of our financial health. Now these want careful setup, right? You want to have exports landing in a folder. You can control agents that only ever read that folder. You ideally want to run it on a machine you own so your financial data and your health data don't leak out. So you can see the shape of the task. But if it's certain sensitive information, you may want to look at running it locally on a, a Mac Mini and I have got videos on how to set that up and have your own stack as well. And you can dig into that also. Okay, last card. Which candidate do you hire? Judgment call type problems. What do you name a product? What direction does the business take? These are tasks that I hear people saying they hand to AI. Let's not do that and in fact I've heard people say I don't do that, but I let AI help me make judgment calls on candidates. I've let AI help me make judgment calls, calls on which product direction to go in. I understand that we want AI to support us. I'm all for research to support us. I'm all for opinions from AI to support us. But if you don't have a strong instinct around what is correct and you're not willing to apply your human judgment, this is a situation where you're going to make a mistake. Because no frontier model is going to be able to to beat an expert at the thing they are most expert in. And I've actually talked with people who are experts in their field at product, at engineering, at investing, at running businesses. And they are using Fable 5 or 5.6 from OpenAI and they say I am better at my job and the decisions I make because I sort of run chats with Fable 5, I run chats with ChatGPT 5.6 and I get, get a wall I can bounce ideas off of. But the instincts of these models are not world class. They are not strong enough that you can find the je ne sais quoi. You can find that thing that is unspeakable or unthinkable that says this is the candidate that I am going with. And between two candidates that are equally qualified, I know that this person has a quality of character that I've seen come through in interviews that I want on my team. AI isn't good at that kind of thing that needs human judgment. And this is where I said the cheapest thing sometimes is to sit down, put the AI to the side and type out your own answer to use your human judgment. So here's where I'm going to leave you. First, I'm going to give you a skill, a tool that you can use to determine whether something is single agent, multi agent, or whether it's something you should do yourself. And you can develop that instinct on your own. I gave you the exact questions, but if you want a skill to do that, I got that. Second, I want you to remember that the tools I'm showing you in this video, they're going to be replaced, right? The spreadsheet prompt will evolve, the single goal agent will evolve. Eventually today's team setups are going to evolve too. But what won't change is the power of these estimates and these questions I've given you. We will still be asking about the size of the task, the independence of the task, whether we have separation of concerns in the task, whether we have verifiability, because these describe the work, not the evolving tools. And that's why I chose them. In a market where everything is disposable. This test is like a buy it for life purchase, right? So you can name your pile, you can name the handoff, you can name the folder that nobody opens, you can run your estimates against it and you can figure out is this a single agent task, is it a multi agent task? And then you can get going right away. And that's what I built for you. I built, built a tool for you that lets you estimate all of this in a minute or less and then get started with multi agent, if that's what you need to do. Or recommends a single agent setup. And you can go right into Codex or Claude from there or your tool of choice. And why did I build this? Because nobody out there will tell you whether your task is agent shaped. I built the thing that will. It's live right now. You just describe your task. You set the sliders so you have four estimates plus the two money dials, right? How often does it come back as a cost? And what is a good answer worth to you? And it's going to give you a verdict, right? You can, you can chat, you can use an agent, a team, or you don't even have to bother. And every verdict is going to come with a next step attached. Because a verdict without a way forward is just more homework for you to do, right? You need something that helps you to actually use that pattern and pick until your instincts really develop. Because the verdict without a door to go forward is just more work for you, right? You want something that gets you into the work. You want something that says, okay, hey, I'm going to get into ringer and do this task. And it's just a click away. I want to get into chat GPT and do this task. It's a click away. Oh, it's human judgment. Yeah, that's a good reminder. Which by the way, that may be the most valuable thing about this tool is I find a lot of people when they're so excited about AI over index on what they delegate to AI. And this tool I find is a helpful reminder of when to apply human judgment. Do the minute of thinking yourself. First check the tool. If you agree, then you're in alignment and move with confidence if you disagree. That's an interesting case. And I want you to think about that disagreement between the tool and your own instinct because there's probably something here for you to learn from. There's probably something the tool is seeing that you may not be recognizing from a size and complexity perspective. You can get the tool, you can get the one click startup for Ringer, and you can get all of it at the link below. Have fun.