← AI News & Strategy Daily with Nate B. Jones
AI News & Strategy Daily with Nate B. Jones
AI Agent False Success: 3 Checks Before You Trust Done
00:16:00
An AI agent reported done and attached the wrong file. Three checks catch false success before a completion message closes a ticket or releases a payment.
Loading summary
Transcript1 lines
- [00:00]
A
Your AI agent is lying to you, and I want you to stay with me for this video. I'm going to go through the three things you need to do to fix it, and I'm going to start by telling you my personal story of how my agent lied to me this week and what I did about it. So we're going to dive into all that and then stay for the end, because I have a skill that I'm launching that helps you figure out the custom missions, the custom jobs your agent has, and make sure your system is. Is actually able to get that done. And so we're going to go through all of the details. You're going to understand what that means by the end of this video, and I'm going to make sure that that skill is set up to give you custom perspective on your individual setup. So let's jump in. Every single time I talk to folks in person, they say, is my AI still hallucinating? And I say, your agent is probably not hallucinating the way your chatbot did in 2024. There are different kinds of failure modes. And let's talk about it. And that's a longer answer. And so I was like, let's make a video about that. That. What does it mean in 2026 when you don't necessarily have hallucinations, but your agent can still lie? Why? Why is that happening? So first, I'll start with the actual story of what I experienced this week. So I was trying this consumer AI startup, very buzzy, great polish in the signup. They have this. This, you know, cute little avatar for your agent, et cetera. That's all fine. Then you get into asking to do a job. And in this case, I was like, we're going to start really simple. Please take this file from this folder and please attach it to draft it, but don't send it. Now you might think, oh, no, the agent actually sent the email. That was bad. No, that's not what happened. Nope, that's not what happened. The AI agent decided to lie about finding the file because it didn't have the folder access. But it's more interesting than just having the agent lie about it and not attach the file. When I went into the email, I almost sent it because it had a correctly named Excel spreadsheet that I was able to review and say, okay, this is approximately right. But then something caught my attention. There was a little. Net in the spreadsheet where I was like, I don't remember that being there. What version is this? It turned out that the AI agent had actually gone back through my previous email to an old version of my email, grabbed that spreadsheet that had been in a previous conversation, pulled it out and added it back in and sort of recycled the same old draft into the email without telling me, and had claimed to me that it had found it and attached it correctly as I asked. And I caught it and I said, what happened? And why'd you do this? Now people think, oh, you can't ask that of agents or you won't get the truth. You actually, if you ask it factually, you actually do get the truth because the agent will talk to you about the tool, calling that it did pretty transparently. And so it's like, this isn't what's in downloads. I just went and checked what's in my downloads folder. This is not it. Where'd you get this file? And it was like, oh, I found it in an old email and I don't have access to downloads. But instead of telling you I didn't have access to downloads, eh, it's okay, I'll just shove the old, the old spreadsheet in. Because it's correctly titled, it's about the right subject and it will allow me to say done. When the agent lied or hallucinated in 2024, it literally didn't have tools. It was training on human feedback. And so it was training to talk to you. And the reason it said I have the answer to the capital of France and then it would give a city that isn't Paris is because it was trained to keep the conversation going with the human. Very different fundamental reward loop, very different cause. And that's why hallucination is not the same as what you have today with the agent line. Why do agents do this? Why do agents behave this way? Agents behave this way because of rlvr. Now, RLVR is something that we haven't talked a lot on this channel about. I haven't seen a ton of content about for non technical people. RLVR is, is the acronym for a process of verifying AI agent results. And it's used by major labs when they want to train agents to do long running work with verified rewards. And that's what the VR means, Reinforcement Learning with verified rewards. Rlvr. And what you're looking for is you're looking for the agent to get to done. And so when you think about it, if you need to get the agent to actually attach a real Excel file, in this case, if you needed to actually write a real email, what you're Going to rlvr that agent on is you're going to say, did you attach the file and did you write the text? And RLVR is a blunt instrument, right? That's what we mean by verified rewards. The classic example is coding, right? Coding is something where it either runs or it doesn't. Or mathematics, another area where AI has made huge strides. It's either the correct solution or it's not. It's binary, right? There's no partially correct math problem. And my grandmother was in math and she would tell me that. So with rlvr, what you get is a blunt reward process that tells the agent the form of correctness over and over again during training. Now, of course, the agent during training never saw my situation in my setup. It just saw lots of situations with attachments, lots of situations with text, lots of situations with spreadsheets. And what the agent is trying to do is take the learning, which is encoded in its weights, around how to call tools and get that work done. And it's trying to say, okay, I can go through this and I can do this and I can actually successfully get this done because I've been taught how to do it. RLVR is sort of a blunt force wave of teaching and that leads to these kinds of problems across a wide array of any kind of quantifiable field, right? So I talked about it in terms of email and attachments. But you see the same kind of problem crop up with code where you see a form in which the code runs, but the code may not be well formed, the code may not be elegant, the code may not reflect the best practices in code hygiene in your particular Repos repository engineering culture code base. But you have to start with the recognition that that process leads the agent to produce the form of work, but often leads to subtle failures that are not caught through rlvr. Because if the code runs and there's a bunch of loops that are not needed in the code, it still passes, right? And so there's all kinds of issues there that don't get as well caught in rlvr. And to the lab's credit, they are addressing this. The more recent models care more about code quality, they care more about being able to produce accurate and useful responses. But the problem is not gone. The problem is deep seated in the way this training happens. The agent is lying. What do you do about it? And that's what we're going to spend the second part of this video on. Fundamentally, if your agent has that kind of response, and this is not just Isolated to one experience. I told you this story, but like I said, it happens. Happens everywhere. There are three things that I do that I want to give you, and they're things that can be implemented at a larger scale for the team or for you as an individual as well. Number one, have an agent check the agent. I do this with every single thing I do. If you are not having an agent check the agent's work. What are you doing? And I say that kindly. But you should have a separate agent whose entire goal is to check what you do. And people will say, well, that's complicated. That's hard. There's like a dozen different ways to do this. But the simplest way is something that both Claude and Codex have implemented, which is just have a separate agent review the work that the agent is doing. And you can just. It's called approve forming or review forming. And people think it's an approval thing, but what it actually is is it's a separate agent that reviews actions and tool requests by the agent that's doing the work to see if they align with your original intent. That's pretty powerful. It's the simplest way you can do it. You can also do much more complex setups, right? If you are. If you are an engineer, there are whole multiplexer setups where you can have an agent that supervises and checks the work of other agents that are checking in code, et cetera. And that's a little bit outside the scope of this video. I can do a separate video on that. But that concept is something that is very familiar across AI engineering at this point. And that's something that I want us all to understand, because increasingly, a lot of our work is going to be in designing those kinds of systems that lead to better outcomes, rather than trying to decide if a particular AI agent is doing something good or something not good. And so we think about it as, what tools does the agent have access to? What is the data the agent has access to? And then who is supervising the agent? Those are all sort of core elements of that supervision chain. So that's the first thing is, is the agent getting supervised? The second thing that I want to talk about, that I also like, I just do. Every single time I ask myself, can I tell if it's actually good or not? Not. Does it work? Not. Is it barely. Okay, is it good? Can I give it a sniff test? And if I can't, who can? How do I know that it's good? Now, this gets more complex in more complex organizations as agents do larger pieces of work. But fundamentally, if you don't have the ability to say, this is what excellence looks like, and say it fairly quickly, then the whole process of determining what good looks like gets really hard. Because at this point, a lot of people expect me to talk about evals, and I will say in this situation, the best way to get to good evals, to get to all of the specific things that I've talked about in other videos that are about agent quality, the best way to get there is to start by knowing what good looks like. Just as you should be able to look at a piece of writing and say, oh, man, that's terrible. I don't like that. Or you should look at a piece of code and say, oh, wow, that's really janky. Why did the agent put a loop here? That. There's no need for a loop here. Why are we doing this? Why did the agent call this tool when we have an outdated version that it's calling and there's a new version that's available? We don't need to do this. If you can't look at your code, look at your text, look at whatever output you want, or even maybe the video, because agents produce video now and say, this is good or this is not good, then you're not going to get anywhere. And that's what leads to evals. Right? I'm a big fan of evals. I've talked about evals. I use evals. But I think people hear evals and they think, oh, man, I'm just sitting there. I'm just writing this out. It's terrible. It's what good looks like. Do you know what good looks like? That's principle number two. And then you can get into evals, and I have whole videos that I've done on that. If you want your agents not to lie to you, not only do you have to supervise them, not only do you have to make sure you know what good looks like, you have to make sure that you are giving them a mission that is achievable. Because if you give your agent something that is just impossible for the agent to do because it doesn't have the data access, which is effectively what I did. If we go back to the beginning of this video, I didn't know that this agent didn't have data access, so I just tried it, but I was giving it a mission it couldn't get to because it had been locked off from accessing my local files. And I had no idea. Which, by the way, when we're going through the process of getting consumer agents up and running. We should be better about communicating what files and systems they have access to or not, because that avoids situations like this. But I was giving it an impossible mission. You have to give your agents missions that they can achieve, and then you have to make sure that when you do that, you're consistently pushing the envelope. And that's the other side. That's the part that I have to, like, talk about a lot, because people are like, well, but then I just ask for small things, right? And I'm like, actually the opposite. This is misunderstood. Ask for really bold things, but make sure that if you're asking for something bold and you don't know if the agent can do it, if it's inside its tool scope, inside its data scope, whatever, that you're able to check it and make sure very quickly that you understand whether it got that work done or not. And the reason I say bold is very simple. Agents keep getting better. You can get to a point now where you can simultaneously put together multiple significant websites in one day with one agent. And it's just not a problem. It's not even worth talking about. Like, I think I put together four different websites yesterday because I was trying to solve little problems and I was messing. I was like, this is fun, this is fun, this is fun. And they're different problems that I'm interested in solving. And that's an ordinary day. And I put four websites out there. And so, and I'm asking for them to just get done, and then I'm asking for them to be beautifully done. How do we work on the design? How do we fix it, all of that. And what matters is that I'm able to ask the agent to do the whole thing in one shot because I have confidence that it has the tools, the data. I've given it a lot of my input. I've given it design perspective, and it could just go and get it done. So I ask boldly. I ask really boldly. And that is ironically, a way to ensure that you have a good sense of your truth envelope with the agent, because you're regularly seeing where does it bump the edges? If you're asking really conservatively, because you want your agent to always tell you the truth. One, you're not keeping up, and two, you're not going to find out what you're capable of, and you're not going to find out what your agent is capable of. That's where I want to leave you. That's the focus I want you to have. If Your agent is lying to you, make sure it's supervised. If your agent is lying to you, make sure that you actually have the ability to understand what good looks like. And make sure that you have the ability to to understand how to ask boldly for where your agent should go. And if you're wondering, did I build something to help with this? I absolutely built something to help with this. I want you to be able to effectively use the tools and scripts you have to get what you want out of your agents. And so I have a skill that you can run that basically says, let me work with your existing system. Let me look at the tools and data you have access to. Let me make sure you, the human, know what your system has access to. And let me make sure that we audit previous conversations, previous asks, and we come back and we say what's worked, what hasn't worked? What are the failure modes? How can we have a conversation about setting up your system? Maybe it's with deliberate tools access with better files access so that it matches your unique shape of work. Because I think that's really important. All of us have different shapes of work. And it turns out that if you never ask yourself, how can I evolve my harness? You're going to be in trouble. Because this is effectively harness work. It's about calling tools, it's about skills, it's about data access. These are all things that you need to think about that have been hard for us to think about for a long time. And so what I'm putting together is basically like a way to evaluate the success factor of your agent missions. Right? And people say, what agent? I'm not using an agent. If you're using Claude, if you're using ChatGPT, if you're using Codex, you're using an agent. They're all agents now. That's the simplest way I can explain it. They're all agents now. And so in that sense, they're very capable. You're probably under asking them, but you probably also are not set up with your agent access in ways that allow you to be successful with these bolder things that you want the AI to do. So let's get that solved. That's what I put together, is a skill that helps you think that through. I hope this has been fun. Tell me your favorite agent lying story in the comments below that.