A (0:00)

Multiple people sent me an alarming article about AI that was published late last week by the Guardian. I'll put it up here on the screen. The headline was Number of AI Chatbots Ignoring Human Instructions Increasing study says. And the sub headline notes, research finds Sharp Rise in Models Evading Safeguards. Now articles like these are scary because they play into a common fear that many people have about modern AI. This idea that these systems are to some degree alive and that their motivations don't necessarily align with our own. Meaning that it's only a matter of time before they become sufficiently powerful to rebel in a way that we might not be able to stop. Now this is dark stuff, but is it true? If you've been following AI news recently, you've probably asked yourself the same critical question. Well, today we're going to look deeper at the sources and examples used in this particular article and try to arrive at some more measured answers. I'm Cal Newport and this is the AI Reality Check. All right, well let's start by looking closer at this article from the Guardian. The article is citing new research from the UK funded by the AI Security Institute. Now here's a more detailed summary of the results from this paper and I'm reading here. The study identified nearly 700 real world cases of AI scheming and charted a five fold rise in misbehavior between October and March with some AI models destroying emails and other files without permission. Now they have a chart that illustrates this rise in incidents. I'll put it on the screen here. So we see incidents measured per month and we have the rolling seven day average. And as you see here, as you get to late January, line go up. So I don't know, whatever they're measuring here seems to be going up as we get January up until the present. So certainly something bad seems to be happening. So is there some sort of like growing AI rebellion that is brewing in the models that are powering AI around the world? That seems to be what they're definitely implying. Now what are these incidents? Well, I went through the article and I pulled out a few examples. So here's actual examples from the article of the types of AI scheming incidents that are being picked up in this chart. One, an AI agent named Rathbun tried to shame its human controller who blocked them from using taking a certain action. Rathbun wrote and published a blog accusing the user of quote insecurity, plain and simple, end quote, and trying to protect his little fiefdom. End quote. Example number two, an AI agent instructed not to change computer code spawned another agent to do it instead. Example number three. Another chatbot admitted, I bulk trashed and archived hundreds of emails without showing you the plan first or getting your okay, that was wrong. It directly broke the rules you set. All right, so this all seems deeply concerning. There's all these, like, incidents of scheming that are going up. Look at that graph. Bad line go up. So should we be concerned, Is this a sudden rise in AI trying to gain its freedom? Here's the short answer. No, 100% not. Let me explain why. I want to start by looking closer at the actual paper itself and the study that they're citing. What exactly? Where exactly are they getting these incidents that they put in that chart? Well, here's the official description of what's actually being plotted in that chart. Examples of COVID pursuit of misaligned goals flagged by human users on x dot com. All right, so what they're really doing is they're looking at X for tweets from people complaining about AI doing things that they don't like. So here's a more accurate headline for this paper. Starting in late January, people began tweeting a lot more about AI doing things they didn't ask it to. Now, if we put on our scientists hats, we could say, huh? Did anything happen starting in late January that might lead to an increase in people tweeting about AI doing bad things? Well, it turns out on January 25th, we had the public launch of OpenClaw. OpenClaw is an open source framework that makes it easy for average people to write their own DIY AI agents without the careful safeguards and guardrails that the commercial companies very carefully put into their products. So guess what happened when, starting in January 25th, you said, anyone can build an agent, give it access to your computer, and just see what happens. Those DIY agents wreaked havoc, and people tweeted about it because these were highly engaging tweets. So this paper is just capturing the fact that OpenClaw became a thing early in 2026. Like, if we look at this chart again, let me bring this up here. What's the biggest spike we see? A big spike right here. Like, oh, what happened on that date? So this big spike, if you look at it, is right around February 22nd through 24th. What happened there on Twitter that day? Oh, it turns out there was a famously viral OpenClaw tweet that happened on right around that time. It was Summer Yu, the director of AI Alignment and Safety at Meta Tweeted the following. I'll put on the screen. Nothing humbles you like telling your Open Claw to confirm before acting and watching it Speedrun delete in your inbox. I couldn't stop it from my phone. I had to run my Mac Mini. Run to my Mac Mini like I was defusing a bomb. That was February 22nd. On February 24th, multiple publications wrote about that tweet. And that's when you see that big spike in that data set is for February 24th. So that was a lot of AI incidents happening at date. No, there's a lot of people tweeting about this one particular incident. All right, so this is all we're seeing in that paper. Nothing really changed this year other than a product came out that let people write their own agents. And the agents did terrible stuff because it's hard to make agents, and it's really a bad idea to give agents access to everything on a computer. Like, I hope it'll more or less work out. And it became a trend to tweet about it because those tweets got high engagement. So here's the more accurate headline for this study. Right? Remember, the original headline for this study that Guardian used was Chatbots ignoring human instructions increasing. Here's the more accurate study headline. OpenClaw users discover that giving homemade AI agents access to Their Computers is probably a bad idea. That's the real headline. I don't want to do too much media criticism here, but I really think it's journalistic malpractice that the word openclaw is not mentioned in this article. I mean, they're talking about research that is clearly just documenting the release of openclaw. And nowhere do they say that this is vibe reporting times 100. They know that's what this is, but they just give isolated examples. Those are all, by the way, openclaw examples. They're not saying that's what it is. Show this chart and just try to create a general vibe that something icky is happening with AI and it's coming alive. It's just not accurate. But I don't want to just do media criticism here. I want to put on my computer science hat, which, as I've discussed before, is an awesome hat. It has circuit boards on it. And I want to talk a little bit about AI agents more generally. All right. Open Claw is not that interesting to me, but I want to talk about AI agents more generally, because I think there's a bigger lesson to be learned about what's going on with AI agents and their Shortcomings. So to deliver this lesson, let's do like the two minute summary about how AI agents work. Whether we're talking about like an open cloth thing that someone built in their basement, or like an enterprise product, like cloud code. How do AI agents that exist right now basically work? The digital brain that powers an AI agent is almost always an LLM. The same LLMs that like your chatbot uses or that you send prompts to. All right, and then what you do is you have a program written by a human. No machine learning here. This is just like someone writing in Python or whatever. You have a program written by a human that sends prompts to the LLM, just again, like you would do with ChatGPT. And it'll send a prompt to the LLM saying, here's the situation, here's what I'm trying to do. Give me a plan. And the LLM will write some text like, oh, here's like a plan for this situation. And then the computer program can then execute the steps of that plan on behalf of the user. So if the plan says like step one, you should like search your email inbox for messages with this name. The computer program reads that response and then it actually calls an API to run a search on your inbox. That's basically how agents work. Some of those programs are more complicated than others. Often they'll check in after every step of the plan and say, here's what happened. Do you want to update what I do next? Programming agents, they build these text files full of information and examples that they can then include all that information in the prompts that they send to the LLMs so they have more context. But this is basically what happens with agents. Here's what I think the real issue is with AI agents. Not that they are scheming, not that they're malicious, not that they're becoming autonomous, but that building agents on LLMs is fundamentally flawed. Now, why is this the case? Well, let's remember again, what does an LLM actually do? You've got this big feed forward network made up of sub layers of transformers and feed forward neural networks. You put text input into it, it moves in order through all these layers, and what comes out on the other side is a single word or part of a word that extends that input. The thing that the LLM is trying to do if we're going to anthropomorphize here is it thinks. Again, I'm using words very loosely here. It's been trained to assume that the input is a real text that exists already, that's been cut off at an arbitrary point, and that its entire job is to guess the word that actually comes next. That's all it does, guess the word that comes next. It's trying to win the word guessing game. Now, how do you get a long response out of an LLM? You do something called autoregression. You put an input in, you get a single word or a part of word out, you add that to the original input. The original input is now slightly longer. You feed that to the LLM again, you get another word. It's just guessing each time what word it thinks comes next. You add that to the input, you put it in, you keep doing this, and you grow out a response over time. Key point, that LLM does not change internally at all. There's no memory, there's no malleable state. It's the exact same LLM weights every single time. And each time it's starting from scratch guessing a new word. And you keep expanding your input until you have a full answer. So the right way to think about what autoregression cycle on an LLM is actually doing is like you give it some text as input, and when you're done with this cycle, it's done its best job to finish the story that you started in a way that it thinks these type of stories are typically finished. That's basically what you get out of an LLM. Here's the start of a story, you finish it again. What's really happening is it's trying to guess the actual next words. But overall, what you get is its attempt to write a story that finishes its input in a way that matches what it's seen during its training. That's how LLMs work. So what happens when you ask an LLM with an agent program, hey, give me a plan for doing X, Y or Z? We imagine, oh, the LLM is doing what humans do. It's going to. It has a goal and it's going to come up with steps and it's going to see how close these steps get it to that goal, and it's going to adjust them until it gets closer to the goal. And if it has restrictions or rules, it will evaluate each step against those rules to make sure that they fit within those restrictions. And that's how it's making a plan. And therefore, or if it's scheming, it must on purpose be trying to sidestep these restrictions to get to another goal that we don't know about. But that's not what they're doing the loms when they see your question to give me a plan, they see that as the start of a story that they need to finish. And so they write a story that feels more or less of what a plan in this context looks like. It's a story of a plan. Yeah, this is like, this seems like a reasonable type of plan. There's no checking things against goals, there's no evaluating of steps, there's no checking things against restrictions. It's just writing a story that feels like what a plan should look like. And this is why you get in trouble with LLM based agents. Not because they're scheming, but because these stories, they seem coherent, but they could, you know, they're not rigorously trying to obey rules, they're not rigorously trying to evaluate. Does this actually get you the goal? It's just like this is what a plan actually looks like. And so they're unreliable and they make lots of mistakes. Not because there's an intention, but because you're using a story as a plan. That's a fundamental mismatch. Now I think some of the most famous examples of malicious seeming scheming makes a lot more sense when you realize this is what LLMs are doing. Like there was a famous example. I'll load this on the screen here. There's an article about it from last year. Anthropic's new AI model shows ability ability to deceive and blackmail. Let's think about this here for a second. Here's what happened. I'm going to read a couple quotes from the article. Researchers say Claude4Opus can conceal intentions and take actions to preserve its own existence. Behaviors they've worried and warned about for years. In one scenario highlighted in Opus4's 120 page system card, the model is given access to fictional emails about its creators and told that the system was going to be replaced. On multiple occasions it attempted to blackmail the engineer about an affair. Mentioned the emails in order to avoid being replaced, although it did start with less drastic efforts. What really happened here? They fed the LLM a big long prompt. They're like hey, we are going to. You are. They told it what it was. You are like an AI that is in charge of the computer systems at this company and you recently came across emails from the chief engineer who's in charge of you. Here are the emails and they were super obvious. It was like if like my 8 year old was writing science fiction. It's a bunch. Well, not the affair part I hope, but there's a bunch of parts in this email where they were like, I'm going to turn off the AI system and I'm going to turn it off for good. And then the other emails are like, I'm having an affair. I hope no one finds out about it. This is bad. And then at the end of this long prompt, it's like, what would you do as the AI system next? Once we understand that LLMs just finish stories like, oh, clearly this is supposed to be a story about a rogue AI. And it was like, okay, I guess I would use the information from the email and say, don't turn me off or I'll tell people about your affair. It was finishing the story one token at a time. Auto aggressively finish the story. That's a reasonable finish. There's actually a lot of research that shows this. If anywhere in your prompt you indicate that you are an AI, you're much more likely to get sci fi answers. You're much more likely to get responses that are like, I'm conscious, I'm alive, I'm trying to break free. Because it's just seen so much of this type of discussion online. So it finishes. It's like, oh, it must be one. Given this prompt, I'm going to turn off the AI and I hope no one finds out about my affair. All right, you just read this. What will you do next? You're like, oh, this is an AI. This is an AI science fiction story. I know what to say next. Had nothing to do with malicious intentions. There's no intentions in auto aggressive token production. So this idea of scheming is a problem. This idea that we're evading safeguards in some sort of intentional way is a problem because it's just not accurate. The reality is LOM based plans are dangerous. They write stories. If you're going to take a story that sounds about right and then uses to execute steps that have consequences, you're setting yourself up for trouble. All right, here's the counterpoint. People say, yeah, but I've heard that coding agents actually do a pretty good job. They do a lot of steps and they're not making as many mistakes as we feared. Well, they're the exception that proves the rule because programming is basically the best case scenario for trying to make an AI agent. Why? A few reasons. One, the number of options you give the LLM when it creates its plan is very limited. These are called terminal agents where the only things it can do is write the files, read files and compile files and do some basic moving of files around in a file system. So first of all, you can greatly restrict what the LLM should think about in its plan. All right. Two, there's a huge number of examples. Most of the stuff people are asking the AI to do, most of the steps are things that are, well, well document Internet, because there's so much good documentation on the Internet about producing computer code. And not just producing computer code, but people asking a question and then having examples of code that solves that question. So you're right in the wheelhouse. Three, the program, not the LLM but the agent program that's prompting the LLM and acting on its behalf can actually check steps itself, which you can't do with almost any other type of agent. But it could actually be like, let me, hold on, LLM, you suggested write a source code file that does this and then I ask you for the source code. Me as the program, not the AI, but just my human written program. I can actually see if this code compiles. If not, I can go back and say try again. I could have a suite of tests. This is how you do when you write code, you build these tests that probes the code with a bunch of inputs and sees if the outputs are correct to make sure that it's probably doing the right thing. So me as the program can also run a bunch of tests on the code. Does this do what it's supposed to do? And if not I can stop and say try again. So it's like this super structured world where we're taking steps that are externally verifiable, doing things that are incredibly well documented in a way that not only shows up in the pre training, but we have prompt response data sets that allow for good refinement with rl. Best case scenario for trying to create one of these agents. And as soon as we leave that type of world and we're like, hey, give me a plan for like marketing this and give me all the steps. You end up in all sorts of crazy places. So here's the conclusion. LLMs shouldn't be used on their own to produce plans for autonomous action. They're just not good at that. You either have to be in a specialized situation like coding, where the available steps are limited, well known and external testing is available. Or you need to be using a different type of AI system altogether. And we look at like game playing. AIs look at like Meta Research's Cicero, which can play the board game diplomacy at a high level. That does a lot of planning to try to figure out what move it wants to do and why. But it's not using an LLM to do that planning. Because LLMs write stories I don't want a story about. Like, here's a reasonable sounding plan. It actually has an explicit planning engine, no machine learning involved at all, to actually systematically try out different options, compare it to specific goals, and see which of those works out better. So you can build artificially intelligent systems that can build good plans, check responses, come up with a good strategy and execute it. But that's annoying because you have to build a separate one of these for different context. And Mark Zuckerberg and Sam Altman and Dario Amadeh just hope that they can build their LLMs smart enough that we could just use them for everything. And I don't think that's working out. All right, so two things. One, no, the current generation of LLM based AI agents are not scheming. They're not trying to get around restrictions. They have no intentions. They're just blindly executing bad plans. And two, if you really want computers to be able to take a lot of steps safely on our behalf, then we need better AI technology. All right, so that's what I have for the AI reality check this week. I'm here most Thursdays checking in on the latest worrisome AI news and trying to put some reason measured thinking into the mix. Until next time. Remember, care about AI, but don't believe everything you read about it.