← Deep Questions with Cal Newport
Deep Questions with Cal Newport
Was the Mythos Ban Justified? (Good Idea. Bad Execution.) | AI Reality Check
00:29:25
Loading summary
Transcript85 lines
- [00:01]
A
Arguably, the only topic that's more controversial than AI criticism at the moment is really any mention of the Trump administration. Well, late last week, both of these topics came together, creating a tidal wave of chaos and recrimination. Here's what happened, if you haven't been following. Back in April, Anthropic announced that their new large language model, which they called Claude Mythos, was so good at finding bugs in computer code that it was too dangerous to release to the public. Here were their exact words. The fallout for economies, public safety, and national security could be severe. Okay, fast forward to last week when Anthropic essentially said, hey, good news. We added guardrails to Mythos, and now it is safe. They call this protected version of the model Fable 5, and they made it widely available. On Friday, the US government said, not so fast. Now, according to David Sacks, who was, until recently the White House aizar, the administration had heard from an independent researcher they trusted who said that he had easily evaded the guardrails that had been added to Fable 5. The Commerce Department promptly placed Fable 5 and its unprotected version, Mythos 5, on an export control list, which means the company must suspend access to the model from all foreign nationals, which presumably includes many of Anthropic's own employees who are here on visas and are foreign nationals. They said they won't lift this restriction until Anthropic fixes the guardrail issue. In response, Anthropic had no choice but to temporarily shut down all access to these two new models. All right, so that's what happened. Almost immediately, the Internet exploded, with most of its criticism focused on the typically haphazard and inscrutable manner in which the Trump administration, as it usually does, acted in this case. Here's a headline from the Economist that I think captures this mood well. It reads, donald Trump's blocking of Anthropic is capricious and chaotic. America's closest allies are shell shocked. Dean Ball, who previously served as a senior policy advisor for artificial intelligence, put it this way, on X. Make no mistake, post Mythos, the United States has a licensing regime for AI it's just informal, with no consistent rules or firm boundaries on state power or public transparency. Even Gary Marcus, who is no fan of Anthropic, was. Was uneasy by the way this all went down saying on X, whatever you may think of Dario or Anthropic, Friday's decision and the impetuousness and arbitrariness of it was a terrible mistake that has left a stain that will last. But this is not a political show. So if we put the politics of the situation aside, there are some key deeper questions that lurk. For example, are these new models actually national security concerns? And two, in the hands of a more competent administration, is something like this more hands on regulatory approach actually warranted? Well, it's Thursday, which means it's time for an AI Reality check episode of this show, which is the perfect opportunity to go seek some measured answers. And before we get into it, I want to add one warning. This is a fast moving story. I'm actually recording this episode on Monday, June 15. By the time this episode airs on Thursday, the situation on the ground with these issues will almost certainly have changed. In fact, my read trying to trying to see the chatter and the writing on the wall. My read is that the government is going to soon reach an understanding with Anthropic and they will release their export control list in the next probably one week or so. I don't know quite when, but I think that's what's probably going to happen. This restriction was done too hastily and Silicon Valley has too much influence on
- [03:54]
B
the current administration for it to stand.
- [03:55]
A
All right, so with this in mind, what I'm going to try to focus on here in this episode are those more fundamental questions which I think are relevant no matter how this particular story
- [04:04]
B
with Fable 5 evolves in the next week or so.
- [04:08]
A
All right, that's enough preamble. Let's get into it. As always, I'm Cal Newport and this is Deep Questions, the show for people seeking depth in a distracted world. All right, so let's start with our first question here. Is the government right to be worried about Fable 5, the consumer version of Claude Mythos with the guardrails added? Now, they haven't actually provided many details about their decision to restrict this model. So our best source of information, at least as of the moment of this recording, is David Sacks. Let me read a little bit more detail about how he explained what he understands happened that led to this restriction. So I'm going to read him here verbatim. A highly credible trusted partner of both Anthropic and the US Government who is testing Fable, came forward with a jailbreak of those guardrails. The admin asked Dario to fix a jailbreak or de deploy the model. Dario refused. In their blog post, Anthropic defended its decision by saying the jailbreak isn't serious. This is not what the trusted partner and the US Government believe. Nor is that kind of minimizing language consistent with Anthropic's brand as the AI safety company, it's difficult to fathom how they could claim a jailbreak. Allowing operability of a cyber weapon could be defined as not serious. All right, so stepping back, the government's argument seems to be the following. Hey, anthropic, you told us that Mythos is essentially a cyber weapon that will create grave national security concerns if bad actors gain access to it. Your protected version, Fable 5, is not really that protected. If we can get around the protections, then the cyber weapon is now available. We have to put a stop to it. All right, so let's break down the government's argument a little bit more. The first part. Do we believe, David Sacks, that a trusted partner of the government was able to bypass Fable 5's guardrails? My answer to that is like, of course, that's not surprising. What do we mean by guardrails? All right, so guardrails, when it comes to applications built on top of large language models, usually means you have done fine tuning of an already pre trained model where you have a bunch of sample answers and responses. So you'll have questions and responses, questions that you don't want the model to answer, and then your sample response will be something like, I'm not going to talk about that. Using techniques that come out of reinforcement learning, you can then fine tune the model so that when they see questions similar to those sample questions in the training set, they will divert the flow of token output logic towards those predefined answers. Right. We've been doing this since GPT 3.5. This is why if you just say, hey, give me the instructions for making a bom, it will say, I'm not going to talk about that. It's because they gave it many different variations of people and asking for BOM instructions and each time reinforce the correct answer. There being, I'm not going to talk about it. Sometimes, if we're going to be clear about it, sometimes guardrails can mean something else. For example, you could have a more old fashioned symbolic pattern matching logic where before the prompt even gets sent to a large language model, the computer program, the control program says, oh, I recognize this as a bad question and I'm just going to short circuit it and not even send this to the language model to answer. There are a lot of credible reports out there that actually that pattern matching guardrail is how these companies have been dealing with some of the more embarrassing
- [07:32]
B
examples that circulate the Internet.
- [07:34]
A
Like asking a language model to count the number of Rs in the word Strawberry, which most of the major models struggle to do. The accusation is they just hard coded a pattern match. If someone's asking that question, just give them the right answer. There was another embarrassing example where with many of the major models, if you
- [07:50]
B
ask them, I only live a quarter
- [07:53]
A
mile from the car wash, should I walk or drive to get my car washed? And most of the major models said, oh that's not that far, you should walk again. Supposedly they have now hardwired with pattern recognition. If we see any variation of that question, don't even send it to the LLM, just say like you should obviously drive because you have to wash your car. All right, so this what guardrails are. We've been doing these since GPT3.5.
- [08:17]
B
They're very available.
- [08:19]
A
So when someone says they've jailbroken a
- [08:21]
B
model, what they normally mean is they are able to get it to give
- [08:25]
A
a response about something even if you
- [08:27]
B
had fine tuned it not to give responses for it.
- [08:29]
A
And there's a couple ways to do this, but to understand how this works
- [08:31]
B
very conceptually and this is rough.
- [08:34]
A
So RL nerds, please restrain yourselves for this explanation.
- [08:39]
B
But essentially when you reinforcement train these models with these sample questions and safety
- [08:44]
A
responses, you can imagine what you're doing
- [08:47]
B
is when the model has recognized the
- [08:51]
A
pattern of one of these type of questions, you have sort of burned its
- [08:55]
B
weights to sort of divert the answer from those type of activated patterns towards the safety answer.
- [09:03]
A
So any sort of question, like the questions you trained it on, when those
- [09:06]
B
neurons are fired up in this sort of virtual parameter space, the model has been adjusted its weights to sort of go downhill from there towards one of these safety answers.
- [09:18]
A
So the way you get around these
- [09:19]
B
guardrails, you jailbreak them, is if you're getting at the question in a way that really doesn't fire up those same neurons, you're coming at it and obfuscated around the bow way that will often evade the guardrail. This is why, for example, I've read a bunch of these papers where they try to convince a, a chatbot to give it information about stuff it's not supposed to. I read a paper recently, I think from last year, Neurips, where they were able to get the major models to give conspiracy theory information.
- [09:48]
A
What they found is it took a
- [09:49]
B
while of prodding it until they finally done until it would finally tell it the earth is flat. And again, that's a similar type of evasion. When you make your context window long enough, it's that the attention space is so complicated that it no longer matches with the patterns from the RL training and now the those diversions towards the safety answers are bypassed. So guardrails, putting the nerd stuff aside, they're available. And this has just been the case with language models up to now. So, yeah, I have no trouble believing that the sort of whatever guardrails they added, from what I understand, there are guardrails to say I will not answer cybersecurity questions. I'm sure they're evadable. We've never seen a guardrail that we couldn't. Jailbreak is one way to think about it.
- [10:29]
A
All right, so the second sub question here is Fable 5 without guardrails or jailbreakable guardrails, a unique national security concern. This is where I'm going to differ with the government. I'm going to say no. Look, I did an episode about the original Mythos model back in April when it was originally announced. Now, remember, when the original Mythos model was announced, Anthropic went on this PR campaign to try to terrify people. They gave briefings to government officials, the banks and the reporters about how scary this model was, how it had this sort of unique, revolutionary ability to find and exploit bugs and software that was going to like, take down the whole infrastructure of the world if we didn't protect it. This is when Tom Friedman wrote a very scared article about kids would now be able to basically shut down nuclear reactors and some such right at the time. Here's what I said. I said, I don't think this is revolutionary. I think it's evolutionary. We have been using large language models to find bugs and security vulnerabilities since the beginning of large language models. With each new generation of models, they've been getting steadily better at this. We have no reason to believe, we've seen no evidence that Mythos is a revolutionary leap in that ability as opposed to just continuing the normal trajectory. In other words, there was nothing shockingly new with Mythos that made it unusually dangerous compared to like Opus4.7 or GPT5. We had some evidence for this point. For example, in its scare marketing campaign, Anthropic talked about the fact that they had found over 1,000 zero day vulnerabilities in code, some of which had been around for decades. This really got a lot of people scared. The problem is, you can go back to one of their earlier models, one of the earlier Opus models, and if you read like I did, a report that Anthropic released on their blog the same day as that earlier Opus model came out, they Said we found hundreds of zero day vulnerabilities, some of which have been around for decades. So it wasn't some brand new thing that Mythos could do that earlier models could not. Then we got multiple independent security researchers that said, okay, well we took some of the marquee bugs that Anthropic said they had found with Mythos and we gave that same source code to other models, smaller models, pre existing models, cheaper models, said, do a bug search on this and they found the bugs as well. Then we got other sort of independent benchmark testing of Mythos once it became a little bit more available. And it really fell into this pattern of like evolutionary incremental increases on these type of capabilities. And of course, maybe the biggest sign that Mythos was not this world changing bug finder is that Anthropic's own software remains very buggy and has security vulnerabilities even post Mythos. So I guess it hasn't been able to fully find all of their bugs. So what was really going on here? My contention was the original Mythos scare campaign was marketing. One of the other biggest pieces of evidence for that is bug finding. Pre Mythos was not what the AI companies were bragging about. These were not the capabilities that they were touting to try to emphasize the power of their software and all of its possibilities going forward. Bug finding is what we were doing with like GPT2. This is not exciting. So the fact that that is what they emphasized by Mythos in my mind was, uh, oh, we trained this new massive model and it got like a little bit better at everything. That's not exciting enough. We need headlines. What can we say? What if we say it's bug? Let's focus on it's bug finding. We haven't talked about that in a while. Oh my God, we built that ultimate bug finder. Nowhere before Mythos were they saying that was their goal or this was one of the big uses of LLM. So to me, that was a red flag. So what I think they did is they were hoping to play us both ways. Say, this thing is really terrifying, get a lot of terrified columns written, get everyone really worried about it, wait a few months, wait a month and a half, and then release it and try to ride on that sense of this is a super powerful model that will we then release a version to the public. They'll be willing to pay the much higher token prices that these larger models command. So they're trying to play us both ways, terrify us, and still just go right ahead without restriction with their plan of releasing and trying to make money off of that software. So I think that was just pure marketing. Again, this doesn't say that LLMs are not good at finding security vulnerabilities. This is not the saying that the new coding harnesses that took off starting last fall aren't good at writing exploits if you explain to it a vulnerability. But what I am arguing is that we never had evidence that Mythos was some sort of revolutionary leap in these capabilities that would merit the way that Anthropic talked about it or it uniquely
- [15:03]
B
being put on an export control list
- [15:05]
A
as compared to other cutting edge frontier models. All right, the second relevant question is, of course, should the government be more involved with AI? So if we are putting aside politics, like, is there a way we could
- [15:19]
B
imagine a more hands on regulatory regime
- [15:21]
A
that would make sense? Well, in order to understand what role the government should have shown at play here, let's look a little bit closer on the question of why the Trump administration did what it did with Mythos, because there's three different explanations that are
- [15:37]
B
out there right now floating around.
- [15:38]
A
The first explanation is corruption. The Trump administration has multiple ties to anthropic's main competitor, OpenAI. They've also already been in a bit of a feud with Anthropic dating back to March and the issues they had with the Department of War, where they
- [15:55]
B
placed it as a supply chain risk.
- [15:57]
A
So this would be very Trumpian. You are my enemies.
- [15:59]
B
Here's a way to like squash or screw with you.
- [16:01]
A
I'm just going to do it. The second explanation given for what happened, the government is legitimately worried about Fable 5 and the idea of China using a jailbroken Fable 5 to find security vulnerabilities in key software that's a part
- [16:13]
B
of our infrastructure and economy really scared them. And they said, we don't want this
- [16:17]
A
to happen on our watch. Reason number three, explanation number three is that they're calling Anthropics bluff.
- [16:23]
B
Anthropic spent six weeks trying to convince
- [16:25]
A
people that they had essentially summoned a demon with Mythos and that it was a grave, powerful source they almost regretted having brought into this world and that they, that they were just trying their
- [16:36]
B
best to steward this inevitable darkness and keep us safe.
- [16:38]
A
And then six weeks they're like, yeah, never mind.
- [16:40]
B
Here you go. It's. It's going to be $0.75 token.
- [16:44]
A
So another way of thinking about this is the government said, no, you don't get to talk about your product. As, you know, the worst thing since
- [16:51]
B
nuclear weapons and then just go and sell it without restriction when you want to.
- [16:55]
A
We have some evidence for this bluff calling rationale.
- [16:58]
B
If we go back to David Sacks,
- [17:00]
A
who said the following. Keep in mind that Anthropic itself widely
- [17:04]
B
promoted the idea that Mythos was a cyber weapon and needed to be regulated as such.
- [17:08]
A
They asked for government regulation of Mythos
- [17:10]
B
and championed the guardrails unfabled.
- [17:12]
A
If there's a vulnerability, big or small, it is Anthropic's responsibility to patch. All right, so let's go through these
- [17:17]
B
three possible rationales for what the government
- [17:20]
A
just did and said.
- [17:20]
B
Are any of these justified? Could we imagine any of these being the foundation for an actual sustainable approach to AI from a regulatory framework?
- [17:29]
A
Well, the first explanation was corruption, and
- [17:31]
B
of course, no, that's never justified. Unfortunately, given the recent history of our
- [17:36]
A
current administration, we have to assume that
- [17:38]
B
corruption or arbitrariness or revenge probably played
- [17:40]
A
some sort of role in whatever they were doing. We cannot have regulatory actions be implemented so capriciously.
- [17:46]
B
I agree with Gary Marcus and Dean Ball about the need for a transparent and consistently applied regulatory regime if we are to create one of those. All right, the second potential justification is the national security concerns. Is the government in general justified for blocking or restricting an AI model? If they have concerns about its ability
- [18:07]
A
to cause harm, yes. That is a good justification for the
- [18:12]
B
government to step in.
- [18:14]
A
You do not, as an American company,
- [18:16]
B
have a right to ship any product you want without interference from the government
- [18:19]
A
if it can cause massive harm, especially
- [18:21]
B
harm to us caused by our enemies. This is where the government should say, hold on again. You are not, as the AI companies like to make themselves seem, inevitable stewards of an inevitable technology. You're building products. If you build a product that's going to screw us. We don't want you to release it, just like you can't release medicine that's poison or cars that'll explode. What about the third just possible explanation here that they are calling AIs, the AI companies bluff? Is this ever a justified rationale for government intervention?
- [18:55]
A
I'm going to say yes. I'm going to say yes because I
- [18:59]
B
think this is a public health crisis.
- [19:03]
A
These companies have been trying to terrify
- [19:05]
B
the public for at least the last two years. And I find this strategy of wanting everyone to be unsettled all the time, unconscionable, strange, and baffling. And it has been incredibly successful. There is a pall of anxiety and fear and uneasiness that tens, if not hundreds of millions of people are feeling right now, because of the direct communication strategies of all these companies, to be continually trying to keep people unsettled and fearful and distrustful of what this technology is going to do, the psychic damage this has caused to our country unquestionably is much larger than any benefits that the AI technology to date has actually provided us. And I think the government has a role to get involved here. You do not get, without restriction to run a psyops on 300 million people because you think either it makes you feel important or you think it's going to help you and a small number of early investors become richer than, you know, Mammon, that is, the government steps in. Now, if you make a claim in your dog food ad that isn't quite right, and yet, hey, you want to just, like, continually see, have a game of who can terrify, like, the average,
- [20:20]
A
you know,
- [20:23]
B
the average citizen more about AI without any restriction or change to what you're doing, we're not just going to applaud it. I think that actually is a reasonable place. These are extreme circumstances. This is a reasonable place for government involvement. So what would it look like for the government to act on those justifications better than what we're seeing right now out of the Trump administration? Well, I want to return to the Trump administration because they did something recently which is a step in the right direction. On June 2, Trump signed an executive order titled Promoting Advanced Artificial Intelligence, Innovation and Security. This report, this executive order requests that AI companies voluntarily provide the federal government access to covered frontier models for a cybersecurity review up to 30 days before their plan released to other trusted partners. That's a step in the right direction because it shows, like, wait a second. The government should have some role in this. Technology is powerful enough. I would go farther, however. I would say it is mandatory, not voluntary. Companies must preemptively establish the safety of frontier models, which you would define probably in terms of parameter size before public release. In these reviews, the communication of the companies itself should be taken into account. So if you have gone around and talked about how terrifying your model is, or if, like, Anthropic did a couple weeks ago, and I talked about it last week on the podcast where they came out with a report, it's like, guys, we're looking at our own cloud code and this thing is getting close to recursively improving itself, we might lose control of it. But don't worry, we have a white paper in which we're going to walk through the possible future. So we're thinking about it at least you don't get to write that article and then release a new version of your cloud code. The government should be like, okay, you said they say technology can get out of control. Just like if you were a virology lab. I think this is the apt comparison. If you're a virology lab talking about your gain of function research, writing reports about this research we're doing could lead to out of control pandemics. And we're going to write reports that walk through the possible ways, like, society could crumble when these out of control pandemics come. The government would have every right to say, you have to stop doing the gain of function research. Then if you think this is a possibility, well, you got to stop right away. All right? So I think you should have to put frontier models up for review. How you talk about them is included in that review. The the government should have the ability to also retroactively go back and say, this model we previously approved, we are now seeing signs that it is causing grave safety concerns and we can revoke its license. Safety is actually going to have to matter. Now, this has to be done in a transparent way. There has to be consistent mechanisms. These mechanisms somehow have to be free from direct influence by Trump as mediated by whoever he happens to be friendly with or whoever gave him a lot of money.
- [23:13]
A
None of that's easy.
- [23:15]
B
But I think it is time for something like this. The onus for safety has to go back on the AI companies themselves, just like with every other consumer product we see. You're welcome to talk all day about how dangerous your products are, but you can't release them. You got to convince us and the government that of course we're not going to release something that does this harm. Of course this is not going to happen. This is a specific product. Here's why it's useful. We are responsible for its harms. Now, if we did have this regime, I think two things would happen that would be immediately positive. One, I think the worldwide psyops about trying to keep everyone terrified so these companies can feel exceptional and push forward to these big IPOs would come to an end because we were not gonna allow them to release products that they're trying to terrify us about. Two, I think it would lead to a narrowing of their products, which would be good. Instead of trying to do this, let's build the biggest possible frontier model and then after the fact, try to probe it and see what it can and cannot do and kind of ring our like. I hope it's not too smart. Build specific products. Here is our coding tool. We have to justify for you why we think it's worth the money it costs. And of course if it we're responsible for any harms. Like any other consumer product company. Right here is our product for, you know, memo writing. And we want to justify why it's worth the cost. And of course we are responsible for any harms you get. Narrow and responsible. That is the way that is the future of AI. That is going to be way more predictable. It's going to be way less distressing, it's going to be way more safe. It's going to have a lot of variety. The giant AI companies hate this model because hey, guess what? When you start releasing specific products, you don't need a 10 trillion parameter model. I don't think the AI companies want you to know that these frontier models are F1 cars. It's like McLaren or whatever or Ferrari or Red Bull, whatever. In F1 you put out these $20 million cars to try to win this sort of ludicrous race. We go 180 miles per hour so that people in general are like, oh, I like that car company. That's a good car company. So I'm more likely to buy the $60,000 car, which is all I need. Right. That's what they're trying to do here. These models are F1 cars for most of people's needs. Much cheaper models would suffice. Remember when I said a lot of independent researchers could duplicate bug identification using smaller models? Yeah, that's probably the case. You could have a bug finding system tuned to do bug finding with a good harness on it that's made to like test and look for bugs and be better at it. They could run on probably like a 50 billion parameter model.
- [25:51]
A
Right.
- [25:51]
B
A lot like coding harnesses. I think Cursor is discovering this. You can build these custom models that are not massive and are perfectly good at producing computer code. And you put the right harness on it, it's very effective. Much cheaper to run. This is the reality. We need the, you know, the I don't know cars very well. I was going to say like the Ford taurus. That's a 90s reference. Right. But we just need the normal consumer cars. And these companies are still working on the F1 cars to try to impress everyone so their IPOs can succeed. They don't like this message being out there because a world of narrow AI application, they don't have an advantage anymore. Like who's going to have the advantage? Computer coding?
- [26:30]
A
Well, Everyone, these models aren't so big.
- [26:31]
B
And then it's who's going to build the smarter harness. And some company that does nothing but think about harnesses is going to have a tool that people prefer. Like, there's no moat. So they want it to be about Mythos. They want it to be about GPT 5 5. Right. They want it to be about these massive models and finding one way or the other to justify them. That's why they have to constantly put out these articles of like, this thing is a national security threat. This thing is going to improve itself until we lose control. They have to make it seem like only these frontier models that only have the capital to make are the only things that are smart enough to be useful. But they're not. We could have much narrower tools that do useful stuff and they're justified and they're affordable and they're not at all something we worry about. That's the future we need and a real consistent and transparent regulatory regime that says you have to convince us your model is safe before we release it. You don't get to just build whatever you want. You don't get to just say whatever you want about a model. The time has come from that. The damages being caused right now psychologically have been massive. We the damages economically that are going to happen if both OpenAI and Anthropic have big IPOs and then the bottom drops out on this, the impact that's going to have on 401ks that are holding index funds is also going to be calamitous. This is time for the government to stand in, but they have to do it with more transparency and honesty than I don't know if the Trump administration is even capable of doing, but that is how I feel about it.
- [27:52]
A
All right, so there's a lot of
- [27:53]
B
things going on in this story. Is the Trump administration handling this in a way that's haphazard and at best and corrupt at worst? Probably. Is Fable 5 a unique danger? Probably not. But what I care about most is
- [28:06]
A
that this incident points towards a potential future in which the government gets off the bench and stops treating these AI companies like some sort of untouchable priestly class and say, wait, who says you get to launder anxiety and potential destruction to your own personal wealth? You don't get to do anything you want to. And if they're not willing to be responsible in both their rhetoric and product design, then someone needs to step in and help put these AI companies back in their proper place as normal consumer product companies. That are beholden to the same type of rules, restrictions, expectations as any other consumer product company. All right, that's all I have for today. Join me on Monday for an advice episode of the show, and I'll probably be back the Thursday after with another AI Reality check. Until then, remember, care about AI, but not everything you read about it. All right, see you next time. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up@calnewport.com ideas and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today. Calnewper.com ideas.