Arvind Narayanan (4:10)

Stop. One thing I strongly believe is that we should not be thinking about this in terms of probabilities at the moment. You're arguing about what the probability is. You're already down a very confusing path that can only lead to, I think, misleading guidance. So I've looked at the most sophisticated efforts that we have for estimating these probabilities. It was led by the Forecasting Research Institute. I actually work with these folks, I know them well, they're incredibly smart and they did a really well thought out effort to get dozens of expert forecasters to discuss, try to change each other's minds and provide these probabilities. And they put out a 753 page report. And I've read that 753 page report. And you could have a room with these so called superforecasters debating and you could have a room with a bunch of people who are high and debating what the future of AI is going to be and you can't tell the difference. And this is no slight to the superforecasters, some of whom I know, they're incredibly smart people. But the thing is, we have no empirical basis for predicting what these probabilities might be. The arguments that people are giving, it's things like, oh, AI might decide to colonize space instead of Earth. So even if we had super intelligent AI, maybe the probability is not as high as we think. Or someone else thinks, oh AI might decide that killing all humans will make the planet cooler and it helps computer chips work better. And so maybe it will decide to kill all humans. And so, you know, they're listing a bunch of reasons like this, assigning some numbers to each of them and then multiplying all of them at the end. Right? So this is the best method that we have. So these probabilities are all bogus. That's my strongly held view. We should not think in terms of probabilities. I do think the risks are potentially real. I'm not advocating for ignoring the risks, but I think the right response cannot be, let's try to stop all this. There are two big problems with that. One, it's just not going to work. The only way it could work is if you have an authoritarian world government that can control every AI developer everywhere.

Matt Clifford (6:27)

Can I pause on that, Arvind, for a second? Because that's an empirical claim. I mean, at the moment, these large language models basically can only be run by some of the largest, wealthiest companies on earth.

Arvind Narayanan (6:39)

Not true.

Matt Clifford (6:39)

With enormous data centers with enormous compute power. So it seems plausible at the moment that if President Trump, driven by Christian nationalists and Xi Jinping, wanted to simply shut down the large language models of two Chinese companies and a small handful of American companies, we would cease to have these LLMs operating pretty quickly.

Arvind Narayanan (7:06)

This is not true at all. It might be true that absolutely the most powerful frontier models can only run on powerful GPUs, but you have, you know, slightly smaller models that are maybe one step below that can run on consumer grade hardware. And the cost of running these models is dropping by something like somewhere between a factor of 10 to a factor of 100 every year. The cost is dropping very rapidly, both because the hardware is getting faster per dollar and because algorithmic improvements are allowing us to squeeze more juice out smaller models.

Matt Clifford (7:40)

Again, I suppose to push this argument to the next phase, the claim only needs to be that the existential risk is posed by the Frontier models. And if one could shut down the full infrastructure that powers the frontier models, then one has less to worry about from the models that you're talking about.

Arvind Narayanan (7:56)

Now again, I completely disagree with this. I think historically this is very easily falsified. When OpenAI built GPT2, which was two generations before ChatGPT, which was when the world started noticing, they thought that model was so dangerous that they were not going to release it for people to download and use. And that's something that my grad students can build today, you know, just for fun and learning in a day or two. Right. And so historically, you know, when we look back, our ideas around what constitutes the threshold level of danger have kind of been comically off. And I don't think there is really any clear relationship between the power of a model in terms of how computationally heavy it is and what dangerous things it might potentially be able to do in terms of enabling cyber attack. In terms of various things that worry about in terms of biorisk, actually some of those models can be much, much smaller and faster than these large language models because those biological capabilities are not about language at all.

Matt Clifford (9:04)

To understand this first argument, this is a the cat is out of the bag argument. It's too late, the stuff's built. If these models are going to become deceptive killer robots, they're already out there and there's nothing we can do about it.

Arvind Narayanan (9:20)

It's not just that, that you can't even prevent these models getting even better in the future unless again, you have this authoritarian world government that stops even small teams of developers from acquiring consumer grade hard and doing research to improve the efficiency of those models and their capability.

Matt Forde (9:38)

I want to say, I want to go on the record as being against authoritarian world government. I mean, I'm very interested to sort of use this as a way to think about what we should do. And you know, I'm a little bit biased because I'm one of the people in the UK who, one of many people who helped set up the AI Security Institute which does these like pre deployment evaluations in UK government on security adjacent risks on a voluntary basis. There's no regulation that requires the companies to do that, although that they have agreed to do that. And you know, like my view has been to your point, I guess in agreement with what you're saying, Arvinda, like it's very, very hard to know. Like you know, when people say we should pause now, we should stop now, I get the impulse, but I'm like, well how do we know? And I think like governments having better knowledge of what capabilities actually are, what it can and can't do feels like a very valuable thing. There's a lot of people that look at what we've done with, with that institute and said it's amazing that government's built that capability, but it has no teeth. But what do you think we should do? I mean, it sounds like you actually, you referred earlier to the idea of having, like, transparency requirements. Do you have a sense of, like, what the right regulatory framework is, given both your sort of skepticism of some of these scenarios, but also you're concerned clearly about some of them as well?

Arvind Narayanan (10:53)

For sure, yes. And I think the AI Safety and Security institutes have been a wonderful thing. I've strongly supported the work that they do, no question on that. And yes, I think transparency and knowledge are the most important thing for now. But I do think there is a lot more that can be done that should be done. One big area is developing AI for defense against these very risks that people are worried about. And historically, this is always how it has worked. We're acting as if AI is a new thing, but AI has been gradually improving out of the public eye, admittedly for decades now. And the reason that things have worked out is because of the attacker defender balance and the efforts that we've taken to shift that balance in a healthy direction. So when it comes to cybersecurity, for instance, automated tools have been in many ways superhuman at hacking for decades now, and yet the world hasn't ended. In fact, things have gotten a lot better. And that's because companies who are developing software use these very same automated tools to find and fix bugs before they put software out there before attackers have a chance to take a crack at them. And that's the critical thing. When these AI models are now becoming even better at automatically finding vulnerabilities, do we have the right incentives in place to ensure that software developers, as well as mom and pop businesses who might be deploying software have the know how, have the funding to use this to protect their cybersecurity? Those kinds of things can benefit from government incentives and can be enormously beneficial?

Matt Clifford (12:29)

This moves us on to a new thing. So the first disagreement you've clearly had with Yoshi sure and Geoffrey Hinton and others who were calling for a stop is your cat's out of the bag. It's too late anyway. The earlier models can do a lot of damage. Stopping the next frontier model is not going to make much difference. Argument. The second argument that certainly people like me would make who are anxious about this is for goodness sake, what is it about the Trump administration or the financial incentives or the psychological personalities of these people running these models that remotely convinces you that they're incentivized in any way to really put proper safety regulations in place? Use the AI to counter themselves. Every time I meet them, they are in an arms race. They're saying, government, back the F off. Government doesn't understand what we're doing. They're never going to understand what they're doing. They don't know what they're regulating anyway. Get out of our way. Here we go. I've got to get there before and then insert the name of somebody running one of the other models who they think is unbelievably dangerous and is going to destroy the world. And I've got to build my model before insert Elon Larry Demis Mark. Right. And if not, I got to build it for the Chinese, because if I know the Chinese can build it right, none of this gives me any sense that we are in an environment in which these people are either they or the US Government are getting remotely behind safety.

Arvind Narayanan (13:57)

I am not saying that we're in a good place right now with regard to regulation, both with regard to safety regulation in the sense of these catastrophic risks as well as the kinds of safety issues we've already seen on a massive scale. Some of the psychological issues that we've talked about, the use of AI models for making non consensual nudes, these are problems that we knew were happening. The nudes one, at least at this point for something like six years. And it took forever for policymakers to start acting on these things. And so the pace at which policymakers are responding is definitely inadequate. But I think at the same time the arms race narrative is a little bit oversold. I think what we're seeing in a lot of cases is that when regulators and policymakers in one company are falling behind, the harms for the most part are felt locally. So for instance, in the US People are realizing that with respect to the safety harms around the psychological impacts of large language models, it's affecting Americans. And now policymakers, both at the federal and the state level are actually taking this very, very seriously. And I think we're going to start to see a mindset shift and we're going to start to see policymakers stop buying this arms race narrative. And they're going to recognize that a lot of these harms are actually going be felt within their own countries. And the way to regulate is not just at the level of the model developers, but how they're deployed. And not just on the companies such as OpenAI and Google, but also everybody else who is using these models within their own companies and try to get ahead of risks like people trying to hand off the running of their companies to large language models. For instance.

Matt Forde (15:53)

One observation I have on that, from both my own time in government in the uk, but also spending a lot of time time in Washington talking to people, is that to paint with very broad brushstrokes, the executive tends to see this technology through a security lens. And there's lots of both institutional and other incentives that make that happen. And typically legislators tend to see it through a sort of what you might call like an online safety type lens. And you know, certainly when I was in number 10 here, we would have these high side conversations about AI and security and then we'd go and see MPs who were worried about their kids or the kids of their constituents talking to chatbots. And I think I may have mentioned this on a previous episode, but you know, in 2025 I, I went to a very interesting conference of faith leaders talking about AI who totally rejected the security framing and really wanted to talk about, you know, what is this doing to kids and their minds. And I suspect that over the course of 26 and 27 we might see this sort of like quite executive security dominated framing kind of actually diminish as, as some of these cases in particular that you've referred to Arvind kind of come to the fore and actually see legislators sort of be a bit noisier voices in this, talking about these sort of more sort of online safety type harms.

Arvind Narayanan (17:10)

That's exactly my prediction, yes. As these models get more diffused into society, we are going to see an increase in the salience of that set of concerns. And I think the good news is that a lot of the interventions, not all of them, but a lot of the interventions for both sets of concerns are similar. And so I think the momentum around one kind of safety issues can also be used to address the other kind of safety issue, Europe.

Matt Clifford (17:34)

If I was speaking on behalf of the Commission or European leaders, they would feel they have been making a good faith attempt to try to work out how the hell you talk about these different type of risks and how you present regulation. And the response that they have got from Silicon Valley, they feel is one of unmitigated contempt that Europe's stuck in the Middle Ages. This is grotesque regulation. This is just another sign of a kind of sclerotic, aging, useless economy. And we will use every tool in the power of the American executive and corporations to punish Europe for trying to regulate us. And we will use every lever we can to break those regulations open to let us in. And again, that doesn't make me feel that this is a culture that is very sympathetic towards people trying to think about safety or security.

Arvind Narayanan (18:27)

Yeah, I agree. I think Europe's approach is one that is democratic. It's not necessarily the only right approach. I understand why the US has chosen a different approach, but I think the hostility of American companies and the government to Europe's approach has been really problematic, and it's not one that I support.

Matt Clifford (18:48)

There's plenty more of that agreeable disagreement. To hear it, sign up@therealDisPolitics.com.