B (5:19)

Undeterred, the companies would say that they are in favor of certain types of regulations. They're not against any and all forms of Regulation, they just want targeted regulation. Or, or they want regulation when we have evidence that there's risk or they want regulation that is narrow in scope. What do you say to that?

A (5:42)

So I think this is funny because, again, emeritus, the tobacco playbook, one to one. So I started my career as a lobbyist. And what I can say is a bad lobbyist will say, we want no regulation. A good lobbyist will say, well, obviously we want regulation, just not exactly the one being proposed now. And every single time, it will always be not the one proposed now. This is fundamentally just what a sophisticated lobbyist does instead of a naive and incompetent one. And this is exactly what we're seeing with these companies, where, of course, they will have many arguments to defend their position. But time and again, when faced with reality, when there is regulation being discussed, they will always oppose it, try to water it down, try to get off the shelf. And they do not propose anything alternative, which, you know, so in the end, look, you know, look at the hands, not at the mouth. It doesn't really matter what they say. This is, you know, mostly pr. What matters are the actions. And again, it's very similar to how the tobacco industry operated, especially on the last one that you mentioned about how we need more evidence. You know, on one hand, you have the CEOs that very clearly know and have admitted that this technology could cause human extinction, the development of superintelligence. And so, you know, regardless, you know, it's a, it's a future event, but it is a risk that they are very well aware of. And some risks are simply not worth taking, especially the ones that involve killing everybody we know, including likely themselves. But, and, but on the other hand, there is this delaying tactic of saying, we always need more evidence. You can always get more evidence. Right? You can eternally get more evidence. There is no natural limit in reality that says, oh, now the evidence is enough. Magically, we have enough. So, for example, what tobacco used to do was to say, well, okay, when the evidence started to become overwhelming that many, many scientists were talking about, obviously cigarettes cause cancer, they started saying, well, Dr. Sure, maybe that's the case. We don't really know. We need more tests. We need more evaluations here. See where you've heard that word before? But of course, if you find the exact chemical that causes cancer, we are very happy to see that regulated and to see that removed from a product. But we haven't found it yet. So unfortunately, we have to keep producing these products and we should not be regulated until the exact Chemical is found, and this is again just fundamentally delaying tactic with tobacco. Governments and the public said a very simple word. No, I don't care. We see with significant amounts of evidence and significant amount of theory and your own admissions that this causes cancer. I don't care that we have not found the exact chemical. You will be regulated now if you want to keep operating, find a way to make your product not cause cancer. And in the end, a lot of large tobacco companies are still there and doing pretty well. Their stocks are doing pretty well. They've diversified into other products that are less carcinogenic and so on. So this is just what companies always say, that they want to dodge regulation. Let's not give too much credit to this and let's focus on dealing with the risk.

B (8:59)

What we're seeing at the moment is that the capabilities of these AI models are jagged, meaning that they are very capable in one domain, for example, mathematics or coding, and less capable in other domains like interacting with the physical world or say, writing might not be fully there. And so while I think it's true to say that the companies want to create superintelligence, how are they doing at that mission? How should we evaluate how close we are to, to them actually developing and deploying superintelligent AI?

A (9:40)

Yeah, so none of us can exactly predict the future, otherwise we will all be billionaires. But I think they're making very significant progress, and I think they know that they're making very significant progress. It is true that AI we are seeing, that it doesn't exactly follow the same basket of skills as humans. And I think it would have been strange to even expect that. It is a very different form of intelligence. It's a form of intelligence that we have developed in the past years and made many breakthroughs in the past years that we also fundamentally, not even the creators of this technology understand how it works internally. So it's normal to see a lot of alien behavior. But I think a lot of this is explained by what the companies are focusing on. And these companies are focusing on one approach to superintelligence, which is to make AI systems that can automate AI R&D, and then to cause what some people call an intelligence explosion or recursive self improvement. And so these companies are allocating billions of dollars to automate that specific task because they think this one task, automating R and D, which is a subset of automating software development, will accelerate everything else to superintelligence very, very quickly. And on that, we are seeing enormous progress. We are Seeing now with cloud code, people that before never wrote a single line of code can create entire apps from scratch. People like nitpickers will always say, yes, but there's going to be one bug in the app. I'm like, oh, it doesn't exactly look as I intended, sure, but have you ever worked with, have you ever developed software yourself or have you ever worked with a human software developer? We don't have telepathy. If you work with a fellow software developer, sometimes they will also make bugs and they will also not exactly design what you do and it will take them much more time. They're human, they need to sleep, they need to eat, they need to take breaks, they get bored. AI has none of that. You can even run multiple AIs in parallel and pick the best app out of 10 you are developing in parallel with the same specs.

B (11:46)

It's always useful when having this discussion to zoom out a bit and, and if I look back just a couple of years ago on this podcast, I was discussing with people who were really in the know about AI, whether AIs would be able to do things that were sort of formal, so coding and mathematics, that seemed like a barrier that was difficult to overcome. And now this is an area in which the models are extremely strong. Perhaps these are the strongest capabilities of models. And so things are moving quite fast and yeah, things can change or capabilities can change quickly.

A (12:22)

Yeah, exactly. And also this side of the capabilities can unlock a lot of the other ones. Precisely. And this is clearly the bet of these companies. These companies are hiring for automating AI, R&D engineers. One of the first companies to openly have roles for this, literally called Automate AI R&D Engineer was DeepMind. And the other companies are following suit. They're giving interviews, talking about how they want to initiate this recursive self improvement loop. Perhaps this year, perhaps next year. This is one of the co founders of Anthropic to the Guardian a few months ago. And again, should we believe them that it will happen exactly at that point? Maybe not. I hope not, given that recursive self improvement is an extremely dangerous path to superintelligence. But the trend line is clear. They have tens of billions, in some cases hundreds of billions of dollars investments are coming in. They are deploying these investments very effectively to automate what they care about, which is software development and AI, R&D. And they're going as fast as they can with no regulation and no oversight. So we can bet against their money, but we'll be taking a very Very risky bet. And at some point somebody will crack this.

B (13:35)

Yeah. For superintelligence to be dangerous, it must interact with the real world. It must change things in the physical world. And I think what we're seeing is that these models perform extremely well on benchmarks. They perform, they quickly saturate many benchmarks such that we have to make up new benchmarks for how smart these models, for measuring how smart these models are. But what we don't see, at least not yet, is evidence of strong evidence of economic impact. So job losses, GDP growth, unemployment numbers, that sort of things, maybe interest rates. What do you think accounts for this discrepancy?

A (14:15)

I think this is much of a feature of the human economy rather than of the AI itself. So in many cases, and I think this is also why a lot of people have a model of how things will go towards super intelligence, where it's some kind of like gradual change, where at first we see the AIs only in toy environment, then we see the AIs doing very productive things in a few settings, then we see them deployed across the economy, all humans essentially replaced out of jobs. And then eventually we die out as a species, either because the AI just doesn't care about us, or the economy is now just run by AIs for AI. So there's no food, no entertainment, and no, no medical help for flesh humans. And while I think that's possible, I don't think that's what we're seeing. And I think it shouldn't be the mainland expectation, because ultimately there are a lot of rules. We forget this on AI because there is no regulation, but there are a lot of rules that constrain how things can be done on the human economy. For example, I expected a lot of current AI systems could automate a lot of entry level jobs. But in a lot of areas these jobs are protected by unions, they're protected by regulation, they're protected by cultural norms where people just want to have a human colleague instead of AI systems. I think some of these things are good and some of these things are things that it is valuable to maintain human culture even in the face of AI. Some of them are not necessary and not relevant for accidental risk. But fundamentally, I think what we're seeing is that the companies, especially inside their development, they have extremely powerful AI systems that could already disrupt a lot of things. And we shouldn't read too much into the adoption on whether these capabilities are not there yet. I think the capabilities are there in a lot of areas and simply adoption will always lag compared to capabilities. So we should be more cautious, not less. We should always expect the frontier. The most powerful AI systems are much further than the ones that are deployed at scale in the economy.

B (16:26)

Tell me a little bit about how you talk about these issues with politicians and members of Parliament. Perhaps we can start with how this works in the uk. What do you say to them? How do they react to this? Because you're coming to them with big propositions and you're potentially making big asks of them. So how do they react? What do you say and what do they say in return?

A (16:50)

Yeah, so kind of for background. So we started at the end of 2024, so a little bit over one year ago, with a very focused and clear theory of change, which was most people have never heard about this problem. We've had top experts, Nobel Prize winners, CEOs, admitting that AI can cause human extinction. We have had multiple of them sounding the alarm about superintelligence. We had letters like the fli, Paul's letter even before that, showing that many, many top experts across the field are concerned. But this just had not reached the vast majority of people and the vast majority of policymakers. It's not that they heard about this and didn't believe it or thought this was ludicrous. They just simply never heard about it. We should never overestimate how much reach one single thing has, especially in the AI field, which is very often very much a bubble of people that know each other. And so what we set out to do was we want to systematically inform every single lawmaker, starting in the UK and then in other countries, about the risks, about where AI is going and about what the solutions are, and make one simple ask of them. Support a public campaign so other people know that you know, so you build common knowledge so you can show others not only you care about this issue, but you know about this issue. And, you know, if you're a politician, it's okay to look into this issue. It's not something weird, it's something that is happening that many of your colleagues know. And when we started, a lot of people told us it's, you know, including in politics told us it's going to be impossible to get lawmakers support a campaign that says the word extinction risk, even though this is what Nobel Prize winners and even AI CEOs have said, or they will never understand superintelligence. This is too far. It's. Talk about something else. Focus on something else in practice now, We've done over 150 meetings, and one year later, we have more than 100 lawmakers, the largest political coalition of this size on the planet, on this issue, recognizing extinction risk, recognizing that superintelligence poses a national and global security threat, and calling for binding rules on this, on these most powerful AI systems, a targeted regulation on superintelligence. And I think there is no secret sauce. The secret is to be honest, to be clear, to find a way to express all of the things that are happening with AI, which are a lot, and some of them are quite complex, in clear, informative and simple terms. And also just to listen, to, not go in with assumptions about exactly what parts they will find interesting, but see where are they coming from. Have they heard about AI before? In some cases they will not. Those are a few cases. In most cases they will have heard of AI, but they will have likely never used any of the most powerful AI models. And they will just even not be aware of what the companies are building. They might be thinking these companies are just building some chatbots. And the chatbots are a bit like a more interactive Google search where you can find some information. And I think this was the front facing product a few years ago. This is not the case now. But most people don't know, again, most people don't use AI on a daily basis. They don't know what these companies are building. And so it's about explaining clearly and straightforwardly what the companies are building, showing the facts, showing the expert concern, showing the plans of this company, showing how already we are seeing that a lot of theoretical concerns that were in the field of AI since its inception, essentially since Alan Turing, are becoming real and happening in the real world. And in testing things like AI models that refuse to be shut down, AI systems that are now so competent at hacking that they can escape their environment and hack out if they cannot solve a task, sometimes threatening to blackmail or even threatening to kill humans in test settings. And of course, this is still happening on a fairly controlled scale. Their AI systems are not smart enough. But I think a very important thing is once people start seeing this, they quickly connect the dots. And most people, including most politicians, do not want a world where humans are overpowered, replaced or exterminated by superintelligence. And they very, very quickly connect the dots from, like, where things are going now to where they will go in the future. And they overwhelmingly support keeping humanity in control and putting regulation on this dangerous technology.

B (21:36)

How useful are demos when you're meeting with politicians? I could imagine that just showing up with the latest, most agentic models, showing them synthesizing documents and putting together a report and doing some work that the politician might have had some of his or her employees do before that might be impactful. Because to some extent seeing is believing. And seeing an AI use a desktop or use a laptop is that impressive to people.

A (22:08)

I think a lot of demos rely more on the communication is more important than the demo itself, because in many cases, ultimately because of what we discussed before, right now the AI systems are at their most powerful on the domain of software development and using code. And fundamentally code just doesn't look that impressive on a computer. Right? It's just a bunch of text on a computer that somebody, many people won't even know what exactly it means. And they're doing extremely powerful things with that. With code, you can essentially access and operate a large amount of the modern economy and a large amount of physical infrastructure as well. You can operate a computer. We've seen the whole cloudbot and Maltbook thing recently, but that just on its own just looks unimpressive without explaining what this actually means and what this actually is doing. So I think what really matters is to just first of all simply show the facts. Now we have an overwhelming body of evidence and a body of warnings from experts that are just undeniable. This is not AI. In the early 2000s, where there were a few people dreaming of AGI in secret, we have again, Nobel Prize winners, even the ICOs themselves, acknowledging literally that AI poses an extinction risk on par with nuclear wars. We have top experts sounding the alarm. We have some of these examples of where AIs are already misbehaving. And also in their experience, they're also starting to see more that in general there is a great unease in the public about AI due to a lot of other things that are happening. We see the deepfake scandals, we see, while we're not seeing it in the statistics, sometimes there is definitely unease about the effect on jobs also because of the stated goal among these companies, which is to just automate most tasks. So the goal is quite clear even if the statistics are not. And so I think fundamentally the most important thing is just to break down the complex and sometimes jargon filled things of the AI field. And demos can help, but they only help if you're actually explaining what they mean, because code on a screen doesn't mean much unless you explain what it is.

B (24:27)

And so when you talk to politicians, what are the types of questions that they're interested in? You mentioned having an open mind about what they might be interested in. And so I Imagine you get a range of questions. What type of questions are you getting? And is there a difference between people from the left and from the right?

A (24:46)

So actually, this issue is extremely bipartisan, and so we're not seeing. And also recently I testified to the Canadian House of Commons, and also there, I didn't really see a difference between questions coming from the left or from the right. I see a lot of concern after hearing about the risks. I think a lot of people are already uneasy about where AI is going. They're starting to hear some of these warnings also from other sources, from tv, from people like Geoffrey Hinton that has been a great communicator about this. But they want to understand, okay, what's happening? What are the companies building? Are these risks as big as they seem? And the unfortunate answer there is, yes, they are. And sometimes they're bigger than they think they are. And then I think, quickly, the crucial part is about. It's less about understanding the risks. I think the risks, they understand them very well. In many ways, the case is obvious. If we build systems that are smarter than us across the board, and we cannot control them, we are screwed. This is just a fundamentally terrible idea. And I think there is many very good human instincts that all point to this. We have many years of human history reflecting on this issue and seeing, like, if you have something vastly more powerful than you that you cannot control, you are in trouble. You should just not put yourself in that position. So that part is very clear. Where the questions often come is, okay, but what can we do about this? And I think this is also where the AI companies are shifting with their lobbying. Rather than focusing on denying the risks, it's becoming harder and harder for them to deny the risks and keep people in the dark. They are focusing on spreading a feeling of impotence, a feeling of nihilism, of, you know, nothing can be done anyways. And we need to have an. We need to race because of some excuse. You know, there will. There will always be an excuse of the day. And so I think more of the questions are, I, you know, I understand this sounds like an extremely big risk. Like, obviously, we shouldn't do this, but what can I do? What can my country do? What can I do in my role? Is it really possible to prevent this? What can we do about this? I think that's a good place to be in, because it's going to be hard. I'm not going to mince words. It's going to be hard to prevent superintelligence and build a different path. But it's very important to move from learning about the risks to actually thinking what action can be done to keep humans in control.

B (27:22)

And so if you are a UK politician or a Canadian politician, maybe you're thinking, well, this seems like a big issue. This seems like it could change the world. But this is mostly about the US and China. We are not really at the cutting edge of this technology. And so what can we do? And what do you say to that? I know you've been thinking about developing a strategy on this.

A (27:47)

Yeah. So I think it's. There are like a few. A few stages. First of all, as we've seen, and you know, this was what our entire theory of change was predicated on. Common knowledge makes things move very fast once it gets going. Right now, the biggest botnet, I think also in the US and likely in China and other places, is that just most people, especially most politicians, just do not know what the risks are. Do not know the companies are building super intelligence and do not know what the experts warn about. And until that changes every, like, doing things will always be very hard. Like, ultimately, people need to understand that super intelligence is being built. This is the goal of the companies. It's not just chatbots. If it was just chatbots, you know, we, you know, I wouldn't be doing this job. I love technology, and I think technology is very useful. I am not, you know, I think there are some social issues with chatbots, but I'm not concerned about the end of humanity from chatbots. I am concerned, like top experts and even the CEOs are about the end of humanity from AI systems that can overpower us and can replace us at all tasks. And so the first step is for them to help build this common knowledge at home and abroad. And information can move very quickly. These cascades can happen very quickly. So this is why, for example, our first ask in the UK is, will you publicly support our campaign? Where they recognize the expert warnings, recognize superintelligence specifically, it's targeted. It's focused as a national and global security threat, and call for oversight and for regulation. And the more this happens, the more their colleagues, the more people outside, the more the public will see and look like, wait a minute, this politician is now talking about this. Wow, I'm going to look it up. I'm going to Google it. Okay, it's real. This politician is talking about this. We have Nobel Prize winners warning about this. We have CEOs warning about this. What the hell? It's actually happening. This is not just something I Dreamt up. I need to do something. And the more this happens, the more this can snowball. What we're seeing now is now that we have all our 100 lawmakers. Just in January, just less than one month after we reached 100 lawmakers, we had two debates in the House of Lords about superintelligence risk. One about what are the threats to the UK itself, to UK national security. As the UK intelligence service MA5 warned about the threat from autonomous AI systems to national security. And one about should we have a international moratorium on developing superintelligence and this is the next step. We need to have this national conversation. We need to have politicians discuss with each other across the political spectrum about what's happening, what can we do about this? This builds knowledge, this builds the ability to have a bigger coalition and the ability to actually act. And then it's true as an individual country, there are limits to what a country like the UK or Canada can do. But this is much more than most people think, often much more than some lawmakers think. Because one, the UK is a major country, it's a G7 country, it's one of the P5 nuclear powers, it's a close US ally. One country starting to openly talk about this issue can quickly lead to a domino effect with other countries realizing okay, this is not something that is just going under our radar. It's fellow major country is discussing this, they're taking measures at home. Should we look into this as well? This happens over and over in history with policy measures where like one country starting to champion an issue will quickly lead to a coalition of others looking into it and can quickly move from nothing is happening at all to there is major action happening around the world, including with the US and China. And you know, I think especially the UK can play a key role in being a champion of this issue. Discussing with the us, discussing with other like minded allies and build a coalition that clearly says superintelligence is a national security threat to each of us. It's a global security threat. We will take measures to make sure it's not developed at home, we will take measures to make sure it's not developed abroad. We don't want have to have the enemy at home or abroad. If you want to join, you know, we can join in a diplomatic compact where we make sure we all monitor each other and make sure that this is not happening. If you don't want to join and you're still developing super intelligence, even though there is this major coalition of countries saying that this threatens Our national security, we will take this very seriously. We will take this as any other major national security violation and we will respond in kind, you know, with pressure, diplomatic pressure, economic pressure, in extreme cases, even more. And this has worked with its limits with nuclear non proliferation. The US does not tolerate countries that are developing their nuclear arsenal that could threaten them illegally. We should get to the same stage with superintelligence, and once we get there, we're going to have a stable situation where we can make sure it's not developed anywhere and it doesn't threaten humanity.

B (33:07)

There's a category of objections here which we could call but others will do it or but others. Right, so if you're a UK politician, maybe you think, well, if we are not at the frontier of AI, maybe DeepMind just leaves the UK for the US and then advanced AI is developed in the US instead. If you're a US politician, maybe you think that if we don't push the frontier on AI in the US well then China will do it. That is sort of a very basic counterargument or objection here. What do we say to that? Or what do you say to that?

A (33:44)

Yeah, I think two things. One is what we're calling for is not to affect the majority of AI development, but it's to focus on superintelligence. So there's a lot of beneficial AI users, especially specialized AI systems. You mentioned DeepMind, things like AlphaFold. This is not what is in scope here. AlphaFold is a model trained on protein data. It's focused on a very narrow task, which is to figure out how do proteins fold. It's very helpful for science. It can have its own risks, but these are not the same risks that we're talking about with superintelligence. And that's not affected and should not be affected with superintelligence. Fundamentally there is a hard point which is if you develop an AI system that is smarter than you, that is the equivalent of, as some call it, a country of geniuses in a data center. And you don't control it, you are creating an adversary. It is just not in your interest to create an adversary that you cannot control. If a country saw dropping next to it the next day, an island with 50 million people that are geniuses, they are amoral, we don't know exactly what they want, and they are plotting to take over the economy. This would be like the number one national security threat for every single person in the country's military, security services, government. The next day, much like in many ways, much smaller terrorist networks become Very quickly, number one threat to deal with. So kind of fundamentally there is no gain from. There are gains on the way, but there is no gain from getting to superintelligence. The only actor gaining is the superintelligence itself. But regardless, I think it's important for countries like the UK or for middle powers to absolutely take some principal stances without crippling their AI development and economy. And some of them are every country has a duty and obligation to protect its citizens. The UK doesn't let people in the UK develop biological or chemical weapons, even if somebody else was developing biological and chemical weapons abroad. So a clear red line of no superintelligence development in the UK and so should go for Canada, for the US and so on and internationally, where I think we will then need more measures beyond just a ban on superintelligence. We will need ways to monitor this. We will need ways to monitor precursors of superintelligence. If somebody's doing it surreptitiously, commit to, to coordinate with allies and see what is a way that we can build an agreement here that doesn't cripple us, but also that makes sure that we can oversee and enforce this. And of course there is a trade off like this will cost some development in the short term. But most security trade offs look like this and we have done the correct ones in the past with nuclear weapons. The reason why we're, you know, here having this podcast is probably to a large extent because the US was very clear and didn't let the majority of countries on the planet just develop a nuclear arsenal. That would have led to a completely unstable world system where one accident, one crazy ruler of a country or one terrorist group getting access to nukes could have led to a full scale global nuclear war. We have a very limited number of nuclear able countries. We have limited stockpiles. We have the US actively preventing new ones from having this. And this is why we in practice there are many issues with it. But we have not seen a nuclear exchange since World War II, which is pretty good.

B (37:20)

You've talked about politicians and engaging with politicians, politicians and getting them to support the cause publicly. What about the public here? What role does the public play? How important is it to have a broader movement where people are informed on this topic and can sort of engage with it?

A (37:38)

Yeah, it's extremely important and I think much like lawmakers, the public is being kept in the dark. The majority of the public just doesn't know where AI is going. There is a general unease about AI, but they just don't know and have not heard that companies are developing super intelligence. And this is the level of risk. And kind of similarly with lawmakers and with others, the first step is to help the public understand, let the public know. And this just means repeating this information over and over, reaching people where they are at, reaching people where, you know, where they get their information from, where they get their information, channels from, their entertainment from and so on and help them, you know, let them know. I think we have a fundamental duty to like let people know about what's happening because this is being done to them without them even being aware or without their consent. And this is, you know, that's also what we do at Control AI so far we've reached hundreds of millions of people with our content so far. And we then find people that are interested in doing more about these risks. We provide them with weekly newsletter and call to actions to help them act. The important thing is not just help them understand but you know, just give them a feeling of great risk without giving them any way to do something about this. What's important is to also empower them to act. And I think the public can play a key role because yes, not everybody can dedicate their entire life or their entire career to this. But a lot of small actions do help and there is a lot of again, fud, fear, uncertainty and doubt from companies and kind of in some ways from the modern world at large about you are powerless as an individual. There is nothing you can do. The institutions will never do anything, nothing ever happens. And this is just fundamentally false. And it's a self fulfilling prophecy. If you think you can do nothing, you will do nothing. And if you do nothing, nothing will happen. And then we will get into more worse and worse crisis. But vice versa, small actions like contacting your lawmaker, just letting them know, hey, I care about this issue. This is what the experts are saying. I want you to take a clear stance on superintelligence. I wanted to take a clear stance to protect humanity. Makes a massive difference. This is something that people can do in five minutes. At Control AI, we've made tools that make it very simple to do this. And so far people have sent over 150,000 messages asking their lawmakers to ban superintelligence in the US and more than 10,000 in the UK. And this is only growing and we're growing this and helping people take more and more engaged civic action to deal with the problem.

B (40:19)

You mentioned that there's a general sense of unease about AI in the public. Do you worry that this general sense of unease might be sort of pushing in the wrong direction. If you're communicating to the public about risks of extinction caused by superintelligence, maybe that's mixed up with other worries about AI, worries about jobs, worries about this AI slob on my feet is unappealing. And if there's a general sense of unease, maybe that is not pointed in the right direction, does that worry you?

A (40:53)

So I think it doesn't worry me, but it just reminds me that I think it's very important to tell people the truth. It's important to give people the tools to act and to understand the situation so then they will do what they believe is best. And I think at the moment there is a general unease. But again, most of them, if you. If you survey them, once you inform them about the facts like the extinction threat from AI that experts warn about, or even just give them brief statements from the companies just describing what they're building, they become extremely supportive of things like banning superintelligence. So once they know about it, they will want to act and they will support it. But the main issue is that we're seeing is that right now, this is just not particularly salient. Again, most people just don't know about these risks and don't know about what the companies are building. This is changing. It's been definitely changing in the past year. We've been working to change it. But also I think there's just more information about this, more and more people speaking out. Just yesterday I saw that Hugh Grant said that AI poses an extinction risk, and this is extremely worrying. So we're just seeing this across a society. Finally, this information is propagating. But I think what's kind of the biggest challenge is just we need to give them this information on time. Otherwise, I think a lot of this concern will either fizzle out or get dispersed across a myriad of issues. So I am not worried. But I do think people should just more and more inform the public about the full scale of the risks, about what's happening, and show them that there is a way forward rather than just building nihilism or not giving them the tools to do something about this.

B (42:36)

On that note, if Control AI is successful, as successful as we could hope, what does the world look like in 2030?

A (42:45)

Yeah. So if we're successful in 2030, we have a ban on the development of superintelligence across all major countries. This is both enforced nationally, so these countries have a law in their own country saying that you cannot do this. At home. And they have an international agreement among them that says we are working together to make sure this also doesn't happen abroad. And with countries that are part of this agreement, they share information, monitor each other, and they make sure that nobody is developing super intelligence. And when countries, outside of disagreement or with rogue actors, they are treated like we treat other rogue actors. And so we make sure that they don't get access to the technology necessary to develop this, they are pressured to join the agreement or to just terminate their development programs. And if we're actually successful, we're in a stable world where no major country is pursuing super intelligence. Also, fundamentally, a very important thing is we're in a world where no matter what's on paper, no matter what's on the text of the law or in a text of an agreement, world leaders and the public deeply understand the level of risk and deeply understand what's at stake. And this deep buy in is fundamentally the end goal of control AI. We want to help people understand what's happening so they can make their own opinion and understand the level of risk. Because ultimately to deal with this issue, it will not be dealt with by one small amendment in the law that then is forgotten. Like if world leaders, or at least if a significant percentage of them, do not fundamentally understand what's happening and understand what the risks are, they will make the wrong trade offs when they're faced with a difficult decision. There will be many hard decisions to make and this deep buy in is needed. And at the same time, while superintelligence development is prohibited, we have a lot of AI systems that are in many ways much more advanced than the ones we have now. But they are just, they're narrow, specialized, used for specific domains. And I think this will still be a world where society is fundamentally transformed the way the economy is fundamentally transformed. Even if we press the magic button and AI development doesn't proceed at all in any area from today, I think society will be fundamentally transformed from the things that we talked about before. There is a big lag in adoption. There are a lot of activities that we do right now that AI already can do better than us. And that's just a very different society with different trade offs. But what the winning work looks like is one where nobody's developing a technology that can wipe out all of humanity. We have a lot of powerful AI systems doing a bunch of other things in narrow, specialized domains. They are scanning pictures for cancer, they are solving problems even more ambitious than protein folding. But they're firmly under human control. They're not so general and so smart. They can overpower people. And we are using them as tools to advance humanity's growth and prosperity, rather than as a new species that takes over the planet and needs to find a place for us.

B (45:56)

Say we fall short of a full global prohibition on superintelligence. Is there a world in which we model through with more limited regulation, like say we have banned recursive self improvement within the leading companies, or maybe we have transparency regulations or mandatory evaluations of the models pre deployment, something like that. Do we have a fallback option here or are we sort of going straight for the big win?

A (46:25)

I think we need to go straight for the big win. I think one important thing is there are some things that are fundamentally important and some things that are different ways to implement them. I think there are many ways to implement this restriction, this prohibition of superintelligence. But fundamentally, if we get to a point where we have AI systems that are vastly more capable than us, vastly more capable than all of humanity combined, they have replaced people across the economy. They make decisions, they are autonomous, they run around utilizing physical and digital infrastructure. This is a point of no return. We are in a world where even if we want to go back, even if we want to retain human control, we cannot. We're facing essentially a major powerful force that now controls our economy and we just cannot afford to get to a point of no return. There is no way out from there. So that's the non negotiable red line, how we get there. In the end, the goal is just to get there and to make sure that humanity stays in control. So the ways to implement it are up for discussion. We have some ideas, but they matter less than the endpoint. And same with this deep buy in. The fundamental win condition is that there is deep buy in about understanding how big the risks are. Understanding the superintelligence under current conditions is a terrible idea for humanity. If enough people have this, we have won even without any specific law, because those people will make the right decisions. Collectively they will. And when they don't, others will explain to them why that's the wrong choice. But vice versa, if we have just one law that passes accidentally that does say all of the correct words, but people don't understand the risks, we will still be in big trouble because enforcement will be weak and leaders will not know what to do about this.

B (48:19)

Yeah, it's interesting how what matters in the real world is not only what what's on paper, it's very much about what people know. And what they think about and how they act in the world. And so sort of information is also, it has, any, any written text has to be accompanied by an informed, an informed sort of bureaucracy for it to matter, I think.

A (48:41)

Exactly. And uninformed public, you know, the public checks, checks the bureaucracy and makes sure the things are on track.

B (48:47)

Yeah, yeah. How much, how much does timing matter here? So maybe we expect superintelligence in say two to five years, but what if it's more like 10 to 20 years? Which actions are robust to shifts in timing of superintelligence?

A (49:07)

I think this is why. Another reason why I like this theory of change is that it's very robust to any timing. Fundamentally, these are the steps that human civilization needs to take to deal with any major risk. There is no meaningful and stable way. There might be ways to delay a risk, but there is no meaningful and stable way to deal with such a level of risk that affects all of human civilization without going through these steps. Going through the steps where a large fraction of the public understands the risk, cares about it, asks their governments to intervene, scrutinizes companies, scrutinizes governments and demands intervention. A large fraction of decision makers know about this, understand that this is a threat to them even before being a threat to their country. It's a threat to their person, it's a threat to their families. Superintelligence taking over. I think sometimes we stay very much on the abstract. That's also what the companies do. But it's very concrete. If human extinction means I die and other people die too, that exist now, today. And fundamentally most people do not want this. And if they understand it's deeply, they will fight back and fight for human control. So fundamentally, the theory of change is the timelines don't matter too much. Maybe there are some situations where we just don't have enough time for the public environments to wake up. And those are words where there is not much to do. But obviously we need to do as much as we can and as fast as we can. And if we have the luxury of having 20 years, we will still have to go through this entire process. We will still have to go through a process where the public becomes informed, where decision makers are informed and they can act. And so the steps are exactly the same. What matters is only like how quickly this needs to happen for things to change. And while I would like for us to have 20 years, but this is definitely not the prediction of the majority of, of experts in the field. And nowadays there's even the joke that so called long timelines are not next year or the year after, even from some of the biggest skeptics.

B (51:19)

Yeah, we talked about recursive self improvement a couple of times here and this is one of the factors that could really speed things up. And so I think it's important to get a grasp on where that project is. We are speaking in early 2026 and it seems like people are. It seems like coding models are getting to a point where they're speeding up the research and development internally at the companies. This is hard to get sort of actual numbers on. So this is more about listening to what people who work at the companies are saying and playing with the models yourself and seeing the benchmarks, how the models are performing on coding benchmarks. So where do you think we are on the path to recursive self improvement? Is that something that's actually beginning to happen now?

A (52:13)

Yeah, so I think definitely the companies are trying extremely hard to get there. Again, it's very clear. They hire roles specifically to get AIs good enough to automate AI R&D. They advertise this very openly. They openly talk about a moment where we can give up control to the AI that improves itself and then hope that it will be benevolent or will keep us as pets or whatever. The cope of the moment is about human relevance. After AI gets there, as you're saying, it's difficult to know exactly where they are internally. What we can see externally is that these AI systems are becoming very competent at software development, which also has another side to it. This is why we are seeing these AI systems becoming very competent at hacking. Hacking ultimately is writing a malicious software and using it to do attacks. And as long as these models become more powerful at developing software autonomously, they will also get more powerful at hacking. So hard to tell exactly where we are. What definitely is happening is the companies are investing billions of dollars into making this happen. They might stumble on a breakthrough, they will accelerate things. It might take longer than they expect. But again, right now, without regulation, we're just facing this very dangerous gamble of somehow hoping that these companies fail at their stated goal, where they're investing billions of dollars and literally the smartest people, some of the smartest people on the planet on this task. So I think this is why it's very important to have a very clear red line of no development of superintelligence and B to restrict and monitor precursors like recursive self improvement. Even if somebody is denying that they're developing superintelligence, this is not the case now, but it might be the case once we have regulation that prohibits it. We need to look at things like, well, are you attempting to get AIs to just fully autonomously improve themselves? If you're doing that, obviously that's a way to get to superintelligence without saying you're doing that. And that's not okay. Much like we have with things like certain drugs, we monitor the drugs and we monitor the precursor chemicals, kind of the chemicals that you can use to make those drugs. Recent example in the US that is very salient is restrictions on fentanyl precursors, restricted drug and restricted chemicals that are used for that. We need to monitor these precursor capabilities for superintelligence, the kind of things that we would see before somebody can get to superintelligence and the kind of things that can accelerate drastically beyond human control. The development of superintelligence.

B (54:54)

We could talk about a spectrum all the way from autocomplete in code, so very simple auto completions, all the way to fully automated end to end research and development internally at the leading companies. How do we determine what is a precursor on that spectrum? Because it seems like, okay, autocomplete is just a faster way to do what you were already trying to do. But then it's sort of not easy to see where there's a red line being crossed here into. Now you're speeding up the development of frontier models by, by using this process. How do we determine this? What's a precursor?

A (55:41)

Yeah, so I agree with you that, for example, just coding assistance would not fall in this category for me. The thing is, the power of the law is that the law is not code. Law doesn't have to be written in exactly the exact specification that can be gamed. This is the case also with existing legislation. Things on like chemical and biological weapons. The definition is quite broad. And then it is on. The burden of proof is on companies, especially when they are checked to show we are not doing the bad thing. Most laws don't exactly specify an exact chemical or an exact thing. They specify a broad class of things and activities. For example, in this case, that clearly it would fall in the category fully automated end to end. You could use words like substantially. AIs are substantially involved in the design and development of other AIs substantially involved in the design and development of AI chips. And these are in practice things that a judge can determine. Judges are pretty good at doing this kind of stuff. You do it all the time with much murkier legal cases elsewhere. Software is not a special category it's just a more new category. And in the same way where we have anti money laundering regulation on banks, you could make similar arguments where like, well, but what exactly is the line for money laundering? Is there an exact amount or is it an exact profile? And generally it's a broader assessment. First of all, it's a normative principle of you should not intentionally do it. You should not intentionally try to make AIs fully automate AI or R and D. If you're just using copilot, that's fine. It should be fine. If you're starting to develop AIs and you let them run autonomously as agents and you check once a week, that's already starting to cross the line. But fundamentally, once we do have these rules in place, the incentives change and it's in the interest of the employee of this company to be protected, not even risk falling afoul of the law. Right now, companies are being extremely reckless because there is no consequence for recklessness with consequences. Even if a company tries to push them to do this, many employees will think, why should I risk this? Why should I risk doing this very dangerous thing? I will be cautious and stick to copilot and not to try experiments like fully automated agent led air.

B (58:12)

And if you have one company or perhaps a number of companies in one country developing superintelligence, it seems like that company or that country will quickly accrue a lot of power. And so they will be able to overpower perhaps other countries or other companies. And so there are two worries with this power concentration. One is that it will happen within one company, such as, say, Google Leadmind or OpenAI or Anthropic will become, will become much more powerful than other companies and perhaps more powerful than other countries. There is also the worry that if we set up a system in which we are able to control AI development, that in itself is a form of power concentration. And so should we worry about the ban on superintelligence turning into a. A sort of extreme form of power concentration where no one is allowed to develop AI in this way because we have an international system that prevents it.

A (59:19)

So all government regulation is some form of restriction on absolute human liberty. But this is the evolution that we have done with our institutions in the past 2000 years or more. We are able to achieve greater and greater feats because we also have decided to pool some of these individual rights in things like governments. Most people do not object to the fact that the government has the monopoly on violence. And this is a very normal thing of modern Society which wasn't normal 1000 years ago or more. This has made our. Of course there is a trade off where you have somewhat less individual liberty. You cannot just kill your neighbors. But this leads to societal outcomes that are better for everyone. And I think in this case, in the two scenarios, the one where superintelligence is developed is overwhelmingly the scenario where there is enormous power concentration compared to the other one. The important thing with preventing the development of superintelligence is this actually reduces the amount of power concentration in the world. Like no entity will be allowed to build an AI system that can literally overpower the government, the military, the police or the entire planet. Which as you said, is sometimes how these AI companies describe their systems. Recently Max Tegmark from FLI was saying in a press conference that he heard some of the AI executives talking about how they want their AI systems to overpower the government and to be powerful enough to or a power of the government. And this is true. You can kind of read between the lines in some of the blog posts and they definitely say it is more in private and this is A unacceptable to governments, B unacceptable to the public. This is a much more highly power concentrated word, one where potentially one company, although with super intelligence, the likely situation is soon just one swarm of AIs, no human will really have control to be like one swarm of AI that has the majority of the power in the world and into can do whatever it wants. So yes, I think all kind of, we need to be honest, obviously all regulation will reduce some amount of freedom. But these are trade offs that we make across many areas. You know, we don't allow private companies to build nuclear weapons. We don't allow private citizens to build nuclear weapons. You know, sometimes even people that do it as a, as a fun science experiment in the backyard, they go to jail because the US government makes sure that people don't just develop enriched fissile material in their backyard. And it's trade offs that we make to then have a prosperous society where we are secure and we maintain our national and global security. And in the world where superintelligence continues to be developed in the least best scenario for a while, we'll have likely not a government, but one company essentially taking over. And then quickly the AIs of the company taking over everything else and leaving both the humans in the company and most governments around the world and most people around the world completely disempowered at the mercy of these AIs. And this is in some ways the worst form and the biggest form of power concentration we would have ever seen in history and we need to avoid that.

B (62:34)

This is a point you've made a couple of times that if you raise the superintelligence and actually get there, you will be handing over power to the AIs. It will not make you powerful. But internally in the companies, they must disagree, right? They must think that they will empower themselves, or maybe they will empower their country, or maybe even sort of optimistically they think they'll empower the entire world by developing advanced AI. So what's the disagreement there? Or how do you explain if you work at say OpenAI or Anthropic, do you think they think that they will lose power? Are they just mistaken?

A (63:13)

So I think a lot of employees are mostly doing their job and it's an extremely highly paid job and it's convenient to look the other way. I think at the executive level we can see there are public statements by some of them. Some of them are just fundamentally transhumanist and it might be kind of hard to believe, but there are public statements from them. One of them is one of the co founders of Anthropic to the Guardian, where he quite openly talks about having this intelligence explosion after which they will hand over control. He's not talking about him personally having control, he's talking about handing over control to the AI. I don't think this is, I don't know if this is the majority of executives in the companies, but there are some prominent ones that essentially just believe we should give the keys over the planet to the AIs and let them do what they want and then either hope they will be benevolent or in some extreme cases the case, for example, of Richard Sutton and probably unfortunately non insignificant fraction of people at these companies, even if they're not benevolent, these AIs, these people consider them as the next stage of evolution of intelligence or some other post humanist ideas of humanity is just a transient phase in history. I completely reject these views. I am human. I care about being human. I care about fellow humans and I don't want us to go extinct. But I think some of these people genuinely think AI is taking over is not a bad idea. I think some other ones, as you're saying, might be deluded or power hungry or willing to take this enormous risk on behalf of everybody else with the sliver of hope that they individually will be able to keep control. But again, I think the views are quite inconsistent and in the end what matters are actions and not words. A lot of these people are people that have recognized that independent experts recognize that this poses an extinction risk. They are still willing to go ahead, maybe because of a momentary flash of glory before the AIs take over from humans, maybe because they genuinely care more about AIs than humans. I think we have seen this sadly more and more, and it's aided by enormous exposure to AI systems, and in some cases it might even be psychosis and other cases where they believe, yes, if others do it, they will not have it, but if I do it, I might keep control. And this small chance of having total domination over the planet is worth risking to sacrifice the lives of billions of people on the planet.

B (65:56)

As a final question here, could you tell listeners what they can do to help Control AI achieve its mission?

A (66:03)

Yeah, absolutely. So first of all, if you're listening and you care about this issue, contact your lawmaker. We have tools that make this very, very easy in many countries around the world. For you to contact Yellowmaker in just a few minutes, go to campaign control AI.com and you'll find everything there. Write to them. If you have time, call them. It's very effective to call them. They get much fewer calls than emails or forms. And especially in the US they will always note down and they will really take seriously what you tell them on a call and they will keep it into account. Also, we are expanding. What we've talked about with Gus in this episode is we have this theory of change that has worked very well. I think now we have a recipe for making this change happen at scale. And so we are scaling up the team. We're scaling up the team here in the uk. We're scaling up the team in the US with some jobs going live soon. So keep an eye out for those roles on ControlAI.com and all other platforms where you can find job postings. We're hiring across a variety of roles.

B (67:05)

Fantastic. Thanks for chatting with me.

A (67:07)

Thank you so much. Great to be on again.