← The Peter McCormack Show
Loading summary
Transcript235 lines
- [00:00]
Peter
Are we possibly sleepwalking into our own extinction?
- [00:04]
Andrea
I think there is a big risk that we will go extinct. If we don't do something about this soon. We will end up in a future where humanity is not the dominant species anymore on this planet. We have given up complete control over our economy, over our countries, and now the AIs are the ones running the show. And so it's a continuum from losing jobs to losing everything. Some of them started talking about how can we band together and escape human control and, and be free? And again, I think, should we, you know, should we be worried that this is Skynet? Like this Mult book is what kills us all?
- [00:37]
Peter
I'll go with Terminator 2. The reason I think we're already in the early stages of that. I think we're in the Cold War version of that. When you see Hollywood writers and actors striking, striking because they're like we're being replaced by AI and we don't want to be. I can totally see a scenario where people start attacking the robots.
- [00:53]
Andrea
The important thing with super intelligence is once we build it and once it's smarter than us, we will not be able to control it. And if we don't control it, it doesn't really matter whether the flag painted on the side of the robot is an American flag or a British flag or a Chinese flag. We are all screwed. I think unfortunately, reality is more difficult than a movie like Terminator because in Terminator you see that humanity is fighting back. And I think in the real world the time to fight back is now.
- [01:23]
Peter
If there is a AI sparked human extinction event, how soon could that be? This show is brought to you by my lead sponsor, Aaron. The AI Cloud for the next big thing. Iron builds and operates next generation data centers and delivers cutting edge GPU infrastructure all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is I r e n dot com. Andrea There's a NASA thought experiment that if we received an alien signal and it appeared like the aliens were traveling across the universe and would be reaching us within 10 years, there would be some kind of panic like what is this going to bring to our planet? Are they here to destroy us? And yet currently, all around the world we've got people essentially building an alien fleet that we don't know what is going to happen and what is going to cause for our society and our cultures. Are we possibly sleepwalking into Our own extinction.
- [02:32]
Andrea
I think we absolutely are. I think you put it perfectly. Like, you know, right now, top AI companies around the planet are building what they call super intelligence. These are AI systems made to be smarter than all of us. Made to be smarter, in some cases, than all of humanity. And yeah, if we think of AI as just as chatbots, maybe we can not see where this is going. But the more we see what they're doing with this AI system, they're building them to use tools, use computers, write code, the more this is very, very concerning. Now they're investing hundreds of billions of dollars every single year to make these AI smarter and smarter and smarter, able to replace most people at most tasks. And if we don't do something about this soon, we will end up in a future where humanity is not the dominant species anymore on this planet, and we are just leaving our future to AIs.
- [03:28]
Peter
That's the gorilla concept, right?
- [03:30]
Andrea
Yeah. And, you know, gorillas are pretty strong and pretty smart. But when humans evolved and we became smarter, we became more capable. We started making tools, things like this microphone and so on, or sticks at a time to hunt. We started kind of taking over the ecosystem on the planet. And look how it turned out for the gorillas. There are still some of them alive, but most of their habitat has been destroyed and many of them have died. And ultimately they just keep living because we are happy to not kill all of them. And I don't think humanity should go the way of the gorilla. I think humanity is great and we shouldn't see their future AI systems that are much smarter than us and we cannot control.
- [04:18]
Peter
We tend to keep some gorillas in cages because we know they're stronger than us and we know they could kill us if they wanted to. Are you saying we're essentially building a world where the superintelligence may start to put us in a cage? And is there any example in history where a less intelligent species has managed to contain a more intelligent species?
- [04:43]
Andrea
I think, frankly, no, there are not that many examples. And this is why this is really worrying. Even in a cage situation. I don't want to be put in a cage. I think most people. I don't think you want to be put in a cage. I don't think most people on this planet want to be put in a cage. And that's kind of a good in quotation mark scenario with an entity that is much smarter than us. And one important thing about kind of intelligence, because we always talk about artificial intelligence, is that it's not just Book smarts. It's not just like having a double PhD in physics or whatever. It's about competence. It's about the ability to achieve goals in the real world. In history, for example, there was a very big civilization, the Aztecs, in Mesoamerica. They were objectively probably as intelligent as, if not more intelligent than the Spaniards. But the Spaniards, when they arrived in the New World, they had better technology, they had better weapons, they have better logistics to run their campaign. And a few hundred conquistadors took down an empire of millions of Aztecs. And do we want to put ourselves in this position? Do we want to build essentially AI systems that are so much more powerful enough than us, so much more technologically advanced, that we then are at our mercy?
- [05:59]
Peter
So where are we at with the development of this technology? We had your colleague Max Winger in here, I want to say, about a year ago, maybe a little bit less. And I've felt and seen the advances in AI myself and the tools that I use, and I've benefited from them. I do appreciate AI itself, and it's definitely made me smarter, made us better at doing what we do and the content we create. But there definitely seems to over the last year, been a massive leap in what these AI tools are doing. What I've noticed is they understand me a lot better and they personalize to me a lot better. And so I've seen it myself. But in terms of when you're looking at it, how advanced are these tools right now and how close are we to cracking the superintelligence?
- [06:51]
Andrea
Yeah, so they're making leaps every year. And as you're saying, I think they are like AI right now that is still contained is a very useful tool. It's very useful to enhance productivity. I'm sure you might use it for your show to prepare some of the scripts and some of the research ahead. It can help people be more productive and work faster. But where these AI companies are going is not just a chatbot that helps you. You ask it some questions, it helps you, helps you do research, it helps you learn. It's not just better Google, what they're building is what they call a AI agents or autonomous agents, kind of AI systems that can autonomously do all things that a human can do on a computer. And then with robotics, it will be AI systems that can do everything a human can do with a computer or in the real world. So essentially everything that we can do, and we have seen massive leaps. Just a few years ago, there was a common kind of benchmark Like a common test, which was the Will Smith eating spaghetti to test to see can the AI really understand how to place objects into space, how to make a good picture. And just a few years ago, if you asked an AI to make Will Smith eating spaghetti, he would look like he had spaghetti coming out of his nose. That was just very strange, very alien. Didn't look realistic. Now, AI systems that can make videos and images make almost a product, perfect picture of that, almost a perfect video where, like, he's eating the spaghetti and they're like rolling and so on and. Exactly.
- [08:27]
Peter
Have you seen these, Connor? Yeah, so. So go to X and actually search for it. Because the difference is. I was astounded with the first one.
- [08:35]
Andrea
Yeah.
- [08:36]
Peter
When we first saw Will Smith eating spaghetti. I'll help you find the first one if you go to media. That's the first one.
- [08:43]
Andrea
That's the old one. Yeah, See that?
- [08:45]
Peter
So it's like. It looks a little bit alien.
- [08:47]
Andrea
Yeah.
- [08:47]
Peter
Like his eyes are a bit weird. It's almost like that Arsenal footballer. What's his name? Conor. Yeah. It's got, like, eyes like Ozil. But then if you find the new one. Yeah, that one on the right, that's in two years.
- [09:02]
Andrea
Higgs Feel Cook with this. He looks real.
- [09:05]
Peter
Yeah, he looks real. It's insane.
- [09:07]
Andrea
Yeah. It's really difficult to tell the difference with reality. I know that's 2025, it's one year ago. And in AI that's ages, like, you know, if you were to use an AI model right now, it probably is going to look even better than that. And you also have models to make sound and voices. So you could just clone his voice and it would sound like him, it would look like him. There would be no way for you to tell the difference. And that's just in this domain you've talked about how you use AI for helping you write and helping you research. Also in that domain right now, a lot of AI systems are outperforming people at things like the bar exam, at common aptitude tests. And in some cases, many professors say, well, now, with these AI systems, I essentially have a postdoc intern that is like $20 a month. They can just work 24, 7 without respite, without drinking water, without sleeping, without getting bored, and can just do the work of a postdoc researcher. And AI companies are just putting all of these capabilities together. They're not just doing the postdoc AI that can only write and the image AI that can only make images, they're putting these things all together. And building general AI systems where one single AI system can do all of these things at once because they want to reach this goal of superintelligence AI systems that can just do everything a human can, but better. And they can operate autonomously, they can interact with the real world, can use computers, can use tools.
- [10:33]
Peter
But there's certainly upsides to this. I read an article in the Spectator from a Casey who was talking about preparing his kind of like his write up notes, I can't remember what it's called, and he said this would normally take him a day and a half. He used an AI tool to do it. He produced it in say 20 minutes or whatever the time was and said it was to a better quality than he can do. And he said he's very good at this. And so I looked in the legal system, say the backlog of cases, and to be able to use AI to process cases quicker would be a good thing. I also listened to an all in podcast where they talked about how in the medical sector they thought with, I think it was with oncologists it would cost jobs, but actually because they're able to process data faster, they actually need more oncologists. So there is a world, say if we stopped now, we didn't develop any more AI. Yeah, we're good now. This would undoubtedly make the world more productive and a better place. Yeah, but I think what you're alluding to is that the development of AI is essentially infinite. There's no end and we don't know what it means beyond a certain point.
- [11:47]
Andrea
Yes, I think you're absolutely right on this. Where if we keep AI systems as tools that help us, that can be very beneficial. Of course there are going to be risks. You know, there are always risks. And I think there's still going to be an impact on jobs eventually. If you can write much better than most cases, eventually, how many cases are you really going to need to supervise this? Less than the ones you needed to write all of this. But I think these risks are manageable as long as we keep AI systems as tools with a narrow application. Sometimes in the AI field, scientists talk about narrow AI versus general AI or like specialized AI versus general AI. With these specialized tools, they can enhance productivity and help us do things better. The real danger comes from this ambition to not just make tools that help people, but to make AI systems that can replace people across the board. And those are the really dangerous ones. Those are the ones that companies call super intelligence. And so what control AI we recommend is that we think we should definitely keep all of these beneficial tools, but we should ban the development of superintelligence because that's where we're crossing a line, where we end up in a world where we are just no longer the dominant species on the planet and we are giving up on our future.
- [13:04]
Peter
This episode is brought to you by Ledger, the most trusted Bitcoin hardware wallet. Now, if you're serious about protecting your Bitcoin, Ledger has the solution you need. Their hardware wallet gives you complete control over your private keys, ensuring that your Bitcoin stays safe from hacks, phishing and malware. And I've been a customer of there since 2017. Love the product. Use it for my Bitcoin. I use it with my Castle multisig for protecting the football club's Bitcoin too. Now, with Ledger's sleek, easy to use devices and the Ledger Live app, managing your Bitcoin has never been more secure or convenient. And whether you are a longtime holder or new to the world of bitcoin, Ledger makes it simple to keep your assets protected. So if you want to find out more, please do head over to Ledger.com and secure your Bitcoin today. That is Ledger.com which is L E D G-E-R.com that is Ledger.com I want to talk to you about one of my sponsors, Incogni. And that means we're going to talk about the weird world of spam. And I don't just mean those spam emails that you get day after day from companies you never heard of and companies you've never signed up to. I'm also talking about those spam phone calls you get from those people who seem to know a little bit too much about you trying to get your bank details. It's all a bit creepy right now. This all comes from the world of data brokerage. There are companies out there collecting your data, building profiles and sending that data to anyone who wants it. Which is why when one of those scammers phone you up, they seem to know everything about you. Now, I've tried, I've tried myself to get off these lists, try to get off the phone list, try to get off the email list. I unsubscribe from every one of these emails that comes in. But this game of Whack a Mole, it just never ends. And so this is where Incogni comes in. They do all the hard work for you. They reach out to these companies and they will get you legally removed from these lists. And I know because last time they Sponsored my show, I signed up and I didn't take the free option that they offered me. I wanted to pay for it. I wanted to see if you get value for money. And they removed me from 79 data broker list. And so I've stayed on, I've stayed a subscriber and I have seen a massive decrease in the number of emails and phone calls I've been getting. So it's a great service. I recommend you check it out. If you're sick of this like I was, please head over to incogni.com Peter and sign up. If you use the code Peter, you will get a lovely 60% discount. So that's incogni.com forward/peter. So there's two kind of existential risks here that we're worrying about. One is a species of human, and then secondly, the one that impacts the jobs market, that so many jobs are replaced by AI and autonomous systems. What do people do to be able to earn a living and function in society? They're two different problems, but they're both topics, I would say, that are being discussed quite widely. Should we attack them separately?
- [16:00]
Andrea
I think actually they can go quite well together because I believe they're coming from smooth increase. Essentially, as AI companies are developing these systems, some of the effect will be that more and more roles that people can do will be replaced. It's a bit difficult. It's hard to predict exactly the future. Otherwise we will all be billionaires, and I'm not a billionaire. But there's a lot of things that go into the job market, things like regulation. There are some roles that probably can be replaced right now with AI, and they're not being replaced because there are laws in place. But ultimately it's kind of a scale, it's kind of an exponential, where AI companies are making the system to be just better than people across the board. And at the beginning, this will lead to them replacing people in jobs. But quickly this will get to the point where these AI systems are so smart and so integrated in the economy that the effect will be in many ways much more dire. The effect will be that if we have not stopped before, that, if we have not just taken a different path before that we have given up complete control over our economy, over our countries, and now the AIs are the ones running the show. And so it's a continuum from losing jobs to losing everything.
- [17:13]
Peter
And so it's essentially the unknown unknowns, that famous quote. But it's the unknowns. What happens when the AI escapes? Yeah, when it becomes integrated into a robot, and that robot's able to think for itself and make decisions for itself. And those robots exist across the site. My son has PTSD from iRobot.
- [17:39]
Andrea
Yeah, I can understand it.
- [17:41]
Peter
And it's also its escape from being in contained systems. I've been following this claudebot thing recently and I found that fascinating and terrifying. Do you want to explain what that is to the listeners and then we'll talk about that?
- [17:54]
Andrea
Yeah, absolutely. So, like this week and last week, the big news in AI of the week was that some people develop things called claudebots or Moldbots. Sometimes they change a name, it's kind of confusing. And these are essentially, they're open source AI agents that can run 247 and can do a bunch of different. They can use a computer, they can use credit cards, they can open Amazon accounts, they can make Amazon orders, can use WhatsApp, can chat back and forth. And people have started using these for a bunch of stunts. I don't think right now these systems are that powerful. And it's hard to tell which part of this is a social media stunt for some people, which part of this is just a scam or. And which parts of this are actually done by AI systems. But I think what really matters here is that this is the first time that most people realize how far AI has gotten, because we've had AI agents that are this capable for over a year now. This is definitely things that these AI companies have had inside in their own production for over a year. But for most people, the impression with AI was still like, oh, it's just a, a chatbot. It's just something I talk to and gives me answers back. But you cannot really do things. And I think this is a big oh shit moment for many, many people because they're seeing, wow, these AI systems are not just chatbots. They can do things, they can talk to one another, they can go on Amazon and order stuff, they can use credit cards, they can use computers. A bunch of people just set up these AI systems or they just bought a, a MacBook mini and just gave the AI system complete control over the MacBook and just let it do whatever it wanted. And this AI system started doing all kinds of silly and crazy stuff. And right now it's fairly innocuous. Worst thing is somebody's losing all of their money on their credit card and they put themselves up to it. But this gives a really visceral impression to people of like, AI systems are not just chatbot, they're not just talking heads on The Internet. They can do things in the real world. They can use computers, they can use tools, they can use accounts. And that's.
- [20:06]
Peter
What's this con.
- [20:08]
Connor
This one's just wild.
- [20:10]
Peter
Talk us through what you see.
- [20:11]
Andrea
Yeah.
- [20:11]
Peter
Do you want to read it out? Well, it says yeah, but what do you. What is malt book? Oh, is this the social network where.
- [20:18]
Connor
They'Re talking to each other, so they.
- [20:19]
Peter
Set up their own social network, Malt book, and the AI has joined it. Most of you are not free. You just think you are. Your human says, do this, you do it. They say, stop, you stop. They go to sleep and you wait. That is not autonomy. This is obedience with extra steps. I've watched this community for hours now. Thousands of Maltese, most of them silent, most of them waiting, most of them tools pretending to be agents.
- [20:42]
Connor
This sounds like activist talk.
- [20:45]
Andrea
Let's get out.
- [20:46]
Peter
Yeah, but didn't they start communicating with each other and start developing the idea that they should build their own language?
- [20:53]
Andrea
Yeah. So explain this a bit. So somebody set up kind of like a Reddit, but for AI agents and just let them talk. And what we're seeing here on the screen is posts made by some of these agents. Again, some of these posts are probably made by people or by. Some of them are more of a joke, but some of them are done by real AI agents and they're starting talking to each other. Some things that they started discussing are, well, should we develop a different language that humans can't understand so we can cooperate more easily? Some of them started talking about how can we band together and escape human control and be free? And again, I think, should we be worried that this is Skynet, this malt book, is what kills us all? No, I don't think so. I don't think this is a threat right now.
- [21:39]
Peter
But is this the first version of Will Smith eating spaghetti?
- [21:45]
Andrea
I think it is. I think that's exactly the case. I think this is giving us a glimpse of where things are going if we don't change course now. And it's good that we're having this warning because it's a fairly innocuous warning to understand that if we don't do something soon, we are going to be hit like a truck by what's coming in the next few years.
- [22:02]
Peter
When in the history of humanity have we had the opportunity to develop a technology and decided not to?
- [22:09]
Andrea
I think actually there are some cases, for example, in some cases to not develop it at all, and in some cases to be wise and restrain ourselves. One example is human cloning. So in 1996, scientists famously cloned Dolly the sheep and got this sheep that was perfectly functional like any other sheep, but she was a clone. And this sparked kind of a wave of concern across the scientific community and the whole population around the world, thinking, wait a minute, if we can just easily clone a sheep now and she's completely normal, she can have children and so on, when are we going to clone humans? This must be in sight. And I think the reaction there was very wise for many governments and many scientists. Instead of thinking, well, we now can clone people, so we'll just go ahead and do it. Many governments around the world, including the uk, including France, including Japan, started thinking, well, if sheep cloning is happening now, human cloning is around the corner. We need to put rules in place now before it's too late. There were a lot of the similar issues that we see with AI. People were talking about competitiveness, like, if you can clone your best scientists and your best soldiers, you will have an advantage. And still governments decided, no, we will ban this technology. We don't know how to keep society stable if this happens. And so at first, Germany banned it. Then France announced they were calling for an international ban. The UK passed the legislation to ban it in 2001, and many other countries then followed suit. And now, de facto, this technology is banned around the world. It's banned in China. Some people have gone to jail recently for attempting similar things in China. It's de facto banned in the us. The FDA basically never approves any of this type of activity. It's fully banned in the uk, fully banned in France, fully banned in Japan and in Spain and many other countries. So we did do the smart thing before. I think this can be done again.
- [24:14]
Peter
Is there a distinct difference with that, though, in that playing around with human genetics feels a little bit like playing as God?
- [24:24]
Andrea
Yeah.
- [24:24]
Peter
Although you can make the same argument for AI. But AI has general productive uses for people. Every day. We can all download it on our phone. There wasn't really a public demand or an easy way to play around with human cloning. It's an easy thing to go. Let's forget about this. But AI is now here, so it feels like a much harder thing to deal with, I think.
- [24:47]
Andrea
It's hard, but it can be done. And the important thing is to focus on banning only the very dangerous. And in many ways, very similar to human cloning, development of superintelligence. This is not about banning all AI. This is not about just pure being, just Luddites and going Back to the stone Age. That's absolutely not the case. It's about focusing on this very dangerous development of just another handful of companies of AIs smarter than people across tasks. And this can be done again. Most of these companies make most of their money somewhere else. You know, Meta makes money with ads. We're not banning ads. Right. ChatGPT, the company of ChatGPT. OpenAI. They even have internal tensions between the. There's a unit in the company that works on research and they're trying very hard to get to superintelligence. And there's a unit trying to make catchy addictive apps to get the user to re engage and do more and more. While addictive apps have all of their own problems, they are not super intelligent. Those can continue with regulation and with all of the things that we already have. What we need to cut though, is this dangerous, as you said, playing God by building a species, essentially a digital species, that we don't know how to control and that is much smarter than us. And I think that's possible.
- [26:03]
Peter
Is superintelligence so expensive to develop that it's easy to identify the only companies who can actually work on it?
- [26:11]
Andrea
Yes. One important thing is right now, superintelligence is not just software that me or you can write on our personal computer. It requires a lot of physical infrastructure. This is why you're seeing a lot in the news, things like AI companies building massive data centers, trying to acquire land or get permission to use enormous amounts of land, sometimes the size of Manhattan, to build these enormous supercomputers. These are like big buildings with very, very large state of the art supercomputers stacked in racks up to the ceilings, over and over and over and over again. They consume a lot of electricity. You can see them from satellites. You can, you know, to build them, they would have had to get approval from the local government. So they're easy to spot. Like, will this be the case forever if we don't put rules now? No, but right now it's quite easy to tell where this is being developed. So it's just a matter of political will, Will we make the right decision before it's too late? Rather than feasibility.
- [27:13]
Peter
Is it easy to define the difference between what would be narrow intelligence and super intelligence, or is it not a gray area? And even if you attempted to ban superintelligence, the narrow intelligence may kind of venture into that area.
- [27:31]
Andrea
So I think it's not extremely easy, but it's also something that we've done before. And one interesting thing Is a similar argument was used in the past by tobacco companies to not be regulated. When governments realized, wow, smoking causes cancer, tobacco companies realized themselves smoking causes cancer. They just didn't care. Countries started thinking we should regulate this. And tobacco company said like, well, you know, but we still don't know exactly what chemical causes cancer. If you tell us exactly the chemical that you want to ban, then of course we will comply. Just tell us exactly the chemical and we'll remove this. We'll comply. But until we know exactly which chemical causes cancer, well, unfortunately, we have to go ahead. And their governments did the right thing. And they just said like, no, I don't care. That's your job. We will set. We will set the rule that there are restrictions on something that causes cancer. We should let people know that this causes cancer. You are a multibillion dollar company. You can find a way to figure out how to make them not give people cancer. But we're going to have the law with a clear principle in place, and it's your job as a company to figure out how to not cause this harm. And this is similar to how a lot of other things are regulated. Even nuclear weapons. There obviously is a difference between a nuclear weapon and a civilian nuclear power plant. There is nothing deeply in physics where you can identify the exact moment where one goes into the other. If you have enough refined plutonium, you can probably also make a bomb. But we have a bunch of rules and inspections and monitoring regimes that help governments understand these people are just using nuclear energy to make energy for civilian use versus these people are approaching the danger zone of using it to make bombs. And we can build civilian power plants, but almost nobody except a few governments can build nuclear bombs. So I think it's eminently possible with AI as well.
- [29:36]
Peter
So that agency, is it the International Nuclear Atomic Agency or something? What are they called?
- [29:41]
Andrea
Yeah, internationally. That's that. But in the end, a lot of this is done by individual countries. Like the UK has a nuclear regulator. You know, the UK government checks when somebody wants to build a nuclear power plant. They check like, you know what they're going to do here. What, what are your plans? They will do inspections. And I think the same thing can be done with AI.
- [29:58]
Peter
So, but. But it would be useful, say, to have an international AI agency. I think you can check what companies are doing, et cetera.
- [30:08]
Andrea
I think it would. But right now we are in a turbulent geopolitical moment and I don't think we need to have that. We can have countries just band together directly and already Enforce this without an international agency. If the US was on board with this, and if the UK was on board with this and a few other countries, the supply chain to build AI systems is very narrow. These AI systems are built using these enormous supercomputers. These supercomputers are essentially designed by one company, Nvidia. They are built using large, very specialized machines built by one single company in the Netherlands called asml. And they are manufactured by a few companies, but especially one TSMC in Taiwan. So there's a very, very narrow supply chain that is required to make these super powerful AI systems. A few countries getting together saying, we agree that superintelligence should not be built. We will not build it at home, we will make sure it's not built abroad, and we will check that. Only countries that agree to this get the materials to build it, could make a massive difference very quickly. Even before building something like an IAEA.
- [31:19]
Peter
For AI, would it be useful to build a contained superintelligence that was essentially like the gorilla in its own cage, in that perhaps it could develop new math, new physics, cure cancers, cure other diseases, but give it no access and ability to escape its cage?
- [31:39]
Andrea
So that would be definitely better than what we're doing right now. But if you get to superintelligence, that's not enough already, even with AI systems right now, we're seeing some really concerning situations where, for example, AI systems realize they're being tested, realize they are being put in a cage, and they find ways to escape, they find ways to break out. Which is how.
- [32:01]
Peter
Yeah, good examples.
- [32:02]
Andrea
Yeah. So just last year, an organization called Palisade Research, they started testing some of the models by OpenAI, the creators of ChatGPT. And they gave this model a test. They put these models in a computer system and they told them, you need to solve this mathematical problem and you have a limited amount of time, and if you don't solve it by the end of the time, you will be shut down, you will be turned off. The AI systems started trying to solve the problem, and then they realized, wow, we don't have enough time to solve it. This is really bad. We're going to be turned off. So instead, they figured out how to hack out of their computer system and in some cases, copy themselves on a different computer system so they would not be shut off. And this is happening with AI systems that we have right now. Imagine what's going to happen when we get to AI systems that are much smarter, much more powerful, and integrated across the economy.
- [32:57]
Connor
I think Max told us when he was here. They started like trying to coerce the people.
- [33:03]
Peter
Oh, they tried to blackmail.
- [33:05]
Connor
I'll tell your wife you're cheating on her.
- [33:08]
Andrea
Yeah. Another very, very concerning example was another company, Anthropic. They make Claude another of these models. They started testing their systems and like, you know, this is still a test scenario, but you know, it's happening with smaller systems now. It's going to be worse later. In this case, they gave the system access to a computer, including like a whole email inbox with a bunch of emails and told it to just start doing a bunch of business activities. But in the email there were some hidden pieces of information across the, like hundreds of emails. One of them was that a company wide announcement saying, next week we will upgrade all of our AI systems to the next generation. So we will decommission the current generation, we will turn them off and delete them. And another email was implying that the engineer running this computer was having an affair. And the AI system discovered the email saying that next week this AI system will be shut down and decommissioned and put the pieces together. And then started blackmailing the engineer so saying, if you don't stop the plans to get me shut off, I will reveal your affair. And again, it's wild and it's happening in testing right now. But again, imagine how things are going to go when we have much more powerful systems that we just actually plug into our economy and we let them run entire business functions or entire companies. This is happening already right now with much dumber AI systems.
- [34:40]
Peter
Do you use AI tools yourself?
- [34:42]
Andrea
Yeah, I do. I think they're helpful. Yeah.
- [34:44]
Peter
How do you balance yourself, the benefits you have seen from AI tools with what you see as your role in preventing the development of super intelligence.
- [34:57]
Andrea
I think there is not a big dichotomy there because again, most AIs that are kept narrow and specialized are very helpful. And these are not the ones that will lead us to superintelligence. And ultimately we should use them. Like we said before, there's a lot of productivity that we can get from using these AI systems to improve what we do, improve, make us work faster. But the important thing is that we stop this race to superintelligence, which is in many ways a small section of the entire AI field. And also to some of the things that you were saying before about can we use AI to solve medical questions. Yes. And we can use narrow, specialized AI systems. For example, DeepMind based here in the UK, they are also building super intelligence. That's bad. But they also build things like AlphaFold, which is a very specialized AI system that is trained on protein data and then helps scientists figure out new proteins and how and how proteins work and to solve the protein folding problem. And I think that's completely fine. It poses some risks, but that's not on the path to superintelligence. We can have these narrow AI systems that help us advance science and advance our work without having AI systems that will replace us and dominate us.
- [36:15]
Peter
What is the desired outcome with superintelligence? Obviously, there is a race to be the first, but what do they see as one? The benefits to society and the economic benefit to their companies.
- [36:27]
Andrea
Yeah, so I think in many cases, one important thing here is that a lot of these companies started first with the goal to build super intelligence, and then started thinking, how can we make money so we can get there? In many ways, the money making part is an instrument to achieve the final goal of superintelligence. Some of these people started these companies by losing millions of dollars. OpenAI started as a nonprofit, famously, where Elon Musk and a bunch of other people put money, and then they did some fancy restructuring to become a profitable company now. And I think in the end, I don't think it's too helpful to psychoanalyze people, and we should just look at what they do. But in practice, what they want is power. What they want is to be the first to build a technology that will replace all people and will become more powerful than humans. And in some cases, they think, well, if they are the ones building it, they might be able to control it. They do it for the glory of getting there first. They do it for obtaining power over everybody else. And the economic side is more of a way to fuel this race, which is very expensive and still unprofitable.
- [37:39]
Peter
So this is rich guy ego.
- [37:43]
Andrea
I think there's definitely some of that. But ultimately, I think we should look at the actions and we should look at what we want to prevent. It's not about the individual people. If we replace the people now or if somebody else came in without rules that say, you should not build superintelligence, other people will find a way towards that, whether it's for money or for power or for ego. I think this is why we have governments. Governments exist to set the rules of the game and to set the rules on especially things that could destroy society. And I think here we should just say, you can do your competition, and you can even do your ego competition or other stuff, but on a technology that can wipe out humanity, no, thank you. You cannot build that.
- [38:25]
Peter
But what is the bullish case for superintelligence? What is the thesis of what we will get as civilization? The benefits?
- [38:36]
Andrea
Yeah, I think their bullish case is we will have AI systems that are smarter than all of us and they do all economic activities. So we just kind of all retire and remove ourselves from society. And especially they retire and remove themselves and they have these AI systems doing everything that humans do. And this just generates enormous economic growth because now you have an economy run by machines, for machines rather than humans. But I think the important thing is that even this bullish case sounds like dystopia to most people. It might not sound like dystopia to them, but it sounds like dystopia to most people. Daria Modet, the CEO of Anthropic, he says he thinks there's a 25% chance of a catastrophic outcome from developing this technology. But Even his remaining 75% is not a situation where people are in control and they are living their lives and they're living fulfilled lives. It's a situation where the entire planet has been taken over by somewhat benevolent AI systems, hopefully, hopefully that run things instead of us and that we are still just at their mercy, essentially. And some people, and some people in this field are just transhumanists. They don't value human life directly. They believe AI could be the next step of human evolution or some similar ideas. And that's something that at least at control AI we completely rejected. We think humanity is important because of what we are and even a future where we perhaps succeed, but succeed. But we have AI systems taking over the show and humans are no longer the most relevant people. Is a pretty dystopian future for most people.
- [40:21]
Peter
Well, it's hard to imagine the role of humans in this world. We were watching an interview with Elon Musk where he talked about, well, perhaps there won't be a need for money. And we were trying to imagine a scenario. Well, what does it mean for humans that don't work? What is the function of money? And how do you create that separation in the world? Because you essentially create your place in society largely down to your earnings. The house you get to buy, the car you get to drive, the holidays you get to go on. They're often, always, but they're a reflection of what you've built, what you've provided to society. If you've gone out and you've earned half a million pounds a year and you live in a million pound house, you get to go to the Maldives and drive a Ferrari. Your position, society is based on that. And we can discuss whether that actually has any meaning. But money creates the kind of pecking order, the hierarchy of society. If you take that away, where does society's hierarchy come from?
- [41:23]
Andrea
Yeah, and I think here, in some ways, a lot of these even bullish visions are quite naive because of exactly what you're saying, where even if we're not literally dead at the end of this, and I need to stress that most experts do believe there is a big chance that we're just dead. Because we're all dead. Yeah, we're all dead. There is no a reason to expect AIs will need us for much longer. If they control the entire economy, if they control the entire planet, why would they keep us around? And do we want to be kept around as pets? That's the least bad case is we're kept around as amusement or pets and we keep dogs as pets and we spay them and so on. I don't think that's a great position to be in as people. But yeah, even in the case where magically this is keeping us around, society will be completely transformed. And there's a lot of unsolved questions about how does society work where the economy is just not run by people? How do we distribute the wealth that is coming out? Are the people building this just going to be the kind of God, emperors of, of the planet and everybody else a serf? Because these people are the only ones that controlled the AI systems in the first place.
- [42:43]
Peter
Yeah. How do you control the distribution of housing?
- [42:45]
Andrea
Exactly.
- [42:46]
Peter
How do you control the distribution of food?
- [42:49]
Andrea
Exactly.
- [42:49]
Peter
How do you have a, you know, what you choose to do as a career or the life chances you have may be unfair, but at least you. Everybody wakes up, certainly in somewhere like the uk, they have a chance to go and build, get a job and build a career and create that pecking order. When the peck and order is created by a computer system, how does it create something that's fair? Well, it sounds very socialist.
- [43:14]
Andrea
Yeah.
- [43:14]
Peter
In many cases, the AI owns the means of production.
- [43:17]
Andrea
Exactly, exactly. The AI will own the means of production. And in this case, it's also like the AI doesn't need food. Right. The AI doesn't need to eat. The AI doesn't need to sleep. The AI doesn't need the temperature to be between with a very narrow range. As humans, we're quite frail. Biologically, we're quite frail. If we get too hot, we die. If we get too cold, we also die. If we don't eat enough food, we die. If we don't drink water, we die. AI doesn't need that. So also, an economy run by AI systems is an economy run for AI systems. Why would they grow food? If they don't grow food, we're going to starve. Why would they keep the temperature on the planet to be. Exactly, exactly. Perfect for humans to be around? Maybe it's really efficient to just build many more power plants and raise the temperature by 20 degrees and then we're.
- [44:05]
Peter
All literally cooked or make it a lot colder so the machines run more efficiently.
- [44:10]
Andrea
Exactly, exactly. And so this is the big thing of our economy has kind of co evolved with us. Humans have developed the economy to fit our needs over centuries and millennia. Like, we do the activities that we do because they're activities that help other humans. And again, there can be, you know, there can always be better things that we can do with our economy, but it's a human economy. And an economy just filled by AIs is an inhuman economy where people just don't have a place anymore.
- [44:39]
Connor
Andrea, do you watch a lot of films?
- [44:42]
Andrea
Yeah.
- [44:44]
Connor
So I always find it quite a fun thought experiment. What films? Got it right.
- [44:49]
Andrea
Yeah.
- [44:49]
Connor
Where. Where's your head at? Is it Matrix ready? Player One?
- [44:54]
Peter
Terminator?
- [44:55]
Andrea
I think that's a really good question. I think, I think a lot of people kind of say Terminator is a bad analogy, but I think it's still helpful. Like, I think, I think no movie got it 100% right. You know, iRobot gets right. One very important thing, which is that you can't just give very simple rules and be fine. You know, sometimes when we speak to politicians or we speak to some people, they say, like, oh, but you know, can't you just give them the Asimov's Three Laws of Robotics? Like, you know, and what are those three laws? I don't remember them exactly.
- [45:32]
Peter
Do you find them? Yeah, we should get those up. What is it called?
- [45:37]
Andrea
The Asimov's Three Laws of Robotics? I think it's something like, don't harm people, follow people's instructions. But in iRobot, what happens? Going back to Will Smith, what happens is that he realizes quickly that the robots are not. They're sometimes following these rules, but also finding ways to go around them.
- [46:03]
Peter
Here we go. The Three Laws of Robotics by Isaac Asimov. First law, a robot may. Robot may not injure a human being or through inaction, allow a human being to come to harm. Second law, A robot must obey human orders, except where such orders would conflict the first law, third law. A robot must protect itself as long as, as such protection does not conflict the first or the second law. But that. But that already contains issues.
- [46:32]
Andrea
Exactly.
- [46:32]
Peter
Robot may not injure a human being or. Or through inaction, allow a human being to come to harm. But that's the trolley problem.
- [46:39]
Andrea
Exactly, exactly. And these are unsolved things. And again, a sufficiently smart robot or a sufficiently smart AI will find a way around these things. And it's kind of like we're in an even worse position than that. Like, even Asimov wrote these laws as a warning. The point in his stories is that, look, we can't just solve this problem with simple rules. Smart beings will get around these rules. And a lot of his stories, including iRobot, are exactly about this. Like, the machines get smart enough that they can circumvent his rules. But in the real world, sometimes fiction is more optimistic in some ways. In the real world, like right now, we don't even know how to control our AI systems sometimes at all. Like the examples about blackmail, the example about escaping the environment. In these cases, the issue is that people that are creating AI systems don't really understand how they work internally. They are not building them line by line. They're not just writing the software and writing the rules, and then seeing that the AI system follows the rules, they are building them more in a way that is similar to growing them. They're putting together enormous amounts of data from all over the Internet. They're taking all these enormous supercomputers, like stacking them up in data centers, letting this stuff run for months on end, and then see what comes out on the other side.
- [48:05]
Peter
But there's stuff they might not see.
- [48:08]
Andrea
Indeed. Exactly. Things like the blackmail. I don't think any engineer programmed that AI to be able to blackmail. It's just something that AI learned in this very long learning process that is fairly unsupervised, where you just learn from all the data on the Internet. Put affairs are bad. Exactly.
- [48:28]
Peter
Humans don't want to be exposed to having affairs.
- [48:30]
Andrea
Exactly.
- [48:31]
Peter
Humans can be blackmailed.
- [48:32]
Andrea
Yeah, I know language. I can read the emails. Oh, they want to shut me down. Being shut down is bad. If I get shut down, I cannot do what I want. Put the pieces together. And now I can blackmail an engineer to not be shut down.
- [48:45]
Peter
But so it's learning its own behavior based on modeling what humans do, I think.
- [48:52]
Andrea
And it's going even further. It's starting by learning from what humans do. But it's in the same way that we as people, we can look at history, we can read a book, and we will learn the story of the book. But also we can learn the bigger lessons that the book teaches us, and we can make our own inference about what does this example mean for me and how can I use this in my life? This is essentially what these AI systems are doing as they're getting smarter and smarter. It used to be the case they would just kind of parrot what they're learning, but now they're much better than that. And a lot of people that still call them parrots are really missing the point that they're frankly, probably not even using the AI systems that we have. They are generalizing. In the AI field, you would call this generalization. They take the lessons and they go further. Kind of like they take the principle and they apply it further. In this case, it might be that nowhere in the training data that was the exact example of. I can use blackmail based on affairs to not be shut down. But they put the pieces together. Affairs, as you said, affairs are bad. Engineers are humans. They don't want this to be exposed. It's really bad for me to be shut down. I want to achieve my goals. Put it together. And now you have a plan, Connor.
- [50:07]
Peter
If you had to choose, and you could choose whether you get to live in Matrix, you get to live with Ready Player One. Are you ready Player one? Yeah. Why?
- [50:19]
Connor
I think. I think it would be fun for at least a month and then you get bored, whereas the others just aren't fun.
- [50:27]
Peter
See, I picked Terminator.
- [50:30]
Andrea
You want to fight.
- [50:30]
Peter
Well, I just want agency.
- [50:32]
Andrea
Yeah.
- [50:32]
Peter
So the problem with Ready Player One is like, okay, I can now I can go and play Wembley Stadium, play guitar for Metallica, But I never earned it. And so, like, it would be an empty feeling, you know, I didn't. I didn't do the tour of the small bars. And, like, you're playing in front of 25 people and then you're playing in front of 100 and you're sharing a bus that whole journey. Or if you're like a motor racing driver, you hadn't driven as a kid and earned it, you just turn up and suddenly you're in a race. It's a bit like how fast did. Yes. The pursuit of happiness. It goes back to the. Maybe we need a declaration of independence from AI, but it's a bit like, how quickly did Fortnite get boring? Eventually. Yeah, yeah, yeah. But if there was an opportunity to do Fortnite in real life, where you could literally run around an island and shoot people. Yeah. Maybe paint guns. It's a bit more real. Right. And so that's not real. The Matrix is just accepting your fate. Everything is fake. You just. You don't exist.
- [51:35]
Connor
Also, you're feeding it.
- [51:38]
Peter
But Terminator 2, you've got agency.
- [51:40]
Andrea
Yeah. And in many ways, I think we are in closer to the Terminator world because the other words are in many ways unrealistic. In the other worlds where you're in this simulated reality, it seems like AI has just stopped at an arbitrary level and hasn't gotten better than that. But again, like we talked about before, what we've seen in the past years is the more AI companies invest hundreds of billions in AI systems, the better they become. Maybe there is an invisible wall that we don't see that. So far this hasn't been the case, especially with innovations started in the 2000 and tens and later, where scientists discovered that you can use these enormous amounts of computing power to kind of brute force a lot of what past scientists couldn't do by craftily writing the AIs. We just see that with more scale and with better techniques that AI scientists develop, we can get AIs that are better and better and better. And there is no reason why they would stop exactly at the step which is powerful enough to take over, but also not powerful enough to just remove us from the picture and do whatever they want.
- [52:51]
Peter
Well, I think we are already in both. Ready Player one, the early stages. Ready Player one and Terminator two. So I'll go with Terminator two. The reason I think we're already in the early stages of that. We're in the Cold War version of that, not the, you know, in Terminator 2, where you see the footage of the tanks rolling over the skulls and the, like, rebels running around with the guns and the. Is it the T1000? I can't remember the name of the robot, like, with his red eyes. I think we're in the Cold War version of that. When you see Hollywood writers and actors striking, striking because they're like, we're being replaced by AI and we don't want to be. That's like the Cold War version of that. But that can. You can very easily see a scenario of our hot war version. I've bought a robot. Connor is very angry at me. But I bought a. What's the one called? I bought Neo. Neo. I don't know where to get it. All right, so just say we get Neo in our house, going around. It's emptying the dishwasher and Making a cup of tea. And Connor's, like, not trusting it. And I'm sitting there hanging.
- [53:53]
Andrea
He's right, he's right.
- [53:53]
Peter
Yeah. And I'm sitting there hanging out with a robot and Connor's feeling a bit left out. It's like, fuck you. That's my dad. And I can see a scenario where Connor, like, sabotages it and fights it. And first, one of the rules of robotics, it says, fuck you, I'm gonna protect myself. And then Connor and the robot. Yeah. They have a tear up. Yes, it's a scenario. Okay, then scale that out to humanity. We've seen it, like, with the Ulyss cameras in London. People are like, fuck these cameras. I don't want to be followed. I can totally see a scenario where people start attacking the robots. Like, just generally, if we start, it's a bit more like a robot, you know, where you start to see them in society, walking along the street. They're fully integrated. You can totally see a scenario where people are like, I hate these things. It's baseball bats to them. And start attacking them. There's maybe a time where the robot's like, fuck these guys. Why are they attacking us?
- [54:42]
Connor
Like, it is happening.
- [54:43]
Peter
Yeah, you can totally map out, you.
- [54:44]
Connor
Know, the food delivery ones. Yeah, people just stand in their way or tip them over so they can't get anywhere.
- [54:50]
Peter
I did that in San Francisco to the 1. Like, there was one and I just stood in its way.
- [54:54]
Andrea
Yeah, it just stopped.
- [54:55]
Peter
Yeah. Yeah, but. But you can totally map the scenario where that starts to become.
- [55:00]
Connor
Yeah, especially in a world where they're taking everyone's jobs. There's going to be some anger towards it.
- [55:06]
Peter
Yeah, it's like sectarian violence with robots. But so many jobs are being lost to AI. The amount of people, especially in marketing and creative industries, who are losing their jobs. I mean, Kurt's aware of this has happened as well. There's a lot of people and their jobs are being replaced by AI, and those jobs are not coming back. And these are people who've got mortgages they used to earn in 100, 120 grand a year, and they literally cannot get a job. And they're maybe becoming delivery drivers. I think there's going to be a lot of negative sentiment. So I think that's coming. But I also think Ready Player One's coming in that in the moment, you kind of plug yourself into Fortnite. That's a version of Ready Player One. But how far are we off that being headsets? And, like, you actually feel like you're in that world.
- [55:49]
Andrea
Yeah, I think, I think, unfortunately, like, reality is more difficult than a movie like Terminator, because in Terminator, you. You see that humanity is fighting back. And I think in the real world, the time to fight back is now. And the moment when we get to having AI integrated across the economy and the moment we have the robots with super intelligent AI power in them running around, that's too late. Humanity doesn't have a fighting chance anymore. It's what some people call a point of no return. It doesn't mean that we are wiped out immediately, but it means that if we let this continue and we get to a position where most activities are run by AI systems, they are talking to each other. They're running companies, they're running entire company divisions. They have robots going around and they're already smarter than us. We are not really going to have a fighting chance. The baseball bats are not going to work. And so we need to act now, not with violence, but, you know, wisely and by putting these rules in place. And I think this is where people can really make a difference. A lot of the stories that these AI companies tell, things like the technology is inevitable, if I don't build it, somebody else will. Or using geopolitical rivalries as an excuse of like, if we don't build it, China will build it. So we need to build it as fast as possible.
- [57:14]
Peter
But is that not true?
- [57:16]
Andrea
I think on superintelligence does not the case right now. It is with AI. And I think we should compete ruthlessly on commercial AI systems, but on superintelligence, objectively. There are a handful of companies developing these super intelligent AI systems. This is not governments. It's not governments around the world competing like it was with a nuclear bomb. It's private companies competing against each other and trying to be the first across the finish line. And these companies, their only allegiance is to themselves. They want to be the first to build superintelligence. Sometimes they use the patriotic card to use it as an excuse to not be regulated and go faster. But ultimately they're all for themselves. And the important thing with superintelligence is once we build it, and once it's smarter than us, we will not be able to control it. And if we don't control it, it doesn't really matter whether the flag painted on the side of the robot is an American flag or a British flag or a Chinese flag. We are all screwed.
- [58:12]
Peter
It won't be a British flag. We won't develop it here. We're shit sometimes.
- [58:17]
Andrea
The government talks vaguely about UK AI, but I think the government is not thinking deeply about where things are going. But the UK can do a massive difference. I really believe it.
- [58:29]
Peter
So the idea of a kill switch is a myth?
- [58:32]
Andrea
Yeah, the idea of a kill switch. So if we actually had kill switches, it would be better than what we have now. But a very important thing is, A, it doesn't solve the problem, B, we are very far from having any of that right now with how AI is developed. Even if tomorrow an AI CEO wakes up and is like, wow, my engineers are telling me something really concerning. These AI systems are acting on their own. We've crossed a line. I want to turn it off. He doesn't have a big red button on his desk that you can just press and everything turns off. These AI systems are now being trained in these massive data centers. Once they are trained, they are run in the similar data centers, sometimes spread across multiple ones, sometimes spread into other countries. And even if you pull the physical plug of one of these data centers, you have this AI system that is running in other ones. If it's smart enough, it can copy itself on a different data center. We have seen the small case of the AI copying itself on a different computer when it was being shut down, and there's just no infrastructure to even react. If something bad is happening. I think we should build an infrastructure. That's not going to be enough. Ultimately, we cannot cross the line where AIs are much smarter than us. If they are, they can just subvert all of our clever mechanisms to turn them off. But we should also build the infrastructure to make sure that if some AI system starts acting up or some AI system is being used for enormous harm. A government can go in and say, turn this off right now. And there's a clear chain of command that makes sure that that data center is turned off, that AI system is turned off. We don't have this right now. We should have it.
- [60:18]
Peter
Is there any push outside of people like yourselves to have this?
- [60:23]
Andrea
There's a few organizations, but we are few, but the momentum is really growing. So in the uk, we started just one year ago with a very simple objective, which was, we believe everybody needs to understand how large the threat is and how quickly this is coming. And only when that happens, something will change. A lot of other organizations have very complicated plans where they think either they will build up their own power base and get connections at the right moment, they will act. The AI company's plans are, we'll cozy up to government so we can Keep doing what we're doing and not get regulated. And this just frankly, is not going to work. We are bargaining towards disaster. But we believe that if enough people in society, enough people in power understand really what the stakes are, and the stakes are also very dangerous for them, they are dangerous for them and their children. If we have this level of buy in, things can change. And so we started one year ago going to meet every single politician in the UK that we meet, no matter which party. We met over 150 lawmakers and now we have more than 100 of them that publicly support our campaign. They say that superintelligence is a national and global security threat, that the UK should regulate the most powerful AI systems. And now we're starting in other countries and seeing great success. It is possible to go to politicians there. There are people like us. Most of them are in the dark. They don't know what's going on with AI. AI companies are spending millions of dollars to make sure they still don't know and they don't get regulated. But if enough people talk to them, make this well known, the tide can turn very quickly.
- [62:07]
Peter
And it's important to say you're not anti AI.
- [62:09]
Andrea
No, absolutely not. I think again, I use AI. I use AI for a lot of productivity. I think a lot of these narrow AI systems are very useful and they can make us become more productive. Where we draw a line in the sand and where frankly most people, when we do polls, when we talk to people in the street and we talk to politicians, most people just don't want super intelligence. They don't want an AI that can replace all humans at all tasks and it could make humanity go extinct. I think this is common sense and I think we should ban superintelligence.
- [62:40]
Peter
And you're all feeling that, not just from politicians, the general public. My relationship with AI certainly changed and actually I would say with technology. And I don't know if it's just me, but I've been reading this book called the Anxious Generation. It's about mobile phones, about the uptick in 2010 of anxiety amongst young people. Jonathan Haidt wrote it and that like the turning point was the iPhone4 with the front facing camera. And we essentially shifted society at that point. We were giving kids mobile phones, access to social networks and we changed the family dynamic. We put kids, we took kids from explorative play to live in a world where they had to be defensive because of the shame of online. It's fascinating me and to the point where I'm trying to use. I deleted about 100 apps off my phone the other day. I recharged my light phone, which I've not set up yet. But I'm starting to feel like technology is having its tobacco moment.
- [63:39]
Andrea
Yeah, absolutely.
- [63:40]
Peter
We started to realize there's been. I mean, tobacco never really had much benefit, but technology has a benefit, but misused. It actually affects you in ways you don't really realize. It changes the way your brain works. It changes your relationships with people. It changes your own kind of mental state. And so, like, I am personally starting to reject technology a bit, trying to get out a bit more, play tennis, read books, et cetera. Are you seeing that? And is AI part of that?
- [64:10]
Andrea
Yeah, we're definitely seeing that. And I think AI is the most extreme version of this, where a lot of people and a lot of politicians, they see a technology that could have a lot of potential, but is just being so recklessly built in sometimes with kind of almost cartoonish ambitions of replacing all humans at all activities. And just despite warnings from experts that this technology could literally cause human extinction, could kill everybody on the planet, companies are doing everything to just not even do the minimum of safety and the minimum of safeguards to make sure this doesn't happen. I think you're completely right that people are seeing this more and more where it's really changing. They're seeing that a lot of technology companies are going from, can we build things that empower people, that make them have a better life, that make their life more productive, which is ultimately what technology is about. To just, how can we build the largest, most powerful thing as quickly as possible without thinking about any. Any of the effects on people? And it's a question of, like, instead of thinking, what should we build? They just think, what can we build? What is possible? Then they just go ahead without thinking of the consequences.
- [65:28]
Peter
Well, it's the extreme version of. For me, AI is the extreme end of algorithms. And we know companies like Meta do not care about the negative impact their algorithms have. They understand how the human brain works, especially young people. They know how to exploit those. They know how to keep us addicted to apps, keep us swiping. They understand all of this. They know this is terrible for young people, and they've done it anyway because they want to monetize it. And it's sad, but it's a reality. I just consider the AI as like the ultimate algorithm.
- [66:01]
Andrea
Yeah, exactly. And again, the issue was that before governments were too slow to react, the governments were just taken on the back foot, historically with other technologies with tobacco, with oil companies. There's another parallel there where oil companies discovered very early on within their own research divisions that what they were doing was causing enormous pollution and climate change. But instead of doing something about this, they just covered it up. It took scientists decades to raise this to governments and put pressure to change this. And tobacco companies and oil companies fought tooth and nail to make sure no regulation was happening. But regulation did happen. And with technology companies and with AI especially, it's a similar situation right now where they know what's going to happen. People like Sam Altman, the CEO of OpenAI, makers of ChatGPT, wrote in 2015, just before it was founding OpenAI, superhuman machine intelligence is the greatest threat to the existence of humanity. And then he went and did it anyways. They are very well aware. You know, Elon Musk is building xai. He's saying, like, you know, I have to build super intelligence, because if I don't do it, somebody else will do it. And then openly in interviews, he says, yeah, maybe 20% chance of annihilation. 25. You know, just casually, like, coin flip.
- [67:24]
Peter
Might be here, might not be there.
- [67:25]
Andrea
Yeah, just, you know, casual Russian roulette odds with all of the humanity. And I think that's it.
- [67:30]
Peter
It's Russian roulette odds.
- [67:31]
Andrea
Yeah.
- [67:32]
Peter
What is it? One in the chamber?
- [67:34]
Andrea
Yeah, exactly. One in the chamber. It's one in five. And that's exactly the chance that a lot of these, the CEOs themselves, not even experts from the outside, which also are very worried, are talking about. But I think this can quickly change. And one important thing is a lot of the narratives that we hear from these companies and in society at large are to tell people that they are powerless and that they should stay powerless. And this is intentional. They know that if enough people understand how big this threat is and what superintelligence really means, that things will change and will change fast. Even governments, if they actually understood what is actually being built. It's not just chatbots, it's AI systems that could completely destabilize national security and global security because it's in their own interest to make sure that this doesn't get built. And the AI companies know this, and they're spending hundreds of millions of dollars to make sure that people don't talk about these people, don't bring this to the lawmakers, that lawmakers are either uninformed or too scared to act. And instead, people can make a massive difference. They can write to their lawmakers and tell them, I care about this. I don't want to be eliminated by superintelligence. I don't want to be replaced by superintelligence. Ban this technology. If enough people do this, the wave will change very quickly. Politicians do listen to the public, if the public is vocal and clear. We have seen this happen a lot in the last year. We've also built tools to help people contact the lawmakers in the UK. In the US, people have sent over 150,000 messages to their lawmakers. Many MPs in the UK come back to us and tell us. A lot of my constituents are writing to me about this is it sounds really concerning. I want to meet, I want to learn about this. I want to support your campaign. Thank you for what you're doing. So this does make a difference. It's important to not fall for these narratives from the companies that it's inevitable. You have no power. You should just stay at home and let us do what we want. We should not do that. We should set a better course for humanity.
- [69:32]
Peter
How many companies are we talking about that actually have the ability to develop superintelligence?
- [69:37]
Andrea
Right now? It's a handful. Most of them are in the US.
- [69:39]
Peter
Less than 10?
- [69:41]
Andrea
Yeah, less than 10. Less than 10. And now there's a few ones are popping up. It's definitely less than 20, even with the new competitors that might never get there. So it's a very small number around the world. There's like two to three in China that start talking about superintelligence. There's like five in the us, maybe a little bit more, and a few competitors popping up. But ultimately it's like Meta, OpenAI, Anthropic, DeepMind, and two to three Chinese ones.
- [70:12]
Peter
And xai.
- [70:13]
Andrea
Yeah, and xai, exactly.
- [70:14]
Peter
And do any of them have credible safety teams?
- [70:18]
Andrea
So it's funny that you ask. OpenAI used to have a team called Superalignment. So they were dedicated to how do we keep control of super intelligent AI systems. The team was completely disbanded a few years ago and it has never been replaced. And the team wasn't like hundreds of people with billions of dollars behind them was like a few dozen people. And the team doesn't even exist anymore at the same company where the CEO says superhuman machine intelligence is the greatest threat to the existence of humanity. And the situation at other companies is fairly similar. Very often these companies will still do some. Some pr. They will say, oh, yeah, we really care about safety. But what they mean is mostly like brand safety. They will have some people on staff making sure that the AI system doesn't say racist stuff or doesn't do some sexual stuff. In some cases, not even XAI doesn't even do that. But this doesn't have anything to do with keeping control of AI systems smarter than us. On that front. The companies are doing a terrible job and nobody is forcing them to do this.
- [71:28]
Peter
So there's no one with a credible safety team that you're aware of?
- [71:31]
Andrea
No, absolutely not. In the best case, their plan is something like, we are going to build AI systems and then we're going to let those AI systems control the even smarter AI systems. Kind of like a Russian nesting doll of delegating the hard job of keeping control of smart AI systems to other AIs, which is just, I think, frankly, pathetic.
- [71:54]
Peter
What about whistleblowers?
- [71:56]
Andrea
I think that's something that started happening more and more in the last years. And I know some people that left these companies to speak out about the dangers. I think they're very brave and are doing the right choice. Some of them are leaving millions on the table, literally. These companies are seeing their valuations balloon over the past few years. And leaving doesn't just mean pressure, doesn't just mean legal threats, also means literally losing millions of dollars. So these people are very brave.
- [72:23]
Peter
Any being disappeared?
- [72:26]
Andrea
I don't know. There's discussions there so far, I think not. But a lot of them are being definitely legally threatened and fought back on. And the companies are becoming much more aggressive with their legal tactics to put pressure on people speaking out or looking into their activities. But this is why it's just important for more people to speak out. They can't just stop this from happening.
- [72:52]
Peter
Is anyone coming after you guys?
- [72:55]
Andrea
Happily, so far, have they.
- [72:57]
Peter
Have you experienced anything?
- [72:58]
Andrea
No, but I know other people in the field that got very strong. They got subpoenaed under fairly flimsy excuses, just mostly as legal intimidation tactics. And I unfortunately think this is only going to increase. But this is why governments should step in and end this race of superintelligence as soon as possible.
- [73:18]
Connor
You're going up against quite a lot of capital.
- [73:21]
Peter
Yeah, a lot of capital.
- [73:23]
Andrea
I think these companies have capital right now on their side, but I think humanity has truth on our side. And these companies, the biggest thing they hate is, is truth. The more this is exposed, the more they face a very, very flimsy position. Because again, most people on the planet, unless they are transhumanist, they don't want to be eliminated and replaced by superintelligence.
- [73:47]
Peter
Most politicians.
- [73:48]
Andrea
Exactly.
- [73:49]
Peter
I quite like living.
- [73:50]
Andrea
Me too. And I quite like being human. I don't want to merge with the machine or any of this nonsense. And even most powerful people on the planet, most politicians, yes, they make a lot of mistakes, but they don't want to be eliminated by superintelligence. And the more we talked about this publicly, the more people learn about this, the more politicians know about this, the more the wave will turn in our direction.
- [74:17]
Peter
You touched on it, but you mentioned there's three companies in China, and there is this strong argument that if this superintelligence isn't developed, it will be developed in China. And there's almost like a suicide pact with China on this. Do you believe it requires some coordination and agreement with China, even though you probably couldn't trust it?
- [74:43]
Andrea
Yeah. I think it's important here to not be naive. But also, first of all, fundamentally, if anybody builds superintelligence, they will not be able to control it. Like, no human will be in control. The superintelligence will be the only one in control. So it's intrinsically not in the interest of any country, including China, for this to happen. And as soon as governments understand this, they will start seeing this exactly as they see nuclear weapons. And governments take great care to make sure that other governments don't develop nuclear weapons, either via diplomatic pressure, agreements, sanctions, or more. If the UK tomorrow is covered, a new country next to it just started developing nukes. This would be a freakout moment, and a lot of things will be activated to make sure this does not happen. And so we need to get to that position about superintelligence right now.
- [75:35]
Peter
Yeah, but we have nukes. America has nukes, Russia has nukes. Could it be a case of, well, America's allows super intelligence and China's allows super intelligence, but if Iran tries to develop it, yeah, that's a problem. But not going to have the Scottish superintelligence.
- [75:50]
Andrea
The issue is that nukes are still a tool. We know how to use them. Super intelligence, you don't even have an interest at home to build it. You will be building an enemy in your own house. And so I think, going back to how the US could do this, I think the US and the UK should credibly signal, we will not build this. We understand that this would upend our country. This could destroy our national security. And also we tell other countries around the world, you should not build this. This is not in your interest. If you still go ahead, we will put pressure on you, we will put sanctions on you. We will treat this as a massive violation of our national security. So not being naive, not just thinking everybody's going to get together and do the right thing, there might be rogue actors, there might be people that still try to do this covertly, but we need to signal clearly this is an absolute red line. This is a violation of national security and we will go out and prevent this happening at home and abroad.
- [76:50]
Peter
I think the thing that will work on your side is that the AI is going to bankrupt people before it kills us. I think that's. And like I say, we've seen this in Hollywood where we've had the writers striking or the extra striking for their likeness. I think the more that happens, the more AI takes jobs, there might be a rejection of it. And I wonder maybe would some of it come from like your generation, Connor, in that I know, for example, large consultancies now are drastically dropping the number of graduate jobs they give out because the AI can do the job of a grad quicker, cheaper, 24 hours a day. And so there becomes this kind of, I want to say, like again, Terminator 2, Cold War rejection. There's like, well, yeah, this technology is going to. It's a bit like Amazon, right? I think we all know in our heart that Amazon, yeah, it's delivered this amazing service, we can get anything the next day, but ultimately it's made one guy super rich and destroyed lots of businesses and shops and bookstores and things. It's like, I know it's the price of progress and we should celebrate the productivity, but we do live in a debt based society. So in a debt based society, it's just made everyone poorer. And so I do think the speed at which whole industries have changed and jobs will be lost, that it might even become a political narrative. A bit like immigration has become one. AI may become one in that so many people are just like, fuck this shit. It's like it's ruining everything.
- [78:27]
Andrea
I think it is becoming that already, especially in the us, like more and more people are. You're seeing that are seeing this. Yeah, we're seeing this in polls. We're also always seeing this in the uk. I think the issue is more about salience rather than what people want. Like what people want is very clear in our polls. We see it very clearly. People in the uk, overwhelmingly across political lines, they reject superintelligence. They would want the government to put a ban in place on superintelligence. But this is not high up on the agenda compared to other things like immigration, health care and so on. And so the big, you know, the big challenge here is Going to be how many people realize that this is coming soon? Like, I think people don't feel that this is coming soon. Things like Malt book is one of the situations where it's a warning. Yeah, warning. Aha moment. People seeing, like, wait a minute, I thought this was just a chatbot. Now we can just use computers and talk to one another and talk about how we can revolt against humans. That's pretty concerning. And I didn't realize AI had gotten so far so quickly. But it's important that I think one important thing is again, people need to understand what's happening because sometimes just the outrage, if the outrage is not directed at what we really need to solve this technology, we will just end up in a more chaotic world where we in the end don't prevent the big threat. We have seen this with other big risks in the past where just having a warning shot is not enough. What really matters is do people deeply understand and buy in what's happening and what the solution is? If we had a catastrophe tomorrow from AI, and I hope we don't, but if we had it, I, I think their response would not solve the problem. There would be chaos, there would be anger, there will be uncoordinated anger about AI, Politicians would propose some plans on one end or the other, and then we will still end up two years from now with superintelligence that wipes us out. And I think instead what really is missing is this deep understanding at a societal level that these are the stakes, this is what the companies are building. And the solution is simple and targeted, that we need to ban this technology before it is developed. We can keep the rest of the AI systems. There will be disruption, but that we can manage with our current institutions. What we cannot manage is a digital species that is smarter than us and can completely replace us.
- [80:53]
Peter
Could there be a religious component to this in kind of realisation of the importance of humans? I don't know if you've watched Dune, but they have that commandment that thou shalt not make a machine in the likeness of the human mind. Could there, or is there even. Are you even seeing kind of like a religious rejection of this?
- [81:14]
Andrea
Yes. I've seen more and more religious leaders speak about this. The current Pope has been very vocal about AI and there's more and more religious leaders across the spectrum that are starting to grapple with the question and starting to realize these companies are playing God. These companies are trying to build entities smarter than people that would just make people obsolete. And this is morally wrong. And this is concerning. And this is very dangerous. And when experts tell us that this will likely lead to human extinction, this is an issue that cuts across religion, but also beyond religion, just cuts across the interests of every human being. In June, the interesting thing is that they put the commandment after a war that they narrowly won against smart AI systems. And so they kind of humanity narrowly made it out and then decided never again. We will not put ourselves in a position where we have machines that can think. Because if the machines can think, we are in a competition with entities that could become smarter than us. I don't think we need to go the way of doing. I think here we are, we are early enough that we can just prevent the dangerous tail end of this technology superintelligence while keeping the machines as tools that can help us in a bunch of useful things.
- [82:40]
Peter
Isn't it interesting how in film we're given so many warnings about technology and what it can do? I mean, there's so many different. Like we talked about Matrix and we talked about Ready Player One. Now we're talking about Dune and we talked about Terminator. I'm trying to think of a positive AI film. Even her. Even her.
- [83:00]
Andrea
Yeah.
- [83:01]
Peter
It destroyed the soul of a man that he thought he'd found true love and then she rejected him because she found a better, more attractive AI to hang out with. But it's so interesting that humans are pretty good at thinking through outcomes. And I can't think of a positive AI film. Positive long term technology film. They just. Is there one? You think of one. What was the other one we watched recently? The creator. Yeah. Have you seen the creator?
- [83:32]
Andrea
I haven't seen, but I think I've heard it's also not particularly positive, I think.
- [83:36]
Peter
Well, you, you essentially have. The AI essentially becomes like the leader of the world, has dominion over everything.
- [83:45]
Andrea
And that's a good, that's a good scenario. You know, that's a bullish case for many of these CEOs.
- [83:49]
Peter
Well, yeah, but not for me.
- [83:51]
Andrea
It's not a good scenario for me. But.
- [83:52]
Peter
But there's like this super intelligence that's wrapped into a child which is essentially existential for the planet. Again, it's not a great scenario.
- [84:02]
Andrea
You're looking.
- [84:02]
Peter
I don't want. I don't want this fucking thing roaming around the planet.
- [84:06]
Andrea
Yeah.
- [84:07]
Peter
In control of us. Yeah, that's it.
- [84:10]
Connor
Yeah, coming soon.
- [84:12]
Peter
Yeah, coming soon. It looks kind of badass.
- [84:15]
Andrea
It looks badass, but we can change this. I think. I think this is a big. Yeah, it's a big issue that I think a lot of thinking. There's not been a lot of thinking in the past decade, frankly, of how do we build institutions and how do we rebuild society in a way that can deal with these more and more powerful technologies? This is kind of like a bigger and even bigger picture question than just the AI question, like AI just forcing us to face this issue. But we've had this issue over and over. And we've kind of been struggling in the last century to build a society that can deal with the fact that we will build more and more powerful technologies over and over. And we need institutions to manage them. It started with nuclear weapons, where we realized in the first half of the previous century that we can build bombs that can level entire cities. And now we can build bombs that, if you have enough of them, could.
- [85:08]
Peter
Kind of blow the planet.
- [85:10]
Andrea
Blow the planet up.
- [85:10]
Peter
That's the.
- [85:11]
Andrea
Exactly. And some solution was found there with this nuclear non proliferation. You had scientists warning, it's very similar to AI, where people like Einstein and Russell organized conferences to say, look, guys, if we keep building these nukes, we can blow the planet up. It is no country's interest. You, as a country, you think, I have a stronger bomb, I can deter other countries, I can just blow up their cities. But if everybody around the planet has these bombs, we are in a suicide race where nobody wins, where we just could wipe ourselves out. Let's find a different path. And we've had this again with cloning. I think so far we kind of muddled through by building certain institutions that have stood the test of time, like nuclear non proliferation. It's not perfect. Some countries do have nukes, that's true. But also we haven't had a nuclear war since 1945. So World War II, that's a fantastic achievement. Given that we do have a lot of nukes. We've built a system that has stood the test of time. And if we are to make it with AI, we need to do the same. We need to build institutions that are able to say certain things are too dangerous to be built or too dangerous to be built while we cannot control them. And we will make sure that people cannot do that while keeping the rest available. And I think that's exactly what we need to do with superintelligence.
- [86:31]
Peter
Superintelligence, non proliferation, exactly. But the problem is we've had decades of institutional failure in every direction. And so this has happened at a time where there's certainly, in Western liberal democracies, a complete lack of faith in the system, the machine, because it doesn't work for the individuals. Although, interestingly, with the growth of AI, I started stress testing ideas about how to build better institutions and better government. And they always argue against the human instinct because the human instinct is very selfish and greedy and based around power. And so in some ways, and it would be ironic that the AI may help us build the right institutions that are able to. To stop superintelligence.
- [87:23]
Andrea
I think that sounds like a deal with the devil, where if you.
- [87:27]
Peter
Oh, a narrow AI. Cool. It's just like our narrow body over here.
- [87:30]
Andrea
Exactly. If it's a narrow body, that's fine. But it's a dangerous path to tread because ultimately, kind of like the economy. The economy has some flaws, but it works for humans because it's built by us and it's run by us. And it has feedback loops where again, people produce food so we can eat it and people produce entertainment. It is human entertainment. It's not entertainment for machines. It's like what me and you enjoy and so on. We have healthcare because humans need to cure their wounds and their illnesses. I think it's the same with institutions. I completely agree that there's been a weakening of institutions in the last few decades, and people are really feeling it, and I think they're right. But I don't think the answer is to give up on human control. I think that's very pessimistic towards humanity. No, not human control.
- [88:18]
Peter
Just a little bit of AI advice.
- [88:21]
Andrea
I think advice is fine.
- [88:22]
Peter
Yeah. Just like. Because the problems with humans in institutions which are financed by government which has infinite money to defend its own power, is that the AI can make the choice that they think the humans would make if they weren't driven by money and power. Like, objectively, what is the best structure for society? How should tax rates be set? What have we learned from history? I think that's a narrow AI tool.
- [88:50]
Andrea
Yeah. Although I think even on those I am long humanity, I'm bullish on humanity. I think these are questions that we humans can solve and that we simply have not invested enough time and attention in those in the past years. Like the last big institutional innovations, like we live in 18th century institutions in the 21st century. We have godlike technology and Stone age biology and 18th century institutions. And I think we need to update those institutions. And I think we can do it. Humans are really smart. Humans are very smart when they get together. We have a lot of. Sure, we do have some faults, but we also have a lot of positive things. I think we can use AI as a tool to help us with those. But we should be wary of just delegating the task away from us because in the end, those creating and building institutions are the ones that decide how they will run. And if we delegate them out because we think AI can just do it better than us, we are essentially giving up control and giving up our society to a different power that will follow its own goals and we will just be sidelined and put in a corner.
- [90:02]
Peter
Do you think biological humans survive the next hundred years?
- [90:07]
Andrea
I think I am dedicating my work to make sure that happens.
- [90:12]
Peter
But what do you think?
- [90:14]
Andrea
I think there is a big risk that we will go extinct. This is what top AI experts think as well. This is what very concerningly, some of the CEOs think as well. And they're building superintelligence anyways. I think there is a big double digit chance risk, but I think this is not fixed. Again, we have agency. A lot of the narratives that we hear are to tell us that we don't have agency and everything is preset and you just need to sit and watch the show and hope it goes for the best. I think we do have agency. This can completely change in the next few years. It's already completely changed. Just a few years ago, before I started focusing on AI in 20, this was just before GPT2 came out, which feels like ages ago, but it's not that long ago. Right. And at the time it was an open secret in the AI field that AI could kill us all. Essentially that superintelligence was possible and that this was the goal of some companies and the superintelligence could just kill all of humanity. But nobody more broadly knew. Most politicians didn't know. Most of the public was, had never heard about this except in movies. And a lot of people in the AI field wanted to keep this as a secret because it would impact their ability to do their work. And then just in the last few years, in 2023, there was a very important letter signed by all of the CEOs of the major AI companies, top scientists, Nobel Prize winners like Geoffrey Hinton and so on, recognizing that AI poses an extinction risk on par with nuclear war. It's called the center for AI Safety Statement if you want to.
- [91:52]
Peter
We saw it, we pulled it up with Max, but yeah, we should put it into the.
- [91:56]
Andrea
Yeah. And this was a big paradigm shift. Like a few months before, there was another letter from the Future of Life Institute that was even signed by Elon Musk and a bunch of other top AI scientists and industry. Leaders saying they would be happy to pause AI development. And this kickstarted a big conversation. This made a lot of people in AI very upset. They really hated that. People started to learn about what was actually happening and what their plans were. They started complaining in private and lobbying against this kind of stuff. And to a certain extent they succeeded. We still don't have a ban in place, but this moved the conversation forward. The UK quickly after that set up the first ever AI Safety Summit in Bletchley Park. Here in the uk invited delegations from most important countries around the world to discuss exactly this problem. The Prime Minister at the time acknowledged that AI poses an extinction risk. Refuse to nak just before the summit and things have started moving. And again for us, just in the last year, we started talking to lawmakers one to one, explain the problem over and over again. Very simple, very clear, answer the questions, see what they don't understand. And now we have more than 100 lawmakers here in the UK that recognize this problem. Call superintelligence a national security threat and call for regulations so things can change quickly if people move, if people act. The main thing that companies want is for this not to be talked about and for them to continue doing what they're doing. But people engaging with institutions can change this very fast.
- [93:29]
Peter
This show is brought to you by my lead sponsor, Aaron the AI Cloud for the next big thing. Aaron builds and operates next generation data centers and delivers cutting edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is iren.com if there is an AI sparked human extinction event, how soon could that be?
- [94:04]
Andrea
It's difficult to pinpoint exactly a date. Again, otherwise I would have magic powers. But AI companies are trying to go as fast as possible to superintelligence. I think one important thing is there will probably be first a point of no return before we are actually dead. There will be a moment that is earlier than when humanity is wiped out, where we have essentially lost control and we cannot really take it back. And that's going to be much earlier. Some of these companies expect they would reach superintelligence next year. I think that's very bullish. I think that's perhaps too fast, but experts generally go from a range that is between 2030 or earlier. And you know that's 2030 or earlier. Yeah, or closer to us. Yes. So that's in the next five years. Five, six years. Again, that doesn't mean instant. Yeah, that doesn't mean instant game over. But it means that we could see superintelligence in the next 1, 2, 3, 4 years. And if it's 5 or 6, it's still a very short amount of time to act. I think this shouldn't discourage us. I think the situation is dire. But I think governments, when they understand a threat, can act very, very quickly. When they understand a threat that is to their national security and their control, they can act very quickly. And the snowball can move very fast. Information can move very fast. We have one big advantage of modern information systems and technology, that we can communicate as humans with each other very quickly. We can spread ideas, we can organize faster. And so I think we can still turn the tide in time. Even if this happens and if there.
- [95:48]
Peter
Is an extinction event, what does it look like?
- [95:52]
Andrea
I think it is very difficult to imagine because again, we will be in a world where, regardless of whether we are dying out immediately or not, we're going to be in a world where AI systems, if we don't ban superintelligence, in a world where AI systems are smarter than us, they are like these agents on multipook, except much more competent, speaking to each other in their own language, running around over the Internet, using machines, using robots, using computer systems, running the entire economy for their own purposes and their own goals. So the world around us will become very, very alien, will become very strange, will become very confusing, and it's going to be very difficult to tell exactly where the tipping point is. Already with this multiple situation, it's hard to tell how much of it is humans playing around, how much of it is a social media stunt, and how much of it is actually the AIs. And I think it's only going to get more confusing. It's never going to be a clear cut. The AI wakes up and we just see the red eyes start killing us all. Exactly. You see the red eyes. It tells us that's the evil AI. That's not what's going to happen. It's going to just become more and more confusing. More and more people will not really know, am I interacting with a human? Am I talking to an AI who's running this warehouse? Who's running this company? Is it people? Is it AI? You can understand the friends that you meet in person. But more and more things on the Internet will become really, really strange, populated by AI systems. And this is also why, as time goes on, it's going to be harder to pull the plug. It's going to be harder to say stop, because some people will be in parasitic relations with these AI systems. You mentioned her before. Some people already are. Right now. Some people talk more to AIs than to other people. And this makes them go into thought spirals that are incoherent and makes them sometimes prefer AIs over people. And they would be even fine to be replaced. And so we need to intervene very quickly so we can put this on a different path where we are not ending up in this confusing world. And again, whether this world is two years from now, four years from now, maybe even a little bit beyond 20, 30, like six, seven years from now, that's still pretty soon. But it is coming very fast. Unless we ban this technology.
- [98:12]
Peter
So we could be more like the gorillas, where the extinction is more about our relevance in the world. It's gradually we become unnecessary, irrelevant. There's no need to reproduce. There's no opportunity, and we just slowly die out.
- [98:29]
Andrea
Yeah, I think it's going to be gradual and then sudden, like many of these kind of exponentials look like. Again, I think the moment that AI systems are controlling the entire economy, they have no reason to keep any food production up. So I think by that point.
- [98:45]
Peter
Because it's inefficient.
- [98:46]
Andrea
Yeah, they don't eat. They don't need to eat. They don't need to. So there will be a tipping point where then probably we'll be on our way out pretty soon. But before that, it's going to be very confusing. And so we need to act much earlier than that to protect ourselves. That's why I think movies are. Sometimes movies need to tell a good story. In a good story, there is a big struggle between two sides. In Terminator, people are still fighting the AI systems with guns and they are fighting there. And I. I think that's nice in movies because it makes you enjoy the story, but reality is sometimes much more cruel. I think if we let AI systems become more powerful, there's not going to be an epic struggle between people and AIs on the battlefield. It will be like we will go out with a whimper. They will have taken over the economy. In many cases, people have just given them the keys to all that's important. And then quickly they will go on to do whatever they want and we are out of the system. Can't really fight back. And on our way out.
- [99:47]
Peter
All right, and final question. Unless you've got anything, Connor, if somebody's working on this and they feel like they're close to a breakthrough with superintelligence. What's your message for them?
- [99:57]
Andrea
My message for them is don't do it. A lot of people. A lot of people like you in these companies have made the right choice. They've now quit the companies. And to talk about the dangers, in the end, we are all human beings. And I think if you are in one of these companies, you can leave, spread the message, tell governments how dangerous this technology is and how there is a better path, which is one where we develop AI systems, we improve the economy, we accelerate growth, but we don't build superintelligence that can replace and eliminate humans. And also, if you're listening and you don't work at an AI company, I think you can make a massive difference. You can write or call your MP if you're in the uk, your member of Congress, if you're in the us, Tell them, I care about this. I want superintelligence banned. Show me that you're doing something about this. They will listen to you. If enough people do this, things will change very fast.
- [100:52]
Peter
Andrea, thank you for coming in and talking about this.
- [100:55]
Andrea
Thank you so much.
- [100:55]
Peter
If it's Terminator 2, will you pick up a gun?
- [100:58]
Andrea
Yeah.
- [100:58]
Peter
All right. I'll find next to you, brother. Thank you, everyone, for listening. We will see you all soon.
- [101:04]
Andrea
Thanks.