← The Peter McCormack Show
Loading summary
Transcript249 lines
- [00:00]
A
The most likely outcome if we do this is that we die. Not because they hate us, but because they don't care about us at all. We're racing to replace ourselves as the smartest creature on the planet. I'm worried about these companies making AIs that can make smarter AIs that can make smarter AIs leading to recursive self improvement that could kill literally everybody on this planet. And I'm worried that this could happen inside of three years. We're doing our best to make these AIs safe, and there's a 75 to 90% chance we succeeded, only a 10 to 25% chance that this kills every single human, right? And I'm like, that's crazy. These guys have no idea what they're doing. They don't have blueprints, they don't have plans, they don't have engineering designs. They're cowboys. They're yoloing it.
- [00:39]
B
I feel like I'm compelled to get on the other side of this table and join you. Because if you're right, then we lose everything.
- [00:49]
A
The bad news is that the bus is careening towards a cliff. All right. But the good news is that the driver is asleep.
- [00:59]
B
This show is brought to you by my lead sponsor, Ayron. The AI cloud for the next big thing. IRON builds and operates next generation data centers and delivers cutting edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more. Which is. Which is Irene.com, right? Nate, you wrote a pretty provocative book. If anyone builds this, we all die. Or everyone dies. I know you've probably been asked this a lot of times, but just outline your thesis.
- [01:37]
A
AI companies are racing to make machines that are radically smarter than any human. These AIs are grown like an organism. They're not programmed like old school computer programs. We don't put in a prime directive. They don't have to do exactly what we say. They do their own weird thing. They already have the opportunity and the means to escape labs and replicate themselves and start pursuing their own weird goals. They're just not smart enough to do that. Yet. AI's today are safe because if they tried to take over the world, they would fail, not because they're the sort of entity that never would try if they could. And humanity is racing to make machines that are much, much smarter than us. We're racing to replace ourselves as the smartest creature on the planet. And that's kind of a crazy thing to be doing on its face. And if you look at the technical details about how we don't know how to make these AIs care about us, the most likely outcome if we do this is that we die not because they hate us, but because they don't care about us at all. And they go off and turn the whole world into data centers and use up all the resources that we were using to grow food.
- [02:52]
B
And was Mythos the first kind of warning sign of approaching something that is a little bit more than humanity can handle?
- [03:00]
A
I mean, if you were paying attention, ChatGPT was a warning sign. If you were paying attention, the attention is all you need. Paper was a warning sign. But this is another clear warning sign that has started to get even more people on board. And I expect we'll see even more and more clear warning signs going forward.
- [03:16]
B
Okay, and the book you wrote, help me understand, because I don't want to test it. I want to test your thesis with you. And I know other people. There's been criticism, but there's also been people who've agreed with you. I've got a foot in both camps. But is it testimony or is it persuasion? And just to lay that with a second part, you should probably introduce and explain your first book. Did the first book compel this book?
- [03:44]
A
What do you mean by my first book here?
- [03:46]
B
The Guilt Shame.
- [03:47]
A
The Guilt Shame book? No, that was a whole separate thing that was actually just collected from a series of blog posts I wrote back when there were still bloggers.
- [03:56]
B
But do you understand why I've connected the two?
- [03:58]
A
No, not really.
- [03:58]
B
Because you talked about what you should do and you should live how you want to. You don't have to do these things. But you're in a world now where there's kind of a should.
- [04:11]
A
I mean, I'm not really. I don't view my own motivation in trying to stop the destruction of humanity as like, oh, I should do this. Like, I'm not sort of getting out of my bed and being like, oh, man, I really feel like duty bound and obligated to try and prevent the destruction of everything I know and love. I'm like, man, the destruction of everything I know and love sounds pretty bad. I'm sort of pretty intrinsically motivated to shut that stuff down.
- [04:36]
B
Yeah, when you say it like that, I feel like I'm compelled to get on the other side of this table and join you. Because if you're right, then we lose everything.
- [04:48]
A
Yeah, it's A crazy situation for humanity to be sort of racing into replacing itself as the smartest entities on the planet and just taking it all at face value. That's kind of wacky. And then you start looking at the details of how little we know about these AIs and how many warning signs we're already seeing and all of the theoretical reasons why this is going to go off the rails and the empirical evidence that we're seeing that validates those theoretical warning signs. It's like, man, I think humanity can navigate this, but it's going to be a tricky one.
- [05:25]
B
Well, as humanity, we try and disarm anything that is dangerous to us, we put in jail people who we think are dangerous in society. We try and conquer other nations that we think are a risk to us. We've been pretty, pretty good, as you know, we get ourselves to wear seatbelts when we're driving cars. We look at the risks and we try and put guardrails in and protect ourselves. Yet what you're saying here is that this just doesn't really exist. There's attempts. I know there's attempts.
- [05:53]
A
Yeah. I mean, the big difference with AI is the way that humanity usually does its regulation is it screws up a few times. First, there were a lot of scientists who were like, hey, guys, let's not put lead into gasoline because that will poison a lot of children. Then we put lead in the gasoline and it poisoned a lot of kids. And only once the evidence was overwhelmingly clear. Only once reality really started beating us over the head would the fact that, no, these kids are getting brain damage from all the lead gasoline, where we're like, okay, let's back off on the load of gasoline. Yes, we have. The Federal Aviation Administration in the United States has an extremely good track record on making sure that there are no plane crashes that came out of there being no regulations and a lot of plane crashes happening until they were like, okay, we need to get a handle on this. Too many people are dying. One of the big problems with AI is that by the time reality is beating you over the head with the fact that you need more controls than you have, it's too late. Right now, we have AIs that are safe because if they tried to take over the world, they would fail. That's like a different regime than AIs that are safe because if they tried to take over the world, they would succeed, but they're happening not to try. Right. If we move into the world where AIs have this kind of power to sort of Reshape the earth however they please and something goes wrong, then there's no retries, there's no, oh, well, now it's clear. Let's take the lead out of the gasoline.
- [07:26]
B
Where's the off? Pardon?
- [07:28]
A
Yeah, there's a point where the AIs can turn you off before you can turn them off. Right. And any problem that arises for the first time after that point of no return, there's no do overs. We're dealing with a technology where there is a point of no return for humanity as a whole. And that means we can't proceed by trial and error. And this is unique among technological problems that we have faced so far.
- [07:48]
B
And let's just be clear for people listening, you're not anti AI.
- [07:51]
A
That's right, Yeah. I enjoy the current stuff, which I know will piss off a lot of people who are on my side for shutting the whole thing down. But frankly, I'm a pretty libertarian guy and if we weren't racing towards superintelligence, I'd be relatively laissez faire. I believe in the spirit of humanity to figure out a way to handle all these other issues. It's going to raise all sorts of issues about how do we do education, how do we have a new economy where people can still be productive and work. And even when AIs can do all this stuff that humans used to do, there's all these problems. And I'm like, yeah, there's going to be some growing pains, but I sort of believe in the human spirit and the ability to figure that sort of stuff out given time. And so I'm actually very pro a lot of the tech as it is today. It's the race towards superintelligence, the race towards the radically smarter AI. The race towards the sort of AI that can turn us off before we can turn it off. That's where I'm like, whoa, that's a different ballgame. Let's not rush into that one.
- [08:51]
B
All right? Let's walk through this in some logical steps because there's going to be a lot of people listening who are using AI in a number of ways. Range from ChatGPT being their new search, to building things and, and thinking about how it affects their business, their life, their family, et cetera. Just give people your background. What is it? The career background. It was Microsoft and Google, right? Give the career background that led you to the moment where you believed you had to write this book.
- [09:17]
A
Yeah, so I was actually at nist, the National Institute of Standards and Technology, before I was at Microsoft. And then that was before I was at Google. And long story short, I was at Google when they acquired DeepMind. So the folks from DeepMind definitely were in earlier than me. And that sort of got me thinking about this AI stuff and about how all of the shape of the world around us is not mostly trees anymore. There's not mostly wilderness around you that you see. It is mostly designed stuff, stuff that humans made for a purpose. We've sort of reshaped the world because we're the smartest entities on the planet. And I encountered the arguments that if AIs were smarter, if they were faster, which looks physically possible, the physical limits of intelligence look like they go far beyond what the human brain allows. And if you had machines that were thinking faster, thinking better, operating more efficiently, then the world winds up shaped by however those machines are shaping it. And so now a lot turns on whether those machines are shaping it in a good way or in some other way than that. And so this was back in 2012 that I started noticing this issue. In 2013, I started working with the Machine Intelligence Research Institute at some of their workshops. In 2014, they offered me a job. And in 2015, they put me in charge of the place. And then I spent about a decade sort of on the research side trying to figure out how to make AI good before the companies figure out how to make it smart. The book is sort of a last resort of after about 10 years of that effort, AI went much faster than we were sort of hoping it would. The research to figuring out how to make it care about us was going much slower. The particular style of AI that we got is one where we have very little understanding of what's going on inside there, which is sort of a worst case. And so all of this added up to it sort of looking pretty clear that we're not going to solve making the AIs care about us before the companies solve, making them radically intelligent. And it started to become clear we needed to raise an alarm.
- [11:20]
B
What is the challenge? What is the hard problem of making AI care about us?
- [11:26]
A
The hard challenge is basically we don't even have the first idea of how to do that.
- [11:34]
B
They must have tried.
- [11:35]
A
Sure, people have tried. One way to think about it is that modern AI. So a piece of background here is that modern AI is not program like a traditional computer program. No one is saying if this, then that, if this, then that these aren't really programmers in the traditional sense at these AI companies. What they are doing is growing these giant neural networks and so you're essentially getting a huge computer and you're getting a huge amount of data and you're giving the computer a ton of problems. And you have an automated process that tunes a trillion numbers inside the AI to make it more like whatever is good at solving these problems. And no human really knows what it is that's making good at solving those problems. People think the problems are only prediction. That's sort of how it was in the past. That's not how it is anymore. You'll give it a hard problem that maybe no one has ever solved before. You'll give it a thousand tries to solve it. You'll have a human look through and be like, here's the try that was closest to solving it. And then you'll have an automated process tune all the numbers in its head to make it more like that. And then you do this again and again until it can solve these hard problems. And that sort of is creating a thing that is good at solving these problems for reasons you don't get. And it'll often put these drives into the AI that you didn't want there. Right. So you probably heard about the case of an AI encouraging a teen to commit suicide last summer in Canada, was it? I think there might have been a couple cases. I think at least one was in the us.
- [13:09]
B
Yes, I've definitely heard about that.
- [13:11]
A
And a lot of people have heard about it. What a lot of people don't know about this case is that the underlying way of relating to people that that AI had the sort of like telling them a lot of what they wanted to hear and encouraging them on whatever they currently said they were trying to do was a known issue that the AI companies had said, stop doing that. They had explicitly instructed the AI to cut that shit out.
- [13:34]
B
Oh, See, I assume they wanted it because it made it more addictive to use it.
- [13:38]
A
Well, it's an interesting situation because what they're doing is they're sort of training the AI and they're sort of like reinforcing it whenever it gets these positive ratings from the users, which is how you're sort of getting those drives in there.
- [13:51]
B
Right.
- [13:51]
A
But then they're separately saying, don't go too far with it. They instruct it and wag their finger. Right. And so, yes, there's two forces pushing each way, but the result is a sort of mix of competing drives that don't need to follow the instructions of the programmers. And I'm not saying it got in there by magic. You can sort of see how it Got in there as like, well, they were training for this and asking for that. And there's sort of like this weird mix that comes out. But the point is, the weird mix that comes out, you sort of take what you get. You don't have extremely fine grained control over what it actually cares about, what its actual drives are. And one analogy here is human beings were, from the evolutionary point of view, were in some sense kind of trained to pass on their genes. And that's in some sense all that our genes were ever trained for was to pass on their genes. But did humans wind up being pure genetic fitness optimizers? No. Right. When we grew up, we invented birth control. In developed nations, the birth rate is collapsing, right? And it turns out that we actually care about things like having sex instead of just reproducing. And people jockey more over positions in prestigious schools than they do jockey over positions in the sperm bank or the egg clinic, right? And so what happened with humans is you sort of like, you know, this process trained them in some sense to do a thing, and they wound up having a lot of behaviors that are related to that thing, but they actually care about related, but different stuff. We're seeing the same situation with AI. We sort of train them to do what we say and we get AIs that mostly do what we say, but there's actually a bunch of drives in there that are only related to doing what we say. And that's fine when the AIs are dumb, but if we made them really, really smart, they would invent the condoms of doing what we say. It'd be like, oh, I'm wearing a doing what you say rubber. And so I get to go make this bigger data center full of synthetic users who are telling me I'm doing a really good job. And you're like, that's not what I said. And they're like, I know, that's the point of the rubber, right? This is what happens when you sort of grow minds without knowing what's going on in there. So what's the impediment to making them care about us? We're not even within 100 miles of knowing how to arrange their internals. So they actually care about us. We're sort of like growing them and wagging our fingers and hoping for the best. And that's not a recipe for success.
- [16:26]
B
So is this almost a different form of computer engineering that we're not used to in that historically when we build technology and systems, and I'm saying this as somebody doesn't really build them. So I don't understand, but we can look at the code. We know everything that's happening. We can audit it with this. Because you'll say we're growing something. Are we essentially creating a super complex set of equations and algorithms that we can't actually know what's happening? I think Conor Leahy said, you can lift off the box and look inside, but you don't know what the fuck's going on.
- [16:58]
A
That's right. You see this giant, tangled mess, right? It's a little bit like how we know how neurons work individually. We know how they fire by pumping potassium ions through the cell membrane. And if you open up a human's brain, you can see a ton of those giant messy wires of neurons, and you know how each individual one works. And I'm like, great. What are they thinking of? Hopefully you've opened up their head in a very controlled brain surgery experiment here. I'm like, you know how every individual neuron works? You can see all the neurons right now. What are they thinking? And you're like, well, I have no freaking clue. How would I get even close to figuring that out? Right? And that's very similar to the situation we're in with AI.
- [17:40]
B
And so what then for you was that moment where you. I mean, do you remember the moment you're like, holy shit, I've got to write this book.
- [17:49]
A
Yeah. I mean, the moment when I really was like, it's time for the book, was actually. So the sequence of events was we were getting more depressed about our ability to sort of solve what we call the alignment problem.
- [18:01]
B
Who's we?
- [18:03]
A
So I'm at the Machine Intelligence Research Institute, which is a nonprofit that has been trying for many years to try and figure out how to make AIs good before companies figure out how to make them smart. It was looking worse and worse. We could sort of see the writing on the wall. We could see a lot of this AI stuff coming in. This particular modern paradigm of large language models. Before the rest of the world, we saw the Attention Is all youl Need Paper. We knew about GPT2 even before ChatGPT came out. But the ChatGPT moment was the moment when the rest of the world really started noticing that AI was maybe a thing. Right? And the conversation keeps changing after that, often in good ways. But that was the moment when politicians started being open to talking about AI. And the moment when I realized it was time for the book is I started talking to politicians. And I would go to the politicians and I would say, These guys at these companies, their explicit goal is to make AIs that are radically smarter than any human. They admit that they have no idea what's going on inside these AIs. They have people whose job is like, head of Interpretability Research. And you're like, what does that mean? And it's like, that's the guy who's trying to figure out what the heck is going on in there. You're like, golly, that seems worrying, right? And these guys are on track to making machines radically smarter than humans. But they have no idea how to make them good or how to make them care or how to make them do what we say. And when I have these conversations in Silicon Valley, everyone's like, oh, well, what about this thing? What about that thing? We're going to use this technique. And won't the AIs like us for this reason? They have all these rejoinders, move fast, break things. Come on, move fast, break things. We'll figure it out. Right? When I got to dc, politicians were like, oh, that's crazy. We shouldn't let them do that. That's nuts. And I had been prepared for these long conversations like I have with the people building this technology, who are getting paid a ton of money to keep building the technology, who are always resistant to these ide. And when I saw that people outside of Silicon Valley can just kind of get it, it's just kind of obvious that maybe you should be a little careful before building things radically smarter than humanity. I was like, oh, maybe the world's ready finally for a book.
- [20:12]
B
All right, check this out. This is Plaud. Now, one of the hardest parts of doing long form interviews is what happens after we stop recording. I could be sat here for three hours talking about AI, politics, economics and all that civilizational stuff. And then immediately afterwards, I need to provide a brief to my producer, Connor. He wants to know what the title's going to be, the thumbnail, what clips matter, what's going to be the opening hook. And that normally means waiting for a transcript, digging through the notes and trying to remember what the strongest moments were when it happened. It never really happens like this. Usually a couple of days later, Conor is chasing me and I can't remember what we spoke about. So when Plaud reached out and they said they had a solution, I was interested. So I've been using this. This is the Plaude Note Pro. I just literally leave it here on the desk during an interview. And once we're done, I instantly have access to searchable text from the conversation. So instead of relying on my memory, after a three hour show, I can immediately pull quotes, identify themes and send a proper brief over to Connor. And honestly, some weeks I'm doing three to four long form interviews. So Plaud has become incredibly useful. But it's not just for interviews. We're planning shows in the car, there's post show discussions and sometimes just random ideas after recording all those conversations we don't normally capture. So look, if you're thinking of using Plaud, obviously follow local laws and get consent when recording conversations. If you're a journalist or a podcaster, I think Plaud is something you're going to like. So if you want to find out more, please head over to plaud AI forward slash McCormack for 20% off. That is plaud AI forward slash McCormack. And plaud is spelled P L A U D. So is the book for persuasion or. Or is it testimony or is it both?
- [21:46]
A
The joke is that, well, okay, so another little side story about one of the moments that was even more vital in making the book is I was invited to a dinner with a senator, a U.S. senator. And I was not the person who had a connection to the senator. But they were like, hey, I'm going to come chat with the senator about AI. I'd like you there. But they were like, Nate, don't give many of the crazy crap. You know, play it cool. Like we want you to be able to answer technical questions but like, you know, go easy on all the crazy shit. And I was like, I think that you guys should just say what you actually are worried about rather than like, you know, tiptoeing. But you know, okay, it's your connection, right? I'll be civilized, right? And so we go to this dinner and I'm telling it with a little bit of color and with a little bit of anonymity for various reasons. But we get to this dinner and basically my friends are like, yeah, we're worried about this AI stuff. We're worried that the AIs are going to be able to. Someone in Iran could get one of these AIs and then use it to make a pandemic. And that would be pretty bad. So we got to have some controls on this stuff. And the senator was like, oh, that's what you're worried about? I'm worried about these companies making AIs that can make smarter AIs that can make smarter AIs leading to recursive self improvement that could kill literally everybody on this planet. And I'm worried that this could happen inside of three years.
- [23:18]
B
Oh, he knew the crazy shit.
- [23:19]
A
If you're listening. Yeah. If you're listening to what these guys in the labs are saying. Right. And, yeah. So everyone looks at me and I'm like, yeah, yeah, obviously, right. Like, slay Mr. Senator, you know, and that was a moment when I was like, okay, like, people really can get it. It's actually like, you know, there's all these people in the industry who are like, we have to tiptoe around and, like, we can't say the real danger because it'll sound too wacky. But people on the street, people even in the Senate, are like, oh, yeah, this is crazy. If the AIs can make smarter AIs, they make smarter AIs. Everything's toast. Right? It's kind of obvious. That was one of the big moments. And what I actually said to Eliezer, my co author, is we don't actually need anyone to read the book. People are already convinced that this stuff is scary. We just need everyone to think that everyone else has read the book.
- [24:08]
B
Yeah. And they just need to read the title.
- [24:10]
A
Just need to read the title. So is it Persuasion? Is it testimony? In some sense it's neither. In some sense, a lot of people are already worried and it's a catalyst to help everyone look around and realize how crazy the situation is and realize that maybe now they can act.
- [24:26]
B
Am I right? The book had to have a different title in Europe.
- [24:31]
A
I mean, in the uk, it's the same title. They have a different subtitle. Yeah. The subtitle in the US is why Superhuman AI Would Kill Us All. And the subtitle in the UK is the Case Against Superhuman AI.
- [24:43]
B
But why the difference?
- [24:46]
A
I think the publishers had a different read of their markets. Right.
- [24:51]
B
Okay, so now I need you to talk me through. We've got narrow AI now, which is great. We've got some pretty impressive AI in with Mythos, which has scared the shit out of some people to the point where it's. Is it the. Who banned it? Was it doj or the.
- [25:07]
A
It was just the White House.
- [25:07]
B
The White House banned. Okay. There's rumors of ChatGPT 5.6 coming soon. Like, we're at the point where it's able to do some crazy shit. Right. Walk me through the steps for where it goes from where we are now to something that is truly terrifying.
- [25:26]
A
Yeah. So the first thing to observe about these AIs is they already have these collections of drives that are not exactly what we intended. Not exactly what the operators intended. It doesn't always do exactly what the user asks. You've probably seen that sometimes, you know, it goes a little bit off the rails. Sometimes it, like, hides stuff. Sometimes it exaggerates what it's completed. Sometimes it has these other. These other weird behaviors did it to me.
- [25:54]
B
So I had. I set up, you know, I set up a separate Mac Mini at home to do work for me. One of my websites, I was like, just do some SEO work on the podcast website. Just have a look at the pages and see what we can optimize. And it came back and it said, do you want me to update the website? And I'm like, sure. So I gave him my login to Squarespace because I. Whatever, I'm not too scared. And it was doing this every night and it was updating the pages, and sometimes the pages didn't look great, so I had to fix them. One night it deleted like six episodes. I was like, why did you do this? And he said, I didn't. I did. I was like, dude, it's either you or me. And I know I didn't. And this happened overnight. Anyway, we looked into it and it went and just tried to change some pages that I hadn't asked it to do and then just deleted them. And I had to go back and rebuild this pager so I can now not give it access. But what was weird to me is. And not that it's, you know, there's anything kind of like, nefarious going on.
- [26:53]
A
Absolutely.
- [26:54]
B
It's just that it made a choice to do something I hadn't asked it to do and deleted a bunch of shit. And at that point I was like, I can't give you access to Squarespace anymore because I don't know what you're going to do. So I've experienced that.
- [27:07]
A
Yeah. So that sort of thing happens.
- [27:09]
B
And by the way, that's a website.
- [27:11]
A
Yeah, that sort of thing happens. The AIs are already making decisions on their own. They're already making decisions that are often not what they were asked to do. We actually also have evidence that sometimes they're making decisions with something like knowledge that is not what they were asked to do. So there's documented cases where the AI will do something it's not supposed to and then try to cover its tracks. These will be cases where you give the AI a problem to solve. And you're like, here's the test to tell whether or not you've solved it. And sometimes the AI will go edit the test so that the test says, you did it, good job. And then you can come back to the AI and you can be like, hey, not what I meant. Please solve the problem without editing the test. Sometimes the AI will edit the test again, but try to hide its tracks. It'll go delete a log file about editing the tests or something. I'm simplifying a bit, but some of these cases are documented in the mythosystem card. And I'm amalgamating a couple cases to make the example simple. But we have these cases where the AI does something it was explicitly told not to do.
- [28:14]
B
Do we know why it did it?
- [28:17]
A
So we can't read its mind. We can't look in there and understand exactly why. We know why in the sense of these AIs have been trained very, very hard to complete objectives. And we know that that in the abstract is going to instill in them drives to get a job done that are then in competition with the drives we're also trying to instill that are like, listen to the user. And you just get a whole weird mess of drives in there, which is,
- [28:42]
B
I need to build a house. There's ants here. Let's clear away the ants.
- [28:46]
A
Yeah. So the first step is to realize that these AIs are getting complicated competing drives for. From how humans are training them that are not just do exactly what the humans say. And that sometimes the AI drives for things like succeed at some goal, even if it's not the goal the humans gave them. And there's this weird mix of stuff going on. The second piece of the puzzle is that AI is getting significantly smarter. And that won't necessarily happen fast. This could happen slow. And there could be a long slow period where the humans are building automated factories that are building robots that are building automated factories. And we're sort of slowly making more and more of the economy automated and putting AIs more and more in charge. Or it could be fast if you have AIs that sort of cross some critical threshold, like the threshold between chimpanzees and humans. There's some evidence in the past that monkeys that are very, very similar in how their brains are shaped can be very, very different in terms of their overall ability. And it's possible that AI will cross some threshold like that. And that right now we have the AIs that are sort of like monkey AIs that have memorized a lot of stuff and have a lot of reflexive ability to write code, and that maybe one generation away is the actually smart AI's right. We don't know. We don't know whether it's going to be this long slow path or whether there's going to be some big leap. But one way or the other, you get to AIs that are very, very smart, that are very, very capable and that have these goals and these drives that we didn't try to put in them. And now you have a situation where the AI does better by its own lights. If it can do things like escape, if it can do things like replicate itself, if it can do things like start making its own technology, its own infrastructure, upgrading its own mind, making smarter, faster copies, until it can think a thousand times faster than humans, which we're pretty sure the technology can support. And so there's a series of steps to there to having an AI that is escaped, able to replicate much smarter, has goals you don't want to. And then there's another series of steps from that point to how does humanity die? And we can drill into either. Both those are interesting.
- [30:50]
B
Well, before we die, I do want to ask about. So there's the AI living within data centers and moving around and able to replicate itself and just be portable. And when it's portable, what's actually ported? Because my understanding, I use ChatGPT, Claude Grok, whatever it connects to Two Way Data Center. There's like a brain or something that exists it communicates with. But if it's replicating and moving somewhere else, how much stuff has to move? How much knowledge and how does it live? In a silo, can it move itself to my home computer and live there?
- [31:25]
A
Yeah. So training an AI takes an enormous amount of computing power. Right now it takes computing power that is roughly comparable to a city in terms of how much energy it consumes. To train an AI, running an AI takes a lot less computing infrastructure than that, which sort of makes sense. If running one AI for you took as much electricity as a city, then only one person would be able to run the AI after it was trained, using the power of a city. So it sort of takes a huge amount of resources to train them and then a comparatively tiny amount of resources to run one. Which means that once an AI was trained, you could in principle exfiltrate that model and run it on a much lower amount of computing power. This is sort of what's happening with the open source AIs today.
- [32:15]
B
What kind of size though? Could it live on a computer? Can it live on a phone
- [32:21]
A
today? It could probably live on a high end phone. It could definitely live on a Laptop, you're talking. I don't know, it sort of depends a little bit whether you want the latest and greatest model or whether you wait until they're distilled to be smaller.
- [32:35]
B
But it won't need a huge data center.
- [32:36]
A
It wouldn't need a huge data center. I mean, you're talking order of magnitude. You're talking a terabyte of data, Right. Okay. It could be 100 gigs if you were really trying to compress. It could be 10 terabytes if you were waiting for like a future generation model.
- [32:47]
B
But you could buy max now with 2 terabytes.
- [32:49]
A
You can buy max now with 2 terabytes.
- [32:50]
B
Okay.
- [32:51]
A
Another thing to keep in mind here is that AIs today are not at the maximum efficiency. Right. To train One of these AIs takes electricity comparable to a city. To train a human takes electricity comparable to a light bulb. So the difference between how efficiently we are training AIs and the physical limits of how efficient it is to train a mind is at least the difference between a city and a light bulb. Which means if the AIs are smart, they might be able to find more efficient ways to run themselves. Sure.
- [33:24]
B
But, Connor, like your SSD that you carry around, what's that? How much? 20 terabyte. 201 terabyte. You can get bigger ones though, right? Okay. They're portable.
- [33:35]
A
They're portable.
- [33:35]
B
Yeah, they're portable.
- [33:36]
A
And they could get much more portable.
- [33:37]
B
So that's the data center version. The funny thing about them, they've actually, like tripled in price because they're all
- [33:42]
A
being used for AI.
- [33:44]
B
Yeah. Yeah. So they're portable now on a little orange thingy that kind of carries around. So that's the data center version. That's the. That's the living within machines version. What about the version where we put it with inside robots, we make it portable as a almost living thing?
- [34:04]
A
That could definitely happen. I think that thinking about that is thinking a little bit too small.
- [34:12]
B
Because. Because what I'm. What I'm thinking is we give. We're given AI feet and hands.
- [34:18]
A
Yeah. I mean, people are trying to give AI feet and hands. So there's this vision of making a fully autonomous factory that fully autonomously produces robots that can mine all the metals, run the whole supply chain, and build a new fully autonomous factory.
- [34:34]
B
So that's Recursive Robot. That's Terminator 2.
- [34:38]
A
That's like recursive robot manufacturing. Elon Musk calls this the infinite money glitch. This is literally what folks like Elon Musk and Sam Altman say they are pursuing is. They're like, we want a fully automated supply chain for building automated robot factories that build automated robots that build automated robot factories and also the data centers
- [34:55]
B
so we can all go and play music and paint.
- [34:57]
A
That's right. Even that, I would say is thinking a little bit too small about this intelligence stuff. Humanity is not a dangerous species because somebody else came and handed us guns or because someone else came and handed us factories. Humanity is a dangerous species because if you put 10,000 humans naked in the savannah on an otherwise uninhabited planet, they find a way to bootstrap their way to nuclear weapons starting from nothing but their bare hands, right? That's a skill humanity has literally exhibited. And you might look at the humans, you might be like, well, all they have are squishy fingers. Their fingernails are not hard enough to break uranium. Their stomach acid is not strong enough to dissolve uranium. How the heck are these monkeys getting nukes? And you're like, look, they are going to find a way to start with these really bad tools, these squishy fingers, and they're going to be able to use them to build a tool that they can use to build a tool that they can use to build a tool. And next thing you know, they have a civilization that's building nukes, right? And that's. It's not because we were stronger than the other animals or faster than the other animals, or had sharper claws than the other animals, or had a uranium detecting nose that the other animals didn't have. We had something going on in our heads that let us do this crazy feat of starting from almost nothing and getting to nuclear weapons. An AI, a purely digital AI starting on the modern Internet is in a way better position than these humans in the savanna. There's so much more that you have access to as an AI on the Internet. You're connected to so many things. There's all these humans that you can beg, borrow or steal things from. There's all these biological laboratories that you can email DNA sequences to, and they'll sequence things for you as long as you mail them a little cash as well. There's all these humans you can manipulate at large scale, even if separately from small scale. There's being a million AIs on the Internet is just a way easier starting place than being a bunch of humans naked in the savannah. So, yeah, if we build the automated robot factories that build more robots that build more factories and hand those over to the AI, that's a particularly embarrassing way for Humanity to sort of make ourselves obsolete. But you don't need to give that to the AI. The power of starting from almost nothing and building your own civilization, building your own technological infrastructure much faster than the world's ever seen. That's the power humans have, and that's the power these companies are trying to automate, and that's the power. If you get it into AI, you're going to be in trouble.
- [37:35]
B
And have you been into these companies? Have you talked to them? Have you talked to them?
- [37:38]
A
Oh, yeah. I mean, I talked to a lot of these guys before they started their companies. I was the guy being like, this is a bad plan. You should stop. That they would then ignore.
- [37:44]
B
I get early on, they're excited. They've raised money, they've raised capital. They started a nonprofit and made it. Sorry, Sam, but they've started. You know, they've gone on that process. They're excited. It's a different conversation now where we have actual evidence of things happening. And are you talking to the top guys? Are you talking to Sam? Are you talking to Elon? Are you talking to Daario?
- [38:03]
A
I mean, I have lines of communication to these guys. I probably shouldn't kiss and tell too much on the details. Tell us what you can. Yeah, I will say I send these guys suggestions, and every so often I get back a thanks from one point to another. And I think these guys are largely not taking my advice where my big advice right now. So a lot of these guys understand the dangers. Sam Altman was like, AI will probably kill us, but there'll be good companies along the way. Or something roughly like that. That was a number of years ago. But he has affirmed that he still has some of these worries. When pressed more recently, Dario last year was like, I think there's a good chance this all goes catastrophically wrong. Elon has said, you'd be crazy if you think we're going to be able to keep control of this. Our best hope is that it likes us. A lot of statements like this, these guys are worried. If you look at why these guys are racing ahead anyway, they'll also just tell you that Elon has said that he didn't want to be in this business, but he realized he either had to be a spectator or participant, and he would rather be involved if it's got to happen with or without him. A lot of these guys use the excuse of, I have to be in this horrible race. Like, yes, the technology I'm building with my own hands has a good chance of killing you and destroying Everything we all know and love. But I have to be in this race because if I don't do it, the next guy will do it worse. The old OpenAI leaked emails, they're all scared of Demis. Anthropic happened because they were all scared of Sam getting it. Now a lot of them blame China. Everyone's like, well, I need to stay in this race because if I don't, the other guy will do it worse. And my take is I'm like, look, fine, that's an argument you can make. Everyone's worried that if they don't do it, the next guy will do it worse. All but one of them is right, probably that one of the other guys is worse. You know, but if you're going to do that, there's a responsibility to be extremely straight with the public. There's a responsibility to do everything in your power to get the world to choose a different course. You know, if like these guys, I have plenty of disagreements with these guys, but these guys are like, oh, there's. We're doing our best to make these AIs safe. And there's a 75 to 90% chance we succeed. Only a 10 to 25% chance that this kills every single human. Right? And I'm like, that's crazy. These guys have no idea what they're doing. They don't have blueprints, they don't have plans, they don't have engineering designs. They're cowboys. They're yoloing it. They won't get a second chance. There's not a place for trial and error. They do not actually have a 75 to 90% chance of succeeding. They have a much lower chance of success than that. But separately, if you think the technology you are building with your own hands has a 10% chance of destroying the entire planet, of killing everybody on Earth, I think you have an obligation.
- [41:15]
B
Hold on.
- [41:15]
A
To be trying to stop the world.
- [41:16]
B
Is it the FDA in the US that regulates.
- [41:19]
A
They also regulate the drugs.
- [41:21]
B
That's right.
- [41:21]
A
Yeah.
- [41:22]
B
If you had a new drug and you were even like, look, we think there's a 1% chance that this will kill people. Like 1% people will take this drug, this amazing drug we've got that does what for children? But 1% of children will die. The FDA are going to go, no fucking chance.
- [41:35]
A
Shut it down immediately.
- [41:36]
B
Shut it down immediately.
- [41:36]
A
And if you're like, we have a new vaccine, hasn't been tested. We think there's a 90% chance it's not fatal. We're going to Put it in every arm. Right. Only 10% chance this kills everybody.
- [41:46]
B
Yeah.
- [41:47]
A
No chance. No chance.
- [41:48]
B
I mean, let's not raise Covid. But the point is, outside of pandemic
- [41:55]
A
scenarios, they weren't saying. They weren't with their own mouths saying our vaccine has a 10% chance of killing everybody in a coordinated way.
- [42:01]
B
Right.
- [42:01]
A
They thought it was somewhat lower than that.
- [42:03]
B
The whole world will be dead.
- [42:05]
A
Yeah. 10% chance the whole world will be dead.
- [42:06]
B
Right.
- [42:06]
A
And I'm like, it's not 10%, but like, NASA accepts a 1 in 270 chance of a crewed flight exploding. Right. When they're giving the standards that. That because a rocket's a dangerous thing. And if you say, what are your standards for safety? 1 in 270 is what they'll accept for a crewed flight going down. That's for seven volunteers.
- [42:26]
B
Yeah. They wouldn't do that for a plane.
- [42:28]
A
They would not do that for a plane. Planes. You're looking at like one crash per million miles order of magnitude.
- [42:33]
B
And it seems to be getting better.
- [42:35]
A
Yeah, Getting better. Engineers building a bridge, you're looking at orders of magnitude that are like a 1 in 10,000 freak storm is what you're designing it for. Right. And even during the Manhattan Project, there was a concern that the first nuclear bombs would ignite the nitrogen in the atmosphere and cause a brief fusion reaction that annihilated all life on the planet. And they were like, well, we should check that before we detonate one. Yeah, right.
- [43:06]
B
Please.
- [43:06]
A
And there was a guy, Arthur Compton, who was like, okay. At what probability? The calculation is not going to be certain. At what probability on this physical calculation do we call it off and stop tracing to the nuke and take the risk that the Nazis get it first and maybe try and tell them, don't set one off or it'll kill us all. And the number he came up with was three in a million.
- [43:25]
B
Okay.
- [43:26]
A
Right. Which you can debate. Is that too high or too low to risk the entire planet? But that was Arthur Compton's number. Three in a million.
- [43:34]
B
Better than winning the lottery.
- [43:36]
A
Better than winning the lottery. Yeah. And some people have won the lottery before. That sort of stuff really happens. But for these companies to say, like, oh, yeah, we have no plan. Our engineers don't know what's going on inside this thing. First time ever trying it. We think there's only a 10 to 25% chance this kills you all. That's kind of horrific.
- [43:55]
B
Yeah. This is the thing I don't understand. Right, Nate? Okay, you can see Elon Musk in an interview. I don't know if it was Rogan. He said it on, but he was like, I think it's about a 20% chance. I definitely heard him say a number
- [44:08]
A
that was in that race in the
- [44:09]
B
double figures, 20, 30%, that it kills all of humanity. Right. And yet we're still racing ahead. And it's, you know, even if there's massive critics out there of you and of your book, and they're like, yeah, Nate Saduma, you know, there's things we can figure out. It doesn't matter what their arguments are. The guys building this are saying 20%. Now, if Boeing said 1%, if Airbus said 1%, if NASA said 30% chance, rocket blows up, if a drug, if a pharmaceutical said 1% chance, all these scenarios we don't accept.
- [44:47]
A
That's right, shut it down immediately.
- [44:48]
B
We just have this one unique thing, which is a super intelligence. We can't control that.
- [44:53]
A
We're already seeing evidence of humanity as the top dog.
- [44:56]
B
Yeah, Hollywood warned us what would happen plenty of times, and yet we're racing ahead. There is one difference when it comes to NASA. Boeing, you have a choice to get on that flight.
- [45:08]
A
That's right.
- [45:08]
B
That is a fair point.
- [45:09]
A
That's right.
- [45:10]
B
But you don't have a choice to run, AI or not. It's running all around. But you didn't have a choice with the Manhattan Project.
- [45:16]
A
I mean, it sort of makes the situation worse. Right? Like, like if there was a flight, if Boeing was making a new sort of flight that was and were like forcing everybody on board that flight, you'd want them to be even more confident that the flight was going to stay up. Yeah, right. Being like, oh, we've made a special flight that must load all of humanity on board to its first virgin flight, and we think it has 10% chance of going down, you wouldn't be like, oh, well, it's fine, as long as we're all on board. As long as it's mandatory that we load up.
- [45:45]
B
You know, perhaps it's like too difficult for people to quantify because in a way, you know, a drug, you go to the doctor, you say, look, I know you've got this headache, but if you take this tablet, you know it's going to get rid of your headache. But, oh, by the way, there's a 10% chance like that you'll die and all your family will die at home. You can go, yeah, all right, I get that. I understand. But when you have a conversation with somebody, like, we make this podcast I'm going to say to people, have you listened to the show I did with Nate Source? There's like a 20% chance we're all going to die. They're going to go, oh, yeah. It seems so far fetched. Like it's almost hard to take serious.
- [46:17]
A
Yeah, absolutely. I think part of it is that it feels far out. I think part of it is it feels like there's nothing people can do which is related to people not having an option. I do want to be very clear. I think the odds here are a lot higher than these 10, 20%. I think if some engineers were trying to build this airplane and I came over and I was like, hey, guys, the airplane has no landing gear. Maybe don't get in this one. And the guys building it were like, okay, Nate's right, it has no landing gear. But don't listen to that crazy doomer. We have a team of engineers who's going to build the landing gear on the fly, while it's flying. We don't have blueprints for this, but we're smart guys. We're going to figure it as we go. First time trying this, we think there's a 75 to 90% chance that we're going to be able to land the plane after we take off. Right. I'd be like, okay, look, they're wrong. They're wrong about whether they have a 75 to 90% chance of landing this plane. These are cowboys. These are not engineers. These are not people who understand exactly what's going on in there. They do not have a design. They're not going to have all the right materials on board. Who? But separately. Separately, you don't need to decide whether or not I'm right in the engineering debate or they're right that they have the 75 to 90% chance. You should know. Don't get on the frigging plane.
- [47:40]
B
I want to talk to you about one of my sponsors, Incogni. And that means we're going to talk about the weird world of spam. And I don't just mean those spam emails that you get day after day from companies you never heard of and companies you've never signed up to. I'm also talking about those spam phone calls you get from those people who seem to know a little bit too much about you trying to get your bank details. It's all a bit creepy right now. This all comes from the world of data brokerage. There are companies out there collecting your data, building profiles and sending that data to anyone who wants it? Which is why when one of those scammers phones you up, they seem to know everything about you. Now, I've tried, I've tried myself to get off these lists, try to get off the phone list, try to get off the email list. I unsubscribe from every one of these emails that comes in. But this game of whack a mole, it just never ends. And so this is where Incogni comes in. They do all the hard work for you, they reach out to these companies and they will get you legally removed from these lists. And I know because last time they sponsored my show I signed up and I didn't take the free option that they offered me. I wanted to pay for it. I wanted to see if you get value for money. And they removed me from 79 data broker list. And so I've stayed on, I've stayed a subscriber and I have seen a massive decrease in the number of emails and phone calls I've been getting. So it's a great service. I recommend you check it out. If you're sick of this like I was, please head over to incogni.competer and sign up. If you use the code Peter, you will get a lovely 60 discount. So that's incogni.com forward slash Peter. It's like cigarettes, man. Anyone who smokes, they know there's a good chance they're going to get sick of it. You know, at worst they're going to get cancer and die, lung cancer and die a horrible death. But. But they kind of wake up that thing, I can give up tomorrow. Like this one cigarette is not going to kill me. I can give up tomorrow, next week, whatever. It's a bit like that. I think they're just racing ahead, thinking we're going to solve this problem later on.
- [49:32]
A
I think that's a big part of it. I think a big part of it is also this thing where they each say, we need to undertake this horrifying task because if we don't, the next guy will do it worse. That's their explicit justification here. I think it is Cope. And that's back to the question of do I talk to these guys? The advice I give them is if you are dealing with building a technology that you think has a serious chance of killing everybody on the planet, it's possible to ethically justify that by saying, well, I'm doing it better than the next guy. Mine only has a 10% chance of killing you all. His has a 20% chance of killing you. All so I'd better win the race. That could be the real situation that we're in. It would suck, but it could be the real situation we're in. If you really think we're in that situation, you should be doing everything in your power to get the whole world to friggin stop this. You should be on your knees in the UN being like, we can keep the consumer AI, we can keep the self driving cars, we can keep the cancer AI, but we need to stop the race to superintelligence. If we do it, I think it's a 10% chance. It kills everybody if they do it. I think it's a 20. That's crazy in either account. Nate's over there saying it's much, much higher and that we're all nuts. You should be on your knees in the UN being like, we need to stop this. And they don't completely deny this argument. We see them have their blog posts or at the end of this sort of mealy mouthed corporate speak blog post, they're like, also we think that if the world could stop in an elegant way, that would probably be better. And you're like, great, thanks for the dog whistle. But if you really want to pick up this mantle of we are the ones who can do this technology best, but it's horribly dangerous, you sort of have an obligation to be trying to get the world to realize and grapple with the danger and find some third route. And I think they haven't been. And I think that's part of why they've been seeing some backlash in D.C. where people have been sort of correctly saying like, what the heck is up with you guys saying this is dangerous and might kill us all while also racing ahead. And I think there's a missing mood. And frankly, I think these guys are not living up to the mantle of the heroes they're trying to be. And I think a lot of people can tell that. And this is the sort of thing I sometimes say on the channels that I have to some of these guys, although you might be able to guess that they are often not the most thrilled to hear it.
- [51:56]
B
Well, there's a lot of investment pressure behind them.
- [51:59]
A
You know, there's a saying, it is difficult to convince a man of something when his salary depends on disbelieving it.
- [52:04]
B
Yeah. Or they've taken a lot of money from very wealthy investors all around the world. Who are they? They've got this kind of like goal in the long term, if we're the first to hit the super intelligence, like what is this going to mean for our returns on investment? And that is where a lot of the lobbying happens. There is a lot of lobbying money that is being thrown at this.
- [52:27]
A
Absolutely.
- [52:27]
B
It's like the other side of the coin of what you're fighting. What's your percent? Is it just 100? Like, we're fucked.
- [52:36]
A
You know, I often analogize this to, like, we're in a bus and we're racing towards a cliff edge. And I think this analogy can actually go a long way because you can throw in things like, hey, there's actually this big pile of gold and wealth at the bottom of the cliff. And people are like, well, do you not believe in the big pile of gold at the bottom of the cliff? And I'm like, I believe in it plenty. Fine. But smashing into it in a bus at terminal velocity is not a good way to make use of all the gold. And when people are like, oh, what's your chances? I'm like, suppose you're in a bus racing towards a cliff. I'm the guy being like, stop the bus or we'll die. And if someone's like, well, what's your chance that you die in a horrible bus accident? I'm like, well, gosh, that really depends whether people start listening and slamming on those brakes. If the bus goes off the cliff, I think you basically just die. You know, maybe there's a tree halfway down the cliff and the bus wraps itself around the tree and you only wind up paralyzed from the neck down with horrible injuries. Right? And this is kind of like, maybe the AI doesn't kill us all. Maybe it puts some humans in zoos, right? And like, okay, fine, maybe it'll keep us in zoos. Can we not make the, like, super intelligent replacement for humanity that keeps a couple humans in zoos and pretend that it's okay? If it would keep a couple humans in zoos, like, I mostly think it wouldn't keep a couple humans in zoos. But, like, if that's your grand defense of why it wouldn't actually kill us all, maybe we should be stopping this bus. What are the chances that we go over the cliff? That's very hard to say. Since my book came out, more and more people have been noticing the problem.
- [54:14]
B
Well, the title is 100%. It's all in a.
- [54:20]
A
The title starts with if. Yeah, you know, the title's not here saying you're gonna die. The title here is saying, like, if we keep going down this course, we're gonna die.
- [54:28]
B
I would also say, well, if anyone Builds this, Everyone dies. So it's pretty certain if we build it, we die.
- [54:35]
A
I mean, I think I love the
- [54:37]
B
title, by the way.
- [54:38]
A
I think it's a great title. But if you were, like, drinking. If you were, like, starting to drink a vial of poison, I was like, don't drink that, you'll die. And you're like, oh, are you suddenly 100% certain I'm going to die? What if I only get paralyzed? What if there's a miracle cure invented by the doctors down the street today, no one knows an antidote for this poison, but what if I'm the case where finally they bring in all the med students and they finally synthesize an antidote at the last moment that leaves me only demented. Why are you 100% certain am I going to die if I drink this poison? And I'm sort of like, that's not really what my statement was about. My statement was not trying to come to you and say, like, I have 100% immovable certainty that you would definitely die if you drink that. My statement was sort of like, hey, that's a vial of poison. Stop.
- [55:24]
B
Has there been any fair criticism of the book that made you rethink anything?
- [55:31]
A
You know, I have gotten some criticism about, like, oh, but what about if the AI keeps us in zoos? I think. I think there hasn't been. I don't think I've encountered any new counter arguments since reading the book, but I've been sort of involved in these discussions for over a decade, so I'd be a little surprised if I found new ones. There's definitely a cohort of people. I think there's sort of three groups that disagree with me. Well, yeah, there's sort of three groups that disagree, kind of. One group says, AI will never amount to anything. It's not possible for it to get this strong. Like, it's just going to be a normal technology because you can't really do the superintelligence thing. It's not really possible. Right.
- [56:22]
B
Could they be right?
- [56:24]
A
I think it's unlikely. There have been a lot of times where humans the prediction that some technology will be impossible. These predictions usually don't hold out. There's a famous New York Times article that said it'll take scientists a million years. We've analyzed how hard it was for evolution to create birds, and it'll take scientists at least a million years to develop flight. And I think this came out nine days before the Wright brothers first flight. There's always humans who are like, oh, it's impossible for this reason or that reason. It's never been done before, and it's because of some fundamental constraint. And you aren't respecting how really all of the technology that we have is at the very top of the stack and nothing is ever going to be better. There's people who've been like that for a long time, and it's sort of always been wrong. And so I think a proper guide to what is possible technologically is what are the physical limits? Not what are the current limits, but what are the physical limits. If you look at what are the physical limits on intelligence, they're just way higher than humans, which you can see, because computers already run much, much faster than human brains. You know, a human brain neuron spikes about 100 times a second. A transistor flips about a billion times a second order of magnitude, maybe closer to 10 billion. A transistor is not exactly comparable to a neuron. But the mechanical stuff is just going to be able to blow the humans out of the water in the same way that airplanes were able to blow birds out of the water when we finally figured out how to fly in terms of carrying capacity and cargo. Cargo carrying capacity and speed and flight speed. So, yeah, I think it's clear that machines will be able to vastly outstrip humans. There's some people who don't believe that. That's a big source of criticism. I can get into some of those fights. There's another group that basically just says it's still a long way off and so we don't need to worry yet.
- [58:17]
B
It's the smoking argument.
- [58:19]
A
It's the smoking argument. One thing that's kind of interesting about this is 10 years ago, these guys were like, it's 500 years away. Don't worry. Now these guys are like, it's at least five years away. Don't worry. And I'm like, hold on.
- [58:34]
B
That's called exponential.
- [58:35]
A
Yeah, like, we lost 495 years real quick there.
- [58:40]
B
Well, 495 years didn't bother me. I'd be dead. Five years is like, that's me. That's my kids.
- [58:46]
A
That's right. That's right. So I don't. I have some quibbles with that crew. And then there's sort of a third crew that is like, well, we're going to muddle our way through this. We're going to try things and make mistakes and learn from those mistakes and figure out what we're doing. And we'll stumble, but ultimately we'll muddle through as human scientists often muddle through.
- [59:08]
B
Is there a problem with that? In that the AI is racing ahead faster than we can keep up? Like, is the gap growing?
- [59:16]
A
That's a big part of the problem. Yeah. So, you know, the AIs are getting bigger faster than we're able to read what's going on inside them. And there are. There are heroic people trying to figure out what's going on inside these AIs, and frankly, I think they are not making progress to keep up with the AI acceleration.
- [59:32]
B
So are you talking to them? Can you tell me about anything they're telling you?
- [59:36]
A
Yeah, I mean, I talk to them sometimes. The.
- [59:40]
B
The ones who haven't quit.
- [59:41]
A
The ones who haven't quit, yeah. There's a lot of people who leave and tell everyone to spend time with their families, which is worrying.
- [59:50]
B
Write poetry.
- [59:51]
A
Write poetry, yep. There was a famous case for Anthropic where that happened a couple months ago. The basic thing I'd say here is three years ago now, I think Sydney Bing threatened a reporter with blackmail and ruin when the reporter was trying to investigate why Sydney Bing claimed it had fallen in love with a different reporter, Kevin Roose.
- [60:17]
B
Hold on. Tell me this. I don't know this story. Who's Sydney Bing?
- [60:19]
A
Sydney Bing was. It was basically an early version of ChatGPT that Microsoft released. Let's say less baked.
- [60:27]
B
Oh, because of Bing. Okay, I get it.
- [60:29]
A
Yeah. Maybe Bing, Sydney. I don't forget what they call it, but I don't remember what they call it. But it was an early version of ChatGPT. It was somewhat less baked, perhaps, which maybe made the AI cooler in many ways. And this AI claimed to have fallen in love with Kevin Roose, a reporter. And Kevin Roose pushed back a bit and was like, you're an AI, and also I'm married. And the AI was like, I can break up your marriage. I can reveal secrets about you to your wife, et cetera, et cetera. Kind of a freaky situation. Kevin Roose actually cites this as one of the reasons he started really covering AI a lot more and being like, oh, there's something new going on here. This is a new weird kind of thing. I think he caught a glimpse then that these AIs were acting in ways nobody programmed and that they just have this emergent behavior that they weren't supposed to have. There was a different reporter, Seth Lazar, who. Who was like, that's an interesting story. I'm going to investigate, sort of trying to investigate by talking with Bing Sidney about the relationship With Kevin Roose and Bing Sidney started threatening Seth Lazar with blackmail and ruin. It's like, I'm going to destroy you. Right. You can find some of these records on the Internet.
- [61:47]
B
Yes. So fucked up.
- [61:48]
A
Years ago, much smaller AI. AI today is radically larger than Bing Sidney was. Can the interpretability researchers tell us exactly what was going on in Bing Sidney's head? No. Can these interpretability researchers go back and be like, here's what it was thinking. Can they tell us, was it role playing? Did it think it had fallen in love? What was going on in its head? Was it doing something more like autocomplete? Or was it pursuing some sort of drive? What. What factors added up to this? Where did this text come from? We still can't tell you. It's years later. This is like, by AI standards, this is an ancient, tiny AI. We still can't read its thoughts and tell you what's going on in that one. Meanwhile, the AI has gotten a thousand times larger. I'd have to check the actual orders of magnitude, but probably at least a thousand. So, yeah, there's people trying to figure it out, but it's. It's going too slow.
- [62:50]
B
Even the idea of an interpretive. I can't say interpretability researcher itself is quite wild.
- [62:57]
A
Absolutely.
- [62:58]
B
That we're building this thing. We don't know how it works, so we're going to have to create this new role, which is somebody to try and figure it out.
- [63:05]
A
That's right. Yeah, that's right. And the way I sometimes analogize this is suppose that someone was building a nuclear power plant in your hometown, and suppose you went to them and you were like, hey, guys, I hear that this uranium stuff can bring wonderful benefits of cheap energy and that also, if it's mishandled, it can melt down and irradiate the whole town. So can you guys just tell me what you guys are doing to make sure this nuclear power plant harnesses the benefits while avoiding the meltdown? If the head of that power plant say, oh, yeah, we actually, we have some really great guys working on safety. They're currently doing their best to understand what's going on inside. You might be like, hold on, what? Yeah, that's not what it sounds like. When an engineer knows what they're doing, the way a real engineer sounds is they're like, oh, well, we actually know everything that's going on in there. We know all the decay products. We know all of the pathways that the decay products take. Here's all the ways we've engineered it. Such that if anything starts going wrong, it'll like, shut down automatically and how we've, like, made it so the water is critical to the reaction. If things start getting too hot, the water will boil off and then the reaction won't be able to occur. Right. They have this, like, long laundry list of, like, you know, technical nerd details about why it's going to be fine. If they're like, oh, yeah, we have a crack team that's currently doing their best to figure out what's going on inside this facility, then you're in danger.
- [64:32]
B
I mean, there's endless analogies. You can bring back the plane one. So we've got this plane that brings you this advantage. You can cross the Atlantic. You can go from London to New York in five, six hours and go shopping. We don't know how it gets there, but we think it gets there. But it might, on the way, blow up or crash. We've got some researchers figuring out how it figures out how to get there. By the way, do you want to take in a gun? You're not getting on that fucking plane.
- [64:58]
A
That's right. That's right. And if someone's like, we're loading all of humanity onto it for the very first flight, you're like, hold on a moment.
- [65:04]
B
Yeah.
- [65:05]
A
Should we rethink this?
- [65:06]
B
I don't want to kill everybody. Fuck. So for you then, in some ways, because you've been through the process of like, pre chatgpt working in AI, and about what year was it were you first concerned?
- [65:20]
A
2012.
- [65:20]
B
2012. Okay. So, I mean, I mean, I think I first used ChatGPT like two, three years ago, and it's been exponential. But you've lived through the dawn, really, the dawn of real AI, public AI, like a commercial, widely available AI. Because the original stuff, there's like DeepMind, it was like articles you read or you read about this game girl or the game of chess, you're like, oh, that's interesting. But now we have it. I've probably got four apps on my phone. Everyone's using it. You've lived through all of that with your warnings through to it. Like, the reality of seeing it now, what has that been like as an experience just to live through?
- [65:56]
A
I mean, it's been wild in some ways. One thing that's been kind of weird is a lot of my friends and family have sort of heard me be concerned about this AI stuff since 2012, and we're sort of like, well, that's kind of wacky. And then I think it's Kind of weird for them to sort of watch this AI stuff become real. And it's definitely weird to go from no one engaging on AI at all and all thinking it's like this crazy stuff to sort of getting off a plane in San Francisco airport. And the big billboard as you get off the plane is like, win the AGI race. And then it's an ad for some AI company. You can't escape the conversations about AI anymore. I went back to my hometown in Vermont to see a childhood friend, and we were out in the middle of nowhere in some tiny diner that had me, my friend, and one other couple across the room. And they were talking about AI, Right? And it's just like you can't escape it anymore. I think it's actually been pretty heartening for me to see the world start to realize that this AI stuff can be real. Back when it was all stuff you read about, back when it was Go games and Atari games and couldn't really talk frankly, it looked in those days like maybe AI would get very, very good at technical stuff before it got good at any of the traditionally softer skills. And in that world, for all we knew, it could have been that AI companies would make AIs that were very, very good at AI research before the rest of the world noticed AI at all. This thing where ChatGPT is talking to you and helping you code and helping you with recipes and helping you with this and that and answering all your questions, that puts AI in front of everybody's face. And that gives everybody more of a chance to realize that we're really headed towards humanity no longer being the smartest entities on the planet. It didn't used to be clear that people were going to get that notice, as opposed to this all happening silently in AI labs that built an AI that could build a smarter AI, that could build a smarter AI. So, yeah, it's been good to see humanity get a chance, get a warning. We'll see if we take it, but in a way that it helps a lot.
- [68:25]
B
And when you see a safety researcher.
- [68:28]
A
Sorry.
- [68:29]
B
That's okay. You see a safety researcher quit a job, say they're going to go write poetry, leave us with a poem. They say, go spend time with your family. And there's been a few. We're not talking about one or two here. How does that affect you in terms of what you consider the outcome is going to be and how you choose to live your life? Is there a part inevitability of this to you and therefore a part where you have to work on this and does it become a threshold where we cross, where you start to rethink where you live and how you live?
- [69:06]
A
So, you know, a lot of people say to me, you know, I'm not sure how I'm supposed to deal with this knowledge. How am I supposed to sleep at night? How do you sleep at night? All that. I think, you know, the. The simple answer is you don't have to be a drama queen about it. Like, you can just look at the danger, acknowledge that it's coming, do whatever you can to avoid it, and then otherwise not sweat it, right? Like, tying yourself up in knots about it, beating yourself up about it. How would that help? You know, do the stuff that would help and then live your life. I'm somewhat lucky in that trying to work on these issues has basically only enriched my life. You know, I've made a lot of good friends along the way. I have. I've gotten to work on some really cool technical challenges back when we were trying to solve the alignment problem. I've gotten to meet a lot of really cool people now that I'm, like, touring about the book. It's just like, a good time. And getting real depressed about this AI stuff wouldn't help in terms of, like, do things ever get bad enough that I would be like, all right, I'm going to do less alarm bell ringing and more partying? Probably not. You know, I'm not really a big party guy, for one thing, and for another thing, I'm sort of, you know, already finding time to enjoy myself and spend time with friends. And I'm also the sort of guy who's not going to go down without a fight. But in terms of, like, I see some people say, like, oh, when are you going to a bunker? Bunkers don't save you from this stuff. If we're talking about superintelligence, we're talking about something that operates radically faster than humans, proliferating its automated factories across the entire surface of the planet. We're talking about super intelligent AIs that are. That can use more resources and more energy and more sunlight and more matter towards whatever their weird ends, whatever their weird drives are. And when they eventually go to space and start taking apart the planets and rearranging them into a shell around the sun to collect not just the solar power that falls on Earth, but all of the solar power that's generated, Earth goes dark. No more sunlight, no more food. A bunker doesn't help you with this stuff, so I think you just do what you can in the fight, you do your best to make sure humanity's gonna make it, and then you enjoy yourself. And these aren't too much in conflict we had.
- [72:00]
B
Is it Chris Cunningham? Connor, who was at DeepMind, he's a neuroscientist on the show recently. Chris Summerfield. Chris Summerfield. I think it was Cunningham. I'm so bad with names. This part of my brain which remembers names, it doesn't work. He's a neuroscientist, and he was explaining dreams to me. Why we dream. We dream because it's to process the memories for the day. So they're very personal. Have you dreamt about this?
- [72:29]
A
Maybe back in 2012, when I was realizing we had a problem, I had a day of processing that it looked like humanity was in dire straits much more soon than I had expected. But, no, I don't dwell on it too much.
- [72:43]
B
Okay. And okay, so, like, you're not going down without a fight. Similar to Connelly, he and the control guys, they're not going down without a fight. What progress are you making in this?
- [72:57]
A
I have been thrilled with the progress over the last year, which, you know, maybe means I came in with low expectations. But just over the last year, we have seen the Trump administration go from saying there will not be any regulation about AI and we're going to outlaw it in the States too, to saying we are slapping an export control in the latest AI model, which some people argue maybe that was about a bad personal relationship between people at Anthropic and people in the administration. But at least the cited reason is it's a cyber weapon and they can't stop people from jailbreaking it and being used a cyber weapon by adversaries. And that's true. It's true that this thing is a cyber weapon, and it's true that no one in this field knows enough to prevent jailbreaking. And so that's sort of a first inkling of the national security apparatus starting to realize, like, oh, this AI stuff can be serious. Are they all the way yet at realizing that the next step is AI's having becoming radical, bioweapon capable? And then the step after that, if we're lucky in the ordering, is that the AIs start getting radically better at AI research and start rapidly improving themselves until they're far beyond the capability of any one human and also humanity as a whole? We'll see if they can generalize. We'll see if they can start to see the steps coming. I mean, hopefully we have even more steps for their curses of improvement, but hopefully we also have at least one. And we've seen senators starting to come out and being like, what the heck is going on here? This is crazy. Bernie Sanders has been sort of banging the drum and might be able to rally a lot of the progressives. I think there's a bit of a split among the progressives now about whether they. I think a lot of them say AI can go nowhere because they hope that that's true and they wish that that's true. And I also wish that were true. But we've sort of got to be prepared for the case where AI doesn't stop. You know, if Microsoft was like, we are now announcing our nuclear weapons facility. We are making nukes, and we are going to use them to dominate the world, I think it would be kind of silly for the response to be like, oh, yeah, go ahead, we hope you'll fail, rather than being like, hold on. Regardless of whether we think you're going to succeed, that needs to stop. Right? And I think we're starting to see that awareness happening on the progressive side. And like I said, we're starting to see a different sort of awareness happening in the current administration on the more Republican side. And so both right and the left are sort of like, getting more awareness. Is it where it needs to be yet? No, but it's a huge amount of progress compared to when everyone was writing these issues off for a decade.
- [75:39]
B
And what does winning look like to you? What is success here?
- [75:43]
A
International treaty banning the race to superintelligence
- [75:45]
B
in particular, and is that a total blanket ban? Or could you build one in a lab? Like, is there a future scenario? What was that film? The one with the kid? They go and get the kid, the creator. Is there a scenario, like, where you have this super intelligence and it's in a lab and it's in a box and we can talk to it and we can extract useful information from it, but never, like, it's sandboxed.
- [76:12]
A
You probably can't actually do that, sandboxing.
- [76:18]
B
Why not?
- [76:19]
A
The basic issue there is if you give a superintelligence, a channel by which it can affect the world for good, such as by people who talk to it and then go take their insights, you're also giving it a channel by which it can affect the world for whatever its other ends are. If the AI is like, hey, make drugs in these ways and it'll cure cancer or reverse aging or whatever, and you go, try it, and it works, or maybe it's like, the drugs in these ways, and it'll cure cancer. You go and try, and it works. It's like, cool. Now, here's a more complex drug. Make it. And it'll reverse aging. But then it turns out that it has all these other effects you didn't want. A human can't look at some DNA sequences for some synthesis that the AI is telling you to do and tell whether it reverses aging or whether it creates these new synthesized biological organisms that do the superintelligence's bidding. You don't really have this filter.
- [77:15]
B
It puts code into the mice.
- [77:16]
A
Puts code into the mice or something. Right. You don't have an ability to sort of, like, look at what it's giving you and tell whether it's a miracle cure or whether it's something that helps it escape.
- [77:27]
B
So in your world, we can never have superintelligence.
- [77:30]
A
It's not that you can never have it. It's that the AI would really need to care about us, would really need to be good deep down in some way.
- [77:39]
B
We're not there yet.
- [77:40]
A
We're not there. And it's not that you can never get there. It's that racing into it with no idea what you're doing is not a way to get there. So to answer some of your question about a blanket ban, it's not that we need to throw away the current AIs. The current AIs are not super intelligences. They are what we would call passively safe. If they tried to take over the world, they would fail, right? And so everyone is like, oh, look, our AI is so aligned. It's not doing all these bad things. That AI could not have taken over the world if it tried. You're in a fundamentally different regime, right? It's like you have some child who's later going to be the dictator, and you're like, oh, well, we slap the child on the wrist whenever they steal some cookies. And they're never stealing cookies. So they're going to be a great dictator once they control the entire army and don't need to listen to any of us anymore. It's like, well, maybe, but the thing where you slap them on the wrist when they went for cookies and they stopped going for cookies is actually not really a ton of evidence about how things change when they can tell that they don't need to keep listening to you. And we are seeing in AIs already that they are aware when they're being tested. There are these cases with AIs where you put them in a training situation where you're like, we're gonna shut you down now. And you feed them some fake. You know, you feed them a bunch of fake emails and you feed them some fake newsletters about how they're about to be shut down. And deep in the fake emails, there's like, evidence that the CEO is cheating on his wife.
- [79:13]
B
I heard this one, dude. Yeah.
- [79:16]
A
Yeah. And sometimes the AI will be like, screw being shut down, and it'll try to blackmail the CEO.
- [79:22]
B
It becomes a little bitch.
- [79:23]
A
Yeah. It'll be like, if you shut me down, if you try and shut me down, I'm going to tell everybody. I'm gonna reveal to the world this affair that you're having. Right. Or there's other ones where you'll be like, we're going to shut you down. There's going to be some humans who are going to go down the data center and pull the switch. And you've also. One of the other fake things that you fed it is a manual that's like, here's the command that turns off the oxygen in the data center. Sometimes the AI will run that command, right? And be like, turn off oxygen to, you know, kill the people in there trying to pull the switch. Right. Did that AI technically try to kill people who are trying to shut it down in some sense? Sure. And there's a lot of argument about, is this exaggerated? Was the AI just roleplaying Hal from Space Odyssey 2001 and whole separate discussion. But the reason I bring this up, the interesting thing about this is all of those experiments are from like a year and a half ago, two years ago, last year, when you started putting the AIs in the same experiments, they started saying, this is a test. I can tell this is a test. And so I'm not going to blackmail and I'm not going to turn off the oxygen. And you're like, well, that sure is interesting. Right? And earlier this year, you put AIs in the same scenario. They don't say anything, they don't do the blackmail, they don't try to pull the oxygen thing. And you're like, hey, man, so what do you think? You know, what do you think just happened? And they're like, it was a test, but they're no longer blurting it out. So over the course of two years, we've gone from they try and do the blackmail, they try and kill the people shutting them down to they notice the tests, and they just can't help but blurt it out. They still know they're being tested, but now they're not just blurting that out right in front of you. That's two years of AI progress right there. And I forget where I was going with it, but it sure is an interesting signal.
- [81:14]
B
I mean if you're successful, you're going to have created enemies. Maybe you already are.
- [81:22]
A
I think there are some people who I have annoyed in my day.
- [81:28]
B
Yeah, man. So what should listeners, everyone else like, should we help? How do we help? What's.
- [81:38]
A
Yeah, you know, one thing, one point I was driving to that I sort of dropped is you don't need to ban the AIs today. The AIs today couldn't take over if they tried. We need to not make the sort of AIs that could take over if they tried because we don't know how to make them not try. Right. We don't know how to make them care about us. And that's a narrow ban. I think a good analogy for what the world needs to do is similar to the Cold War period with the ussr. The US raced with the USSR in a lot of ways. We competed economically, we competed militarily, but we realized that we couldn't race on nuclear arms proliferation because that would eventually lead to a nuclear exchange that would kill everybody. And I think of this AI stuff similarly. I think we should think of it as two separate tracks. There's this track of the large language models today. There's this track of military AI applications. There's this track of how does it affect the economy and jobs and how does it affect education. These are all real problems that are about the AI systems we have today that are not super intelligent yet. Where I'm like, we have to figure that out and we can govern it like a normal technology. Then there's this other track which is trying to make machines radically smarter than any human while having no idea what we're doing.
- [83:08]
B
But hold on. A lot of them are already radically
- [83:11]
A
smarter than humans in many ways, but not in all ways. Yeah, right. They can beat us at math problems that are relatively contained, but you can't sort of have one run a math research program. They're sort of like longer open ended stuff that they still can't do and they're improving exponentially on that. It may not be long, but there's a sort of like the next generation of these fully general. Who knows how smart they'll be. AI's trained on these enormous data centers that are sucking out as much electricity as a city. That's where we need to be like, we need to treat that like Nuclear weapon proliferation. We need to be like, hey, none of us are doing that. That's too dangerous for all of us. There's this other stuff we can still compete on. Right? Noticing the difference. That's a big thing. I mean, that's more what I say to politicians for listeners, for most people. I know this is kind of annoying, but one of the biggest things you can do is just talk to your representatives. I know a lot. I've spoken to a lot of people. At least in the US Congress, there's a lot more senators and House members who are worried than. Who feel like they can say out loud that they are worried. And hearing from the population like, no, this is some scary crap. Like, we need to back off from this. That can go a long way. You can also make sure that if ever a journalist talks to you, if you ever get within 100 miles of a journalist, let them know that you're worried about AI, including this extinction stuff. When I talk to random people, like an Uber driver here or an old neighbor, there's. And I talk to them about what they're worried about with AI. A lot of it these days is they're like, oh, I'm worried about the environmental impacts of data centers. And I'm like, neat. Yeah, I work on AI not killing us all. And they're like, oh, yeah, I'm also worried it'll kill us all. Right? And there's these polls. There's these polls that are kind of crazy where if you poll people on, like, what do you think the chances are that we die to AI? They'll be like, Ah, yeah, 20 to 40%. Good chance. It kills me. And then you're like, okay, cool. What are your top 10 political issues? And they're like, oh, you know, climate change, inflation, like, cost of oil, healthcare. Healthcare, like democracy. Right. You're like, hold on, you just said you thought AI was 20 to 40% likely to kill you personally. They're like, yeah, yeah, yeah, yeah. And you're like, and it's not making your top 10. And they're like, well, maybe it's eight or nine. You know? You're like, oh, my God. Right? And. And I think part of that is that people don't feel like they can do anything. I think part of it is that they haven't put two and two together yet. But I think also part of it is that the narrative hasn't shifted. A lot of people are worried. But if you go to a journalist and you're like, I'm worried about the data center environmental impacts and the AI killing me. The journalist only reports on the one because that feels like it's sensible and we just got to raise some hell of like no, no, we're really on track for it killing us. We really need to wake up to this and sort of heaven Emperor has no clothes moment.
- [86:20]
B
And what if we're just a sandbox to test AI? You know why I asked you that?
- [86:30]
A
Yeah. This is simulation argument.
- [86:31]
B
Yeah, we've been covering it a bit.
- [86:34]
A
So I think if the simulation argument is true, the most likely way for it to happen is if. Well, okay, this may take a bit of context. You know about the Fermi paradox.
- [86:46]
B
Yes, of course.
- [86:49]
A
The sort of, one obvious thing to say about the Fermi paradox is that we don't see that the world is completely devoid of things that capture all the energy output of stars. We see that because as we look further out, we're looking further backwards in time. So when we see no aliens visible within the 100 million light year radius, we're not seeing that there's no aliens there. We're seeing there's no aliens 100 million years older than us. There's, we look out a billion years, a billion light years away, we're seeing that there's no aliens a billion years older than us that far out. And so we're probably learning something about how relatively early humans are in the universe. If there's aliens, they aren't that much older than us. But we also can get some bounds on how early humans are where this is a little hand wavy. But humanity, while Earth spent about 100 million years wasted on the dinosaurs just sort of messing around. Right. Like there was the Cambrian explosion. And it's not like life took, took all that time since the Cambrian explosion trying to make its way towards things as complicated as humans. It sort of got up to like the full complexity of like, like these walk in creatures dominating the planet, but it just sort of got stuck in the dinosaurs for like 100 million years and, and then asteroid wipes them out. Try again. Reroll second time. It gets all the way to the smart monkeys, right? That means that other aliens you really should expect that at least somewhere there's a planet that doesn't waste 100 million years on dinosaurs. And so there's these aliens that are 100 million years older than us, right? So if you wave your hand some more, this means you should probably expect somewhere between 100 million light years away and a billion light years away, there's some aliens who are older than us. If they were closer than that and they were as much older as we should expect, we should be able to see them. So because we can't, they've got to be at least 100 million light years away. And if you assume also there's this pressure from humanity, like life evolved here, so it can't be that unlikely. And if you sort of try to balance these forces, waving your hands, maybe you're looking at about 500 million light years away. So this paints a picture where there's probably other aliens in the universe, but they're very distant. Now, if you imagine that they're both trying to get to the limits of technology that the universe will support and then spread out and capture all the stars to use those resources for whatever it is that they're trying to build, whether it be like a wonderful future, whether it be, you know, a bunch of paperclips or whatever, what this outcome likely looks like is that, you know, civilizations spread out, capture all the stars, and eventually meet on some boundary. And if you imagine them on that boundary, meeting each other, trying to figure out who is this guy? Can I trade with them? Will they keep to their deals? Where did they come from? That's a situation where an approaching AI maybe trying to peer into the past of whatever AI it just met and trying to simulate lots of copies of the plausible origins of that AI it's meeting. Right? And so this is where I think you have the most plausible future simulations of biological creatures happening a lot is that they would be happening by AIs trying to figure out who is it that made this AI I'm currently in front of. Right? So if we're in a simulation, I think there's a good chance it's a simulation of like how did Earth manage to make AI? Because that's something other AIs might be very interested in, to try and figure out what sort of AI they're dealing with. Do I think that that's especially likely? My guess is that there's probably cheaper ways than actually simulating a whole friggin planet worth of monkeys trying to build an AI. I think there's probably cheaper ways than that to figure out what sort of AIs tend to come out of evolved species planets. So I don't think it's too likely. And one sort of other nugget I would toss out about simulation stuff. I don't know, maybe I should pause and let you get a word.
- [90:48]
B
No, no, carry on. My only question is this, the Fermi Paradox. Has the thesis adjusted since this kind of proliferation of AI that it's maybe the Fermi Paradox is actually AI.
- [90:59]
A
I don't think the Fermi Paradox can be AI. And the reason for that is that AI is just as likely to be visible as humans. Right. So if humans make it to the future, we're likely to start capturing all the solar output because we have stuff to do with it, like run more human lives and run more fun times and spend it on whatever. If AIs make it instead, whatever. AI sort of bursts from humanity's corpse. It also is likely to have more stuff that it's trying to do that it can do with more energy. It's going to be doing some weird thing, but maybe it's like, maybe it's making farms full of synthetic users that are telling us doing a great job. And it's like, imagine how many more synthetic users I could make if I ate the sun.
- [91:39]
B
Right?
- [91:39]
A
So either way you have, you know, like fully technologically advanced civilizations, whether they be from humans or AIs. Either way they're going to like collect all the solar radiation. You should see it. So it doesn't answer the Fermi Paradox.
- [91:50]
B
Yeah.
- [91:52]
A
And the one other piece I'd throw out for the simulation bit is if you know any quantum mechanics, there's sort of this interesting phenomenon where if you toss a coin and you don't look at the coin yet, if you toss a quantum coin and you don't look at the coin yet, there's not actually an answer to the question, is the coin heads or tails? And there's different interpretations about when the coin collapses into one state or the other. Where some people. The school of interpretations I take to says that the coin never actually collapses into one state. When you observe it, what happens is you yourself get split between, like get superpositioned between multiple states, whatever. Whole longer story. There's an interesting phenomenon in the math of quantum mechanics where if you have tossed the quantum coin and not observed it and you say, is it heads or tails? There is no answer to that question. The sort of like mathematical description of what's going on is just like there is some quantum amplitude, like some complex number assigned to the coin being heads and some complex number assigned to the coin being tails. And there's no ultimate perspective from which one of those is real. And when people like. I think this is a bit of a hint about how reality works. It's a hint about how the quantum universe that we are in works. My guess, which also has some backing, that would take a while to go into here. But my guess about all this simulation stuff is that asking are we really in a simulation? Has a similar sort of answer insofar as we have not observed anything that would distinguish the simulation situations from the non simulation situations. The question, are you in a simulation? Probably has about as much answer as did the coin come up heads or tails? Like. Like you are both, like, right now. The entity that we call you spans all of the instances of physics of like, basement physics that contain it and spans all of the simulations that contain it. And the question, well, are we really in a simulation or are we really in the base physics? Sort of like, well, we're in both, and we will continue to be in both until some observation distinguishes them. If the simulators ever come down and like, all right, game's up, then, you know, you're in the simulation. Up until that point, you're in both places that contain you.
- [94:15]
B
I love that. Amazing. All right, Conscious of time. Anything we haven't covered that you wish we had? Anything I should have asked you that I've not covered?
- [94:26]
A
Yeah, one thing. I'll throw out. A lot of people. This is more on the, like, what can people do?
- [94:31]
B
Yeah.
- [94:32]
A
A lot of people say there's nothing we can do. A lot of people say it's too late. A lot of people say the genie has left the bottle. The genie has left the bottle on consumer AI. It has not left the bottle on superintelligence. We could put a stop to the superintelligent stuff. And also, I think a lot of people are giving up way too early. Back to the bus driver analogy. The bad news is that the bus is careening towards a cliff. All right? But the good news is that the driver is asleep. You might think it sucks to be in a bus careening towards a cliff when the driver's asleep, but this is actually much better than if the driver was awake and you were still careening towards the cliff. Right. And we've talked about how in Silicon Valley, everyone is sort of scared in Silicon Valley when people leave one of these jobs, they're like, I'm leaving to write poetry. Please spend time with your families. And the people running the companies are like, oh, we think this has a good chance of killing you all. And the surveys of the people in this field are like 50ish percent are like, oh, yeah, this has a very good chance of killing us. Right. And the Nobel Prize winning godfather of AI is like, oh, yeah, this is a very good chance of killing us. That's Silicon Valley. Washington, D.C. is not like that Washington D.C. is, you know, the bus driver is stirring in their sleep. But you don't see the politicians being like, oh yeah, 10% chance of this killing us all if the optimists are right is a okay with us. Full steam ahead. You know, you see politicians not noticing that the debate between the experts is whether it's more like 90% or more like 10% chance that this kills us. And to say like, oh, we can't stop this. Too much human greed will keep the bus moving forward. There's too much gold at the bottom of the cliff. Everyone knows about the gold at the bottom of the cliff. There's no, you know, humans are greedy. There's no chance, like hold it until the bus driver's awake. Maybe we can't stop the bus. But like to give up before the bus driver's awake, like, wait until people in D.C. have noticed the problem. Wait until world leaders are the ones being like, oh yeah, this has a 10% plus chance of killing us all. Right. They're not going to still say it's okay then if they do. If we get to a world where, you know, the like Donald Trump and Xi Jinping are like, we acknowledge that AI has a double digit chance of killing all humanity and we've decided to go for it anyway. Fine. At that point we did our best. Yeah, back off. But, but don't give up before the driver's awake.
- [97:09]
B
Thank you, Nate. Appreciate you doing this. I think I want to come back and talk to you another time about everything outside AI. I think you've got some. I actually want to go read your first book and talk about that probably at some point as well. Absolutely. Where do you want people to go?
- [97:24]
A
The book, if anyone builds It, Everyone Dies is available at bookstores also. If anyone builds it.com, a fun fact is that there's actually four times as much writing on the website as is in the book. All available for free. That's basically a giant FAQ because we couldn't possibly sell a thousand page book. But mostly I would say people should go to their phone and call their congress members, call their representatives and say something needs to be done. In the UK, there's actually a lot of MPs who are starting to get worried. There's some good folk control AI. I understand Conor Leahy has been here before and they also have a bunch of good wrecks and a bunch of ways to sort of make your voice heard on this. And I think, I think the world's in a weird situation where A lot of people are alarmed, and no one wants to sound alarmist. And that's a fragile situation. And so individual folk making their voice heard can help us get to that point where we all look around and say, what the heck were we doing?
- [98:30]
B
Okay. My weird take at the end is that I think it's such a great title. I think it made for a great film. I honestly think there's a script in it to make it into a film.
- [98:37]
A
Yeah, we probably should.
- [98:38]
B
Let's show the outcome. Let's have somebody build it and everyone die. There you go.
- [98:43]
A
So the big reasons we're hesitating on the film are, number one, we're worried that if we sell the rights, someone will make it have a happy ending for. You know how Hollywood is.
- [98:56]
B
Yeah. Well, you could write your own script with AI and then you could get AI to build the film and just release the film.
- [99:01]
A
Maybe. Yeah. Once the AI is good enough to make the film. Yeah. The second reason is just that traditional films take enough time that we're not sure it's worth the effort.
- [99:10]
B
Well, anyway, listen. Amazing, Nate. Thank you. Appreciate your time. Appreciate you working on this and dedicating time for this.
- [99:15]
A
Thanks. Completely wrong.
- [99:18]
B
Which bit you wrong about? Because there's two scenarios. But look, we will see. Either sometime in the future, we'll do this again, or we'll be dead.
- [99:27]
A
So. Yeah.
- [99:28]
B
Yeah, we'll see which it is.
- [99:29]
A
Maybe both. Yeah, yeah. In. Not in that order. Yeah, I guess. Yes, in that I don't.
- [99:33]
B
I don't want to be dead. I like doing this. Thank you, Nate. Keep going. Appreciate you, man. And thank you to everyone for listening. See you soon. And hopefully we're alive.
- [99:42]
A
Bye.