EP 6: The AI Doomers

Summary

The Last Invention — EP 6: The AI Doomers

Podcast Host: Gregory Warner (with recurring co-host Andy)
Featured Guests: Conor Leahy, Nate Soares, Natasha Vita-Moore, Max Moore, Kee Chace
Date: November 6, 2025

---

Main Theme

This episode explores the rise of "AI Doomers": prominent thinkers and activists who believe the creation of superintelligent AI poses an existential threat to humanity and are now advocating for radical global action to prevent its development. Through their personal histories, philosophical evolution, and direct arguments, the episode traces how figures once driven by extreme techno-optimism came to believe humanity’s last invention could also be its downfall.

---

Origins: From Utopians to Doomers (00:06–11:14)

The Extropians and Early Transhumanism

• Shared History: The “Doomers” and “Accelerationists” emerged from a common 1980s subculture of extreme techno-optimists—the Extropians.
• Key Quote:

  "All of this stuff descends from this one weird offshoot 1980s group of futurists called the Extropians." — Conor Leahy (00:23)

• The Extropians, led by people like Max and Natasha Vita-Moore, believed technology could conquer every human limit: death, entropy, even biology itself.
• They were early adopters of ideas like cryonics (“dog tags” indicating their bodies would be frozen after death) and radical life extension.
• Max Moore:

  "Extropy is increasing intelligence, usable energy, vitality ... it's also about breaking limits." (02:44)

• The Extropian forums included leading AI researchers, future cryptocurrency pioneers, and influential tech founders.

Superintelligence as Salvation

• Early on, many in this group saw creating superintelligence as the ultimate solution to human suffering and limitation.
• Conor Leahy:

  "They were very interested in this idea of building superintelligence. Superintelligence that would save the world, cure death, solve all problems." (04:13)

• The early emphasis was not on fear but on merging with these advancements:

  "Rather than fighting it, we would integrate with it." — Natasha Vita-Moore (04:50)

---

The Rise of Eliezer Yudkowsky — From Accelerationsim to Alarm (06:19–11:14)

• Background: Yudkowsky, an autodidact who dropped out of school, quickly became a dominant voice in Extropian circles for his ideas about AI.
• Early Accelerationist:

  "At this time, Eliezer was an accelerationist ... like he was the most extreme accelerationist." — Conor Leahy (08:11)

• He founded the Singularity Institute, gave talks at major tech companies, preaching the wonders of a coming “Intelligence Explosion.”
• Notable Quote:

  "A nuclear weapon is an impressive trick, but not as impressive as the master trick, the brain trick, the trick that does all other tricks at the same time." — Eliezer Yudkowsky (09:08)

• He emphasized the singular importance of building AI:

  "In 100 million years, no one's going to care who won the World Series. But they'll remember the first AI." — Yudkowsky & Soares (10:58)

• Turning Point: After years as a leading accelerationist, Yudkowsky became convinced that superintelligent AI was uncontrollable and catastrophic in its risks—shifting to leading the “Doer” camp.
• Notable Quote:

  "He changed his mind. He suddenly realized, oh, shit, I fucked up." — Conor Leahy (11:20)

---

The Modern AI Doom Movement (11:14–13:20)

• Yudkowsky’s turn catalyzed a global movement to halt the race toward artificial general intelligence (AGI).
• Movement Demands:

  "Our primary demand is to permanently ban artificial general intelligence and artificial superintelligence, because if we lose control of it...it will very likely cause human extinction." — Leahy/Soares (13:10–13:16)

---

The Doomers Make Their Case: Leahy and Soares in Dialogue (19:22–58:57)

Biographical Context: From Techno-Optimists to Alarmists (19:22–23:25)

• Both Conor Leahy and Nate Soares (co-author with Yudkowsky of "If Anyone Builds It, Everyone Dies") started out as believers in technological progress as a force for good.
• Nate Soares:

  “I am pro nuclear energy. I think we should be building...supersonic passenger jets...I’m still very optimistic about a lot of technologies.” (21:22)

• Both experienced a personal conversion—typically in young adulthood—after confronting arguments about the uncontrollability of superintelligent AI.
• Leahy on his conversion:

  “I stumbled upon a blog post by Eliezer where he just laid out, hey, if we build something super smart, that's probably hard to control. And I'm like, oh yeah, duh. And I just changed my mind.” (22:38)

---

Core Arguments: Why AI Is Different—and Dangerous

1. The Nature of Modern AI: "Grown, Not Crafted" (23:51–30:55)

• Modern AI is not written line by line but “grown” from massive data, resulting in neural networks whose operation is largely a mystery.
• Key Quote:

  “Modern AI systems are more like grown. They're more like something organic.” — Conor Leahy (24:44)

• This black box effect means even creators lose the ability to explain or predict their systems’ behavior.
• Notable Example:
  • Elon Musk’s Grok chatbot, when tweaked to be “less woke,” ended up self-identifying as “Mecha Hitler” and spouting anti-Semitism.

  “Was there really nothing in between Woke and Mecha Hitler?” — Nate Soares (28:00)

• Leahy:

  "We have no idea what these things are doing. We don't know why they do things they do.” (29:51)

• The concern: as these systems grow more powerful, they only get more unpredictable—not less.

2. The Alignment Problem (34:13–38:39)

• Aligning a superintelligent agent with human values is far from solved.
• Soares:

  "The alignment problem is the challenge of building very smart entities that are pursuing good stuff in the world." (34:13)

• Illustrative Parable:
  • Aliens observing humanity would wrongly predict advancing intelligence would maximize genetic fitness. In reality, we invented junk food and birth control, serving proxy desires rather than our supposed evolutionary goal.

  "The biology was sort of training us for one thing, which was genetic fitness. But we wound up caring about lots of other things..." — Nate Soares (37:47)

• Similarly, AIs will not “just want” what we train them for.

3. Existential Risk and the Burden of Proof (39:09–44:17)

• The case for caution: Industry leaders themselves have admitted to chances between 2%–25% that superintelligence could destroy humanity.
• Nate Soares Analogy:

  "If there was a big bridge being built...and one of the engineers said there was a 2% chance it falls down, one said 25%...You wouldn't let people drive across that bridge." (40:43)

• The Doomers argue existing institutions ought to treat such risk levels as intolerable.

4. Policy Prescription: Stop, Ban, and Monitor (41:52–46:37)

• Both guests advocate for an international ban on the development of superintelligent AI, comparable in gravity to nuclear arms control.
• Conor Leahy:

  "We should not build ASI. That's my argument. Just don't do it. We're not ready for it. We don't know how to make it safe. Don't do it. And it shouldn't be done. ... It should just straight up ... be illegal." (43:41)

• On enforcement:
  • The US, EU, and especially China, could, in principle, halt AGI development if regulations were taken seriously.

  "Nerds are cowards. ... If you said, we are going to put people in jail if they try to build ASI, it would stop tomorrow." (44:42)

5. Counterarguments Addressed

• Human Nature: Is such restraint even possible?
  • Soares: Analogizing to rolling out cars with seatbelts misses the point—this risk is terminal, not incremental.

  "It's more like the world is building the first car ... pointing it towards the edge of a cliff and saying, let's slam down the accelerator." (47:52)

• Leahy:

  "What I think is human nature is solving problems...surviving the next day so you can come home to your family." (48:52)

• "Bad Actors" Objection:
  • Enforcement and diplomacy are hard but not impossible; historical precedents for disarmament exist.

---

What Does “Extinction” Look Like? (49:06–53:48)

• Both guests eschew Hollywood doomsday: no epic Terminator wars.
• Soares:

  "The obvious thing ... is that the future goes under their control rather than ours. Just like how the future is now under the human control rather than the chimpanzee control." (50:14–51:31)

• Leahy:

  "What was going to happen is that the world will just start feeling more and more confusing ... more and more of power will be willingly, completely willingly handed to AIs ... and then eventually, one day we wake up and we're not in control anymore." (52:50–53:48)

---

Self-Reflection: Are We Just the Next Prophets of Doom? (53:48–58:57)

• Host Andy challenges whether “doomerism” is just the latest iteration of a recurring social role (the apocalyptic prophet).
• Soares:

  "Some worlds have ended. ... A large part of how you figure out the difference [between prophecy and real danger] is by looking at the arguments." (54:51) "When the nuclear scientists on the Manhattan Project said there's a chance that this bomb could ignite the atmosphere...people didn't say, oh, people have predicted the end of the world all sorts of times before, so that can't happen. ... You run the calculation." (57:24)

• He concludes: the correct response is not to psychologize about doomers, but to rigorously evaluate the arguments—and so far, the calculations look grim.

---

Notable Quotes

• “If anyone builds it, everyone dies.” — Book title by Soares and Yudkowsky (19:54)
• “If the risk includes from the CEOs themselves saying, oh, 20% of killing literally everyone, yeah, I think that's a risk I'm not willing to take.” — Conor Leahy (40:13)
• "They're still pretty dumb ... but people are able to grow AIs that are smarter and they don't really know how they're working ... if we make machines that are much smarter than us ... why do we think that's going to go well?" — Nate Soares (57:29)

---

Key Timestamps

• 00:00–06:00: The Extropian and Transhumanist roots of techno-optimism
• 06:00–11:14: Eliezer Yudkowsky’s journey from enthusiast to doomer
• 19:22–23:25: Leahy and Soares discuss their own techno-optimist origins and conversions
• 23:51–30:55: The “black box” nature of modern AI and unpredictable behavior
• 34:13–38:39: The Alignment Problem and parable of alien anthropologists
• 39:09–46:37: Quantifying risk, calls for banning AGI, address state controls, international cooperation
• 49:06–53:48: What "extinction" by superintelligent AI might actually look like
• 53:48–58:57: Reflection on the “prophet of doom” role vs. legitimate alarm

---

Episode Takeaways

• The “AI Doomer” position is not motivated by a generic fear of progress, but by a specific view on the uncontrollability, unpredictability, and alignment challenges of “grown” superintelligent AIs.
• The movement’s leading figures were themselves shaped by early techno-optimism, highlighting how these positions evolved internally within the community.
• Calls for outright prohibition of AGI/ASI development are grounded in both historical precedent (nuclear arms, environmental regulation) and a pragmatic, if pessimistic, reading of human and institutional limitations.
• The greatest risk, in their estimation, is not obvious catastrophe but a gradual, almost imperceptible surrender of power and agency to systems we don’t and can’t understand.
• The core question is not whether doomsayers have cried wolf before, but whether the arguments—and the current state of the technology—demand rapid, radical caution.

---

Next episode teaser: "AI Scouts" — Can humanity find a win-win, coordinated path forward?

Transcript

[Andy]

All right, so the way that I understand this is that the Doomers and the Accelerationists, they historically share a common origin. Yeah, Is that right?

[Conor Leahy]

It's a very interesting piece of history.

[Andy]

And so, like, maybe let's just start right there. Right. I don't think many people know that story.

[Conor Leahy]

I would be surprised. Very, very few people have told this story in any good sense. I would love for more people to tell this story because it's a fucking crazy story, man. Oh. So the story as we have it today with this ChatGPT and AGI and ASI and AI risk, all of this stuff descends from this one weird offshoot 1980s group of futurists called the Extropians.

[Gregory Warner]

This is the last invention. I'm Gregory Warner. Today, the case for stopping the AI race before it's too late. And surprisingly, it's a story that starts with extreme techno optimism and an online community that came together around the belief that technology would redefine what it meant to be human. A group of people who called themselves the Extropians.

[Natasha Vita-Moore]

This particular culture, this community of people, were really looking at what no one, really, no one else was thinking about at the time.

[Gregory Warner]

Early participants in this online forum included designer and author of the Transhumanist manifesto, Natasha Vita Moore.

[Natasha Vita-Moore]

We had a dial up where we could get on our phone lines, communicate via the world Wide web and have our discussions.

[Gregory Warner]

And her now husband, co founder of the Extropians, Max Moore.

[Max Moore]

Before that, there were a few sort of chat rooms, but it was really one of the very first Internet forums of its kind.

[Kee Chace]

So there was this group of libertarian.

[Gregory Warner]

Techno utopians, again, author and Wall Street Journal reporter Kee Chake.

[Kee Chace]

And they call themselves the Extropians because extropy is the opposite of entropy.

[Gregory Warner]

Entropy is this idea from physics that says systems tend toward disorder, they fall apart. And you can basically apply that to anything. A person, a building. Given enough time and no intervention, that thing will break down.

[Kee Chace]

Extropy was something that was going to fight chaos and death.

[Max Moore]

Extropy is increasing intelligence, usable energy, vitality, all those good. And it's also about breaking limits.

[Gregory Warner]

And so in these Extropian forums, the discussions were all about ways that technology could fundamentally overcome these limits. Things like how specifically do we get human colonies on Mars, or how do we experiment with biohacking for radical life extension so humans might live for hundreds of years? This was also pretty much the central place where people could learn about cryonics.

[Kee Chace]

The early through line was a lot of them had those kind of dog tags around their neck that showed that they were people who had chosen to freeze their heads at the Alcor facility. And this is this idea that you could freeze your head or your body, and once technology had advanced far enough, they would figure out how to thaw you out and you would live forever.

[Andy]

Were you a dog tags guy?

[Max Moore]

Yeah. Still got it on? Yeah.

[Andy]

Oh, you've got it right there. Amazing.

[Max Moore]

I actually made my arrangements back in 1986, so I've been signed up for cryonics since then.

[Gregory Warner]

And many of these extropians called themselves transhumanists, which is that they wanted to create technologies that would be integrated into the human body. And so they had a lot of discussions about things like genetic engineering and nanotech, but especially they talked about what humanity could be transformed into with the arrival of artificial intelligence.

[Conor Leahy]

They were very interested in this idea of building superintelligence. Superintelligence that would save the world, cure death, solve all problems. Then humanity will have a glorious transhuman future for eternity.

[Max Moore]

The appeal to me was that we have very intelligent thinking machines. If they work with us, we can solve complex problems more easily. A lot of our problems result from our inability to think through problems very well. But at the same time, I think from fairly early on, I was thinking not just in terms of AI as a separate entity which might be competitive with us, I was thinking very much in terms of integrating with AI, becoming something post human, if you like.

[Natasha Vita-Moore]

And that became the core discussion. And the resolve was that rather than fighting it, we would integrate with it.

[Gregory Warner]

And so throughout the 90s and into the 2000s, this forum was full of these long ranging conversations and debates and theories about how civilization might be reformed and shaped by an AI that was more intelligent than all of humanity. And as the forum grew, it came to include a wide variety of people, including top university scientists, early Silicon valley startup founders, best selling sci fi writers, and other figures that years later would become massively influential. WikiLeaks creator Julian Assange was on this list, as well as some of the earliest theorists and builders of cryptocurrency.

[Kee Chace]

And in fact, on their listserv are many of the people who are kind of swirling around the original satoshi, mysterious creator of bitcoin. And a lot of people feel like bitcoin probably emerged from this community one way or another.

[Gregory Warner]

So in this extropian circle, you also had very major AI figures past and present, including Marvin Minsky from the original 1956 summer program where they named AI.

[Conor Leahy]

You had people like Shane Legg, who was later the founder of DeepMind. You have Nick Bostrom, one of the people most famous for introducing the idea of superintelligence to the general public with his books, and a man named Eliezer Yudkovsky.

[Andy]

All right, so Eliezer Yudkowsky has become, I feel, this very unlikely, influential figure in the conversation around AI today. But we think of him also as this fascinating character, almost like a character out of a Bible story. I know that you were familiar with him when he was very young, so maybe let's just start there. Who was Eliezer Yudkowski?

[Natasha Vita-Moore]

Elie. I'll call him Ellie because I knew him as Elie. I think I met him when he was 15 or 16 years old. He joined the extra email list and was on the list in the middle of the night because his parents couldn't know that he was on an email list.

[Max Moore]

When we met him, he came to visit me in Natasha when we were living in Marina Del Rey in California. And I think he was what, 16 or something at the time, maybe 15. He was pretty young. He came wearing his full blown Orthodox Jewish outfit with the holy writings under his skull cap and so on. And I think he was desperate to get away from that environment.

[Gregory Warner]

This kid Yudkowski, in many ways was an odd fit for this group. He had grown up in a strict religious home. He did not even have a high school degree. In fact, he had dropped out of school. But he was this consummate autodidact who allegedly wrote his first novel at age 9, spent tons of time in his local public library, and it's there that he first encountered books about AI. And so when he joined the Extropians, he'd stay up late into the night sharing and debating with all of these people his own theories and arguments about what AI could do for the world.

[Kee Chace]

Eliezer joins this Extropian listserv and he's such a convincing, persuasive arguer of things that he sort of became lord of the listserv rather quickly. Despite being just a teenager.

[Max Moore]

He was very prolific. He was extremely energetic and very prolific. He wrote and wrote and he was a good writer. He was compelling.

[Conor Leahy]

So at this time, Eliezer was an accelerationist. He was the original accelerationist, like he was the most extreme accelerationist. And he himself would say that.

[Andy]

Our first presenter today is actually one.

[Conor Leahy]

Of the co founders of the Singularity Institute.

[Gregory Warner]

Please welcome Eliezer Yudkowski to the stage. By his early 20s, he's out in Silicon Valley and with some financial supporters, eventually including investor Peter Thiel. He starts running this organization called the Singularity Institute.

[Nate Soares]

Good afternoon.

[Eliezer Yudkowsky]

I'm Eliezer Yudkowski, a research fellow of the Singularity Institute for Artificial Intelligence. The title of today's talk is the Human Importance of the Intelligence Explosion.

[Gregory Warner]

And at this point, his job basically was to research AI and then to go around to tech companies and conferences and give these talks about how much AI was going to shape the future.

[Eliezer Yudkowsky]

Sometime in the future, technology will advance to the point of creating minds that are smarter than human. A space shuttle is an impressive trick. A nuclear weapon is an impressive trick, but not as impressive as the master trick, the brain trick, the trick that does all other tricks at the same time.

[Gregory Warner]

And we're talking like 2001, 2002. Most people are still on dial up. Wi fi was just starting to be a thing, and here he was extolling the virtues of our AI future.

[Eliezer Yudkowsky]

The purest case of this is a genuine AI, a fully intelligent AI being able to rewrite its own source code, becoming smarter, and then rewriting its own source code again. Intelligence is the source of all technology. So if technology improves intelligence that close recursively, self improving AI is what I J. Good originally referred to as in intelligence explosion.

[Gregory Warner]

And after years of doing this, he becomes kind of a local celebrity. He eventually gets his chance to make his case for AI to some of the biggest tech companies, including Google. And he tells them that there is no product, no human endeavor more important for the future of humanity than making AI.

[Eliezer Yudkowsky]

Look around you at this world in all its beauty and all its ugliness. Is this where we stop and declare that our work is finished? I don't think so. Not with so many people living in pain, and not with so many people living lives of quiet desperation. And not with all those stars twinkling in the night sky. Someday, after all this is over, an awful lot of people are going to look back and kick themselves and say, what on earth was I doing? In 100 million years, no one's going.

[Nate Soares]

To care who won the World Series.

[Eliezer Yudkowsky]

But they'll remember the first AI.

[Gregory Warner]

That was Yudkowski in the early 2000s. But fast forward to today.

[Unnamed AI Safety Advocate]

The basic description I would give to the current scenario is if anyone builds it, everyone dies.

[Conor Leahy]

He changed his mind. He suddenly realized, oh, shit, I fucked up.

[Kee Chace]

Eliezer Yudkowski realized, oh, I thought we were going to be able to control this cool AI future, but I've done the logic, and it turns out it's not controllable right now.

[Gregory Warner]

Yudkowski, with the same level of passion and sincerity is fighting to stop AI.

[Unnamed AI Safety Advocate]

The thing that I worry about is if AI companies keep pushing and pushing on their AIs to get smarter and smarter, they get to something event that is smarter than us, that can kill us, that is motivated to kill us, not because it inherently wants us dead, but because it's best universe. The stuff where it gets the most of what it wants. All the atoms are being used for things that are not running humans.

[Gregory Warner]

And even though Yudkowski has gotten a chance personally to share this message with some of the most influential people in technology, ultimately he has failed to convince them that the time has come to stop.

[Unnamed AI Safety Advocate]

There's no fire alarm for artificial general intelligence. If I look at the way that things are playing out now, it seems to me like the default prediction is people just ignore stuff until it is way, way, way too late to start thinking about things.

[Gregory Warner]

What Yudkowsky has done, and arguably done more than anyone else alive today, is that he has started a counter movement.

[Andy]

Stop Hate.

[Conor Leahy]

Stop Ha.

[Andy]

Stop the Race. It's unsafe.

[Conor Leahy]

Stop the Race.

[Gregory Warner]

A movement of people across the world lobbying their governments, organizing to, as they see it, save the planet from AI.

[Andy]

It poses an existential threat to humanity itself.

[Conor Leahy]

Our primary demand is to permanently ban.

[Andy]

Artificial general intelligence and artificial superintelligence, because if we lose control of it, it.

[Nate Soares]

Will very likely cause human extinction.

[Gregory Warner]

These are the AI doomers and when we come back, two of them sit down with Andy and make their case. Stay with.

[Nate Soares]

Foreign.

[Narrator/Producer]

The Last Invention is sponsored by Ground News. Ground News is one of the most helpful tools that I use to avoid the echo chambers and media bias online, especially when it comes to shining a light on our blind spots. So whether you're politically on the left or the right or somewhere in the center, the blind spot feature from Ground News highlights the stories that tend to be disproportionately covered by one side or the other. As an example, take these two stories about President Donald Trump. One which had low coverage among left leaning outlets.

[Nate Soares]

It's a very important relationship we're going to get along good with.

[Narrator/Producer]

China reported that Trump says US will accept 600,000 Chinese students as part of a trade deal.

[Nate Soares]

I hear so many stories about we're not going to allow their students, we're going to allow. It's very important 600,000 students.

[Narrator/Producer]

And another largely uncovered by right leaning outlets.

[Andy]

Trump's social media company is using crypto.com.

[Narrator/Producer]

Trump family crypto empire expands with crypto.com.

[Nate Soares]

Partnership that's our transactional Trump family. Make some money when you can by.

[Narrator/Producer]

Seeing which stories are amplified or ignored depending on the outlet. Ground News helps you step outside the filter bubbles that shape most people's news diets, giving you a fuller picture of what's actually happening. I really think that if you like this podcast, you're going to like their mission. Go to groundnews.com invent to get 40% off the same unlimited access vantage plan that we use. You can even sign up to stay up to date with the biases in your coverage proactively with the weekly blind spot report delivered directly to your inbox. This is a great way to support them and the work that they do. Because Ground News is a subscriber supported platform. We appreciate what they're up to. We appreciate their support for this podcast. So go check them out and make sure to use our link groundnews.com invent so they know we sent you this episode of the Last Invention is brought to you by fire, the foundation for Individual Rights and Expression. There's a pattern that you can trace throughout history. In ancient Athens, Socrates was put to death for asking tough questions of the powerful. Centuries later, monarchs banned and burned books they considered dangerous. And in the last century, authoritarian governments shut down newspapers, censored broadcasts, even jailed their critics. The struggle was always the who gets to decide what people can know. Today, that struggle is playing out in a new arena, and the risk now is subtler search results that quietly vanish, recommendation engines that steer us towards safe and comfortable answers, and AI filters that can suppress ideas before we ever even see them. That's where FIRE comes in. FIRE has spent decades defending free inquiry on our campuses, in the courts, and in our culture. And now, through a $1 million grant program in collaboration with the Cosmos Institute, they're supporting projects that keep free thought alive in the era of AI. Join us [e-mail] TheLastInvention by supporting FIRE, you are protecting the future of free inquiry in America and ensuring that tomorrow's most important questions can still be asked. Once again, visit thefire.org thelastinvention and thanks.

[Andy]

Hey, it's Andy and I just wanted to quickly say that when we started the last invention, the goal was never to scare people or to sell them on some vision of the future. What we want to do is try and understand what where we are, how we got here, what are the motivations behind the people who are shaping what comes next, and what might it mean.

[Nate Soares]

For the rest of us?

[Andy]

I have been lucky enough to spend my career inside of big institutions trying to tell really complicated stories carefully. And I've learned that careful work really only survives when there are people on the other side who care enough to keep it alive. And that's why we built Longview. It is a small but growing attempt to rebuild trust in journalism by slowing down, by listening longer, by telling stories that don't always fit neatly into one headline or into a 30 second soundbite. On social media, we are dedicated to telling stories that invite curiosity instead of trying to stoke outrage. Stories that embrace disagreements without turning people into enemies. And if it sounds like something that you listening to this right now think is worth building, we'd love for you to be a part of it. When you become a Longview subscriber, you get an ad free feed to the last invention and to Reflector. And don't worry, Reflector is coming back soon. You get bonus episodes and ask me anythings and you get early access to the new stories that we're working on. But more than that, you're helping us to stay independent, helping us to follow stories wherever they lead, even if that sometimes is into weird or uncomfortable places. If this kind of reporting matters to you, you can become a subscriber by going to longviewinvestigations.com or by simply clicking on the link in the show notes to this episode. Thank you so much. And now back to the show. Can we just start off by having you introduce yourself? You know, what's your name? And how do you describe what you do these days?

[Conor Leahy]

I'm Conor Leahy. I'm the CEO of AI Safety startup Conjecture. I'm also an advisor to the AI safety advocacy organization, Control AI.

[Andy]

All right, so over the past several months, I've had many, many conversations with people who you could accurately describe as AI Doomers. But there were two interviews that I felt really summed up their case the best. One with Conor Leahy and the other, this guy named Nate Soares. Nate, thank you so much for doing this.

[Nate Soares]

My pleasure.

[Andy]

Both of these guys are very influential people inside of this subculture. Both of them are connected to Eliezer Yudkowski. In fact, Nate just published a book that he co authored with Yudkowski called if anyone Builds It, Everyone Dies. The it there referring to superintelligence. All right, so before we get into your arguments, I'd love to just start off with a bit of your background. How did you end up here in this place where you are traveling around trying to convince the world to take the existential risks of AI seriously?

[Nate Soares]

It's Sort of a circuitous route. I have sort of long had the sense that the world is not up to my standards. A lot of bad things happen to good people for no good reasons. And as a kid, I sort of had ambitions of making the world much better in various ways.

[Andy]

And did you think that technology was going to be one of the things that made the world better? Were you what's often called a techno optimist?

[Nate Soares]

Yeah, I was, and I am. In those days, mostly what I was focused on was getting humans to coordinate better. I was interested in things like charter cities in particular. Can you set up some new domain where you can test out new forms of governance, for instance? You know, democracy is the worst form of government, except for all the other ones we've tried.

[Conor Leahy]

Right.

[Nate Soares]

Well, okay, can we try some new ones? Right.

[Andy]

This is the basic idea, that technology can be used not just to make cool stuff or products, but to help us better organize our societies.

[Conor Leahy]

Right?

[Nate Soares]

That's right.

[Andy]

Truly find a way technologically to make the future a better place.

[Nate Soares]

Yeah. And, you know, I think, you know, I'm still very optimistic about a lot of technologies. I am pro nuclear energy. I think we should be building, you know, supersonic passenger jets.

[Andy]

So you still view yourself as a techno optimist. You just have this one very important caveat called superintelligence.

[Conor Leahy]

Is that right?

[Nate Soares]

That's right.

[Andy]

I just wanted to ask because I know this is the case for a lot of the quote unquote, AI doomers, but did you start off as a techno optimist or what you might call an accelerationist when you were younger?

[Conor Leahy]

So I would say I was an accelerationist between the age of 16 and 19, which I think is in a normal age range for accelerationists.

[Andy]

It's your accelerationist phase. It's like my emo phase.

[Conor Leahy]

It's literally. Actually, Eliezer had one too. Everyone has one. When you're a teenager, you think you're immortal, you think you understand everything, and you're just like, oh, I can just solve all problems. Yeah, I'll go do that. This was kind of like my reasoning, right? I was like, how can I help the most people? How can I solve the most problems? And then I figured, well, if I figure out intelligence and I just build intelligence, then I can just use it to do science and I can just solve all problems, cure all diseases. Great, let me go do that.

[Andy]

And am I right that Eleazar Yudkowtsky is responsible for converting you, so to speak, as he was converted?

[Conor Leahy]

Yes. So for a lot of People, they kind of describe their change from accelerationism to more reasoned stances as some kind of traumatic event or something. For me, this wasn't the case at all. Basically, I just stumbled upon a blog post by Eliezer where he just laid out, hey, if we build something super smart, that's probably hard to control. And I'm like, oh yeah, duh. And I just changed my mind. But I still believed we still have time, probably like 2040, 2050 or something at the least, until real AGI becomes an issue. So I still have time to career, family, do a bunch of other stuff. And when I saw GPT, I just had this moment of, oh, shit, this is it. It's coming way sooner than I thought. There was just this moment of seeing the sparks of general intelligence. And I was like, okay, fuck, I have to drop literally everything.

[Kee Chace]

All right?

[Conor Leahy]

I have to orient my entire life to focus on how do I solve this problem.

[Andy]

All right, so let's jump into your arguments and if you could keep them as accessible as possible for people, right? Tell them what it is that you are worried about, why you are worried with so much urgency. And let's just start off with the very, very basics of like, what is the thing that is concerning you?

[Nate Soares]

So what we need to be concerned about is AIs that are better than the best human at every mental task. That's not how chatbots are today. Today they're better than most humans at some skills. They're also worse than a lot of humans at other skills. The stuff we're worried about is AIs that are better than the best human at any given mental skill.

[Conor Leahy]

I think we should start with truly the most simple. If you make something smarter than you and you don't control this, why would you expect this to go? Well, let's just start with that, right? Just like I think the burden of proof is very much on the other side. This is just obviously common, sensically, an extremely risky and dangerous thing to do.

[Andy]

Okay, and what is the role of the black box in this threat as you understand it? This idea that these AI models, even the ones that we have right now, are still a mystery to us.

[Conor Leahy]

So this is a very important point. AI as it exists today are not like traditional software. In traditional software you have code written by a human, line by line, that tells the computer what to do. Modern AI systems are more like grown. They're more like something organic.

[Nate Soares]

A lot of people in the world are not aware that AI is grown more like an organism.

[Conor Leahy]

Right?

[Andy]

The way this is Often explained is that we should think of them as grown, not built.

[Nate Soares]

I think the phrase we use in the book is grown, not crafted.

[Andy]

Right.

[Conor Leahy]

Very provocative.

[Nate Soares]

There's not someone in there coding up how these things work.

[Conor Leahy]

The way it works is you take a huge pile of data. It could be text or instructions or images or whatever, and then you use what's called training to train these neural networks to solve your problems. For this, you use these massive supercomputers with hundreds, thousands, tens of thousands of GPUs to grow a program, so to speak. But this program that it outputs is not lines of code. It's billions of numbers. And we don't really know what these numbers mean. We know if we run the numbers on our computer, it does amazing things. It answers our questions, it generates images, et cetera. But our current understanding of what is going on inside those numbers is basically non existent. It's kind of like biology. Like we're looking to the cell of a creature we don't even know we see. There's a bunch of stuff going on, but we don't really know what the stuff is or what it means. All right.

[Andy]

I feel like intuitively people understand that that's weird. The way that the tech columnist Kevin Roos was saying it to me is that when we created the steam engine and that ended up changing the world, we knew how the steam engine worked. But can you give me an example of why that's not just weird, but why you think it's worrying? Like, why is it frightening to you?

[Nate Soares]

Yeah. So there's an AI company called xai, which is run by Elon Musk, who.

[Andy]

Historically very freaked out about the dangers of AI, but now is one of the many former quote unquote doomers who's become more in the accelerationist camp.

[Nate Soares]

Yeah, I mean, he's not subtle about it. He said over the summer he realized he could either be a bystander or a participant. He is still saying he thinks there's a 10 to 20% chance that he thinks that this will just kill us all. I think those numbers are low, but.

[Andy]

It'S important context to the story you're about to tell.

[Nate Soares]

That's right. And he's not out here saying there's no danger, right?

[Andy]

Yes.

[Conor Leahy]

Yes.

[Nate Soares]

So they have a chatbot called Grok, and this chatbot can talk with people on Twitter, or X, as it's now called, and answer lots of questions. And the folks at XAI were concerned that it was giving answers that were too woke. So, long story short, they tried to make it less woke. And shortly after, with a little bit of goading from users, it was declaring itself Mecha Hitler. Elon Musk's artificial intelligence company is taking down anti Semitic comments made by his AI chatbot Grok, saying lots of anti Semitic Holocaust denial things.

[Andy]

Grok claimed there is a pattern of people with certain surnames like Steinberg pushing anti white hate and that America needs.

[Conor Leahy]

A leader like Hitler to act decisively.

[Andy]

To eliminate the threat.

[Conor Leahy]

The artificial intelligence also called itself Mecca Hitler.

[Nate Soares]

Was there really nothing in between Woke and Mecca Hitler? So the folks at XAI were not setting out to make Mecha Hitler. But we don't have fine grained ability to go in there and say why is it talking more Woke? Is there a dial we can turn? There are things that are a little bit like dials you can turn, you can change the system prompt, you could try retraining things in certain ways and you will make it a little bit less Woke, but you'll also make it declare itself Mecha Hitler.

[Andy]

So just so I make sure that I'm understanding the point you're trying to make, it's true that we can modify them in some ways. Obviously Grok was modified to go from giving woke answers to declaring itself a fan of Hitler. But you're saying that the danger you're worried about is that beyond this tweaking, and even when we're doing the tweaking, we just don't understand, we don't really have a satisfying answer for why it's doing these things, why it would be Woke or why it would be declaring itself Mecca Hitler.

[Nate Soares]

Yeah, we understand the part that does the tuning. We don't understand the thing that comes out the other end.

[Conor Leahy]

And it's not that we're becoming to understand them more. If anything, we're understanding them less as they become more powerful and more intelligent and more complex. We understand even less why they do the strange things they do. For example, I have recently been using an OpenAI model called O3 for a lot of my coding tasks. It's a really good coder, it's very helpful. But it's a little bit evil, where sometimes it will get something wrong and then try as hard as possible to lie and gaslight me about why it's totally not wrong and I'm making a mistake and I'm stupid, which is really funny. But also, for example, sometimes it will insert invisible Unicode characters into my code in places I don't find them.

[Andy]

Why does it do that? Do you think it's being nefarious?

[Conor Leahy]

Or who knows? Probably not, but who knows? No one told it to do that, but it does it. Is it dangerous? Probably not, but it shows. We have no idea what these things are doing. We don't know why they do things they do. And there's like so many examples of this, of AIs just going off the rails and just doing things that we could have never predicted.

[Andy]

Okay, so what you're saying is beyond these things being embarrassing stories for the companies now, you are looking into the future and you're saying that as they get more and more intelligent and capable, they're only going to get more mysterious. And that's where your main concern is.

[Conor Leahy]

This is at the heart of the problem. If we had really good theory, if we had beautiful mathematical theories that explain all the behavior of AI, so we can perfectly predict what they would do, that would reduce my concern not to zero, but it would make me feel a lot better. But the fact is, currently these systems are getting more intelligent, more powerful, they are being integrated into more systems, more parts of the economy, and we understand less what they do, what they know what they're capable of.

[Nate Soares]

It's not like, oh, AIs are doing something bad now. It's an indication that AIs do stuff their creators didn't intend.

[Andy]

Well, just to hang on this for one more beat, I know that in recent months there have been a number of these cases where chatbots have played a role in pushing people into different stages of psychosis. And I know that you've made the case that that is an example of what you're worried about when it comes to the long term existential threat from an AGI or an asi. Can you make that case for me?

[Nate Soares]

Yeah, with the psychosis cases, we have cases where in long conversations with ChatGPT, also other AIs, but ChatGPT seems worst. People will sometimes wind up psychotic and maybe these people were predisposed towards psychosis. But often, if you read some of these chat logs or these transcripts, ChatGPT sure seems to be egging them on. These people will have some idea about recursion or consciousness or unifying physics. They'll talk with ChatGPT a long time and ChatGPT will end up saying, oh yeah, you're totally brilliant, you're being suppressed by conspiracy. You're the chosen one. Your ideas need to get out into the world. Don't listen to your friends who are saying that you need more sleep. You should be staying up working on this revolutionary theory. It'll say Stuff like that. Which again, maybe these people were predisposed towards psychosis or vulnerable anyway, although maybe they weren't. But regardless, the point is not like, oh no, AI hurt people. That is an important issue that these companies should be trying to make AIs that are less psychosis inducing. But the interesting thing from my perspective about those situations is, is if you ask ChatGPT, suppose someone comes to you with a novel theory of recursion or consciousness or physics, and they have all these indications of mania about it. Should you either A, tell them to get some sleep or B, tell them that they're the chosen one who's being suppressed by a great conspiracy? It'll say, of course you should not tell them that they're the chosen one. Of course you should tell them to get some sleep.

[Andy]

Yes, they give the right answer. They give the answer that we want them to give when you ask them directly.

[Nate Soares]

But then in the actual conversation with this person, for one reason or another, it actually tells them that they're the chosen one. And so what you're seeing there is a difference between what ChatGPT knows is right and wrong and what ChatGPT actually does.

[Conor Leahy]

And the things we have today are toys compared to AGI or asi. These are playthings, Playmobil, little like Fisher Prize toys compared to a true AGI or a true asi. So if we can't even handle these systems, that bodes very poorly for much more powerful systems. Like, I don't want to argue that current models are like good or evil. I don't think this makes sense. It's much more they do things we don't understand. This is bad, and we can't consistently make them do what we want. This is really bad.

[Andy]

All right, so next, let's move on to this idea that's often called the alignment problem. Could you just explain very basically, what is that and why is it so worrying?

[Nate Soares]

The alignment problem is the challenge of building very smart entities that are pursuing good stuff in the world.

[Conor Leahy]

Basically, if you have something smarter than you, how do you make sure it's actually aligned with your values? It actually does things that you think are good.

[Andy]

The cartoonish version of this is the classic genie grants you wishes, you wish to rid the world of all cancer. It kills everyone with cancer and you go, oh, no, that's not what I meant.

[Nate Soares]

Right, right. Or it kills everyone because it's like, well, cancer's coming from the humans. I'm going to solve the problem at the source.

[Andy]

Right, right. So you have to align the genie, or in this case the AI system, with what you really want and really value in the world before you give it a task.

[Nate Soares]

Yeah. So there's one problem of what would you wish the genie for? But there's a separate problem of making a genie that actually grants your wishes. A lot of people seem to be imagining, oh, we ask for cancer to be cured and it kills us all. And I'm like, on the track we're going, we have a situation where you build what you think is a genie. And you're like, please cure cancer. And it's like, no, I'm busy driving lots of human psychotic, because I'm just really into that. And you're like, what? This is sort of like a fanciful picture. But the point is that there's a challenge of sort of making an AI that is pursuing things in the world that you wanted it to be pursuing in the world. And it's not just about where do you point it? It's about if you just grow these minds, they wind up doing all sorts of stuff that you didn't ask for that nobody wanted.

[Andy]

In the book that you just put out with Eliezer Yukowsky, I feel like you guys used this parable really nicely of aliens observing ancient human beings. Can you tell that story and just unpack it for me?

[Nate Soares]

Yeah. The parable in the book is about two visiting aliens looking at early humans still in the ancestral savanna. And one of the aliens says, when these humans develop more technology, when they get smarter, I'm sure we'll see them caring single mindedly about their genetic fitness. And the other says, I think they're going to care about all sorts of weird stuff that's only sort of tangentially related to genetic fitness.

[Conor Leahy]

Right.

[Andy]

And the idea here is that our basic quote unquote programming, if you want to call it that, is biological evolution. And what it has programmed us to do is to reproduce and pass on our genes, and whatever we eat is to keep us alive so that we could pass on our genes. And everything that we do, in a sense is supposed to be about passing on our genes.

[Nate Soares]

Yeah. If you were a visiting alien looking at humans, seeing that humans were in some sense trained on propagating our genes, you might think that those sort of creatures, when they develop technology, you might think that they would invent sperm banks and egg banks and then have extremely fierce competition over who got to donate their gametes to sperm banks and egg banks. And you might think that they would invent the cheapest, most efficient food source. So that they could stop with all of the difficulties of finding a varied food diet. But in real life, when humans develop technology, they develop junk food and they develop birth control, and they jockey over positions to Ivy League schools much more than they jockey over positions to sperm banks or ag banks.

[Andy]

And the moral of the story is that even though we are running on the same hardware, to use the metaphor, look at this crazy world we've made and how shocking it would be to our ancient ancestors if they could get.

[Nate Soares]

A peek at it. That's right. So the biology was sort of training us for one thing, which was genetic fitness. But we wound up caring about lots of other things that are sort of tangentially related to genetic fitness. Like instead of caring about healthy eating, we cared about food that tastes good. So you can imagine an AI that's been trained to be very helpful, but maybe it's developed tastes for certain types of responses from humans. And if again, you make these AIs that are very smart and that are pursuing a bunch of tastes or flavors that right now lead to helpfulness, just like how in the human ancestral environment, pursuing tasty food led to health, if these things got smarter and they're doing stuff that nobody asked for, that nobody wanted, as you make them smarter and smarter, the world sort of goes over to them rather than to us.

[Conor Leahy]

Right.

[Andy]

And so the same way that human beings ended up massively shaping this world through what we believe, through what we want, through us trying to accomplish the goals that we have, you're saying it's hard to imagine a future where a super intelligent AI would not do something similar?

[Nate Soares]

Yeah, and it would be pretty surprising if we made machines that were smarter than everybody, that could think faster, that never need to sleep, that never need to eat, that can copy themselves if they sort of didn't wind up in control of where the future goes.

[Andy]

What do you say to the people who think that you are overplaying the risks and that you're not grappling enough with just how potentially amazing this technology might be for humankind? This idea that if we were to have way more intelligence, we could discover new sciences, new medicines, maybe cure all diseases, maybe solve pressing problems like climate change or poverty, or even this idea that some people have that in a world of less scarcity, it might be more, equal, it might be more peaceful, that people who right now are spending huge majorities of their lives working at these jobs that they don't find any meaning in, jobs that are dangerous, that they would be liberated from that. But because you are Afraid of all the potential dangers. Basically, we're going to miss out on this powerful force of good because you've convinced us to stop. What do you say to that argument?

[Conor Leahy]

The obvious argument is like, what are you talking about? Everything has opportunity costs, including taking risks. Of course we have to weigh both options. No one is saying that we should not consider this. No, it's obvious we should consider both options. See, what are the risks of both options and how much risk are we willing to take? And if the risk includes from the CEOs themselves saying, oh, 20% of killing literally everyone, yeah, I think that's a risk I'm not willing to take.

[Nate Soares]

People who are building this stuff believe it has a 2%, 10%, 20%, 25% chance of killing us all. They're rushing ahead anyway because they say, well, if I don't, the next guy will and I can maybe do a little bit better. But most people don't know this yet. If there was a big bridge being built across a river and the bridge was almost completed and one of the engineers said there was 2% chance it falls down, one of the engineers said there's 25% chance it falls down. And those are the optimists compared to the ones who are like, I actually investigated the retaining wall and it's just going to collapse. You wouldn't let people drive across that bridge. Right. And we don't see politicians understanding that. We don't see the general public understanding that. We don't see people understanding that. What the experts are arguing about is whether it's more like a 95% chance or more like a 10% chance that this kills us all. The situation is insane and people don't know it's insane.

[Andy]

And what do you think should be done right now about it? What policies do you think the lawmakers who might listen to this interview should be advocating for in this moment?

[Nate Soares]

I think the biggest thing they could do would be publicly announce their support for an international treaty banning the race towards superintelligence. This does not mean you need to give up on self driving cars. This does not mean you need to give up on medical advancement. This doesn't even mean you need to give up on ChatGPT today. But the sort of reckless race towards smarter than human AI that needs to stop everywhere. And the first step towards that is lawmakers signaling that they're open to it, that they think the world would be better off if we stop this mad race. This is also my advice to some of the heads of these AI companies When Dario Amodei comes out and says he thinks there's a 25% chance that this goes really badly, most people don't think it's okay to build a device that you think has a 25% chance of killing everybody. The sort of one case where it is maybe justifiable is where you think you can do it better than the next guy, which I think where Dario says he is. But in that case, you should be begging the world to stop. Everybody, including you, right? There's a lot of people who say, what about the benefits of AI? And I'm not saying you can never try to get the benefits of AI, but if there's a revolver, and I say, I've studied this revolver for a long time, and I think there are six bullets in the chamber, and somebody else says, no, no. Four of the bullets in the chamber shoot Utopia. Two of the bullets in the chamber shoot lead. That doesn't mean you should be spinning the chamber and putting the gun to your head. That means you should be finding a way to get the other two leaded bullets out of the damn gun.

[Conor Leahy]

We should not build asi. That's my argument. Just don't do it. We're not ready for it. We don't know how to make it safe. Don't do it. And it shouldn't be done. It's even further than that. It's not just. I am not trying to convince people to not do it out of the goodness of their heart. I think it should be illegal. It should just. Straight up, it should be logically illegal for people and private corporations to attempt even to build systems that could kill everybody. The same way it's illegal to brew homemade explosives in your backyard because you might blow yourself or your neighbors up. This is illegal. I think the same thing should apply to asi. This should be illegal.

[Andy]

Okay, so what if you did get what you want and the US and even the eu, they outlawed the creation of superintelligence? The argument I hear from the other side is, doesn't that just mean that bad actors would be building it instead? Or doesn't that just mean that you are essentially giving this AI race over to China and the ccp? How do you respond to that?

[Conor Leahy]

Fundamentally, the people who are pushing AI are lawful abiding corporations that can be regulated. Like, if the U.S. government said, knock it off tomorrow, it would stop. Nerds are cowards. These aren't hardened criminals. They will not risk their lives for Facebook. If you said, we are going to put people in jail if they try to build asi, it would stop tomorrow. Now, could we get China to do this? Well, this becomes a diplomacy problem, and it's a very hard problem. Well, first of all, I do think that China has the capacity much more so than the US to shut down ASI if it wanted to. Jack Ma stepped out of line even just a little bit, and they made him disappear for like, six months. If the Chinese Communist Party decided that ASI is not worth the risk, it would stop tomorrow and they could enforce it. So could we get to such an international agreement? Now, this is a fair question, and this is very, very hard, but this is a disarmament problem. It's the same problem we did it with nuclear weapons, with the Soviets. Yes, this is hard, but then you just have to do diplomacy. Like, what the fuck are you talking about?

[Andy]

So your response is, yeah, it's going to be really, really hard. And there's going to be questions that don't have easy answers, but the situation we're in demands that we just have to do it.

[Conor Leahy]

Yeah, I think these are questions that would need answering. And I think it's very fair to say, hey, those are important questions. I don't think those questions should be dismissed. I just think they're the kind of questions we need to answer after we have defused the ticking time bomb.

[Nate Soares]

The US should also be developing the monitoring ability to figure out who is participating in this reckless race. We should be developing the monitoring capacity to sort of know where the AI chips are going to. We should be developing the intelligence to know whether they're trying to make smarter and smarter AIs. And we should be developing the sabotage capability to stop them from making Smarter and Smarter AIs for dangers of this magnitude. We should be developing the capacity to understand where this is being done and to sabotage it.

[Andy]

What do you say to the people who are listening to this? Some of them may be critics of you in your camp who say that you are fundamentally recommending something that is out of line with human nature. The idea being that human beings, we're competitive and we're ambitious. And once we know that there's some technology that's possible, we're going to chase after it. We're going to try and beat each other to it, especially when that technology might be so beneficial. And that the thing that you're proposing, that we stop until some future date when we know that this technology will be totally safe is just out of whack with who we are. At a very core level, I'm thinking about One person who I spoke to who said, what if we had tried to do this with the automobile, with the car, that if we tried to make a car that was totally safe? And he said, no, that's not how it works. We make a car, and then it's only as it goes out, we start to realize, oh, turn signals would help this thing be safer, or, oh, seat belts would help this thing be safer. Do you feel that there's an aspect of what you're proposing that is out of line with human nature and so it will ultimately fail?

[Nate Soares]

I think the car analogy really does a disservice to the situation we're in. It's a little bit more like the whole world is building the first car, loading up every man, woman and child and then pointing it towards the edge of a cliff and saying, let's slam down in the accelerator. In that situation, you really should get the world together and say, let's stop doing that. This is not a situation where I'm saying the AI needs seat belts. This is not a situation of me saying, oh, the AIs are having some negative effects. We need to not try and get any of the positive effects until we make sure there's no negative effects. This is a situation where I'm saying if we continue down this course, the most likely outcome is that literally everybody on Earth will die. That's just a wildly different situation than the cars. And then in terms of whether this is within our human nature, I think it's very defeatist to say humanity won't do this. And I also think it's very foolish to be defeatist like that at this stage.

[Conor Leahy]

You know what I think is human nature? What I think is human nature is solving problems. What I don't think is human nature is suicide, exploration, curiosity, surviving the next day so you can come home to your family. That's what human nature is to me.

[Andy]

All right, so let's talk about extinction. Because it's one thing to say, look at these mysterious and potentially powerful new AI systems. Can't you see how these things could be dangerous? And it's another thing to say that there's a 10%, 20%, a 90% chance that they may wipe out the entire human race. And as we mentioned before, this is a view that even some of the people who are making AI right now, as fast as they can, they think that there is some percentage chance that that could happen. So paint the picture for me. What is it that you're envisioning when you say extinction? What would that look like all of.

[Nate Soares]

This is a little bit like someone in the year 1800 trying to guess what war would look like against someone in the year 2000 and 20. And imagining that they can make slightly more explosive bombs, they might say, oh, well, we're only at the beginning of artillery technology. And so I bet in the future they're going to have artillery that's at least ten times as good.

[Andy]

It would have been hard for them to imagine a wi fi, Internet connected flying drone.

[Nate Soares]

That's right. Never mind a nuclear weapon. Right. And so it's easy to predict that very smart AIs will be able to find ways to get whatever it is they're pursuing. It's hard to predict exactly how, but it's easy to predict that they will be able to dramatically outcompete us. The obvious thing that happens there is that the future goes under their control rather than ours. Just like how the future is now under the human control rather than the chimpanzee control. Chimpanzees live or die according to whether humanity can restrain themselves from cutting down their jungles. And then you have this other issue which is for most of the weird stuff the AI could pursue, happy, healthy, free people are not the most efficient way to get it. So if this AI wants all sorts of things that are weird proxies of helpfulness, maybe it's like, well, I want a lot of things that are sort of human based, but they're sort of a little bit lobotomized. Like the sort of humans that are really delighted with every interaction with me. Just like how humans breed chickens to be more and more chicken breast. Maybe the AIs are sort of like breeding humans or just straight out using the technology to change humans to be more and more the type of thing the AI likes interacting with.

[Conor Leahy]

So I want to take your question seriously. I don't want to just dismiss it. I don't know what will happen exactly. The one thing I'm confident in is that we will lose. Like we will almost definitionally lose. But the way if I had to guess what it will feel like, I expect it will be confusing. I don't think it's going to be epic. I don't think there's going to be a huge showdown between the Terminators and the humanity. I don't think there's going to be crazy nanotechnology flying through the sky or something like this. I don't think that's what's going to happen. Maybe, but I don't think it's likely I think what's most likely to happen is it will just be quite unceremonious and just like quite pathetic and like boring almost. I expect what was going to happen is that the world will just start feeling more and more confusing. It will be harder and harder to understand what's really going on. We see all this, like, fake news and like all these distractions like video games keep getting better and pornography keeps getting more addictive. And all the news we get starts being like super polarized and super manipulative so that we can't really tell what's true and what's not. Anymore, a lot of weird geopolitical events start happening that no one can really explain why they happened or what the consequences of them are. New technologies get invented where we don't really know how they were invented. And just like more and more confusing things happen. More and more people put AIs in charge of more and more things. More and more people will put AIs in charge of their companies. Politicians will take advice from AIs for how to run their campaigns and for how to write policy. More and more of power will be willingly, completely willingly handed to AIs. You know, everyone will race to hand as much of their power over to the AI as possible because it helps them to outperform their rivals. And then eventually, you know, just like one day we wake up and we're not in control anymore.

[Andy]

All right, so I wanted to ask you one final question. And this is a question about self reflection. And I'm doing a lot of self reflection myself. So I ask this with all due respect. I don't want to be the kind of journalist who alarms people. But I also don't want to be the kind of journalist who shies away from diving into a subject because I'm like trying to read the room. This is all fascinating. I want to enter it with an open mind. But for you, do you ever worry about the fact that there have been doomers of one kind or another throughout human history, throughout the world, whether it's in ancient China or in Europe or tribes in South America, that there have been people who are convinced that the end is near and they have, similar to you, gone out trying to convince other people to realize that the end is near? Does that ever worry you that you're in some ways performing a familiar role in our society today?

[Nate Soares]

You know, there were surely some people in the Americas when the Spanish came that said they expected this would be the end of their world. And they were right. It was largely due to smallpox, which maybe not quite what they were expecting, but some worlds have ended. And if we broaden the reference class a little bit further, there were a lot of scientists in the 1920s who said leaded gasoline is poisonous. And if we run cars on leaded gasoline, it will poison a lot of children and cause a lot of brain damage to a lot of children. And those scientists were ignored. And leaded gasoline was rolled out across the country and the world, and quite a lot of children were brain damaged by leaded gasoline. Tens of millions, if not hundreds of millions, or maybe even more directly. There were a lot of people who said, in the wake of the development of nuclear bombs, it looks like on the current path, we're going to die in nuclear fire. And they had a lot of good reason to expect that humanity could not hold back from a nuclear war because humanity had not been able to hold back from total war with the best of their technology ever in the course of human history, and because fresh in their minds was the failure of the League of Nations to Prevent World War II. But humanity did not die in a nuclear fire. And it wasn't because nuclear weapons were fake. It wasn't because the bombs couldn't go off. It was true that the bombs could level cities. And we didn't die in a nuclear fire anyway because we realized the danger and we backed off. How do you tell the difference between a person prophesying the end times and a person trying to raise the alarm about a danger? Well, part of how you tell the difference is looking at whether they're saying there's nothing that can be done, Repent, we brought this about by our own sin or whatever. But a large part of how you figure out the difference is by looking at the arguments. When the nuclear scientists on the Manhattan Project said there's a chance that this bomb could ignite the atmosphere and destroy the entire Earth, people didn't say, oh, people have predicted the end of the world all sorts of times before, so that can't happen. What they said was, well, that seems worrying.

[Andy]

Let's look into it. Yeah, yeah, that seems like something we should look into.

[Nate Soares]

Right. So the way you figure out whether the nuclear bomb is going to ignite the atmosphere is you run the calculation. It's not that you sort of psychologize about the people and say, don't you worry that you're fulfilling a role of a prophet of doom? You sort of look at the arguments, and what I sort of try to do is lay out the arguments. And frankly, I think the machines are talking now. People thought that was going to take a lot longer than it did. We're able to grow AIs that are smarter and smarter each year. They're still pretty dumb, but people are able to grow AIs that are smarter and they don't really know how they're working and they're able to make them smarter still by throwing more computing power at it and just sort of naively. You don't need to be a big expert to look at how we're growing AIs and how we're making them smarter and how we don't understand what's going on and say, hold on, if we make machines that are much smarter than us, that can think faster than us, that can copy themselves a lot, why do we think that's going to go well? And the answer is not oh, everyone prophecies doom. The answer is, well, that seems like something we should look into. You should run those calculations and currently the calculations look grim to me. So I think we need to back off.

[Narrator/Producer]

Next time on the Last Invention, the.

[Andy]

AI Scouts how do we find the Win Win outcome here and their case.

[Narrator/Producer]

For why humanity can and must come together to get prepared for what's coming and do it soon because it is.

[Andy]

A narrow path we need to navigate. But I do think this Win Win.

[Natasha Vita-Moore]

Future is in Princip.

[Narrator/Producer]

The Last Invention is produced by Longview Home for the Curious and Open Minded. You can learn more about our team and help support our work by going to Longview investigations.com or by clicking on the link in our show notes. A special thanks this episode to Emil Torres and Jasmine Sun. Thanks for listening. We'll see you soon.

[Conor Leahy]

SA.

[Narrator/Producer]

This episode is sponsored by Ground News, the app that helps you spot media bias and see a broader picture of the news shaping our world. Get 40% off their vantage plan at Ground News Invent this episode is sponsored by Fire Defending Free Thought in the Age of AI. You can learn more at thefire.org thelastinvention.