They warned about AI before it was cool. They're still worried

Summary

Podcast Summary: "They Warned About AI Before It Was Cool. They're Still Worried"

Podcast: Consider This from NPR
Host: Juana Summers, NPR reporters
Date: September 25, 2025
Duration: ~15 minutes

---

Overview

This episode of NPR's Consider This explores the evolving conversation around artificial intelligence—specifically, the fears and warnings from researchers who have long sounded the alarm about AI's potential dangers. The episode weaves together history, recent AI advancements, and varied perspectives—from "AI doomers" to students using AI in daily life—while examining whether society is ready (or willing) to put meaningful brakes on AI's rapid development.

---

Key Discussion Points & Insights

Science Fiction Origins & AI Doomerism

• AI in Sci-Fi:
  The episode opens with iconic AI fiction references, like HAL from "2001: A Space Odyssey" and Skynet from "Terminator," setting the stage for fears of runaway machine intelligence.
  • "Skynet becomes self aware at 2:14am Eastern Time, Aug. 29... In a panic, they tried to pull the plug. Skynet fights back." — (00:26–00:36)
• The Singularity:
  Keith Roedersheimer, a former fellow at the Singularity Institute (now MIRI), describes the classic "self-improving AI" scenario, emphasizing the danger of recursive, exponential self-enhancement.
  • "It's able to look at its own source code and say, ah, if I change this, I'm going to get smarter...that creates an extraordinarily intelligent thing." — Roedersheimer (01:05)
  • "They called this the Singularity because that intelligence could grow so fast, our human minds might not be able to keep up." — Juana Summers (01:22)

AI in Everyday Life: Ambition, Irony, and Fatalism

• AI Developers’ Mindset:
  At a San Francisco AI demo night, entrepreneur Jonathan Liu (creator of "Cupidly," an AI dating agent) embodies the duality among techies: hope for utopia clashing with genuine doomsday anxiety.
  • "I think once we do get super intelligence, hopefully we'll live in a utopia where nobody has to actually work ever again." — Jonathan Liu (05:09)
  • "What's my P doom [probability of AI doom]? I would say around 50%." — Jonathan Liu (05:36)
  • "And yet you're smiling about it." — Martin Kosti (05:40)
  • "I'm smiling about it because there's nothing we can do about it." — Liu (05:42)
• Alignment Problem:
  AI boomers and "doomers" alike wonder: if we create something smarter than humanity, how do we ensure it's aligned with our goals and values? The fundamental challenge is popularly called "the alignment problem."
  • "If they were to build something that's smarter than us, how would they keep it on our side? That problem is called alignment..." — Martin Kosti (05:44)

Extreme Caution: Warnings from Longtime Critics

• MIRI & Nate Soares:
  In Berkeley, MIRI president Nate Soares gave up on technical "alignment" efforts and is now focused on a stark warning: stop building advanced AI altogether.
  • "I spent quite a number of years, maybe about 10 years, trying to figure out how to make AI go well. And for a bunch of reasons that’s been going poorly." — Nate Soares (06:40)
  • "I would not call it AI safety. I would say, you know, safety is for seatbelts. And if you're in a car sort of careening towards a cliff edge, you wouldn't say, hey, let's talk about car safety here. You would say, let's stop going over the cliff edge." — Soares (07:10)
  • MIRI’s new book is pointedly titled:
    "The title of the book is 'If Anyone Builds It, Everyone Dies.'" — Soares (07:54)
• Timeline Anxiety:
  Soares refuses to put a precise date on when catastrophe could strike, but he warns the window could be "a couple years, could be a dozen years." (08:04)

Pushback, Criticism, and Policy Inertia

• Critics of AI Pessimism:
  Some believe the concerns are exaggerated, arguing that current AI is still far from even human-level intelligence. There’s also the risk of doom-talk helping to inflate AI’s reputation.

  • "Some critics say it's overblown... Others say the doomers are unwittingly hyping AI." — Martin Kosti (08:15)
  • The Atlantic labeled MIRI figures as "useful idiots" for making AI look more powerful than it is.

• Government Regulation and International Race:

  • Governmental efforts lag behind the rapid pace of AI development.
    • "Right now the conversation on AI is still very, very early." — Mark Beale, AI Policy Network (08:46)
  • Game theory dominates the larger conversation: No single nation or company is willing to pause for fear others will press ahead, creating a classic "race" to potentially catastrophic outcomes.
    • "If I end up killing everyone, I've maybe taken off a couple of weeks because OpenAI would have done it a week later. And then Trump and Vance can say, yeah, maybe this will kill everyone, but if we don't do it, China will." — Jim Miller (09:23)

• Personal Stakes:
  Economist Jim Miller, so convinced by AI doom calculations, has postponed a risky brain surgery, betting on (or against) the arrival of a superhuman AI in the next few years. (09:42)

Younger Generation and Practical Risks

• Campus Voices:
  At UC Berkeley, student club members are skeptical of immediate doom but note that AI is already deeply integrated into daily life (assignments, automation of thinking), raising subtler concerns.

  • "I can't remember the last time I did an assignment without using AI." — Adi Mehta (10:38)
  • "It's automating a lot of our thinking away, which personally, that's like a pretty big fear." — Student (10:38)
  • "I think many things are possible but it seems like it's not the most likely scenario at this stage..." — Natalia Trounce (10:39)

• Difficulty of Persuasion:
  MIRI’s Soares laments the difficulty convincing people of existential risk when AI is so normalized.

  • "With AI already such a normal part of life here, it's hard to convince people that we're about to go over that cliff." — (11:25)
  • But he hopes for a gradual enough progression to enable a warning and response:
    "Maybe the AI is doing a little better, getting a little smarter, getting... more reliable. Maybe that'll make people a lot more spooked. I don't know." — Soares (11:25)

• A Wish to Be Wrong:

  • "And maybe, just maybe, he and his fellow AI doomers are wrong about the danger. He says he would love to be wrong, but he doubts he is." — Martin Kosti (11:37)

---

Notable Quotes & Memorable Moments

• "What's my P doom? I would say around 50%."
  — Jonathan Liu (05:36)
• "I would not call it AI safety. ... if you're in a car sort of careening towards a cliff edge, you wouldn't say, hey, let's talk about car safety. You would say, let's stop going over the cliff edge."
  — Nate Soares (07:10)
• "The title of the book is 'If Anyone Builds It, Everyone Dies.'"
  — Nate Soares (07:54)
• "If I end up killing everyone, I've maybe taken off a couple of weeks because OpenAI would have done it a week later."
  — Jim Miller (09:23)
• "I can't remember the last time I did an assignment without using AI."
  — Adi Mehta (10:38)
• "Maybe the AI is doing a little better, getting a little smarter, getting... more reliable. Maybe that'll make people a lot more spooked. I don't know."
  — Nate Soares (11:25)
• "He says he would love to be wrong, but he doubts he is."
  — Martin Kosti on Soares (11:37)

---

Timestamps for Key Segments

• 00:00–01:22: Sci-fi fears, origin of AI doomerism, Singularity
• 04:21–05:44: AI demo night in San Francisco; Jonathan Liu’s optimism and fatalism
• 05:44–07:23: Alignment problem explained; interviews with researchers at MIRI (Nate Soares)
• 07:23–08:15: Why some have given up on alignment, “If anyone builds it, everyone dies”
• 08:15–09:12: Criticism of doomer stance; policy debate and regulatory inertia
• 09:12–09:42: The international AI race and personal stakes (Jim Miller’s health decision)
• 10:38–11:17: Student perspectives at UC Berkeley; everyday uses and emerging risks
• 11:25–11:52: Soares on gradual risk, hope to be proven wrong

---

Tone and Takeaway

Throughout, the tone is alternately wry, anxious, and urgent—reflecting both the uncertainty and the near resignation among some AI experts, offset by skepticism from younger users and critics. The episode leaves listeners with a sense of precariousness: the future of AI could be either mundane, utopian, or extinction-level disastrous, and the outcome may hinge as much on policy and social response as on technical breakthroughs.

The final word is a mix of hope and doubt: the doomers, as the host notes, "would love to be wrong, but [doubt] they are." (11:52)

---

Transcript

[Juana Summers]

AI used to be a thing of science fiction.

[Martin Kosti]

I know I've made some very poor.

[NPR Announcer]

Decisions recently.

[Martin Kosti]

But I can give you my complete assurance that my work will be back to normal.

[Juana Summers]

And the genre is full of superhuman AI machines that become so smart they turn against the humans that created them.

[Narrator/Storyteller]

Skynet begins to learn at a geometric rate.

[Keith Roedersheimer]

It becomes self aware at 2:14am Eastern Time, Aug. 29. In a panic, they tried to pull the plug.

[Juana Summers]

Skynet fights back.

[Keith Roedersheimer]

Yes, that's an AI that could get out of control. But if you really think about it, it's much worse than that.

[Martin Kosti]

Much worse than Terminator.

[Keith Roedersheimer]

Much, much worse.

[Juana Summers]

That's Keith ROEDERSHEIMER Talking to NPR's Martin Costi. Back in 2011, he was a research fellow for what was then called the Singularity Institute for Artificial Intelligence. It's now the Machine Intelligence Research Institute, or miri. At the time, Roedersheimer was looking into the idea of a computer that was not only smart, but capable of improving itself.

[Keith Roedersheimer]

It's able to look at its own source code and say, ah, if I change this, I'm going to get smarter. And then by getting smarter, it sees new insights into how to get smarter. And then by having those insights into how to get smarter, it modifies its source code and gets smarter and gets new insights. And that creates an extraordinarily intelligent thing.

[Juana Summers]

They called this the Singularity because that intelligence could grow so fast, our human minds might not be able to keep up. In 2011, that still seemed like a long, long way off. But in 2025, artificial intelligence is seeping into everyday life with ChatGPT and the like. Even proponents of AI, like developer Jonathan Liu, joke about the estimated probability of AI doom.

[Jonathan Liu]

What's my P doom? I would say around 50%.

[Martin Kosti]

And yet you're smiling about it.

[Jonathan Liu]

I'm smiling about it because there's nothing we can do about it.

[Juana Summers]

Consider While the AI boom continues and companies across the country are heavily investing in the technology, some researchers are begging humanity to pump the brakes. From npr, I'm Juana Summers.

[Sponsor Announcer]

This message comes from Saatva, official mattress and restorative sleep provider for Team usa. All athletes have to clear hurdles at some point, one of which is the need to get quality restorative sleep to perform at their peak. That's why SAATVA is proud to be providing athletes with mattresses and bedding for the LA28 Olympic and Paralympic Games. You can enjoy that kind of sleep, too. Visit saatva.com NPR and save $200 on.

[Martin Kosti]

$1,000 or more this message comes from Carvana. Explore Carvana's quick and easy financing and browse thousands of car options all within your bud. Get pre qualified [e-mail] financing subject to credit approval. Additional terms and conditions may apply.

[Narrator/Storyteller]

On the Throughline podcast from npr, Immigration enforcement might be more visible now, but this moment didn't begin with President Trump's second inauguration or even his first, a series from Throughline about how immigration became political and a cash cow. Listen to Throughline in the NPR app or wherever you get your podcasts.

[Juana Summers]

Hey, a quick request before we rejoin today's episode. We have heard from listeners who say consider this has become part of their daily routine, a way to make sense of things. If that is true for YouTube, take a couple minutes and leave us a review. It is a small thing, but it really does help people find this show. Thank you so much. It's Consider this from npr. A superhuman artificial intelligence so smart it can decide to get rid of slower witted humans is a pretty terrifying concept. What was once strictly the stuff of science fiction is now closer than ever to being a reality. And if it does, some AI researchers have gloomy predictions about humanity's chances of survival. NPR's Martin Costi caught up with these so called AI doomers.

[Martin Kosti]

This is the main event of the evening. Welcome to a demo night in downtown San Francisco. Competitive events like this are a big part of the AI boom in this town right now. A chance for new developers to show off their new AI apps and maybe attract investors.

[Jonathan Liu]

Yeah, my name is Jonathan Liu and I'm the founder of Cupidly, which is an AI agent that swipes for you on Hinge.

[Martin Kosti]

You describe your ideal mate to the Cupidly AI and it goes into the dating app for you to find a match. Or it did. Lou has now shut it down because app users were getting banned by the Hinge dating app. But Lou is typical of this crowd in his bullishness about AI and the prospect of AI eventually becoming as smart or even smarter than humans.

[Jonathan Liu]

I think once we do get super intelligence, hopefully we'll live in a utopia where nobody has to actually work ever again.

[Martin Kosti]

But almost in the same breath, Lou also says that he sees a possibility that superhuman AI could end up killing off all of humanity. And he's not kidding. Just ask him for his P Doom. It's a term he'll recognize because it's sort of joking AI slang for estimated probability of AI doom.

[Jonathan Liu]

What's my P doom? I would say around 50%.

[Martin Kosti]

And yet you're smiling about it.

[Jonathan Liu]

I'm smiling about it because there's nothing we can do about it.

[Martin Kosti]

This strange mix of optimism and fatalism has long been a part of the AI world. Even the CEOs of OpenAI and Anthropic, two of the most important AI companies, signed a public statement a couple of years ago that acknowledged the risk of extinction from AI. And the reason for this is a pretty straightforward, logical problem. If they were to build something that's smarter than us, how would they keep it on our side? That problem is called alignment, as in how to align AI with human values. And here in Berkeley, near the UC campus, there's now a cluster of people working on that problem and related AI questions. Nate Soares is president of miri. That's the Machine Intelligence Research Institute. That's the newer name for an AI alignment organization that NPR first visited back in 2011.

[Nate Soares]

I spent quite a number of years, maybe about 10 years, trying to figure out how to make AI go well. And for a bunch of reasons that's been going poorly.

[Martin Kosti]

Soares has now given up on trying to figure out that alignment riddle. He says the machine learning revolution of the last few years, which created ChatGPT and the like, is now moving things too fast towards superhuman AI. And he gets little comfort from the fact that this also means there are now many more researchers here who are focused on AI safety.

[Nate Soares]

Yeah, I mean, for one thing, I would not call it AI safety. I would say, you know, safety is for seatbelts and if you're in a car sort of careening towards a cliff edge, you wouldn't say, hey, let's talk about car safety here. You would say, let's stop going over the cliff edge.

[Martin Kosti]

That cliff, as Soares puts it, is a scenario in which AI gets more closely involved in helping to improve AI, accelerating a kind of feedback loop of self improving artificial intelligence that ends up leaving humans behind as uncomprehending spectators and then perhaps just obstacles to be swept aside. And that's why Soris and another MIRI colleague have given up on alignment and are instead going the last ditch route of publishing a book that begs humanity to slam on the brakes.

[Nate Soares]

The title of the book is if anyone builds it, Everyone dies.

[Martin Kosti]

Let that sink in. And look around you. Does this all go away really in a few years?

[Nate Soares]

I mean, I can't tell you when, but could be a couple years, could be a dozen years. But yeah, this, this around us is what's at stake.

[Martin Kosti]

It's an extreme vision. Some critics say it's overblown that the current AI training methods can't even achieve human level intelligence, let alone super intelligence. Others say the doomers are unwittingly hyping AI. One writer in the Atlantic ridiculed Soris and his co author as useful idiots whose doom saying makes AI look more powerful than it really is. And it's also just a lot to ask to get a booming new tech sector to restrain itself. Maybe with government intervention in D.C. right.

[Mark Beale]

Now, the conversation on AI is still very, very early.

[Martin Kosti]

Mark Beale is president of government affairs for the AI Policy Network, a lobbying organization.

[Mark Beale]

There does seem to be at least an appetite to start measuring these the risks and start to examine more carefully, you know, what threshold, what alarm bell might need to go off that would change that assumption about whether or not we ought to consider something as drastic as a pause.

[Martin Kosti]

Government restrictions seem unlikely to Jim Miller. He's an economist at Smith College who's focused on the game theory aspect of AI development. He sees this as quickly turning into a race.

[Jim Miller]

If I am Elon Musk, I can say, you know what? I don't know if racing is super intelligence going to kill everyone or not, but if it is going to kill everyone and I don't do it, someone else will. And if I end up killing everyone, I've maybe taken off a couple of weeks because OpenAI would have done it a week later. And then Trump and Vance can say, yeah, maybe this will kill everyone, but if we don't do it, China will.

[Martin Kosti]

And for Miller, this isn't just an academic question. In his own life, he's decided to put off a risky surgery to correct a potentially fatal condition in his brain. Because he's an AI doomer and he's convinced that a superhuman AI is likely to end human civilization in the next few years. Or if he's very lucky, that superhuman AI will spare us and then offer him a safer treatment. On the campus of UC Berkeley, the generation with the most at stake are setting up the information tables for their student clubs. At the table for the club devoted to AI safety, Adi Mehta says he has heard the doomer argument, but he's focused on AI's more immediate risks. One thing that is more apparent for college students is that I can't remember the last time I did an assignment without using AI.

[Nate Soares]

It's automating a lot of our thinking.

[Martin Kosti]

Away, which personally, that's like a pretty big fear. Another club member, Natalia Trounce, says she's also just not that focused on doom. I think many things are possible but it seems like it's not the most likely scenario at this stage. If I were to ask the average Berkeley student, is this like, my life's gonna be over in three years, just have fun now? Or is it just, I feel like if it was like three years, it's gonna be over, like it would have happened already. Walking back to the offices of Miri, Nate Soares admits that with AI already such a normal part of life here, it's hard to convince people that we're about to go over that cliff. He says one hope is that maybe the rise of superhuman AI will be just gradual enough to be noticed and give people some time to react.

[Nate Soares]

Maybe it doesn't take a ton. Maybe, maybe the the AI is doing a little better, getting a little smarter, getting a little bit more competent, getting a little bit more reliable. Maybe that'll make people a lot more spooked. I don't know.

[Martin Kosti]

And maybe, just maybe, he and his fellow AI doomers are wrong about the danger. He says he would love to be wrong, but he doubts he is. Martin Kosti, NPR News, Berkeley, California.

[Juana Summers]

For a more detailed look at AI and its risks, listen to our NPR Explains podcast, only available on the NPR app. This episode was produced by Mallory Yu with audio engineering by Ted Mebane. It was edited by Gigi Duban and Courtney Dorney. Our executive producer is Sammy Yenigan. It's Consider this from npr. I'm Juana Summers.

[NPR Announcer]

Sources and Methods, the crown jewels of the intelligence community, Shorthand for How do we know what's real? Who told us? If you have those answers, you're on the inside, and NPR wants to bring you there. From the Pentagon to the State Department to spy agencies, listen to understand what's really happening and what it means for you. Sources and Methods, the new National Security Podcast from npr. Military commanders, intelligence officials, diplomatic power players. They know things you may not about where the world is headed. And we will pull back the curtain on what they're thinking on sources and methods, NPR's new national security Podcast. Our team will help you understand America's shifting role in the world. Listen to Sources and Methods from npr.

[Sponsor Announcer]

Want to hear this podcast without sponsor breaks? Amazon prime members can listen to Consider this sponsor free through Amazon Music, or you can also support NPR's vital journalism and get consider this plus@ +npr.org that's +npr.org.