Jonathan Liu (1:47)

What's my P doom? I would say around 50%.

Martin Kosti (1:51)

And yet you're smiling about it.

Jonathan Liu (1:53)

I'm smiling about it because there's nothing we can do about it.

Juana Summers (1:56)

Consider While the AI boom continues and companies across the country are heavily investing in the technology, some researchers are begging humanity to pump the brakes. From npr, I'm Juana Summers.

Sponsor Announcer (2:17)

This message comes from Saatva, official mattress and restorative sleep provider for Team usa. All athletes have to clear hurdles at some point, one of which is the need to get quality restorative sleep to perform at their peak. That's why SAATVA is proud to be providing athletes with mattresses and bedding for the LA28 Olympic and Paralympic Games. You can enjoy that kind of sleep, too. Visit saatva.com NPR and save $200 on.

Martin Kosti (2:43)

$1,000 or more this message comes from Carvana. Explore Carvana's quick and easy financing and browse thousands of car options all within your bud. Get pre qualified now@carvana.com financing subject to credit approval. Additional terms and conditions may apply.

Narrator/Storyteller (3:03)

On the Throughline podcast from npr, Immigration enforcement might be more visible now, but this moment didn't begin with President Trump's second inauguration or even his first, a series from Throughline about how immigration became political and a cash cow. Listen to Throughline in the NPR app or wherever you get your podcasts.

Juana Summers (3:27)

Hey, a quick request before we rejoin today's episode. We have heard from listeners who say consider this has become part of their daily routine, a way to make sense of things. If that is true for YouTube, take a couple minutes and leave us a review. It is a small thing, but it really does help people find this show. Thank you so much. It's Consider this from npr. A superhuman artificial intelligence so smart it can decide to get rid of slower witted humans is a pretty terrifying concept. What was once strictly the stuff of science fiction is now closer than ever to being a reality. And if it does, some AI researchers have gloomy predictions about humanity's chances of survival. NPR's Martin Costi caught up with these so called AI doomers.

Martin Kosti (4:21)

This is the main event of the evening. Welcome to a demo night in downtown San Francisco. Competitive events like this are a big part of the AI boom in this town right now. A chance for new developers to show off their new AI apps and maybe attract investors.

Jonathan Liu (4:40)

Yeah, my name is Jonathan Liu and I'm the founder of Cupidly, which is an AI agent that swipes for you on Hinge.

Martin Kosti (4:46)

You describe your ideal mate to the Cupidly AI and it goes into the dating app for you to find a match. Or it did. Lou has now shut it down because app users were getting banned by the Hinge dating app. But Lou is typical of this crowd in his bullishness about AI and the prospect of AI eventually becoming as smart or even smarter than humans.

Jonathan Liu (5:09)

I think once we do get super intelligence, hopefully we'll live in a utopia where nobody has to actually work ever again.

Martin Kosti (5:17)

But almost in the same breath, Lou also says that he sees a possibility that superhuman AI could end up killing off all of humanity. And he's not kidding. Just ask him for his P Doom. It's a term he'll recognize because it's sort of joking AI slang for estimated probability of AI doom.

Jonathan Liu (5:36)

What's my P doom? I would say around 50%.

Martin Kosti (5:40)

And yet you're smiling about it.

Jonathan Liu (5:42)

I'm smiling about it because there's nothing we can do about it.

Martin Kosti (5:44)

This strange mix of optimism and fatalism has long been a part of the AI world. Even the CEOs of OpenAI and Anthropic, two of the most important AI companies, signed a public statement a couple of years ago that acknowledged the risk of extinction from AI. And the reason for this is a pretty straightforward, logical problem. If they were to build something that's smarter than us, how would they keep it on our side? That problem is called alignment, as in how to align AI with human values. And here in Berkeley, near the UC campus, there's now a cluster of people working on that problem and related AI questions. Nate Soares is president of miri. That's the Machine Intelligence Research Institute. That's the newer name for an AI alignment organization that NPR first visited back in 2011.

Nate Soares (6:40)

I spent quite a number of years, maybe about 10 years, trying to figure out how to make AI go well. And for a bunch of reasons that's been going poorly.

Martin Kosti (6:49)

Soares has now given up on trying to figure out that alignment riddle. He says the machine learning revolution of the last few years, which created ChatGPT and the like, is now moving things too fast towards superhuman AI. And he gets little comfort from the fact that this also means there are now many more researchers here who are focused on AI safety.

Nate Soares (7:10)

Yeah, I mean, for one thing, I would not call it AI safety. I would say, you know, safety is for seatbelts and if you're in a car sort of careening towards a cliff edge, you wouldn't say, hey, let's talk about car safety here. You would say, let's stop going over the cliff edge.

Martin Kosti (7:23)

That cliff, as Soares puts it, is a scenario in which AI gets more closely involved in helping to improve AI, accelerating a kind of feedback loop of self improving artificial intelligence that ends up leaving humans behind as uncomprehending spectators and then perhaps just obstacles to be swept aside. And that's why Soris and another MIRI colleague have given up on alignment and are instead going the last ditch route of publishing a book that begs humanity to slam on the brakes.

Nate Soares (7:54)

The title of the book is if anyone builds it, Everyone dies.

Martin Kosti (7:59)

Let that sink in. And look around you. Does this all go away really in a few years?

Nate Soares (8:04)

I mean, I can't tell you when, but could be a couple years, could be a dozen years. But yeah, this, this around us is what's at stake.

Martin Kosti (8:15)

It's an extreme vision. Some critics say it's overblown that the current AI training methods can't even achieve human level intelligence, let alone super intelligence. Others say the doomers are unwittingly hyping AI. One writer in the Atlantic ridiculed Soris and his co author as useful idiots whose doom saying makes AI look more powerful than it really is. And it's also just a lot to ask to get a booming new tech sector to restrain itself. Maybe with government intervention in D.C. right.

Mark Beale (8:46)

Now, the conversation on AI is still very, very early.

Martin Kosti (8:49)

Mark Beale is president of government affairs for the AI Policy Network, a lobbying organization.

Mark Beale (8:54)

There does seem to be at least an appetite to start measuring these the risks and start to examine more carefully, you know, what threshold, what alarm bell might need to go off that would change that assumption about whether or not we ought to consider something as drastic as a pause.

Martin Kosti (9:12)

Government restrictions seem unlikely to Jim Miller. He's an economist at Smith College who's focused on the game theory aspect of AI development. He sees this as quickly turning into a race.

Jim Miller (9:23)

If I am Elon Musk, I can say, you know what? I don't know if racing is super intelligence going to kill everyone or not, but if it is going to kill everyone and I don't do it, someone else will. And if I end up killing everyone, I've maybe taken off a couple of weeks because OpenAI would have done it a week later. And then Trump and Vance can say, yeah, maybe this will kill everyone, but if we don't do it, China will.

Martin Kosti (9:42)

And for Miller, this isn't just an academic question. In his own life, he's decided to put off a risky surgery to correct a potentially fatal condition in his brain. Because he's an AI doomer and he's convinced that a superhuman AI is likely to end human civilization in the next few years. Or if he's very lucky, that superhuman AI will spare us and then offer him a safer treatment. On the campus of UC Berkeley, the generation with the most at stake are setting up the information tables for their student clubs. At the table for the club devoted to AI safety, Adi Mehta says he has heard the doomer argument, but he's focused on AI's more immediate risks. One thing that is more apparent for college students is that I can't remember the last time I did an assignment without using AI.

Nate Soares (10:38)

It's automating a lot of our thinking.

Martin Kosti (10:39)

Away, which personally, that's like a pretty big fear. Another club member, Natalia Trounce, says she's also just not that focused on doom. I think many things are possible but it seems like it's not the most likely scenario at this stage. If I were to ask the average Berkeley student, is this like, my life's gonna be over in three years, just have fun now? Or is it just, I feel like if it was like three years, it's gonna be over, like it would have happened already. Walking back to the offices of Miri, Nate Soares admits that with AI already such a normal part of life here, it's hard to convince people that we're about to go over that cliff. He says one hope is that maybe the rise of superhuman AI will be just gradual enough to be noticed and give people some time to react.

Nate Soares (11:25)

Maybe it doesn't take a ton. Maybe, maybe the the AI is doing a little better, getting a little smarter, getting a little bit more competent, getting a little bit more reliable. Maybe that'll make people a lot more spooked. I don't know.

Martin Kosti (11:37)

And maybe, just maybe, he and his fellow AI doomers are wrong about the danger. He says he would love to be wrong, but he doubts he is. Martin Kosti, NPR News, Berkeley, California.

Juana Summers (11:52)

For a more detailed look at AI and its risks, listen to our NPR Explains podcast, only available on the NPR app. This episode was produced by Mallory Yu with audio engineering by Ted Mebane. It was edited by Gigi Duban and Courtney Dorney. Our executive producer is Sammy Yenigan. It's Consider this from npr. I'm Juana Summers.

NPR Announcer (12:26)

Sources and Methods, the crown jewels of the intelligence community, Shorthand for How do we know what's real? Who told us? If you have those answers, you're on the inside, and NPR wants to bring you there. From the Pentagon to the State Department to spy agencies, listen to understand what's really happening and what it means for you. Sources and Methods, the new National Security Podcast from npr. Military commanders, intelligence officials, diplomatic power players. They know things you may not about where the world is headed. And we will pull back the curtain on what they're thinking on sources and methods, NPR's new national security Podcast. Our team will help you understand America's shifting role in the world. Listen to Sources and Methods from npr.

Sponsor Announcer (13:14)

Want to hear this podcast without sponsor breaks? Amazon prime members can listen to Consider this sponsor free through Amazon Music, or you can also support NPR's vital journalism and get consider this plus@ +npr.org that's +npr.org.