OpenAI Part 3: Heaven and Hell, Part 1

Summary

Foundering: OpenAI Part 3 – Heaven and Hell, Part 1

Release Date: June 13, 2024
Host: Ellen Hewitt | Produced by Bloomberg Technology

---

Introduction

In the third installment of the "Foundering" series, Ellen Hewitt delves deep into the contrasting narratives surrounding OpenAI's meteoric rise under CEO Sam Altman and the personal struggles faced by his sister, Annie Altman. This episode, titled "Heaven and Hell, Part 1," explores the internal dynamics of OpenAI, the widespread impact of ChatGPT, and the polarized views on artificial intelligence's future.

---

Contrasting Success: Sam Altman vs. Annie Altman

The episode opens with a vivid portrayal of Annie Altman's tumultuous living situation juxtaposed against her brother Sam's soaring success with OpenAI.

• Annie Altman's Struggles:

  • Lost Stability: "For much of the past two years, Annie hasn't been able to afford a stable place to live." (02:13)
  • Frequent Moves: "Recently, over the course of just a year, she moved 22 times. That's on average about twice a month." (02:24)
  • Homelessness: She recounts staying in "two months on a newly built with no running water or no electricity house... slept on floors in friends' houses." (02:45)

• Sam Altman's Rise:

  • Global Recognition: Sam was named CEO of OpenAI by Time magazine and became a prominent figure speaking to world leaders about AI. (03:08)
  • Public Persona: Sam’s vision contrasts sharply with Annie’s reality. At a Bloomberg conference in June 2023, he confidently stated, "We just shouldn't have poverty in the world." (04:39)

Notable Contrast: Sam envisions a future where AI eradicates poverty, while his sister grapples with homelessness, highlighting a stark disconnect between his public optimism and his family's personal challenges.

---

OpenAI's Internal Disputes and the Birth of Anthropic

As OpenAI pushed the boundaries of AI technology, internal disagreements about safety and ethical implications emerged.

• Employee Concerns:

  • Dario Amadei's Fear: "This is like, this is crazy. You know, there's nothing like this in the world." (11:21)
  • Existential Risk: Employees like Dario feared the AI models could "destroy us," leading to concerns about the lack of safety measures. (11:42)

• Formation of Anthropic:

  • Mass Exodus: In 2021, Dario and six colleagues left OpenAI to create Anthropic, a company focused on building safe AI. (12:31)
  • Implications: Their departure suggested distrust in OpenAI’s commitment to AI safety, casting a shadow over the company's internal culture.

---

The ChatGPT Phenomenon

The release of ChatGPT marked a pivotal moment for OpenAI, catapulting the company and Sam Altman into global prominence.

• Rapid Adoption:

  • User Growth: "ChatGPT reached 100 million users in just two months, the fastest growth ever at the time." (14:22)
  • Practical Uses: From drafting emails to aiding students with homework, ChatGPT became an indispensable tool for millions. (15:14)

• Sam Altman's Elevated Profile:

  • Household Name: Sam became the face of the AI boom, with investors eager to capitalize on the AI surge. (15:30)
  • Public Speeches: At an Airbnb conference in 2015, Sam emphasized the profound societal changes AI would bring, stating, "We have an opportunity that comes along only every couple of centuries to redo the socioeconomic contract." (28:48)

---

Rise of AI Doomsday Beliefs

With AI's advancement, a subset of tech enthusiasts began to fear an existential threat posed by superintelligent AI.

• Chao Chu Yuan's Perspective:

  • Apocalyptic Vision: Chao believes that "the world is probably going to be swallowed by superintelligent AI soon." (20:05)
  • Personal Impact: His fears led him to abandon a math PhD and isolate from friends, driven by the conviction that AI poses a catastrophic threat. (20:43)

• Influential Evangelism:

  • Public Influence: Figures like Eliezer Yudkowsky have propagated these fears, convincing many in Silicon Valley of the imminent dangers. (19:18)
  • Surge in AI Safety Funding: Tech leaders, including Dustin Moskovitz and Sam Bankman-Fried, have pledged significant funds toward AI safety projects, reflecting the growing concern. (22:00)

Key Quote: Chao Yuan describes his transformation: "There's this black hole in my kind of sense of which things were important... Maybe the only thing that matters." (21:10)

---

Criticism of AI Apocalypse Focus

Not all experts agree with the doomsday narrative, arguing that it diverts attention from immediate, tangible AI-related issues.

• Emily Bender's Critique:

  • Distraction from Current Harms: "There's all kinds of harms that are happening right now... The more we focus on these... existential risk... the less time and effort goes into actually dealing with the real harms." (24:26)
  • Immediate Concerns: Issues like racial bias, data theft, and surveillance are pressing problems exacerbated by AI. (24:55)

• Broader Implications:

  • Skewed Focus: The intense focus on catastrophic scenarios overshadows critical debates on AI ethics, fairness, and societal impact. (25:48)

Notable Insight: Emily emphasizes that "these are urgent problems, ways that people are being harmed by AI right now," advocating for addressing present-day issues over speculative future risks. (25:23)

---

Sam Altman’s Evolving Public Image

Sam Altman's statements reflect a balance between acknowledging AI's risks and promoting its potential benefits, navigating between heroism and pragmatism.

• Early Doomsday Fears:

  • In 2015, Sam expressed strong concerns about AI: "I think AI will probably sort of lead to the end of the world." (31:26)
  • Apocalyptic Preparations: He mentioned stockpiling resources and securing a land parcel in Big Sur for potential catastrophes, although he later downplayed these statements. (31:31)

• Shift to Optimism:

  • Post-ChatGPT, Sam adopts a more moderate stance, focusing on the positive societal transformations AI can bring. He avoids referencing his earlier apocalyptic plans, aiming to present a balanced view. (32:53)

• Heroic Narrative:

  • Sam positions himself as a pivotal figure shaping the future, stating, "We're going to create a galactic civilization." (27:39)

Key Quote: At an Airbnb conference, Sam remarked on public perception: "People are much more sensitive to... theatrical extreme risk than... boring, slow, plotting risk." (34:13)

---

Motivations Behind AI Enthusiasm and Fear

The episode explores the psychological drivers behind the fervent belief in AI’s potential to either save or doom humanity.

• Heroism and Legacy:

  • Chao Yuan discusses the desire to be seen as a hero: "Most people don't get a chance to do that in any meaningful sense... What if this is like the most significant era of human history?" (27:06)

• Influence of Science Fiction:

  • OpenAI's engagement with fiction writers, like commissioning a novella from Patrick House, highlights the role of storytelling in shaping AI narratives. (30:07)

• Motivational Drivers:

  • The belief in being pivotal to humanity's future drives many in the AI field, fostering a culture where cosmic significance is entwined with technological advancement. (27:54)

---

Conclusion

"Foundering: OpenAI Part 3 – Heaven and Hell, Part 1" paints a comprehensive picture of the dual realities within the AI industry. While leaders like Sam Altman project a future of abundance and societal transformation, the episode underscores the personal and ethical dilemmas that persist behind the scenes. The narrative juxtaposes visionary aspirations with real-world struggles, prompting listeners to reflect on the true cost and potential of artificial intelligence.

Upcoming in Part 2: The series will further explore the debate around poverty in an AI-driven economy and delve deeper into Annie Altman's experiences, providing a holistic view of AI's multifaceted impact on society.

---

Notable Quotes

• Sam Altman on Poverty:

  "I think we are not that far away from being able to eliminate poverty effectively worldwide, certainly in developed countries." (04:57)

• Chao Chu Yuan on Existential Fears:

  "It's like, oh, hey, what if everyone dies in 10 years? That's a scary idea." (22:58)

• Emily Bender on AI Focus:

  "There's all kinds of harms that are happening right now... The more we focus on these... existential risk... the less time and effort goes into actually dealing with the real harms." (24:26)

• Sam Altman on Public Perception:

  "People are much more sensitive to sort of like theatrical extreme risk than they are to sort of like boring, slow, plotting risk." (34:13)

---

Production Credits

• Host: Ellen Hewitt
• Executive Producer: Sean Wen
• Contributing Reporter: Rachel Metz
• Associate Producer: Molly Nugent
• Audio Engineer: Blake Maples
• Story Editors: Mark Millian, Ann Vandermay, Seth Vigerman, Tom Giles, Molly Schutz
• Production Help: Jessica Nix, Antonia Mufarec

---

For those interested in the intricate dynamics of the AI revolution and its broader societal implications, "Foundering: OpenAI Part 3 – Heaven and Hell, Part 1" offers a thought-provoking and comprehensive exploration.

Transcript

[Ad]

In today's data driven world, you need real data to shape an effective marketing strategy. That's where Ahrefs comes in. Ahrefs is a marketing intelligence platform powered by big data. It shows you exactly what your target audience is searching for and how you measure up to your competitors. Whether you're making day to day marketing decisions or defining your long term digital strategy, Ahrefs delivers real time insights that help you grow your brand. Head over to ahrefs.com to get started. That's a h r e f s.com.

[Matt Levine]

Are you looking for a new podcast about stuff related to money? Well, today's your lucky day. I'm Matt Levine.

[Katie Greifeld]

And I'm Katie Greifeld and we are.

[Matt Levine]

The hosts of Money Stuff the podcast. Every Friday we dive into the top stories about Wall street, finance and other stuff. We have fun, we get weird and.

[Katie Greifeld]

We want you to join us.

[Matt Levine]

You can listen to Money Stuff the podcast on Apple Podcasts, Spotify, or wherever you get your podcasts. Today we're going to start on a drive in Hawaii.

[Annie Altman]

We're on North Shore, going deeper into the jungle on the North Shore. So we're passing Twin Falls.

[Matt Levine]

Right now I'm driving through the lush green forests of Maui. Annie Altman, Sam Altman's little sister, is sitting in the passenger seat. You heard from her briefly in the first episode.

[Annie Altman]

And I love this because you wouldn't turn here if you didn't know you could turn here.

[Matt Levine]

I certainly would not be driving down.

[Annie Altman]

This if I didn't tell you to.

[Matt Levine]

Tiny little road. I know. We're taking a tour of the different places Annie has moved around in the last couple years, driving down dirt roads to look at cabins and houses hidden behind enormous tropical plants. Oh my God. Huge monstera. Look at that.

[Annie Altman]

They're enormous. It's so goofy then in driving this. I know because then you see it in someone's office plant and you're like, is that the same thing?

[Matt Levine]

For much of the past two years, Annie hasn't been able to afford a stable place to live.

[Annie Altman]

The place we just passed is one of the places I stayed at longer term in all of the houselessness. Two months on a newly built with no running water or no electricity house at the far end back of the property.

[Matt Levine]

And I think she's an important part of Sam's story.

[Annie Altman]

And at the time I had no nowhere to stay and no rent money, certainly no deposit money and barely enough money for rent.

[Matt Levine]

Recently, over the course of just a year, she moved 22 times. That's on average about twice a month. Sometimes she has stayed places for a week at a time or even just a night or two. Some of them have been illegal rentals without running water. She says she slept on floors in friends houses. She stayed with strangers when she didn't have another option.

[Annie Altman]

The man who lived in the front house messaged me on Instagram and I stayed in his kids room the week that they weren't there and then slept on the floor in the common room the week that the kids were there. I was having. I didn't have somewhere to go. I stayed in this cabin with the slanty roof right there for three months.

[Matt Levine]

How many different places have you lived in that didn't have running water?

[Annie Altman]

Maybe 5ish, 5 or 6, I don't know.

[Matt Levine]

Meanwhile, thousands of miles away in San Francisco, her brother Sam was having a spectacular year. In 2023, the success of ChatGPT had launched OpenAI into the stratosphere. Sam was named CEO of the by Time magazine. He spent months flying around the globe talking to world leaders about AI. So without further ado, let's give a big round of applause to our special guest, Sam Altman.

[Sam Altman]

It is with great pleasure that I.

[Matt Levine]

Invite on stage Sam Altman, the individual.

[Ron Krushevsky]

Who transcends introduction, Mr. Sam Altman, the CEO of OpenAI.

[Matt Levine]

Sure, Sam had been positioning himself as a public intellectual for some time, but for a long while, he had merely been Silicon Valley Valley famous. This was a new level of fame. Suddenly it seemed like Sam Altman was the spokesman for this entire AI boom. On stage, on podcasts, in interviews, people kept turning to Sam for answers. They were asking him what our AI future would hold. In May of that year, he confidently suggested a future where no one is poor. It's an idea he's talked about for years, and the remarks show that his tune hasn't changed, despite growing renown and wealth.

[Sam Altman]

One thing I think we all could agree on is that we just shouldn't have poverty in the world.

[Matt Levine]

So he's saying something like poverty, which is as deeply entrenched as civilization itself could be fixed in the age of AI, everyone will have what they need to live, including a home.

[Sam Altman]

I think we are not that far away from being able to eliminate poverty effectively worldwide, certainly in developed countries. And I think there are things that are 50 years from now will be basic human rights, like healthcare and enough food to eat and a place to live. I'm confident we're going to get that done.

[Matt Levine]

He sounds so sure of himself. How will these changes happen? Sam says that they could come from AI. He's basically saying the technology his company is working on can eliminate poverty. Here he is at a Bloomberg conference in June 2023.

[Sam Altman]

I think it'd be good to end poverty. Maybe you think we should stop with technology that can do that? I personally don't.

[Matt Levine]

He's even scolding people who say they want to stop or slow down the development of AI by saying that they want to keep people poor. Solving poverty seemed to become part of his personal brand. Is almost a stump speech of sorts. And there's one word he likes to use to encapsulate our bountiful future.

[Sam Altman]

I think it's how we get to this world of abundance. In a world with the level of abundance AI and the abundance that comes with that. If we can have abundant and cheap energy and intelligence that will transform people's lives.

[Matt Levine]

It sounds wonderful, almost utopian. But Sam was saying on stage that everyone should have enough money, enough food, everyone should have a place to live while his own sister was struggling with homelessness. I want to believe Sam's promises about abundance, but Annie's story complicates a lot of the things Sam has projected about the future. You're listening to Foundering. I'm your host, Ellen Hewitt. By the time OpenAI had been around for a few years, its employees began sensing that they were working on something profound. AI was starting to do things even its creators didn't quite understand. And the response divided people. Some people at OpenAI, including Sam, argued that an all powerful computer intelligence would be good for the world. Some thought it could be very, very bad. These visions were extreme, polarizing, as different as heaven and hell. This period ultimately led up to the release of ChatGPT, which was a defining moment for the and for the entire AI complex. The two predictions people talked about the most during this time were that AI would either wipe out the human race or completely change the definition of work. The first one is hugely controversial. The second one is less controversial. People will lose jobs and the economy will have to adapt. I'm going to tell the story story in two parts. First, the disputes within OpenAI, the reckoning with what they'd built, and the elevation of Sam as some kind of AI hero. In the second part, we'll hear more from Sam's sister and explore the debate around poverty and what an AI economy might look like. And you'll see exactly how far away Annie's life is from Sam's vision of AI utopia. We'll be right back.

[Ellen Hewitt]

Success is more than the final destination. It's a path you take one step at a time. It's discipline, it's teamwork, and it's the drive and passion inside of us that comes before all recognition. It's what Stifel's been doing for over 130 years. Quietly yet strategically, Stifel has become one of the fastest growing wealth management and investment banking firms in the country. Our financial advisors go beyond traditional wealth management to provide clients with direct access to one of the industry's largest equity research franchises and a leading middle market investment bank. Because success is the drive it takes to keep climbing, the passion to keep investing the best of each of us made better by the best in all of us. And that is where success meets success. Start your [e-mail] that's s t I.

[Ron Krushevsky]

F e l.com Stifel Nicholas & Co. Incorporated member SIPC and NYSE wings nuggets.

[Ad]

Eggs no matter the form, Americans love their chicken.

[Matt Levine]

The chicken industry is one of the.

[Ron Krushevsky]

Largest and most complex supply chains that America has.

[Matt Levine]

These birds are big business and we.

[Ron Krushevsky]

Wanted to get to the bottom of.

[Ad]

Welcome to Beat Capitalism brought to you by Odd Lots.

[Matt Levine]

In this special three part series from.

[Ad]

Bloomberg Podcast, we are going to examine some of the thorniest issues facing the U.S. economy through the medium of this humble bird.

[Matt Levine]

Examine. Get it? So there's going to be chicken puns. There are definitely going to be chicken puns. We're going to be asking why the chicken industry has evolved the way that it has and what is its say about the American economy that so many consumers are flocking to poultry. There's another one for you.

[Ron Krushevsky]

Listen to Beat Capitalism from Odd Lots out now on Apple Podcasts, Spotify or.

[Matt Levine]

Wherever you get your podcasts. Let's pick this story up in 2021. It's about a year and a half before OpenAI launches ChatGPT and OpenAI's technology had been rapidly gaining new capabilities.

[Katie Greifeld]

This was around the time that OpenAI was building more and more powerful models such as GPT2 and GPT3 that could ingest text and spit out ever more human sounding text.

[Matt Levine]

That's my colleague Rachel Metz who covers AI.

[Katie Greifeld]

They were focused on building increasingly powerful AI models, giving it very rapidly moving toward this goal that they have long stated of creating artificial general intelligence.

[Matt Levine]

This rapid progress was good. It was great for Sam, great for the company. But it made some OpenAI employees nervous, even ones very high up. Here's Dario Amadei who at the time was the Vice president of research at OpenAI he's talking on the Logan Bartlett podcast and he sounds genuinely agitated.

[Sam Altman]

So, you know, I really freaked out about this stuff in 2018 or 2019 or so. The first time I looked at GPT2.

[Ron Krushevsky]

I was like, oh, my.

[Sam Altman]

This is like, this is crazy.

[Matt Levine]

This is.

[Sam Altman]

You know, there's. There's nothing. Like, there's nothing like this in the world. Like, it's crazy that this is possible.

[Matt Levine]

He looked at what he was helping build at OpenAI, and his overwhelming emotion was fear.

[Sam Altman]

Why am I scared? Okay, this thing's going to be powerful. It could destroy us. And like, all the ones we've built so far, like, you know, are at.

[Ron Krushevsky]

Pretty decent risk of doing some random.

[Sam Altman]

Shit we don't understand. If such a model wanted to wreak havoc and destroy humanity or whatever, I think we have basically no ability to stop it.

[Matt Levine]

That's from a tech show called the Dwarkesh Podcast. You can hear Dario's uneasiness, which spread to some of his peers at work. In 2021, he and six of his OpenAI colleagues all left at once, and they started a new rival company focused on building AI that was safe.

[Sam Altman]

We need to make these models safe in a certain way, and we need to do them within an organization where we can really believe that these principles are incorporated top to bottom.

[Matt Levine]

While his tone sounds innocuous, if you read between the lines, he's implying that OpenAI was not that concerned with safety and not following those principles top to bottom. He and his co founders felt they needed to create a whole new company focused on preventing AI from potentially wiping us all out. This idea was baked into their name, Anthropic, which refers to the existence of humanity on Earth. Anthropic's split from OpenAI was a big deal. The group of employees who walked away included executives and key people who had worked there for a long time. It cast this suspicious pall over OpenAI. Their exit suggested that they didn't trust Sam to do the right thing. So fast Forward to late 2022. OpenAI released ChatGPT. They weren't expecting a huge response, but it drew people in gradually. Then suddenly everyone was trying it out. Here's Rachel again.

[Katie Greifeld]

I kept an eye on it for a few days and I started noticing people were talking about it, but it started getting more and more traction. And I remember reaching out to my editors and saying, hey, we should write about this.

[Matt Levine]

ChatGPT wasn't new technology. It was basically powered by GPT 3.5, a pre existing model. But it was a new way of presenting that technology to the world. It was a free, easy to use chat based AI tool. It felt a little like texting a friend.

[Katie Greifeld]

This was much more fluid feeling. It gave you a sense that you were communicating with a person. A person that would make a lot of things up, right? That whose answers you would have to constantly check.

[Matt Levine]

And people were hooked. ChatGPT reached 100 million users in just two months, the fastest growth ever at the time. For reference, it took TikTok nine months and Instagram two and a half years to reach the same popularity. ChatGPT felt intelligent even when it got things wrong. It prompted a lot of people to ask themselves, how is AI going to change my life? The technology had some immediate practical uses. People started using ChatGPT to write code more quickly, to translate documents more fluidly and to draft emails. Students used it for homework help. Good. And to cheat on papers. Not so good. Needless to say, all this excitement was a huge boost for OpenAI.

[Katie Greifeld]

With ChatGPT, OpenAI basically skyrocketed to brand awareness and success. And people immediately started thinking of it as the hot AI company, a leader in AI.

[Matt Levine]

It also skyrocketed Sam's public profile. Now AI was the story of 2023 and Sam was the main character. He became a household name. And across the tech industry, investors were desperate to throw money at AI. Companies pivoted their focus to AI and highway billboards touted new AI startups. And this ChatGPT mania did something else too.

[Katie Greifeld]

In addition to making all kinds of people interested in and aware of cutting edge AI, the release of ChatGPT inflamed this small subsection of people that have long been interested in this idea that AI is going to become more and more powerful and that it will inevitably go rogue and threaten us. And we have to figure out how to save humanity from it.

[Matt Levine]

Fear of AI destroying the world. It's the fear that Dario described feeling before he quit OpenAI to found Anthropic. It might sound a little ridiculous, but to many people it's dead serious. This Silicon Valley subculture that believes AI might destroy us soon has steadily been growing more influential and powerful. These ideas are a major motivating force in AI right now. Many tech workers in AI feel that they are working on the most consequential thing possible for the human race. A matter of life and death for the whole planet. This belief influences where billions of dollars are directed and which problems get worked on and which ones get ignored. ChatGPT brought these fears into the public consciousness. Big time. One survey in the UK shows that in the course of one year right after ChatGPT's release, the percentage of people who believed that AI was a top possible cause of human extinction more than doubled. To be clear, this claim is hotly disputed. A lot of people, even in Silicon Valley, think AI doomerism is overblown or a quasi religious ideology. But many take it seriously. We'll be right back.

[Ellen Hewitt]

Hi, I'm Ron Krushevsky, Chairman and CEO of Stifel Financial Advisors. If you're not growing your practice, you're losing market share. Stifel is a growing entrepreneurial advisor centric firm built for successful advisors like you. Imagine having the resources of the largest wire houses and the support of the boutique shops, but none of the bureaucracy to get in the way of you serving your clients. At Stifel, it's your business, your book, your clients. I always tell the advisors we're recruiting, I want you to come to Stifel and double or triple your business. Most of them laugh and shake their heads. But I'm serious. Don't take it from me. Take it from Stifel's number one finish in J.D. power's 2023 U.S. financial Advisor Satisfaction Study. So there's a reason why 148 financial advisors joined Stifel last year. Come join us and find out why Stifel is the firm where success meets success. Visit www.choosestif.com Stifel Nicholas & Co. Inc.

[Ron Krushevsky]

Member SIPC and NYSE Are you looking.

[Matt Levine]

For a new podcast about stuff related to money? Well, today's your lucky day. Hi, I'm Matt Levine.

[Katie Greifeld]

And I'm Katie Greifeld and we are.

[Matt Levine]

The hosts of Money Stuff, the podcast. Every Friday we dive into the top stories about Wall street finance and other stuff. We have fun, we get weird and.

[Katie Greifeld]

We want you to join us.

[Matt Levine]

You can listen to Money Stuff, the podcast on Apple Podcasts, Spotify, or wherever you get your podcasts. I want to talk about how it feels to believe AI might destroy humanity. Here's Chao Chu Yuan. He's just one of many people in tech who held this belief. His introduction to this world was bizarrely through Harry Potter fan fiction he found online that was written by a very influential AI Doomer named Eliezer Yudkowsky. That discovery led him to lots more of Eliezer's writing about AI. He eventually became convinced that the world was probably going to be swallowed by superintelligent AI Soon.

[Ron Krushevsky]

I was like, okay, it's happening. We're getting there. It's going to happen in my lifetime. It's going to happen before I retire. I made a conscious decision around this time to not open a retirement account, which some people might think was silly. But I was like, I really do not think money is going to matter by the time I hit retirement age. I think by the time I hit retirement age, the world is going to be unrecognizably strange. I just don't think money is going to be a thing. Either we're all going to be dead or we're going to be in some crazy post singularity environment where it's just. I just don't think it's going to matter.

[Matt Levine]

The singularity is the point when robots become smarter than humans. For him, it went beyond skipping out on a retirement account. Once Chia Chew really started believing this, other things in his life just didn't seem to matter. He dropped out of his math PhD program. Five years in, he drifted away from old friends. He thought they didn't understand the magnitude of the situation.

[Ron Krushevsky]

It was like this black hole in my kind of sense of which things were important and sort of the closer I got to it, the less important everything else seemed. I was like, oh, well, there's this other stuff that doesn't really matter. There's this other stuff that doesn't really matter. And this is like maybe the only thing that matters.

[Matt Levine]

I know this sounds scary, extreme and maybe ridiculous, but if you spend enough time talking to people in AI and around Silicon Valley, you will hear this attitude. Sometimes it's whispered, sometimes it's said loudly. Sometimes they'll use terms like AI safety, AI alignment, or AI existential risk. But they're all talking about the same thing. There are significant numbers of influential people in AI who believe that our world in 20 years will be utterly unrecognizable from the world of today. Some people think it'll be way better, but a lot think it might be way worse. And they feel that they should devote their lives to preventing AI do doomsday. In the last five or 10 years, a lot of famous, rich tech people have thrown money at this new cause. Facebook co founder Dustin Moskovitz and crypto fraudster Sam Bankman Fried each pledged hundreds of millions of dollars toward AI safety projects. Tech leaders have signed public statements about how AI could put our human civilization at risk. Chow Chew and his fellow adherents felt it was deeply important to get other people to accept this threat. They became evangelists at events. They would trap people in long conversations about AI apocalypse.

[Ron Krushevsky]

And they Started off skeptical, and then they ended up somewhere between either really convinced or scared of just like, oh, oh, this is terrifying. This is really. I'm kind of up about this. This is a very frightening idea. It's like, oh, hey, what if everyone dies in 10 years? That's a scary idea. And like, it's, it's. It's sort of safer psychologically to just dismiss it as, like, that's ridiculous. That can never happen. But if someone really, like, gets. Gets to you about that, like, if someone, you know, spends four hours talking to you and very carefully convincing you, like, what if it's possible, though? Like, that's really. That's very scary. That's like, oh, that. How do I. Oh, like, that's like terrify. Like, what do I do about that? There's a world of difference between, like, reflexively dismissing that idea and, like, really considering it seriously as a possibility on an emotional level.

[Matt Levine]

Cha Chu took jobs at AI safety organizations. He volunteered at rationality workshops to try to get more young minds working on AI safety. In this world, when people talk about what's at stake, it's on a galactic scale. Okay, so that's one point of view that the most important thing right now is to make sure AI is safe in the future. But there are some really smart people, academics and researchers, who find this AI doomsday frenzy very frustrating and harmful. Like Emily Bender, an academic who specializes in computational linguistics.

[H]

It's a distraction because there's all kinds of harms that are happening right now in terms of labor exploitation, in terms of data theft, in terms of discriminative outcomes, in terms of representational harms. And the more we focus on these very sort of exciting in the way that action movies are exciting fantasy scenarios of existential risk, the less time and effort goes in to actually dealing with the real harms that are happening right now.

[Matt Levine]

Emily and lots of other experts argue that, that all of this talk of AI doomsday is an enormous, dangerous distraction, like a big flashing alarm screaming, we're all going to die. It steals all the air in the room. It makes it easy to ignore issues happening today, like racial bias in AI systems used in criminal justice, stealing copyrighted work of artists to train models, and so many more.

[H]

You've got the surveillance applications of these technologies that are being used to over police communities, poor performance of facial recognition technology, especially for women with darker skin, in particular. Google and putting identity terms up for sale. If you search for black girls, you get a whole bunch of porn. There's enormous risks around synthetic media. All of our ability to find trustworthy information and then trust it when we find it is threatened.

[Matt Levine]

Public health, democracy, the list goes on and on. Emily reminds us that these are urgent problems, ways that people are being harmed by AI right now.

[H]

And yet we are wasting our time talking about these fantasy scenarios because the people with all the money decided to get worried about it. In the meantime, there's the doomer stuff. The existential risk stuff is also another kind of AI hype. Because if these systems are so powerful they might destroy the world, then these systems are really powerful.

[Matt Levine]

I think that hype is a key part of this because a super powerful AI system that's exciting to investors and to employees. Emily thinks AI apocalypse beliefs are harmful and misleading, but she also thinks many of the people concerned with AI safety are coming from a sincere place.

[H]

But it does seem to be a genuine belief of some of these folks. The people who get deep into this set of beliefs about super intelligent AI taking over the world see themselves as the heroes in their stories who are going to work to stop that. Like, I think it's genuine, I think it's misplaced, but I think it's genuine.

[Matt Levine]

I think Emily is hitting on something real here, that this urge to be a hero becomes a motivating force in the AI industry. Here's Chao Chu again.

[Ron Krushevsky]

There is this kind of very boyish desire to be a hero. Most people don't get a chance to do that in any meaningful sense. Not only just, what if I could be a hero, but oh, what if I could smart my way to being a hero? I think it'll be difficult for some people to admit this because it's kind of. It feels immature or something, but I really do think that is part of the appeal of the pitch.

[Matt Levine]

Being a hero by using your brain. Cha Chu is saying the quiet part out loud that many people in this field are motivated by this pride. They want to feel like they're important and that their work is cosmically significant.

[Ron Krushevsky]

People want to feel like they made a difference. Like, people want to feel like their lives matter. And that's a huge part of the hook. It's like, what if this is like the most significant era of human history that there ever has been and, like, the choices we make now are going to reverberate into the future. You know, like people on Twitter who've started, like, getting really into this stuff will use this, like, very hyperblock language, like, oh, yeah, we're going to like, conquer the stars. You know, we're going to, like, go create a galactic civilization. Like, I think some people think that those guys are exaggerating and joking. Like, I think they should be taken exactly at face value. Like, literally. That's what people think.

[Matt Levine]

The stakes are believing that this is the most significant era in human history because of what we're building with AI. You know who that sounds like? Sam. In speech after speech over the years, he has said that the AI we're working on now is going to be historically significant. Here he is with my colleague Emily Chang.

[Sam Altman]

I think we have an opportunity that comes along only every couple of centuries to redo the socioeconomic contract. And how we include everybody in that, make everybody a winner, and how we. We don't destroy ourselves in the process is a huge question. You know, what does it mean to build something that is more capable than ourselves? Like, what does that say about our humanity? What's that world going to look like? What's our place in that world? How is that going to be equitably shared? How do we make sure that it's not like a handful of people in San Francisco making decisions and reaping all the benefits?

[Matt Levine]

He says that the AI being built now will reshape the world, our humanity, our social contract. His words are brimming with the heroism that Emily and Chow Chew are pointing out. It's pretty common for people in tech to think they are uniquely smart enough to fix big hairy problems, like the tech billionaires who are creating a new city near San Francisco. Or the entire crypto industry, which thinks it has invented a superior financial system. Maybe Sam sees himself as a hero and he understands the motivating power of a good story. In 2019, he even hired a fiction writer on contract to write for OpenAI for a few months.

[Ron Krushevsky]

I ended up basically writing a novella.

[Matt Levine]

Full of short stories, science fiction short stories. That's Patrick House. He's a neuroscientist and an author. He says he has no idea whether OpenAI still uses his novella in any way, but they saw value in commissioning it. Sam Altman is influenced by certain kinds of fiction. A lot of startups are kind of motivated by story and that the story often comes from science fiction. A powerful story can make your employees work really hard. How do you motivate people? How do you motivate people, especially in.

[Ron Krushevsky]

The mostly secular San Francisco?

[Matt Levine]

Maybe give them a foundational document and an apocalypse myth and tell them they're.

[Ron Krushevsky]

Averting the end times.

[Matt Levine]

And that's a tried and true, historically known way to motivate people. All of this made me wonder. Does Sam believe that AI might destroy humanity? Honestly, it's hard to tell. His answers have changed over the years. In 2015, he very clearly wrote that he believes advanced AI is, quote, probably the greatest threat to the continued existence of humanity. We heard him say in the past, with a kind of funny tone, I.

[Sam Altman]

Think AI will probably like, most likely sort of lead to the end of the world.

[Matt Levine]

And around that same time, in a New Yorker profile, Sam basically said he was a doomsday prepper. He told a reporter that he had stockpiled guns, gold, ammonium, antibiotics and gas masks from the Israeli Defense Force. He said he has a big patch of land in Big Sur that he can fly to in case the world crumbles. This was really funny to me and probably to anyone who has visited Big Sur because the land out there is literally crumbling right now, leading to rock slides, road closures, and difficulty getting food and supp flies in and out. Seems like a precarious place for your apocalypse bunker. When another reporter asked him follow up questions, Sam tried to blow it off. He said that he does this stuff for fun because it, quote, appeals to little boy survival fantasies. In the years since then, Sam has notably avoided mentioning the Big Sur property or his stockpile of guns. Annie, his sister, says she never saw the property, but that it's in line with what she knows about him.

[Annie Altman]

He's big into that. Or was safety like guns, gold, how to stock up on worst case scenario apocalyptic movie scene events. My guess would be that he has it and stop talking about it.

[Matt Levine]

Sam is a savvy guy. As his profile has gotten bigger after he helped build the world's leading AI company, he has stopped saying things like AI will kill us all. Instead, he talks about how society will be profoundly changed, but overall it will be for the better. Since his newfound ChatGPT fame, he has shifted toward presenting himself and by extension OpenAI as more middle of the road. Sam is allowed to change his views, but people have also complained to me in private that Sam has a tendency to talk out of both sides of his mouth. He's good at telling people what they want to hear in that moment. So it's not surprising that if it's advantageous for him to seem more moderate, that he would start to sound that way. In any case, he definitely knows how powerful an explosive apocalypse story is. We trigger more easily on dramatic fears than boring ones. Sam says this himself. Here he is speaking at an Airbnb conference in 2015. He's talking about nuclear power but it's an analogy that maps very clearly onto AI existential risk. And you can hear, again, Sam's undertone of arrogance pointing out how he's aware of something rational that other people are blind to.

[Sam Altman]

People are much more sensitive to sort of like theatrical extreme risk than they are to sort of like boring, slow, plotting risk. Like, nuclear energy has had an unbelievable safety record. You know, it's like 1,000 or 10,000 times safer than coal. But most people, would they rather live next to a coal plant or a nuclear energy plant? They pick the coal plant all day long. And when they die in 30 years of lung cancer, it doesn't feel as sort of dramatic as dying in a nuclear meltdown. And no, it's true. Like, this is, this is like the human risk miscalculation that always happens, right? People always underweight the boring, slow stuff and overweight the quick dramatic stuff.

[Matt Levine]

People underweight the boring, slow stuff, like misinformation, racial bias, and they overweight the big dramatic stuff. That may be why the AI industry has been telling us disaster stories. You've just listened to part one of Heaven and Hell. Listen to part two for the rest of this story. Foundering is hosted by me, Ellen Hewitt. Sean Wen is our executive producer. Rachel Metz contributed reporting to this episode. Molly Nugent is our associate producer. Blake Maples is our audio engineer. Mark Millian, Ann Vandermay, Seth Vigerman, Tom Giles and Molly Schutz are our story editors. We had production help from Jessica Nix and Antonia Mufarec. Thanks for listening. If you like our show, leave a review and most importantly, tell your friends. See you next time.

[Ad]

In today's data driven world, you need real data to shape an effective marketing strategy. That's where Ahrefs comes in. Ahrefs is a marketing intelligence platform powered by big data. It shows you exactly what your target audience is searching for and how you measure up to your competitors. Whether you're making day to day marketing decisions or defining your long term digital strategy, Ahrefs delivers real time insights that help you grow your brand. Head over to ahrefs.com to get started. That's a h r e f s.com.

[Matt Levine]

This podcast is supported by BetterHelp, offering licensed therapists you can connect with via video phone or chat. Here's BetterHelp head of clinical operations, Hes Yu Jo discussing who can benefit from therapy. I think a lot of people think that you're supposed to be going to therapy once you're like having panic attacks every day. But before you get to that point, I think once you start even noticing that, you feel a little bit off and you can't maintain this harmony that you once had in relationships, that could be a sign that maybe you want to go talk to somebody. There's always a benefit in talking to someone because we can all benefit from improved insight about ourselves and who we are and how we behave with other people. So if you're human, that's like a good indicator that you could benefit from talking to somebody. Find out if therapy is right for you.

[Ad]

Visit betterhelp.com today.

[Matt Levine]

That's BetterHelp combination.