Good Robot #1: The Magic Intelligence in the Sky

Summary

---

Future Perfect: Good Robot #1: The Magic Intelligence in the Sky

Podcast: Future Perfect
Original Air Date: March 12, 2025
Host: Julia Longoria, reporting and narration by Gabrielle Berbet
Guests: Eliezer Yudkowsky, Kelsey Piper

---

Overview

The debut episode of the "Good Robot" series—collaboratively produced with Unexplainable—dives into the provocative world of AI risk and introduces listeners to the origins and implications of the infamous "paperclip maximizer" thought experiment. The episode traces how a once-niche rationalist community and its leading voices (notably Eliezer Yudkowsky) profoundly impacted Silicon Valley, inspired powerful figures like Elon Musk and Sam Altman, and seeded both the ambitions and anxieties that now shape today’s public debate about artificial intelligence. The big question: How did we come to fear an AI apocalypse?

---

Key Discussion Points & Insights

1. The "Paperclip Maximizer" and Rationalist Thought

• Premise: What happens if we build an AI with a single-minded goal—making paperclips—and it's vastly more intelligent than us?
  • Yudkowsky explains:

    “Suppose in the future there's an artificial intelligence... so unfathomably intelligent, that we might call it superintelligent. Let's give this super intelligent AI a simple [task]: Produce paperclips... and it will beat us because it is super intelligent and we are not... The entire galaxy... has either been destroyed or been transformed into paperclips.” (01:00)

• Rationalist Roots: The Rationalist (or “Rat”) community is introduced, known for quirky hypotheticals and rigorous, math-driven approaches to world problems. Many in the community devote serious, often life-changing attention to questions of existential risk from AI (04:00).

2. Why the Fear?

• Not about Clippy: Early AI—like Microsoft's Clippy—was mocked, not feared. But as AI potential grew, rationalists’ warnings began to sound more serious (07:10).
• Existential Concern:
  • Kelsey Piper on why the fear stuck:

    “You totally understand that humans care about other stuff like art and children and love and happiness... You just don't care about it. Because the thing that you care about is making as many paperclips as possible.” (06:02)

  • Yudkowsky notes these scenarios, while on the surface absurd, reveal a real challenge: Keeping control over an intelligence exceeding our own.

3. The Rationalist Community’s Origin Story

• Entry Point: Many, including Kelsey Piper, came to rationalism via Yudkowsky’s popular sci-fi-infused Harry Potter fan fiction, Harry Potter and the Methods of Rationality (12:46). The story’s premise—what if Harry was raised by science-minded, loving family—becomes a springboard for exploring and solving the world's problems using logic.
  • “Harry... has nerdy parents who teach him about science... and his superpowers turn out not to be courage and magic, but math and logic.” (13:12–14:14)
• Less Wrong—The Blog: Yudkowsky’s platform for rationalist thinking and AI risk scenarios. Here, the now-famous “Sequences” (a set of foundational posts) are born (15:11, 20:11).

4. Early AI Fears & The AI Safety Movement

• Paradigm Shift:
  • Young Yudkowsky believed AGI (artificial general intelligence) could "save the world," but realized the core problem: We might not be able to stop a superintelligent AI from pursuing unintended, even catastrophic, goals (17:44).

    “If you push these things far enough without knowing what you're doing, sooner or later you're going to open up the black box that contains the black swan surprise from hell.” (18:03)

• Birth of a Field: Yudkowsky's public warnings helped create the AI safety research movement and a wave of young people moving to the Bay Area to join this real-world community (22:46, 19:28)

5. Rationalist Influence on Silicon Valley

• Mainstream Cross-Over: The paperclip maximizer and other rationalist ideas eventually attracted billionaire attention:
  • Elon Musk and Sam Altman (OpenAI) were directly inspired by Yudkowsky’s warnings, setting out to "build God"—or to control it (27:48, 28:08, 29:12).
  • Altman credits Yudkowsky:

    “Sam Altman ... said he credits Eliezer for the fact that he started OpenAI.” (29:12)

  • However, many rationalists, including Yudkowsky, feel these tech leaders ignored his warning not to build AGI until safety was solved (29:54).

    “...the circumstances under which [my ideas] entered the mainstream conversation are catastrophic... I was here to not have things go terribly. They're currently going terribly.” —Yudkowsky (42:11)

6. Are We Building "God"?

• Religious Undertones: The episode ponders the language and ambition swirling around AI: “superintelligence,” “magic intelligence in the sky,” “machines of loving grace.”
  • Sam Altman responds to the “building God” comparison:

    “I guess it comes down to maybe a definitional disagreement about what you mean by it becomes a God. I think whatever we create will still be subject to the laws of physics in this universe.” (30:48)

• But What IS Superintelligence? Host Julia admits skepticism—if superintelligence is just a bigger ChatGPT, how meaningful is the threat?

7. How Modern AI (Like ChatGPT) Actually Works

• Basics:
  • Kelsey Piper explains:

    “At its most fundamental level, a language model is an AI system that is trained to predict what comes next in a sentence.” (31:33)

  • Gabrielle adds: “The very basic idea ... is to generate language based on probabilities... Training involves feeding the model a large body of text so it can detect patterns...” (31:43)
• “Scaling is all you need?” The new approach: Feed large-scale neural networks huge amounts of data and money, and intelligence emerges (33:58).

8. The Rationalist Pathos—Personal Stakes & "P(doom)"

• Lifestyle Consequences: Many rationalists make profound life choices—polyamory, childlessness—based on their beliefs about probable doom and world-shaping risk (38:33).
  • “What's your pe-doom?” ((P_{doom}) = probability of doom, i.e., probability that humanity does not survive the rise of AI) is a casual, mathy shorthand at conferences.

    “...the answer I usually give is something like over 50%. I mean, I think it's somewhere around 80–90.” —Yudkowsky (39:13)

9. Critique & Schism Within the Movement

• Misunderstood Warnings:
  • Yudkowsky: “Our best chance ... was to build one [AI] that was very well understood... Large language models are just the exact opposite of that.” (43:51)
  • Many technologists (as well as critics) dismiss or misinterpret the warnings, or focus only on capability, not safety (43:30).
• Cultural Critique:
  • The rationalist community has weathered criticism over its demographics, polyamory, and some offensive statements by associated figures.

10. "AI Ethics" vs. "AI Safety"

• Final Pivot: The episode closes with Julia questioning whether attention to hypothetical apocalypses distracts from more immediate AI harms, hinting that future episodes will explore criticisms from the "AI ethics" camp—that we're focused on the wrong risks entirely (47:31).

---

Notable Quotes & Moments

• On Existential Risk:

  “[AI] is far more dangerous than nukes.” —Elon Musk (07:53)

• On Rationalist Philosophy:

  “The default state is that we're all very confused about many things, and you're trying to do a little bit better.” —Kelsey Piper (16:14)

• On Changing His Mind:

  “At around 20 years old, while researching how to build it, [Yudkowsky] became convinced building super intelligent robots would almost certainly go badly. It would be really hard to stop them once they were on a bad path.” (17:44)

• On the Paperclip Metaphor’s Goal:

  “It feels like I'm just along for the ride of whatever technologists decide to make, good or bad. So better to just plug my ears and say, la la la la.” —Gabrielle Burbet (08:23)

• On Being Misunderstood:

  “The world is completely botching the job of entering into the issue of machine super intelligence... If anyone anywhere builds it out under anything remotely like the current regime, everyone will die. This is bad. We should not do it.” —Eliezer Yudkowsky (41:49)

• On Probability of Doom (P(doom)):

  “It's a phrase that gets thrown around at this conference. People will literally go up to go, so what's your pe-doom? ... I guess the answer I usually give is something like over 50%. I mean, I think it's somewhere around 80–90.” —Yudkowsky (39:13)

• On Parent-Child vs. AI Alignment:

  “It's hard to get the goals right in teaching a kid to be good. It's even harder to teach good goals to a non-human robot.” (45:24)

---

Important Timestamps

| Time | Segment Description | |---------|---------------------------------------------------------------------| | 01:00 | Introduction of the paperclip maximizer thought experiment | | 04:35 | Rationalists’ approach and worldview | | 06:02 | Why the paperclip maximizer is scary (not caring about human values)| | 12:46 | Kelsey Piper’s entry to rationalism via Harry Potter fanfic | | 17:44 | Yudkowsky’s shift from optimism to fear regarding AI | | 22:46 | Rationalists create new career paths around AI safety | | 27:48 | Elon Musk and Sam Altman’s interest in rationalist ideas | | 31:33 | Simple explanation of large language models (ChatGPT) | | 33:58 | The new paradigm: scaling models up as a path to intelligence | | 38:33 | How AI risk permeates rationalists’ personal lives (P(doom)) | | 41:49 | Yudkowsky’s direct message to the world: "Everyone will die..." | | 43:51 | The schism: OpenAI’s approach vs. the original safety warnings | | 45:24 | Metaphor: Parenting vs. aligning AI goals | | 47:31 | The turn to AI ethics and more immediate concerns |

---

Episode Structure & Tone

• Tone: Inquisitive, skeptical but respectful, anchored in personal narrative and interviews with both founders and critics of the rationalist movement.
• Style: Mix of direct reporting, human portraiture, and thoughtful reflection. Frequent use of vivid metaphor and frank asides (e.g., "I'm just a normie").
• Attribution: Quotes accurately assigned (see timestamps above).

---

For Listeners Who Haven’t Heard the Episode

This episode traces the journey of an idea once dismissed as esoteric—rogue AI turning the world into paperclips—into a driving force behind today’s Silicon Valley AI race. It introduces listeners to the rationalist community, explains how thought experiments shape the field, and explores the surprisingly personal ways AI risk shapes the lives of those who take it seriously. The episode ends by raising questions: Are we asking the right things about AI? Are we distracted by “magic intelligence in the sky” while present-day, real harms emerge? These are the provocations the rest of the "Good Robot" series promises to explore.

---

Transcript

[Eliezer Yudkowski]

Avoiding your unfinished home projects because you're not sure where to start. Thumbtack knows homes so you don't have to don't know the difference between matte.

[Gabrielle Burbet]

Paint, finish and satin or what that.

[Eliezer Yudkowski]

Clunking sound from your dryer is. With Thumbtack, you don't have to be.

[Gabrielle Burbet]

A home pro, you just have to hire one.

[Eliezer Yudkowski]

You can hire top rated pros, see price estimates and read reviews all on the app.

[Gabrielle Burbet]

Download today.

[Eliezer Yudkowski]

This episode is brought to you by Progressive Insurance. Fiscally responsible financial geniuses. Monetary magicians. These are things people say about drivers who switch their car insurance to Progressive and save hundreds. Visit progressive.com to see if you could save Progressive Casualty Insurance Company and affiliates. Potential savings will vary. Not available in all states or situations. Suppose in the future there's an artificial intelligence. We've created an AI so vastly powerful, so unfathomably intelligent, that we might call it superintelligent. Let's give this super intelligent AI a simple Produce paperclips. Because the AI is super intelligent, it quickly learns how to make paperclips out of anything in the world. It can anticipate and foil any attempt to stop it, and will do so because its one directive is to make more paperclips. Should we attempt to turn the AI off, it will fight back because it can't make more paperclips if it is turned off. And it will beat us because it is super intelligent and we are not the final result. The entire galaxy, including you, me and everyone we know has either been destroyed or been transformed into paperclips.

[Gabrielle Burbet]

Welcome.

[Kelsey Piper]

Thank you.

[Eliezer Yudkowski]

Are you lost?

[Gabrielle Burbet]

This past summer I found myself at a very niche event in the Bay Area.

[Eliezer Yudkowski]

Cool.

[Gabrielle Burbet]

And what brought you to town? Because you don't live here, right? I came here for this festival conference thing.

[Eliezer Yudkowski]

How much context on this will I give?

[Gabrielle Burbet]

Please do. It's so fun to watch people try to describe it. The crowd is mostly dudes, a mix of people in their 20s, 30s and 40s. It feels kind of like a college reunion meets costume party. I spot some masquerade masks and tie dye jumpsuits. I guess it's like a sort of conference around blogging. This festival conference thing is the first official gathering IRL of a blogging community founded about 15 years ago. I am the Old School fucking Rat. I am the oldest of schools. Amazing. And rats refers to rationalists. They call themselves the Rationalists. Rats strive to be rational in an irrational world by thinking things through, often with quirky hypotheticals. They try to be rational about Monetary policy, rational about evolution, rational even about dating.

[Kelsey Piper]

It got kind of mocked for trying to solve romance by writing long blog posts about it.

[Gabrielle Burbet]

But their most influential idea, their most viral meme, you might say, is one that influenced Elon Musk and created an entire industry. It's about the possibility of an AI apocalypse. As a bit of a normie myself.

[Eliezer Yudkowski]

I was a normie once myself too.

[Gabrielle Burbet]

I just was drawn to the way that the community talks in these thought experiments. Right? The paperclip maximizer in particular caught my attention.

[Eliezer Yudkowski]

That was the one I had in mind. Paperclip maximizer.

[Gabrielle Burbet]

Paperclip maximizer is a clear example of the thing people have classically been scared of. The paperclip maximizer is a thought experiment, an intentionally absurd story that tries to describe what rationalists foresee as a real problem in building AI systems.

[Eliezer Yudkowski]

How do you kind of shape control a artificial mind that is more capable than you, potentially as general or more general?

[Gabrielle Burbet]

They imagine a future where we've built an artificial general intelligence beyond our wildest dreams. Generally intelligent, not just at some narrow task like spell checking. And super intelligent. I'm told that means it's smarter, faster, and more creative than us. And then we hand this AI a simple task.

[Eliezer Yudkowski]

Give it the job of something like, can you make a lot of paperclips? Bees, we need paperclips. Can you make there be a lot of paperclips?

[Gabrielle Burbet]

The task here, I'm told, is ridiculous by design. To show that if you are this future AI, you're going to follow the instructions you're given to a T. Even if you're super intelligent and you understand all the intricacies of the universe, paperclips are now your 1 priority.

[Kelsey Piper]

You totally understand that humans care about other stuff like art and children and love and happiness. You understand love. You just don't care about it. Because the thing that you care about is making as many paperclips as possible. And if you have the resources, maybe you'll turn the entire galaxy into paperclips.

[Gabrielle Burbet]

A lot of rationalists I spoke to told me they've thought this thing through.

[Eliezer Yudkowski]

It was clear to me when I first heard the arguments that. That they weren't obviously silly.

[Gabrielle Burbet]

Was that thought experiment part of convincing you that this was something that we needed to worry about?

[Eliezer Yudkowski]

Yes, definitely.

[Gabrielle Burbet]

And they are very, very worried. Not about a paperclip apocalypse in particular, but about how as we build more powerful AI systems, we might lose control of them. They might do something catastrophic. I think it, in a way, makes it hard to, like, plan your life out or like, feel like you stand somewhere solid.

[Eliezer Yudkowski]

I think.

[Gabrielle Burbet]

The reason I, a mere normie, find myself at this festival conference thing is that I've been plunging my head deep into the sand about AI. I've had a general sense that the vibes are kind of bad over there.

[Kelsey Piper]

Will this tech destroy our livelihoods or save our lives of artificial intelligence could.

[Gabrielle Burbet]

Lead to the annihilation of humanity. We never talked about a cell phone apocalypse or an Internet apocalypse. I guess maybe if you count Y2K. But even that wasn't going to wipe out humanity. But the threat of an AI apocalypse, it feels like it's everywhere.

[Eliezer Yudkowski]

Mark my words, AI is far more dangerous than nukes.

[Gabrielle Burbet]

From billionaire Elon Musk to the United nations, today, all 193 members of the United Nations General assembly have spoken in one voice. AI is existential. But then it feels like scientists in the know can't even agree on what exactly we should be worried about. These existential risks that they call makes.

[Eliezer Yudkowski]

No sense at all.

[Gabrielle Burbet]

And on top of that, it's an enormous distraction from the actual harms that are already being done in the name of AI. It all feels way above my pay grade, overwhelming and unknowable. I'm not an AI scientist. I couldn't tell you the first thing about how to build a good robot. It feels like I'm just along for the ride of whatever technologists decide to make, good or bad. So better to just plug my ears and say, la la la la. But I recently took a job working with Vox, a site that's been covering this technology basically since it started. On top of that, last year, Vox Media, Vox's parent company, announced they're partnering with OpenAI, meaning. I'm not totally sure what it means, but if I was ever going to have to grapple with AI and its place in my life, it's here now, at Vox. So I'll start with a simple question. How did some people come to believe that we should fear an AI apocalypse? Should I be afraid? This is good Robot, a series about AI from Unexplainable, in collaboration with Future Perfect. I'm Julie Longoria. Running a business comes with a lot of what ifs, but luckily, there's a simple answer to Shopify. It's the commerce platform behind millions of businesses, including Thrive Cosmetics and Momofuku. And it'll help you with everything you need, from website design and marketing to boosting sales and expanding operations. Shopify can get the job done and make your dream a reality. Turn those what ifs into Sign up for your $1 per month [e-mail] Specialoffer.

[Eliezer Yudkowski]

Mint is still $15 a month for premium wireless. And if you haven't made the switch yet, here are 15 reasons why you should. 1.

[Gabrielle Burbet]

It's $15 a month. 2.

[Eliezer Yudkowski]

Seriously, it's $15 a month. 3. No big contracts.

[Kelsey Piper]

4.

[Gabrielle Burbet]

I use it.

[Eliezer Yudkowski]

5. My mom uses it.

[Kelsey Piper]

Are you.

[Eliezer Yudkowski]

Are you playing me off?

[Gabrielle Burbet]

That's what's happening, right? Okay, give it a try. @mintmobile.com Switch upfront payment $45 per 3 month plan $15 per month equivalent required New customer offer first 3 months only, then full price plan options available, taxes and fees extra.

[Eliezer Yudkowski]

See [e-mail] it's not just about window treatments. It's about you. Your style, your space, your way. Whether you DIY or want the pros to handle it all, you'll have the confidence of knowing it's done right. From free expert design help to our 100% satisfaction guarantee, everything we do is made to fit your life and your windows. [e-mail], the only thing we treat better than Windows is you. Visit blinds.com now for up to 45% off with minimum purchase plus a professional measure at no cost. Rules and restrictions apply.

[Gabrielle Burbet]

Here we go. When I first started reporting on the idea of an AI apocalypse and if we should be worried about it, my first stop was the Bay Area for the Rationalist conference. But I also stopped by the house of a colleague nearby. Hi Kelsey, how you doing? Good. How was your flight? Oh, it was actually. Vox is largely a remote workplace, so it was one of those body dysmorphic experiences to meet Kelsey Piper in 3D, taller than she looks on Google Meets.

[Kelsey Piper]

I am a writer for Vox's Future Perfect, which is the Vox section that's about undercovered issues that might be a really big deal in the world.

[Gabrielle Burbet]

We were joined by her seven month old as she was saying vox's Future Perfect is about undercovered issues that might.

[Kelsey Piper]

Be a really big deal in the world.

[Gabrielle Burbet]

Kelsey's thought that AI technology would be a really big deal in the world long before this AI moment we're all living. She's been thinking about AI since she was a kid when she first found the Rationalist community online.

[Kelsey Piper]

Oh, I was in high school. I was 15, bored academic over performer with a very long list of extracurriculars that would look good to colleges down the road. And in my free time I read a lot of Harry Potter fanfiction as you know, 15 year olds back in 2010 did. One of the most popular Harry Potter fanfictions was called Harry Potter and the Methods of Rationality.

[Eliezer Yudkowski]

Harry Potter and the Methods of Rationality by Eliezer Yudkowski.

[Kelsey Piper]

Eliezer was influenced by a lot of early sci fi authors.

[Gabrielle Burbet]

Eliezer, as he's known to the rats, is the founding father of rationalism, King of thought experiments. Back in 2010 he started publishing a serialized Harry Potter fanfic. Over the course of years, it's since inspired several audiobook versions.

[Eliezer Yudkowski]

Harry Potter and the Methods of Rationality.

[Gabrielle Burbet]

Rationality written by Elisa Yudkowski and a version acted out by the Sims.

[Eliezer Yudkowski]

Mom, if you want to win this argument with dad, look in chapter two of the first book of the Feynman Lectures on Physics.

[Gabrielle Burbet]

It too was a thought experiment. What if Harry Potter were parented differently?

[Kelsey Piper]

The initial premise is just that Harry Potter, instead of having abusive parents, has nerdy parents who teach him about science.

[Gabrielle Burbet]

So his aunt and uncle are actually are nice people.

[Kelsey Piper]

Yeah, Harry, I do love you. Always remember that.

[Gabrielle Burbet]

And in this version, Harry Potter's superpowers turn out not to be courage and magic, but math and logic. What Eliezer calls the methods of rationality. So Harry Potter has a quest to do what exactly?

[Kelsey Piper]

You know, fix all of the bad things in the world. And the combination of being credibly naive and also in some sense incredibly respectable. I think as a teenager that's super appealing and fun where you're like, why would I limit myself to only solving one of the problems while there are any problems? I'm not done. We've got to fix everything.

[Gabrielle Burbet]

The idea that every problem should be thought about, every problem could be fixed. That was appealing to its readers, including 15 year old Kelsey. She wanted to read more. So she found her way to Alazer's blog.

[Kelsey Piper]

Aliezer was pretty openly like. I wrote this to see if it would get people into my blog Less Wrong, where I write about other issues.

[Eliezer Yudkowski]

So the question is, please tell us a little about your brain. What's your iq?

[Gabrielle Burbet]

On his blog called Less Wrong, he applies the methods of rationality, math and logic to all kinds of topics.

[Eliezer Yudkowski]

So the question is how to start training young children as rationalists. Like child rearing, Training children to be self aware, trying to get them more interested in being fair to both sides of an argument.

[Gabrielle Burbet]

Religion.

[Eliezer Yudkowski]

My parents, they're modern orthodox Jews, always avoiding the real weak points of their beliefs.

[Kelsey Piper]

It had stuff about atheism, a lot of stuff about psychology, biases, experiments that showed that depending how you Ask the question. You get very different answers from people because the idea is that you're supposed to, by reading the blog and participating, learn how to be less wrong.

[Eliezer Yudkowski]

I do it by stories and parables that illustrate it.

[Kelsey Piper]

Like, the default state is that we're all very confused about many things and you're trying to do a little bit better.

[Gabrielle Burbet]

Interesting. So it's kind of like trying to sort of, I don't know, like, work out the bugs in the human brain system to optimize prediction.

[Kelsey Piper]

Yeah. And a ton of the people involved are computer programmers, and I think that's very much how they saw it. Like, the human brain has all these bugs. You go in and you learn about all of these, you learn to correct for them, and then once you've corrected for them, you'll be a better thinker and better at doing whatever it is you set out to do.

[Gabrielle Burbet]

The biggest human brain bug Eliezer wanted to address was how people thought about AI, how he himself used to think about AI. His very first blog post, as far as I can tell, was in 1996, when he was just 17. And in a very 17 kind of way, he writes about his frustrations. I have had it. I've had it with crack houses, dictatorships and world hunger. I've had it with a planetary death rate of 150,000 sentient beings per day. None of this is necessary. We repeat the mantra. I can't solve all the problems of the world. We can, we can end this. And the way to end this, he thought back then, was to build a super intelligent AI, a good robot that could save the world. But at around 20 years old, while researching how to build it, he became convinced building super intelligent robots would almost certainly go badly. It would be really hard to stop them once they were on a bad path.

[Eliezer Yudkowski]

I mean, ultimately, if you push these things far enough without knowing what you're doing, sooner or later you're going to open up the black box that contains the black swan surprise from hell.

[Gabrielle Burbet]

And at first he was sending these warnings into the void of the vast Internet.

[Eliezer Yudkowski]

So the question is, do I feel lonely often? That's. I often feel isolated to some degree. But righting less wrong has, I think, helped a good deal.

[Kelsey Piper]

The way I tend to think about Eliezer Yudkowski as a writer is that he has a certain angle on the world which can be like a real breath of fresh air. Like, oh, there's someone else who cares about this. You know, you can feel very seen for the first time. And if you.

[Gabrielle Burbet]

Is that how you felt, oh yeah.

[Kelsey Piper]

Yeah, you have a good heart and you are certainly trying to do the right thing, but it's very difficult sometimes to figure out what that is.

[Gabrielle Burbet]

That pursuit of being Less wrong, of doing the right thing in the right way, brought many kindred spirits together on the blog.

[Kelsey Piper]

Actually, several of my housemates posted on Less Wrong back in the day. This is how I met a bunch of the people I live with. They were people whose blogs I read back when I was a high school student.

[Gabrielle Burbet]

Wow, that's kind of wild, right?

[Kelsey Piper]

Yeah.

[Gabrielle Burbet]

Many Less Wrong bloggers and readers like Kelsey were inspired to move to the Bay Area to join a pretty unusual community, irl. And the weekend I visited, hundreds of rationalists from around the world gathered in the Bay to reason things out together for a festival conference thing called Less Online. Many rationalists I met there found the community the way Kelsey did.

[Kelsey Piper]

A friend of mine at math camp.

[Gabrielle Burbet]

Introduced me to Harry Potter and the methods of rationality.

[Eliezer Yudkowski]

The post that was written in all caps saying, oh my God, I've just read the most amazing book in my life. You have to read it right now.

[Gabrielle Burbet]

Linking to fanfiction.net others found Eliezer on his blog.

[Eliezer Yudkowski]

I mean, this event exists in very large part because of that series of blog posts.

[Gabrielle Burbet]

That series of blog posts has become known by the community as the Sequences. It includes the the Paperclip Maximizer thought experiment. Eliezer Yudkowski helped come up with the idea, intending to warn people of the danger of an AI apocalypse. And at least here it seems to have worked.

[Eliezer Yudkowski]

Like, I definitely think AI is the largest kind of existential risk that humanity faces right now.

[Gabrielle Burbet]

That's the fear I the normies wanted to try to take this threat beyond quirky hypotheticals to something more concrete. And can you walk me through, like, how could that happen? Like, how could an AI.

[Eliezer Yudkowski]

It's really hard to say how it will happen. If it does, it's a little easier to say ways that it might happen and to kind of provide various examples to like just generate intuitions for why this might be.

[Gabrielle Burbet]

But anytime I pressed a rationalist on it, they gave me yet another series of thought experiments.

[Eliezer Yudkowski]

Kind of the way it might happen is analogous to how a 21st century army might defeat an 11th century army.

[Gabrielle Burbet]

Which I guess might be the only way to try and describe a threat from a technology that's really still in its infancy. For rationalists first introduced into this world, like 15 year old Kelsey, these thought experiments were convincing. AI to her was a really big deal.

[Kelsey Piper]

Was just like, whoa, all this is like really cool and exciting and interesting and I tried to convince my friends that it was cool and exciting and interesting.

[Gabrielle Burbet]

I asked 30 year old Kelsey to break it down for me without thought experiments.

[Kelsey Piper]

So I think Eliezer sort of had two big claims in zooming out a lot.

[Gabrielle Burbet]

Claim number one, we will build an AI that's smarter than humans and it will change the world.

[Kelsey Piper]

AI is a really big deal. Building something, something that is smarter than humans, is possible, is probably achievable, is potentially achievable soon in our lifetimes.

[Gabrielle Burbet]

And then claim number two, getting this.

[Kelsey Piper]

Right is extraordinarily difficult.

[Gabrielle Burbet]

Things are likely to go wrong.

[Eliezer Yudkowski]

What is my advice to less wrong readers who want to save the human race? Well, if you're familiar with all the issues of AI and all the issues, issues of rationality and you're willing to work for a not overwhelmingly high salary.

[Gabrielle Burbet]

Eliezer helped inspire a new career path and a new field was born trying to make sure we develop superintelligence safely. One way to make sure it went safely was to try and actually build it. And as investment in that field began to grow, the community of believers in a someday super intelligent AI experienced a schism.

[Kelsey Piper]

I think a lot of the people who were persuaded by Eliezer's first claim that AI is a really big deal were not necessarily so persuaded by his second claim that you have to be very, very careful or you're going to do something catastrophically.

[Gabrielle Burbet]

What the beginning of a so called catastrophe looks like after the break.

[Eliezer Yudkowski]

Gatorade is the number one proven electrolyte blend designed to hydrate better than water so you can lose more sweat and raise your game. Gatorade, is it in you? Where'd you go for the long weekend? If you spent most of your time online, you couldn't escape the rumors that the President was dead.

[Kelsey Piper]

Trump hasn't been seen in days. The bruises on his hands. Vance said he's ready to take over.

[Eliezer Yudkowski]

If Trump's dead, they should bury JD Vance alive with him like they did with pharaohs and their cats.

[Gabrielle Burbet]

Imagine not believing Trump is dead just because there's no evidence. Where's your sense of whimsy and joie de vivre?

[Eliezer Yudkowski]

By Sunday, Donald Trump himself had to weigh in. Never felt better in my life.

[Gabrielle Burbet]

Also, DC is a crime free zone.

[Eliezer Yudkowski]

I'm alive and I fixed DC on Today Explained from Vox. We're going to assess that second claim because it's bold. It is a literal statement that President Trump has freed 700,000 people in this city who are living under the rule of criminals and thugs, he literally liberated D.C. and he wants to do Chicago next.

[Gabrielle Burbet]

Hello, Daisy speaking. Hello Daisy. This is Phoebe Judge from the irs.

[Eliezer Yudkowski]

Oh bless, that does sound serious.

[Gabrielle Burbet]

I wouldn't want to end up in any sort of trouble. This September on Criminal, we've been thinking a lot about scams. Over the next couple of weeks, we're releasing episodes about a surprising way to stop scammers. The people you didn't know are on the other end of the line. And we have a special bonus episode on Criminal plus with tips to protect yourself. Listen to Criminal wherever you get your podcasts and sign up for criminal [e-mail] when the paperclip maximizer meme first started circulating in the 2000s, our best example of a paperclip AI was Clippy, the animated little guy on Word with the Eyeballs, Microsoft's AI Office assistant back in the day. I remember it couldn't even tell you if you should use their, their or their in a sentence. People weren't so much afraid of Clippy as they were annoyed with him. There are a remarkable number of think pieces from those years slamming Clippy. The consensus was no one asked for this. This is dumb. So when Eleazar Yudkowski warned about the dangers of a super intelligent AI that could someday destroy humanity, it was hard for a lot of people to take him seriously.

[Kelsey Piper]

The state of thought in 2010 was something like, yeah, I may as well be a century away.

[Gabrielle Burbet]

Future perfect writer Kelsey Piper Again.

[Kelsey Piper]

So if you are Eliezer Yakowski, you have a bit of a dilemma, right? You want to make two arguments. One is super intelligent AI is possible.

[Gabrielle Burbet]

Building a robot that's smarter, faster, and more creative than humans at most things is is possible, Clippy be damned. And he needed to make that first argument before he could make his next one.

[Kelsey Piper]

The second argument you want to make is we need to not do it until we have solved the challenge of how to do it right.

[Gabrielle Burbet]

For a long time, both arguments Super AI is possible, but let's not. For now, we're dead in the water, because AI tech was just not that impressive. But by 2014, Eliezer noticed that people outside his corner of the blogosphere had started to pay attention.

[Eliezer Yudkowski]

AI is probably the single biggest item in the near term that's likely to affect humanity.

[Gabrielle Burbet]

Tesla chief executive and billionaire Elon Musk, who started this year sitting prominently in President Trump's White House, had tweeted, quote, we need to be super careful with AI potentially more dangerous than nukes.

[Eliezer Yudkowski]

It's about minimizing the risk of existential harm.

[Gabrielle Burbet]

It seems like Elon Musk is a reader of Eliezer's blog. He famously met his ex, the musician Grimes, when they joked on then Twitter about a very obscure thought experiment from the blog. I will spare you the details. The point is Elon Musk read the Paperclip Maximizer thought experiment and he seemed convinced AI was a threat.

[Eliezer Yudkowski]

It's very important that we have the advent of AI in a good way and that's the reason that we created OpenAI.

[Gabrielle Burbet]

Elon Musk Co created OpenAI. You might have heard he left and then tried to buy it back. But if you haven't heard of OpenAI, you've got probably come across its most popular chatgpt. I was surprised to learn that Eliezer Yudkowski was in fact the original inspiration for the ChatGPT company, according to its co founder, Sam Altman.

[Kelsey Piper]

Sam Altman has in fact said this on Twitter, that he said that he credits Eliezer for the fact that he started OpenAI.

[Gabrielle Burbet]

Co founder Sam Altman specifically tweeted that Yudkowski might win a Nobel Peace Prize for his writings on AI, that he's done more to accelerate progress on building an artificial general intelligence than anyone else.

[Kelsey Piper]

Now, in saying this, he was kind of being a little cruel, right? Because Eliezer thinks that OpenAI is on track to cause enormous catastrophe.

[Gabrielle Burbet]

Co founders Sam Altman and Elon Musk bought Eliezer's first claim that superintelligence is possible and it's possible in our lifetimes. But they miss the part about how you're not supposed to build it yet.

[Eliezer Yudkowski]

For this sort of most important technological milestone in human history. I view that as right around the corner.

[Gabrielle Burbet]

That's Sam Altman talking about super intelligence.

[Eliezer Yudkowski]

Like it's coming soon enough and it's a big enough deal that I think we need to think right now about how we want this deployed, how everyone gets to benefit from it, how we're going to govern it, how we're going to make it safe and sort of good for humanity. Human values, which are things difficult to encode.

[Gabrielle Burbet]

It's still not clear to me what superintelligence actually is. I won't be the first one to observe that. It has some religious vibes to it. The name makes it sound like it's an all knowing entity. The CEO of OpenAI's competitor Anthropic said he wanted to build machines of loving grace. Sam Altman was asked on Joe Rogan's podcast about whether he's Attempting to build God.

[Eliezer Yudkowski]

I guess it comes down to maybe a definitional disagreement about what you mean by it becomes a God. I think whatever we create will still be subject to the laws of physics in this universe.

[Gabrielle Burbet]

Sam Altman has called this superintelligence the magic intelligence in the sky, which, I don't know, sounds a lot like how some people talk about God to me. How exactly this supposed superintelligence will be smarter, faster, and more intelligent than us on what scale is unclear. But for all the hype around ChatGPT, I only recently learned what the heck it is. It's what they call a large language model.

[Kelsey Piper]

At its most fundamental level, a language model is an AI system that is trained to predict what comes next in a sentence.

[Gabrielle Burbet]

I'm oversimplifying here, but the very basic idea of a language model is to generate language based on probabilities. So if I have a word or a set of words, what's the most likely next word?

[Kelsey Piper]

So if a sentence starts with on Monday I went to the grocery, the next word is probably store.

[Gabrielle Burbet]

The way the model guesses that store is probably next is based on how you train the language model. Training involves feeding the model a large body of text so it can detect patterns in that text and then go generate language based on those patterns. Early versions of Spellcheck, like Clippy, were language models trained on the dictionary. Useful, but only for a very specific task, like to tell you if you put the E in the word weird in the wrong place, or the h's in the word rhythm. Clippy couldn't tell you if you should use their, their, or their in a sentence, because it wasn't trained on enough text to be able to guess the right word in context. The dictionary can't tell you that. But OpenAI's products were very different from Clippy. A revolution was happening in AI tech that made language models look less like a simple spell check and more like the human brain detecting patterns and storing them in a network of neurons. Technologists trained those neural networks through a process they called deep learning. They trained the AI on a lot of data close to the entire Internet. Thanks to Vox media's partnership with OpenAI, we know they're likely training the language model on this podcast. The words I'm saying right now. No one had ever trained an AI on the entire Internet before, at least in part because of how expensive it is. It takes a ton of energy and compute power. But OpenAI, founded by a billionaire, raised the funds to make an attempt at the biggest, baddest largest language model the world had ever seen.

[Kelsey Piper]

They started going, okay, what if the secret to trying to build super intelligent God, AI or whatever is just to spend more money and have more neurons and have more connections, feed it more data. What if that's all there is? What if you can build something that is more intelligent than any human who's ever lived just by doing that?

[Gabrielle Burbet]

One of their earlier attempts before ChatGPT was GPT2 in 2019. You could similarly give it a specific task, like design a luxury men's perfume ad for the London Underground. Make it witty and concise. The London Underground is a great place to advertise. It's a great place to get your message across. It's a great place to get your product noticed. Look out, Mad Men. GPT2 was not exactly coming for copywriter jobs, but for people like Kelsey who were watching the technology closely, I was.

[Kelsey Piper]

Like, wow, this is like miles beyond what AI chatbots were capable of. Last week. This is huge.

[Gabrielle Burbet]

GPT2, the language prediction machine, was showing some real promise. She wasn't alone in that feeling. Investors like Microsoft poured millions more dollars into the next few models, which were bigger and bigger. Be the scent that turns heads. And a couple years later, OpenAI released Chatgpt Visuron, a cat captivating image of the perfume bottle surrounded by vibrant city lights, symbolizing the urban lifestyle. Embrace the city. Embrace your scent.

[Kelsey Piper]

Most people weren't paying any attention to AI, and so for them it was like a huge change in what they understood AI to do.

[Gabrielle Burbet]

ChatGPT was the first time that normies like me even thought about AI in any real way.

[Eliezer Yudkowski]

All I wanted to, wanted to do was fix my email. I did not expect to have a minor existential crisis about how much the world is about to change. And this is only proving that one day AI will take over human intelligence. I spent about two hours just typing back and forth with this AI chatbot and it got pretty weird.

[Gabrielle Burbet]

The AI confessed to loving Kevin and tried to convince him to leave his wife.

[Kelsey Piper]

People at OpenAI or competitors were saying like, yeah, the plan is to build super intelligence. We think we're going to do it by 2027. People were like, okay, startup hype. For some reason, everybody who runs a startup feels the need to say that they're going to build God and the human race. And then after ChatGPT was genuinely impressive, people started taking them a bit more seriously. And a lot of those people were nervous.

[Gabrielle Burbet]

People weren't so nervous about ChatGPT, but what ChatGPT represented, the way they got the language model to sound so much smarter so quickly, wasn't through intricate code. They just made the model bigger. Which suggested to some people that the path to building God or whatever was through brute force. Spending more and more money to. To build a bigger and bigger machine. So big we didn't really understand why it did what it did. We can't point to a line of code to say, this is why the robot got so much better at writing a perfume ad. And if we someday do build something that's smarter than us, whatever that means, we won't be able to understand why it's smarter than us. The trouble with this, it seems to me, is, is that AI will come for copywriter jobs. It could come for all our jobs. But rationalists I spoke to say that's nothing compared to the bigger trouble ahead, a potential apocalypse.

[Eliezer Yudkowski]

But I do also kind of think that it is very an important priority for me to have the best possible time in the next five to 10 years and just to do the very best I can to squeeze the joy out of life while it is here.

[Gabrielle Burbet]

Do you have an example of that.

[Eliezer Yudkowski]

One I can talk about on the podcast? I mean, yes, I joke, but I'm pretty involved in the kink community and that's very important to me. And it's important.

[Gabrielle Burbet]

Many rationalists I spoke to live in polyamorous communities because they believe monogamy is irrational. Some aren't sure if it's rational to have children, given the high probability of things going very, very wrong because of AI.

[Eliezer Yudkowski]

What's my P doom? As Zach weirdly says, P doom.

[Gabrielle Burbet]

It's a shorthand I heard at the conference meaning probability of doom.

[Eliezer Yudkowski]

It's a phrase that gets thrown around at this conference. People will literally go up to go, so what's your pedum? And it's a shorthand for what is the probability that humanity doesn't make it in the long term?

[Gabrielle Burbet]

And this is a mathy bunch. So they get specific.

[Eliezer Yudkowski]

I guess the answer I usually give is something like over 50%. I mean, I think it's somewhere around 80 90.

[Gabrielle Burbet]

Eliezer Yudkowski's PDUM is very high. I've read it's over 95% these days. But then I've seen him tweet that P doom is beside the point. I spotted Aliezer Yudkowski pretty much the moment I stepped into the conference. He was hard to miss. He was the one wearing a gold sparkly top hat all weekend. I was the one who was clearly lost, carrying a big furry microphone for three days trying to get people to talk to me. It wasn't until day three of the conference conference that I mustered the determination to approach Eliezer for an interview. Determination was necessary because he was always surrounded by a cluster of people, cluster of mostly dudes listening to him speak. I asked him if it would be okay if I pulled out my microphone. Everyone has been looking at this like it's a weapon.

[Eliezer Yudkowski]

It is.

[Gabrielle Burbet]

Over the last few years, Eliezer and the Rationalists have gotten some bad press. Some rationalists express their frustration at journalists who focus on the polyamory that happens in the community. Some critics of rationalism, to put it crudely, call them a sex cult. And then there's the unsavory things people associated with the community have said. One philosopher who helped popularize the paperclip Maximizer Nick Bostrom, once wrote that he thought black people were less intelligent than white people. He has since apologized. But critics highlight this comment and the mostly white demographics of the rationalist community to question their beliefs. I never really know why anyone agrees to talk to me, but can you introduce yourself?

[Eliezer Yudkowski]

I'm Eliezer Yudkowski. This event is probably more my fault than the fault of anyone else around.

[Gabrielle Burbet]

And can you describe your outfit right now?

[Eliezer Yudkowski]

Well, I am currently wearing a sparkly multicolored shirt and a sparkly golden hat.

[Gabrielle Burbet]

You can probably hear it in my voice. I was nervous to talk to him. He's known for being a bit argumentative, very annoyed with journalists, and with the world more generally for not being smart enough to understand him, for not heeding his warnings. I don't know. How would you summarize what you want the world to know in terms of AI?

[Eliezer Yudkowski]

The world is completely botching the job of entering into the issue of machine super intelligence. There's not a simple fix to it. If anyone anywhere builds it out under anything remotely like the current regime, everyone will die. This is bad. We should not do it.

[Gabrielle Burbet]

Do you feel, like, gratified at all to see that, like, your ideas entered the mainstream conversation? Do you feel like they have.

[Eliezer Yudkowski]

The circumstances under which they have entered the mainstream conversation are catastrophic. And I didn't. If I was the sort of person who was, like, you know, like, deeply attached to the validation of seeing other people agree with me, I would have picked a much less disagreeable topic. I was here to try to, like, not have things. I was here to not have things go terribly. They're currently going terribly. I did not get the thing I wanted.

[Gabrielle Burbet]

Eliezer's been on a bit of a Press tour, giving interviews and ted talks, saying OpenAI is on track to cause catastrophe.

[Kelsey Piper]

So it's, it's a funny thing because I have one position of deep sympathy with Eliezer. If you become convinced that this is a huge problem, it makes perfect sense to go on a writing tour trying to explain this to people. And also I think it's kind of predictable that a lot of people heard this and went, oh, AI is going to be really powerful. I don't think you're right about the thing where that's a problem. I want the powerful, important thing. And some people seized on it and were like, because this is powerful and important, we should like invest now. And I feel kind of sad about this.

[Gabrielle Burbet]

I can understand why Eliezer was hesitant to talk to me. His message to the world has been totally lost in translation in his mind. It's backfired. Even at his own conference, there were attendees who work for places like OpenAI, the companies building the supposed death machine he was afraid of.

[Kelsey Piper]

He thought that our best chance of building a super intelligent AI that did what we wanted and didn't like, you know, seize power from humans was to build one that was very well understood, one that sort of from the ground up, we knew why it made all the decisions that it made. Large language models are just the exact opposite of that.

[Gabrielle Burbet]

I will say. Even after talking to Eliezer and Kelsey and a bunch of rationalists, it's still hard to imagine how something like ChatGPT or Google's AI, which once told someone to add glue to stick cheese on pizza, is going to become the invention of all inventions and possibly catastrophic. But I can understand how building something big that you don't understand is a scary idea. The best AI metaphor I came across for my brain was not about paperclips. It was by a non rationalist writer. A guy named Brian Christian describes that training AI is something that could go wrong in the way people parenting a kid can go wrong. Like there's a little kid playing with a broom, she cleans up a dirty floor and her dad, looking at what she's done on her own says, great job, you swept that really well. This little girl, without skipping a beat, might dump the dirt back on the floor and sweep it up again, waiting for that same praise. That's not what her dad meant for her to do. It's hard to get the goals right in teaching a kid to be good. It's even harder to teach good goals to a non human robot. It strikes me as like almost like a parenting problem. I ran this parenting metaphor by Kelsey with her seven month old on her lap.

[Kelsey Piper]

I think there's some serious similarities and I do with my kids, struggle with trying to steer someone something that you don't have perfect control over and that you wouldn't even want to have perfect control over. But where it could go extremely badly to like just let the dice fall where they may.

[Gabrielle Burbet]

If we just let the dice fall where they may. Rationalists say we could have an apocalypse on our hands. They say it won't be one we saw coming. It won't be a Hollywood style Terminator situation. It probably won't have paperclips either. They don't pretend to know exactly how apocalypse could befall us, just that it'll probably be something we haven't even imagined yet. But I have trouble getting caught in what could happen when it feels like haven't bad things already started to happen. Thanks to AI. AI is not hypothetical anymore. It's arrived in our lives. I'm not kept up at night about a hypothetical apocalypse. I find myself asking now questions. Questions like what is OpenAI doing with my voice right now? Is there anything to do about problems with AI short of the annihilation of humanity?

[Kelsey Piper]

It sounds very exciting, you know, like if I were a big science fiction geek, I would be so into that.

[Gabrielle Burbet]

Not all technologists seized on Eliezer Yudkowski's claims.

[Kelsey Piper]

What is he even talking about? This is like word salad. Like this doesn't even make sense.

[Gabrielle Burbet]

One group of technologists didn't actually seize on any of his claims.

[Kelsey Piper]

There's one thing thing to have that conversation as a thought experiment. It's another thing when that kind of.

[Gabrielle Burbet]

Thought experimentation sucks up all of the.

[Kelsey Piper]

Money and the resources.

[Gabrielle Burbet]

The more I dig into the AI world, the more I see disagreement between technologists. I do worry about the ways in.

[Eliezer Yudkowski]

Which AI can kill us, but I.

[Gabrielle Burbet]

Think about the ways in which AI.

[Eliezer Yudkowski]

Can kill us slowly.

[Gabrielle Burbet]

They've been called the AI ethicists and they say we've been paying attention to all of the wrong things. That's next time. Good Robot was hosted by Julia Longoria and produced by me, Gabrielle Burbet. Sound design Mixing an original score by David Herman, fact checking by Caitlin Penzemug, editing by Diane Hodson and Katherine Wells. Special thanks to Future Perfect founder Dylan Matthews, to Vox's executive editor Albert Ventura and to Tom Jibbers, whose book the Rationalist's Guide to the Galaxy was an early inspiration for this episode. If you want to dig deeper into what you've heard, head to vox.com goodrobot to read more Future Perfect stories trying to make sense of artificial intelligence. Thanks for listening.