EP 1: Ready Or Not

Summary

Podcast Summary: The Last Invention — EP 1: Ready Or Not (October 2, 2025)

Main Theme

The premiere episode of "The Last Invention" sets the stage for a sweeping exploration of the AI revolution that has moved from the speculative margins to one of the most powerful and controversial forces shaping the 21st century. Through investigative reporting and illuminating interviews, host Gregory Warner and his team unravel the increasingly urgent debate over the rise of artificial intelligence, examining Silicon Valley conspiracies, accelerationist visions, the specter of superintelligence, and the existential risks and societal transformations ahead.

---

Key Discussion Points & Insights

1. The Accelerationist Conspiracy: AI Power Ambitions

• Origin Story: Reporter Andy Mills shares how a clandestine tip from an ex-Silicon Valley executive, Mike Brock, introduces him to allegations of a "slow motion, soft coup" within the U.S. government, allegedly led by a tech elite aiming to replace human officials with AI ([00:11-04:38]).
  • Mike Brock: "We are in a democratic crisis. This is a coup. This is a slow motion, soft coup." ([02:44])
  • Details include Elon Musk’s supposed “Department of Government Efficiency,” codenamed "Doge," whose phase one goal was automating government jobs.
  • Though elements of the story could not be verified, Mills discovers that the real phenomenon at play is broader: a powerful faction within Silicon Valley seeking to automate not just government, but "pretty much everyone who has a job."

2. A New World Order: The Vision of AI Utopians

• Accelerationists’ Belief: Leading figures—Bill Gates, Sam Altman, Mark Zuckerberg—see AI as humanity’s "last and most important invention,” heralding an era of abundance and near-magical progress ([04:46-06:49]).
  • William MacAskill: "Imagine that everybody will now, in the future, have access to the very best doctor in the world... This really will be a world of abundance." ([05:51], [06:05])
  • AI is seen as vital for solving major challenges: energy, healthcare, education, and possibly even enabling life extension or space colonization.
  • However, this isn't a secret plot—these leaders are outspoken in their objectives.

3. AGI and Beyond: The Supermind Horizon

• Defining AGI: The ambition goes beyond chatbots; the benchmark is "AGI — Artificial General Intelligence," a digital supermind capable of excelling at virtually any task a human can do ([07:12-08:47]).

  • Kevin Roose: "Many of these people believe that the human brain is just a kind of biological computer. If you could just build a computer that simulated that, you could essentially create a new kind of intelligent being." ([07:59])

• Timeline Compression: A significant shift in industry thought—what once seemed like science fiction (AGI by 2040+) now appears imminent. “It would be surprising... if it took more than about three years for AI systems to become better than humans at at least almost all cognitive tasks,” says Roose ([10:35]).

  • Gregory Warner: "I mean, holy shit, holy shit. That is really soon." ([11:12])

4. The Doomers: Existential Fears and Calls to Halt Progress

• Opposing Faction: Figures like Eliezer Yudkowsky and Geoffrey Hinton ("the godfather of AI") warn that forging ahead may be catastrophic ([11:49-14:52]).
  • Geoffrey Hinton: "The risk I’ve been warning about the most... is the risk that we'll develop an AI that's much smarter than us and it will just take over." ([14:09])
  • The transition from AGI to ASI (Artificial Superintelligence) is described as a tipping point where AI could rapidly outstrip human control ([15:16-16:17]).
  • Geoffrey Hinton: "By default, these systems will be more powerful than us... unless they have a very good reason for keeping humans around, I expect that by default, they will simply not do so, and the future will belong to the machines, not to us." ([17:33])

5. The Ants Analogy: AI’s Indifference to Humanity

• Comparison: Even if the superintelligent AI feels no malice towards us, we may become as incidental as ants are to human affairs ([19:05-20:01]).
  • William MacAskill: "Our survival might be as contingent on the goodwill of those AIs as the survival of ants are on the goodwill of human beings." ([20:01])

6. Solutions Debated: From Global Bans to Preparedness

• Stopping AI (The Doomer Approach): Some advocate making ASI illegal, even floating proposals as extreme as bombing data centers to delay progress ([24:45-26:11]).

  • Geoffrey Hinton: "We should not build asi, just don't do it. We're not ready for it and it shouldn't be done further than that." ([24:45])

• The Scouts (Preparedness Approach): Others (like MacAskill, Liv Baree, Sam Harris) believe humanity must urgently prepare institutions and regulatory frameworks for AGI’s arrival ([27:30-31:54]).

  • Liv Baree: "Our job now... is to collectively figure out how we unlock this narrow path..." ([28:17])
  • Focus on regulatory oversight, stress-testing, whistleblower protections, and international cooperation ([29:37-31:25]).
  • Geoffrey Hinton: "We’d like regulations that say when a big company produces a new, very powerful thing, they run tests on it and they tell us what the tests were... we'd like things like whistleblower protection." ([29:37], [29:55])
  • International cooperation may be possible because no government—China, the U.S., Russia—wants to be replaced by AGI ([30:47-31:25]).

7. The Urgency of the Moment

• Sam Harris’s Warning: Harris, a “Scout,” likens humanity’s situation to a dangerous tightrope walk: “We’re like racing out there in the most chaotic way, flailing our arms... we’re off balance already, we’re looking over our shoulder, fighting with the last asshole we met online, and we’re leaping out there.” ([32:26])

• Galvanizing Perspective: Harris invokes a thought experiment by Stuart Russell: If we learned an alien superintelligence would arrive in 50 years, “just think of how galvanizing that moment would be. That is what we're building, that collision and that new relationship.” ([33:30])

---

Notable Quotes & Memorable Moments

• On the existential risk of building AGI:
  • Geoffrey Hinton: "It really is an existential threat. Some people say this is just science fiction, and until fairly recently, I believed it was a long way off." ([13:50])
  • Gregory Warner: "One shot. Like one shot, meaning we can't update the app once we release it..." ([17:53])
• The Ant Analogy:
  • William MacAskill: "Our survival might be as contingent on the goodwill of those AIs as the survival of ants are on the goodwill of human beings." ([20:01])
  • Andy Mills: "If you're going to build a new house... you're not going to be concerned about the ants that live on that land..." ([19:19])
• On the tightrope of progress:
  • Sam Harris: "We have something like a tightrope walk to perform successfully now, like in this generation. Not 100 years from now." ([32:26])
• On imminent AGI:
  • Kevin Roose: "The majority view of the people that I talk to is that something like AGI will arrive in, in the next two or three years, or certainly within the next five." ([10:35])

---

Timestamps for Key Segments

• Opening Conspiracy & The Accelerationists: [00:11] – [04:38]
• Transformation, AGI, and AI Utopian Views: [04:38] – [07:12]
• Defining AGI & Its Stakes: [07:12] – [11:12]
• Accelerationists vs. Doomers and Existential Threats: [11:49] – [14:52]
• The ASI Cascade & Ants Analogy: [15:16] – [20:01]
• Responses to Risk: Doomers vs. Scouts & Regulation: [24:13] – [31:54]
• Urgency and the Tightrope Walk (Sam Harris): [32:08] – [34:58]

---

Tone & Language

The episode’s tone oscillates between investigative curiosity, incredulity, and sober urgency, with both skepticism and awe at the scale of transformations AI might bring. Speakers, from philosophers to ex-tech insiders, speak with candor about their shifting beliefs, personal anxieties, and the high stakes of an extraordinary future that may be far closer than most believe.

---

Conclusion & Look Ahead

The episode concludes by setting up future explorations—delving deeper into why some fear AI’s risks are exaggerated, how leading researchers changed their minds, and tracing the origin story of AI technology. The battlegrounds of ambition, dread, and societal change are clearly drawn, leaving listeners with both awe and unease as humanity stands on the precipice of, as the podcast puts it, “the last invention.”

---

For a full understanding of the depth and drama of this unfolding debate, this episode is an essential listen for anyone curious—or concerned—about the real story behind the AI revolution.

Transcript

[Gregory Warner]

This is the last invention. I'm Gregory Warner, and our story begins with a conspiracy theory.

[Andy Mills]

So, Greg, last spring I got this tip via the encrypted messaging app Signal.

[Gregory Warner]

This is reporter Andy Mills from a.

[Andy Mills]

Former tech executive, and he was making some pretty wild claims, and I wanted to talk to him on the phone, but he thought his phone was being tapped. But the next time I was out in California, I went to meet with him.

[Mike Brock]

I'm really kind of contending with, like, who I am in this moment. Up until a few months ago, I was an executive in Silicon Valley, and yet here I am sitting in a living room with you guys, talking about what I think is one of the most important things that needs to be discussed in the whole world, Right, which is the nature in which power is decided in our society.

[Andy Mills]

And he told me this story, that a faction of people within Silicon Valley had a plot to take over the United States government, and that the Department of Government Efficiency Doge under Elon Musk was really phase one of this plan, which was to fire human workers in the government and replace them with artificial intelligence. And that over time, the plan was to replace all of the government and have artificial intelligence make all the important decisions in America.

[Mike Brock]

I have seen both the nature of the threat from inside the belly of the beast if it were in Silicon Valley, and seen the nature of what's at stake.

[Andy Mills]

Now, this guy, his name is Mike Brock, and he had formerly been an executive in Silicon Valley. He'd worked alongside some big name guys like Jack Dorsey, but he'd recently started a substack. And he told me that after he published some of these accusations, he had become convinced that that people were after him.

[Mike Brock]

I have reason to believe that I've been followed by private investigators. For that and other reasons, I traveled with private security. When I went to D.C. and New York City last week, he told me.

[Andy Mills]

That he had just come back from Washington D.C. where he had met with a number of lawmakers, including Maxine Waters, and debriefed them about this threat to American democracy.

[Mike Brock]

We are in a democratic crisis. This is a coup. This is a slow motion, soft coup.

[Gregory Warner]

And so this faction. Who is in this faction? What is this, like the Masons or something? Or is like a secret cult?

[Andy Mills]

Well, he named several names, people who are recognizable figures in Silicon Valley. And he claimed that this quote, unquote conspiracy went all the way up to J.D. vance, the Vice President. And he called the people who were.

[Mike Brock]

Behind this coup the Accelerationists.

[Andy Mills]

The Accelerationists. It was a wild story.

[Kevin Roose]

Yeah.

[Andy Mills]

But, you know, some conspiracies turn out to be true. And it was also an interesting story. So I started making some phone calls, I started looking into it, and some of his claims I could not confirm. Maxine Waters, for example, did not respond to my request for an interview. Other claims started to somewhat fall apart. And of course, eventually Doge itself somewhat fell apart. Elon Musk ended up leaving the Trump administration. And for a while it felt like, you know, it was one of those tips that just doesn't go anywhere. But in the course of all these conversations I was having with people close to artificial intelligence, I realized that there was an aspect of his story that wasn't just true. But in some ways, it didn't go quite far enough, because there is indeed a faction of people in Silicon Valley who don't just want to replace government bureaucrats, but want to replace pretty much everyone who has a job with artificial intelligence. And they don't just think that the AI that they're making is going to upend American democracy. They think it is going to upend the entire world order.

[Sam Harris]

The world as you know it is over.

[Andy Mills]

It's not about to be over. It's over. I believe it's going to change the.

[Kevin Roose]

World more than anything in the history.

[Andy Mills]

Of mankind, more than electricity. But here's the thing. They're not doing this in secret. This group of people includes some of the biggest names in technology. Bill Gates, Sam Altman, Mark Zuckerberg, most of the leaders in the field of artificial intelligence. AI is going to be better than almost all humans at almost all things. A kid born today will never be smarter than AI. It's the first technology that has no limit.

[Gregory Warner]

So wait, so you get a tip about, like, a slow motion coup against the government, and then you realize, no, no, no, this is not just about the government. This is pretty much every human institution?

[Andy Mills]

Well, yes and no. Many of these accelerationists think that this AI that they're building is going to lead to the end of what we have come to think of as jobs, the end of what we traditionally thought of as schools. Some would even say this could usher in the end of the nation state. But they do not see this as some sort of shadowy conspiracy. They think that this may end up literally being the best thing to ever happen to humanity.

[William MacAskill]

I've always believed that it's going to be the most important invention that humanity will ever make. Imagine that everybody will now, in the future, have access to the very best doctor in the world, the very best.

[Andy Mills]

Educator the world will Be richer and can work less and have more. This really will be a world of abundance. They predict that their AI systems are going to be the thing that helps us to solve the most pressing problems that humanity faces. Energy breakthroughs, medical breakthroughs, maybe we can.

[William MacAskill]

Cure all disease with the help of AI.

[Andy Mills]

They think it's gonna be this hinge moment in human history where soon we will be living to maybe be 200 years old, where maybe we'll be visiting other planets, where we will look back in history and think, oh my God, how did people live before this? Technology should be a kind of era.

[William MacAskill]

Of maximum human flourishing where we travel to the stars and colonize the galaxy.

[Andy Mills]

I think a world of abundance really is a reality.

[Mike Brock]

I don't think it's utopian, given what.

[Andy Mills]

I've seen that the technology is capable of.

[Gregory Warner]

So these are a lot of bold promises and they come from the people who are selling this technology. Why do they think that the AI that they are building is going to be so transformative?

[Andy Mills]

Well, the reason that they're making such grandiose statements and these bold predictions about, you know, the near future, it comes down to what it is they think that they're making when they say they're making AI. Okay, this is something that I recently called up my old colleague Kevin Roos to talk about. Kevin, how is it that you describe what it is that the AI companies are making? Am I right to say that they're essentially building like a super mind, like a digital superbrain?

[Kevin Roose]

Yes, that is correct.

[Andy Mills]

He's a very well sourced tech reporter and a columnist at the New York.

[Gregory Warner]

Times, also co host of the podcast Hard Fork.

[Andy Mills]

And he says that the first thing to know is that this is far more of an ambitious project than just building something like chatbots, essentially.

[Kevin Roose]

Many of these people believe that the human brain is just a kind of biological computer, that there is nothing you know, special or supernatural about human intelligence, that we are just a bunch of neurons firing and learning patterns in the data that we encounter. And that if you could just build a computer that sort of simulated that, you could essentially create a new kind of intelligent being.

[Sam Harris]

Right.

[Andy Mills]

I've heard some people say that we should think of it less like a piece of software or a piece of hardware and more like a new intelligent species.

[Kevin Roose]

Yes. It wouldn't be a computer program exactly. It wouldn't be a human. Exactly. It would be this sort of digital supermind that, that could do anything a human could and more.

[Andy Mills]

The goal, the benchmark that the AI industry is working towards. Right. Now is something that they call AGI, artificial General intelligence. The general is the key part, because a general intelligence isn't just really good at one or two or 20 or 100 things, but like a very smart person can learn new things, can be trained in how to do almost anything.

[Gregory Warner]

I guess this is where people get worried about jobs getting replaced, because suddenly you have a worker like a lawyer or a secretary, and you can tell the AI to learn everything about that job.

[Andy Mills]

Exactly. I mean, that is what they're making. And that's why there's a lot of concerns about what this could do to the economy. I mean, a true AGI could learn how to do any human job. Factory worker, CEO, doctor. And as ambitious as that sounds, it has been like the stated on paper goal of the AI industry for a very long time. But when I was talking to Kevin Roos, he was saying that even just a decade ago, the idea that we would actually see it within our lifetimes, that was something that even in Silicon Valley was seen as like a pie in the sky dream.

[Kevin Roose]

People would get laughed at inside the biggest technology companies for even talking about AGI. It seemed like trying to plan for, you know, something, building a hotel chain on Mars or something. It was like that far off in people's imagination. And now if you say you don't think AGI is going to arrive until 2040, you are seen as like a hyper conservative, basically Luddite in Silicon Valley.

[Andy Mills]

Well, I know that you are regularly talking to people at OpenAI and Anthropic and DeepMind and all these companies. What is their timeline at this point? When do they think they might hit this benchmark of AGI?

[Kevin Roose]

I think the overwhelming majority view among the people who are closest to this technology, both on the record and off the record, is that it would be surprising to them if it took more than about three years for AI systems to become better than humans at at least almost all cognitive tasks, some people say physical tasks, robotics, that's going to take longer. But the majority view of the people that I talk to is that something like AGI will arrive in, in the next two or three years, or certainly within the next five.

[Gregory Warner]

I mean, holy shit, holy shit, that is really soon.

[Andy Mills]

This is why there has been such insane amounts of money invested in artificial intelligence in recent years. This is why the AI race has been heating up, right?

[Gregory Warner]

This is to accelerate the path to AI.

[Andy Mills]

But this has also really brought more attention to, to this other group of people in technology, people who I personally have been following for over a Decade at this point, who have dedicated themselves to try everything they can to stop these accelerationists.

[William MacAskill]

The basic description I would give to.

[Host/Producer Gregory Warner or Narrator]

The current scenario is if anyone builds it, everyone dies.

[Andy Mills]

Many of these people, like Eliezer Yudkowsky, are former accelerationists who used to be thrilled about the AI revolution and who for years now have been trying to warn the world about what's coming.

[Host/Producer Gregory Warner or Narrator]

I am worried about the AI that is smarter than us. I'm worried about the AI that builds the AI that is smarter than us and kills everyone.

[Andy Mills]

There's also the philosopher Nick Bostrom. He published a book back in 2014 called Superintelligence. Now, a superintelligence would be extremely powerful. We would then have a future that.

[Geoffrey Hinton]

Would be shaped by the preferences of this AI.

[Andy Mills]

Not long after, Elon Musk started going around sounding this alarm. I have exposure to the most cutting edge AI and I think people should be really concerned about it. He went to mit. I mean, with artificial intelligence, we are summoning the demon. Told them that creating an AI would be summoning a demon. AI is a fundamental risk to the existence of human civilization. Musk went as far as to have a personal meeting with President Barack Obama, trying to get him to regulate the AI industry and take the existential risk of AI seriously. But he, like most of these guys at the time, they just didn't really get anywhere. However, in recent years, that has started to change. The man dubbed the godfather of artificial intelligence has left his position at Google and now he wants to warn the world about the dangers of the very product that he was instrumental in creating. Over the past few years, there have been several high profile AI researchers, in some cases very decorated AI researchers. This morning, as companies race to integrate artificial intelligence into our everyday lives, one man behind that technology has resigned from Google after more than a decade, who have been quitting their high paying jobs, going out to the press and telling them that this thing that they helped to create poses an existential risk to all of us.

[Geoffrey Hinton]

It really is an existential threat. Some people say this is just science fiction, and until fairly recently, I believed it was a long way off.

[Andy Mills]

One of the biggest voices out there doing this has been this guy, Geoffrey Hinton. He's like a really big deal in the industry and it meant a lot for him to quit his job, especially because he's a Nobel prize winner for his work in AI.

[Geoffrey Hinton]

The risk I've been warning about the most because most people think it's just science fiction. But I want to explain to people it's not Science fiction, it's very real is the risk that we'll develop an AI that's much smarter than us and it will just take over.

[Andy Mills]

And it's interesting, when he's talking to journalists trying to sound this alarm, they're often saying, yes, we know that AI poses a risk if it leads to fake news. Or like, what if someone like Vladimir Putin gets a hold of AI, it's inevitably, if it's out there, gonna fall.

[Geoffrey Hinton]

Into the hands of people who maybe don't have the same values, the same motivations.

[Andy Mills]

And he's telling them, no, no, no, no. This isn't just about it falling into the wrong hands. This is a threat from the technology itself.

[Geoffrey Hinton]

What I'm talking about is the existential threat of this kind of digital intelligence taking over from biological intelligence. And for that threat, all of us are in the same boat. The Chinese, the Americans, the Russians are all in the same boat. We do not want digital intelligence to take over from biological intelligence.

[Gregory Warner]

Okay, so what exactly is he worried about when he says it's an existential threat?

[Andy Mills]

Well, the simplest way to understand it is that Hinton and people like him, they think that one of the first jobs that's gonna get taken after the industry hits their benchmark of AGI will be the job of AI researcher. And then the AGI will 247 be working on building another AI that's even more intelligent and more powerful.

[Gregory Warner]

So you're saying AI would invent a better AI and then that AI would invent an even better AI?

[Andy Mills]

That is one way of saying it, yes, exactly. The AGI now becomes the AI inventor. And each AI is more intelligent than the AI before it. All the way up until you get from AGI, artificial General Intelligence, to asi, artificial Superintelligence.

[Geoffrey Hinton]

The way I define it is this is a system that is single handedly more intelligent, more, more competent at all tasks than all of humanity put together.

[Andy Mills]

I've now spoken to a number of different people who are trying to stop the AI industry from taking this step. People like Conor Leahy. He's both an activist and a computer scientist.

[Geoffrey Hinton]

So it can do anything the entire humanity working together could do. So, for example, you and me are generally intelligent humans, but we couldn't build semiconductors by ourselves. But humanity put together can't build a whole semiconductor supply chain. A superintelligence could do that by itself.

[Gregory Warner]

So it's kind of like this. If AGI is as smart as Einstein or way smarter than Einstein, I guess.

[Andy Mills]

An Einstein that doesn't sleep that doesn't take bathroom breaks. Right.

[Gregory Warner]

And lives forever and has memory for everything.

[Andy Mills]

Exactly.

[Gregory Warner]

ASI that is smarter than a civilization.

[Andy Mills]

A civilization of Einstein's. That's how the theory goes, right? Like you have the ability now to do in hours or minutes, things that take a whole country or maybe even the whole world a century to do. And some people believe that if we were to create and release a technology like that, there'd be no coming back. Humans would no longer be the most intelligent species on Earth, and we wouldn't be able to control this thing.

[Geoffrey Hinton]

By default, these systems will be more powerful than us, more capable at gaining resources, power, control, et cetera. And unless they have a very good reason for keeping humans around, I expect that by default, they will simply not do so, and the future will belong to the machines, not to us.

[Andy Mills]

And they think that we have one shot, essentially.

[Gregory Warner]

One shot. Like one shot, meaning we can't update the app once we release it, once.

[Andy Mills]

This cat is out of the bag, once this genie is out of the bottle, whatever metaphor.

[Gregory Warner]

Once this program is out of the lab, as it were.

[Andy Mills]

Exactly. Unless it is 100% aligned with what humans value, unless it is somehow placed under our control, they believe it will eventually lead to our demise.

[Gregory Warner]

I guess I'm scared to ask this, but, like, how would this look like a global disaster? Or are we talking about it getting control of CRISPR and releasing a global pandemic?

[Andy Mills]

Yes, there are those fears for sure. I want to get more into all the different scenarios that they foresee in a future episode, but I think the simplest one to grasp is just this idea that a superior intelligence is rarely, if ever, controlled by an inferior intelligence. And we don't need to imagine a future where these ASI systems hate us or they break bad or something. The way that they'll often describe it is that these ASI systems, as they get further and further out from human level intelligence, after they evolve beyond us, that they might just not think that we're very interesting.

[Gregory Warner]

I mean, in some ways, hatred would be flattering. Like if they saw us as the enemy and we were in some battle between humanity and the AI, which we've seen from so many movies. But what you're describing is just like indifference, right?

[Andy Mills]

I mean, one of the ways that people will describe it is that, like, if you're going to build a new house, of all the concerns you might have in the construction of that house, you're not going to be concerned about the ants that live on that land that you've purchased and they think that one day the ASIS may come to see us the way that we currently see ants.

[William MacAskill]

You know, it's not like we hate ants. Some people really love ants. But humanity as a whole has interests. And if ants get in the way of our interests, then we'll fairly happily kind of destroy them.

[Andy Mills]

This is something I was talking to William McCaskill about. He is a philosopher and also the co founder of this movement called the Effective Altruists.

[William MacAskill]

And the thought here is if you think of AI as we're developing as like this new species, that species as its capabilities keep increasing, so the argument goes, we'll just be more competitive than the human species. And so we should expect it to end up with all the power. That doesn't immediately lead to human extinction. But at least it means that our survival might be as contingent on the goodwill of those AIs as the survival of ants are on the goodwill of human beings.

[Gregory Warner]

We'll be back right after this break.

[Geoffrey Hinton]

Foreign.

[Host/Producer Gregory Warner or Narrator]

The Last Invention is sponsored by Ground News. Ground News is one of the most helpful tools that I use to avoid the echo chambers and media bias online, especially when it comes to shining a light on our blind spots. So whether you're politically on the left or the right or somewhere in the center, the blind spot feature from Ground News highlights the stories that tend to be disproportionately covered on by one side or the other. As an example, take these two stories about President Donald Trump, one which had low coverage among left leaning outlets.

[Sam Harris]

It's a very important relationship we're going to get along good with.

[Host/Producer Gregory Warner or Narrator]

China reported that Trump says US will accept 600,000 Chinese students as part of a trade deal.

[Sam Harris]

I hear so many stories about we're not going to allow their students. We're going to allow, it's very important, 600,000 students.

[Host/Producer Gregory Warner or Narrator]

And another largely uncovered by right leaning outlets.

[Andy Mills]

Trump's social media company is using crypto.com's.

[Host/Producer Gregory Warner or Narrator]

Square Trump family crypto empire expands with crypto.com partnership.

[Geoffrey Hinton]

That's our transactional Trump family.

[Gregory Warner]

Make some money when you can by.

[Host/Producer Gregory Warner or Narrator]

Seeing which stories are amplified or ignored depending on the outlet. Ground News helps you step outside the filter bubbles that shape most people's news diets, giving you a fuller picture of what's actually happening. I really think that if you like this podcast, you're going to like their mission. Go to ground news.com invent to get 40% off the same unlimited access vantage plan that we use. You can even sign up to stay up to date with the biases in your coverage proactively with the weekly Blind Spot report delivered directly to your inbox. This is a great way to support them and the work that they do. Because Ground News is a subscriber supported platform. We appreciate what they're up to, we appreciate their support for this podcast. So go check them out and make sure to use our link ground news.com invent so they know we said you this episode of the Last Invention is brought to you by fire, the foundation for Individual Rights and Expression. There's a pattern that you can trace throughout history. In ancient Athens, Socrates was put to death for asking tough questions of the powerful. Centuries later, monarchs banned and burned books they considered dangerous. And in the last century, authoritarian governments shut down newspapers, censored broadcasts, even jailed their critics. The struggle was always the same who gets to decide what people can know. Today, that struggle is playing out in a new arena and the risk now is subtler search results that quietly vanish, recommendation engines that steer us towards safe and comfortable answers, and AI filters that can suppress ideas before we ever even see them. That's where FIRE comes in. FIRE has spent decades defending free inquiry on our campuses, in the courts and and in our culture. And now, through a $1 million grant program in collaboration with the Cosmos Institute, they are supporting projects that keep free thought alive in the era of AI. Join us [e-mail] TheLastInvention by supporting FIRE, you're protecting the future of free inquiry in America and ensuring that tomorrow's most important questions can still be asked. Once again, visit thefire.org thelastinvention and thanks.

[Gregory Warner]

If the future is closer than we think, and if one day soon there is a at least reasonable probability that super intelligent machines will treat us like we treat bugs, then what do the folks worried about this say that we should do?

[Andy Mills]

Well, there's essentially two different approaches to the perceived threat. Some people who are worried about this, they simply say that we need to stop the AI industry from going any further and we need to stop them right now.

[Geoffrey Hinton]

We should not build asi, just don't do it. We're not ready for it and it shouldn't be done further than that. It's not just I am not trying to convince people to not do it out of the goodness of their heart. I think it should be illegal. It should be logically illegal for people and private corporations to attempt, even to, to build systems that could kill everybody.

[Gregory Warner]

What would that mean? To make it illegal? Like how do you enforce that?

[Andy Mills]

Yeah, I mean, some accelerationists joke like what, are you going to outlaw algebra?

[Gregory Warner]

Right. You don't need uranium in a secret center. You can just build it with code.

[Andy Mills]

Right. But you do need data centers. And you could put in laws and restrictions that stop these AI companies from building any more data centers and a number of other laws. There are some people, though, who go even further and say that nuclear armed states like the US should be willing to threaten to attack these data centers if these AI companies like OpenAI are on the verge of releasing an AGI to the world.

[Gregory Warner]

Wait, so even bombing data centers that are in Virginia or in Massachusetts? I mean, like, they see it as.

[Andy Mills]

That great of a threat? They believe that on the current path we're on, there is only one outcome, and that outcome is the end of humanity.

[Gregory Warner]

If we build it, then we die.

[Andy Mills]

Exactly. And this is why many people have come to calling this faction the AI Doomers.

[Liv Baree]

The accelerationists like to call doomer. That was a kind of pejorative coined by them. And very successfully, I must say, I.

[Geoffrey Hinton]

Disavow the Doomer label because I don't see myself that way.

[Andy Mills]

Some of them have embraced the name Doomer, others of them dislike the name Doomer. They often will call themselves the Realists. But in my reporting, everyone calls themselves the Realists. So I didn't think that would work.

[Geoffrey Hinton]

I consider it to be realistic, to be calibrated.

[Andy Mills]

And one of the reasons that they balk at the name is that they feel like it makes them come off as a bunch of anti technology Luddites, when in fact many of them work in technology. Many of them love technology. People like Conor Leahy, I mean, they even like AI as it is right now. I mean, he uses ChatGPT. He just tells me that from everything that he sees, where it's headed, where it's going, we have no choice but to stop them.

[Geoffrey Hinton]

If it turns out tomorrow there's new evidence that actually all these problems I'm worried about are less of a problem than I think they are, I would be the most happy person in the world. Like this would be ideal.

[Gregory Warner]

All right, so one approach is we stop AI in its tracks. It's illegal to proceed down this road we're on, but that seems challenging to do given how much is it already invested in AI and frankly how much potential value there is in the progress of this technology. So what's the alternative?

[Andy Mills]

Well, there's another group of people who are pretty much equally worried about the potentially catastrophic effects of making an AGI and IT leading to an asi, but they agree with you that we probably can't stop it. And some of them would go as far as to say we probably shouldn't stop it. Because there really is a lot of potential benefits in AGI. So what they're advocating for is that our entire society, essentially our entire civilization, needs to get together and try in every way possible to get prepared for what's coming.

[Liv Baree]

How do we find the win, win outcome here?

[Andy Mills]

One of the advocates for this approach that I talked to is Liv Baree. She is a professional poker player and also a game theorist.

[Liv Baree]

Our job now, right now, whether you know you are someone building it or someone who is observing people build it, or just a person living on this planet, cause this affects you too, is to collectively figure out how we unlock this narrow path, because it is a narrow path we need to navigate.

[William MacAskill]

We should be really focusing a lot right now on trying to understand as concretely as possible what are all the obstacles we need to face along the way and what can we be doing now to ensure that that transition goes well.

[Andy Mills]

This faction, which includes figures like William MacAskill, what they want to see is the thinking institutions of the world, you know, the universities, research labs, the media, joined together to try and solve all of the issues that we're gonna face over the next few years as AGI approaches.

[Gregory Warner]

So you mean not just leave this up to the tech companies?

[Andy Mills]

Exactly. They wanna see, you know, politicians brainstorming ways to help their constituents in the event that the bottom falls out of the job market. Right, right.

[Gregory Warner]

Or prepare communities to have no jobs.

[Andy Mills]

I guess some of them go that far. Right. Like universal basic income. And. And they also want to see governments around the world, especially in the US, start to regulate this industry. What are the concrete steps we could take in the next year to get ready?

[Geoffrey Hinton]

So we'd like regulations that say when a big company produces a new, very powerful thing, they run tests on it and they tell us what the tests were.

[Andy Mills]

Geoffrey Hinton, after he quit Google, he converted to this approach. And he was talking to me about the kinds of regulations that he wants to see.

[Geoffrey Hinton]

And we'd like things like whistleblower protection. So if someone in one of these big companies discovers the company is about to release something awful which hasn't been tested properly, they get whistleblower protections. Those are to deal, though, with more short term threats.

[Andy Mills]

Okay, but what about the long term threats? What about this idea that AI poses this existential threat? What is it that we could do to prevent that?

[Geoffrey Hinton]

Okay, so I can tell you what we should do about AI itself taking over. There's one good piece of news about this, which is that no government wants that, so governments will be able to collaborate on how to deal with that.

[Andy Mills]

So you're saying that China doesn't want AI to take over their power and authority. The US doesn't want some technology to take over their power and authority. And so you see a world where the two of them can work together to make sure that we keep it under control.

[Sam Harris]

Yes.

[Geoffrey Hinton]

In fact, China doesn't want an AGI to take over the US Government because they know it will pretty soon spread to China. So we could have a system where there were research institutes in different countries that were focused on how are we going to make it so that it doesn't want to take over from people. It will be able to if it wants to. So we have to make it not want to. And the techniques you need for making it not want to take over are different from the techniques you need for making it more intelligent. So even though the countries won't share how to make it more intelligent, they will want to share research on how do you make it not want to take over.

[Andy Mills]

And over time, I've come to calling the people who are a part of this approach the Scouts.

[Gregory Warner]

Like the Boy Scouts. Be prepared.

[Andy Mills]

Like the Boy Scouts. Yes, exactly. And it turned out that after I ran this Name by William McCaskill, so what if I called your camp the Scouts?

[William MacAskill]

So a little fun fact about myself is I was a boy scout for 15 years.

[Andy Mills]

He actually was a Boy Scout. And so I thought, okay, the Scouts.

[William MacAskill]

Maybe that's why I've got this approach.

[Andy Mills]

But the key thing about the Scouts approach, if it's going to work, is they believe that we cannot wait, that we have to start getting prepared, and we have to start right now. This is something that I was talking about with Sam Harris.

[Sam Harris]

The reasons to be excited and to want to go, go, go are all too obvious, except for the fact that we're running all of these other risks and we haven't figured out how to mitigate them.

[Andy Mills]

Sam is a philosopher. He's an author. He hosts the podcast Making Sense, and he's probably the most impassioned Scout that I know personally.

[Sam Harris]

There's every reason to think that we have something like a tightrope walk to perform successfully now, like in this generation. Right. Not 100 years from now. And we're edging out onto the tightrope in a style of movement that is not careful. If you knew you had to walk a tightrope and you got one chance to do it, and you've never done this before. What is the attitude of that first step and that second step? We're like racing out there in the most chaotic way, flailing our arms. Yeah. And just like we're off balance already, we're looking over our shoulder, fighting with the last asshole we met online, and we're leaping out there.

[Andy Mills]

Right. And you've been on this for a long time. In 2016, I remember you did this big TED talk. Yeah, I've watched it. At the time, it had millions of views. And you were essentially saying the same thing. You were trying to get people to realize that we have a tightrope to walk, and we have to walk it right now.

[Sam Harris]

Well, I wanted to help sound the alarm about the inevitability of this collision, whatever the timeframe. We know we're very bad predictors as to how quickly certain breakthroughs can happen. So Stuart Russell's point, which I also cite in that talk, which I think is a quite brilliant changing of frame, he says, okay, let's just admit it is probably 50 years out. Right. So let's just change the concepts here. Imagine we received a communication from elsewhere in the galaxy, from an alien civilization that was obviously much more advanced than we are, because they're talking to us now. And the communication reads, thus, people of Earth, we will arrive on your lowly planet in 50 years. Get ready. Just think of how galvanizing that moment would be. That is what we're building, that collision and that new relationship.

[Host/Producer Gregory Warner or Narrator]

Coming up on the last invention.

[Andy Mills]

Why is all the worry about the technology going badly wrong? And why are people not worried enough about it not happening?

[Host/Producer Gregory Warner or Narrator]

The accelerationists respond to these concerns.

[Kevin Roose]

Existential risk for humanity is a portfolio. We have nuclear war, we have pandemic.

[Andy Mills]

We have asteroids, we have climate change.

[Kevin Roose]

We have a whole stack of things.

[Andy Mills]

That could actually, in fact, have this existential risk. So you're saying that it's going to decrease our overall existential risk even as it itself may pose to some degree an existential risk?

[Mike Brock]

Yes.

[Host/Producer Gregory Warner or Narrator]

Researchers tell us what they saw that changed their minds.

[Andy Mills]

I was a person selling AI as a great thing for decades. I convinced my own government to invest hundreds of millions of dollars in AI. All my self worth was on the plan that it would be positive for society. And I was wrong. I was wrong.

[Host/Producer Gregory Warner or Narrator]

And we go back to where the technology fueling this debate began.

[Kevin Roose]

Basically, this is the holy grail of the last 75 years of computer science.

[Geoffrey Hinton]

It is the genesis, the ur like Philosopher's Stone of the field of computer science.

[Host/Producer Gregory Warner or Narrator]

The Last Invention is produced by Longview Home for the curious and open minded. To learn more about us and our work, go to longviewinvestigations.com Special thanks this episode to Tim Urban Urban thanks for listening. We'll see you soon. This episode is sponsored by Ground News, the app that helps you spot media bias and see a broader picture of the news shaping our world. Get 40% off their vantage plan at Ground News Invent this episode is sponsored by Fire Defending Free Thought in the age of AI. You can learn [e-mail] TheLastInvention.