EP 7: The Scouts

Summary

Podcast Summary: The Last Invention – EP 7: The Scouts (Nov 13, 2025)

---

Overview

In this episode of The Last Invention, host Gregory Warner investigates the birth of the “AI Scouts”—a group united by the belief that we not only can, but perhaps should, pursue superintelligence, but only if we also come together to avoid catastrophic risks. The discussion starts with the historic 2015 Puerto Rico meeting that brought together both optimists and skeptics in AI, leading to the formation of foundational principles for the industry. Warner then explores the current landscape through in-depth conversations with two leading “Scouts,” poker champion and game theorist Liv Boeree and philosopher William MacAskill, as they debate the peril and promise of AGI, the dangers of race dynamics, and the need for new institutions, wisdom, and long-term thinking.

---

Key Discussion Points & Insights

1. The 2015 Puerto Rico Summit: Birth of the AI Scouts

• Gathering the Camps
  Max Tegmark (co-founder, Future of Life Institute) organized an unprecedented closed-door meeting of AI leaders and critics using a clever, lighthearted invitation:

  “The invitation I sent out… had a photo of a guy shoveling his car out from three feet of snow next to a photo of the beach by the hotel… where would you rather be on this date?” (Max Tegmark, [03:05])

• Disconnection and Dysfunction
  The summit aimed to bridge bitter divides between engineers optimistic about AGI and critics worried about existential risks:

  “The conversation happening was completely dysfunctional… they both thought that the other ones were crazy or reckless or morally unscrupulous…” (Max Tegmark, [03:53])

• Finding Common Ground
  Personal interaction—lunches, debates, drinks—helped diverse attendees find each other more reasonable than presumed:

  “…to see people who both thought the other one was crazy… over lunch and some wine… both updated to think, ‘oh wow, this other person is actually much more reasonable than I thought.’” (Tegmark, [07:06])

• Lasting Legacy
  Elon Musk’s $10M commitment at the summit catalyzed the first major grant program for AI safety ([08:00]). Tegmark notes,

  “[It] went a very long way to mainstreaming AI safety in academia.” ([08:14])
  Safety became a mainstream research topic—no longer taboo.

• Principles & “Race Avoidance”
  Key outcome: signing the Asilomar AI Principles, including #5, “race avoidance,” and a call for superintelligence to be developed for the common good ([09:53], [10:31]).
  Tegmark laments that many commitments have since “been compromised by the current race to be the company that makes it first” ([10:47]) and observes competitive pressures have overcome those early ideals.

---

2. Meet the Scouts: Liv Boeree & William MacAskill

a. Reasons for Optimism—and Catastrophic Risk

Liv Boeree (game theorist, astrophysicist, ex-poker champion):
Asserts the win-win future is possible but “I try not to be a naive optimist… I’m extremely concerned that the current trajectory we are on is actually on a lose-lose path.” ([17:20])

William MacAskill (philosopher, effective altruism leader):

“The attitude is one of taking really seriously the potential benefits of highly advanced AI… but also appreciating… enormous challenges and… we should be preparing now.” ([18:18])

• Risks Outlined:

  • Catastrophe from rogue AI or global pandemics ([20:16])
  • Institutional, environmental, and mental health collapse ([20:52])
  • Intense concentration of power, potentially a “world government” ruled by the US or China ([30:22])
  • Threats to personal freedom in a world where humans have no economic value ([31:13])

• Rewards Outlined:

  • Potential to eradicate poverty, disease, and even war:

  “One is just the ability to make better decisions, to think better, to have more knowledge… If we have superintelligence, then we can get superintelligent advice.” (MacAskill, [21:30])

  • “Enormous abundance, such that everyone in the world… could be millionaires many times over.” ([23:02])

b. The AGI Race Trap

• The Geopolitical “Race”
  Both Scouts express alarm over the intensifying US–China AI race:

  “It’s just a race to who can go off the cliff the fastest. No one wins such a thing…” (Boeree, [27:38])

  • Fear of “value lock-in” if one country or company monopolizes AGI ([28:55])
  • MacAskill warns: “leaders of AI companies could themselves stage a coup if they wanted to… that’s quite a lot more extreme than merely an erosion of democracy.” ([33:27])

• Historical Echoes: The Trap of Moloch
  Boeree explains the "Moloch" metaphor for competition gone malignant:

  “It’s the personification of game theory gone wrong… sacrificing other important values to win a quick thing… when everybody does it… creates these race to the bottom dynamics.” ([38:11])

• Current Incentive Structures
  Boeree describes how even well-intentioned lab leaders are pressured into risky, rushed releases:

  “People I know who work at some of these labs… would love to take more time before releasing their products… but if they don’t… they run the risk of losing their engineers… It’s an incredibly tricky situation.” ([44:09])

---

3. Solutions: Wisdom, Institutions, and Longtermism

a. Diplomacy Over Arms Race

• Boeree on Avoiding the AI Arms Race:

  “My advice… is… for people to put much more energy into diplomacy… No one is trying to do the diplomacy route, it feels like, in the US and China.” ([35:22], [35:47])

  • Suggests inspiration from nuclear arms reduction treaties:
    “The nuclear arms race, at its peak, had over 60,000 warheads… through really clever incentives and diplomacy… they managed to break the game theoretic stalemate.” ([36:13])

b. Building Long-Term Perspectives

• MacAskill's “longtermism” ([45:31]):

  "Longtermism is the view that we should be doing much more… to improve the lives of future generations."

  • Emphasizes humanity is in its infancy; our actions now will affect “billions and billions” yet unborn ([47:01])
  • “There’s only one moment at which [AGI] first gets developed… that is something that is unique about our current situation.” ([48:58])

c. Institutions, Wisdom & Reinventing Society

• Accelerate Wisdom, Not Just Tech
  Boeree advocates for building “new social structures that manage these incredibly powerful technologies” ([56:09]):

  “If you let the technology drive the social structure… that's where you get Moloch. But if you flip that stack… come up with the good memes… the high philosophies… then you get the win-win outcomes.” ([57:43])

  • Calls herself a “wisdom accelerator”:

    “We need to build the wisdom alongside the power. So, I'm a wisdom accelerator… How do we accelerate the wisdom and social structures that support that wisdom?” ([59:12])

---

Notable Quotes & Memorable Moments

• On the Puerto Rico Summit’s Impact:
  “Once people realized AI safety doesn’t just mean shouting from rooftops ‘Stop! Stop!’ but actually means… doing concrete hands-on work… much of the taboo kind of melted away.” (Max Tegmark, [08:12])

• On the Dangers of Incentive Traps:
  “It's this act of sacrificing other important values to win a quick thing… when everybody does it… creates these race to the bottom dynamics. Unfortunately, that's what I see going on in the AI world now.” (Boeree, [38:11])

• On our Place in History:
  “A typical member of human or human-originating civilization will be far in the distant future… they will think of us as people from the distant past… who had enormous responsibility.” (MacAskill, [47:01])

• On the Need for New Institutions: "I 100% think we have to build new institutions. I think most of our institutions are grossly outdated. They're crumbling… I want to accelerate the building of new social structures that manage these incredibly powerful technologies…” (Boeree, [56:09])

---

Timestamps for Important Segments

• [02:22] - Introduction to “The AI Scouts” and the origin of the camp
• [03:05] - Max Tegmark describes organizing the Puerto Rico AI summit
• [05:00] - Nick Bostrom on being an early “AI risk” advocate
• [08:00] - Elon Musk’s $10M pledge for AI safety
• [09:53] - Reading from the Asilomar AI Principles
• [11:33] - How competitive pressures eroded early ideals
• [17:20] - Introduction of Liv Boeree and William MacAskill (“the Scouts”)
• [20:16] - MacAskill on the spectrum of AI risks
• [23:02] - The promise of abundance from superintelligence
• [27:38] - Boeree on the catastrophic nature of the AI race
• [31:13] - MacAskill on AI’s threat to democracy and freedom
• [35:22] - Boeree on the need for diplomacy, not escalation
• [38:11] - “Moloch”—the metaphor for self-destructive competition
• [42:11] - The AI “Moloch trap” and high-profile AI mistakes (Sydney, image gen, etc.)
• [45:31] - MacAskill introduces longtermism
• [47:01] - We are living at the hinge of history
• [50:58] - MacAskill’s AGI timeline: “early 2030s”
• [52:25] - MacAskill on hopefulness due to AI safety becoming mainstream
• [56:09] - Boeree on accelerating new social institutions
• [57:43] - Building wisdom-based societies to drive win-win outcomes

---

Conclusion

The "AI Scouts" episode weaves together the history of AI safety, the high stakes of current geopolitical competition, and the urgent moral imperatives posed by AGI. It frames the challenge as not only technical but as deeply social and philosophical, advocating for diplomacy, new social institutions, and a commitment to future generations—reminding listeners that the narrow path to a win-win future demands wisdom, collective action, and building structures and norms as powerful as the technology itself.

Transcript

[A]

Hi, everyone. This is Andy. And before we get to today's episode, I just wanted to say a quick thank you to everybody out there who has been sharing the series. It is because of you that we have been a success. We're very grateful. And I wanted to talk a little bit about our hope for the future of the press and share a recommendation with you. The field of journalism, it is no secret, is one with a checkered history. You can go back 200 years ago, and Thomas Jefferson was saying that the man who never looks in the newspaper is better informed than he who reads them. And the truth itself, he said, becomes suspicious by being put into that polluted vehicle. Pretty tough stuff, President Jefferson. And we know from polling that right now, again, much of the country shares a version of that sentiment. With trust in the media hitting a new historic low year after year after year, here at Longview, we are working hard to earn back that trust. We aim to build a reputation of fairness and openness and to do our part to forge a more honest and trustworthy press in the future. And one of the organizations that we look up to because they're already paving the way forward for that, are our friends over at Tangle. Every day, Tangle publishes this newsletter and podcast where they summarize the most influential arguments coming from the right, coming from the left, coming from the center. And I think it's a genuine beacon for where our future industry can go. And this is not a paid advertisement. Tangle is legitimately someone who we have collaborated with in the past and we hope to in the future. And as a part of that collaboration, Tangle is offering a 40% discount on a subscription to all of you who listen to the show. And so if you want to go check that out, please go to readtangle.com invention Again, that is readtangle.com invention. I think you're going to love what they're doing over there. And again, thank you so much for listening. Now, onto the show.

[B]

This is the last invention. I'm Gregory Warner. Today, the case for why we can and maybe should build superintelligence. But how we'll need to come together to make sure we don't destroy humanity in the process. And there is a good reason to argue that this worldview, this camp, the AI Scouts, as we call them, was born in the year 2015 with a gathering of true AI believers on the island of Puerto Rico.

[A]

Okay, so first up, can you just tell me how you pulled this off? Like, how did you get all of these different figures in the big AI debate together down in Puerto Rico?

[C]

Yeah. First of all, we scheduled a meeting in Puerto Rico in January. And the invitation I sent out to everybody had a photo of a guy shoveling his car out from three feet of snow next to a photo of the beach by the hotel, and it said the date. And so where would you rather be on this date?

[A]

Very clever, Max.

[D]

Very clever.

[B]

This is Max Tegmark, MIT professor of Physics, co founder of the Future of Life Institute. And back in 2015, what inspired him to organize this meeting of the minds was that while most people at that time still did not believe that anything like true AGI was on the horizon in our lifetimes, the people who did believe it were already starting to fight about whether AI was going to be great for the world or lead to its destruction.

[C]

The conversation happening was completely dysfunctional. On one hand, you had some people outside the research community, like Eliezer Yudkowski, Nick Bostrom and others, who expressed concerns. And then you had people inside the community who either weren't thinking about it at all or felt very threatened by the people complaining about it, worrying it was gonna be bad for funding. And since these two groups mostly didn't talk to each other, they both thought that the other ones were crazy or reckless or morally unscrupulous or something like that. And I felt we have to get the AI community itself into this conversation.

[B]

And so at this time, that, with hindsight, we know was right between these two defining moments. And in the history of AI, when DeepMind's Atari demo got them acquired by Google, and when Elon Musk and Sam Altman started OpenAI, it's right then that all of these AI hopefuls, like Demis Hasabis and Ilya Sutskever were brought together with the AI worried, like Eliezerkowski, Nate Soares, Elon Musk, and also Nick Bostrom.

[A]

So take me back to this moment in 2015, because I want to understand how it felt to be you. I know that for many years, somewhat like Eleazer Yudkowski, you had been going around trying to convince people to take AGI and the risks it might pose seriously, but not really getting anywhere. Am I right?

[E]

Yeah. It was striking because it seemed pretty clear to me that we're going to, at some point, get AGI and then superintelligence, and that this was going to be maybe the biggest thing ever, and that it was going to involve these huge challenges, in particular the technical alignment problem, but also obviously, governance problems and ethics challenges, et cetera. And yet it was completely ignored by academia and by the wider Sort of intellectual world.

[B]

Bostrom, who just published his surprise hit book Superintelligence, was encouraged to get an invitation to this conference where he'd be able to sit down and talk face to face with some of the people actually building it, because the majority of.

[E]

The world dismissed this completely at the time.

[A]

And is it right that the basic pitch of this conference was like, hey, all of you guys may have your differences, but you all agree that AGI is important, that we should take this seriously. So let's stop our bickering, let's get together, let's have some talks, let's have some debates, let's have some drinks and see if we can find some common ground. Is that essentially it?

[B]

Yeah.

[E]

I mean, this conference was like. It brought together a bunch of different important constituents. On the one hand, there were many of the leading lights in the AI field at that time. Like, Rick Sutton was there, Stuart Russell was there, the founders of DeepMind were there, Ilya Sutskever, and then a big contingent from AI safety people and some potential funders. And these communities had previously been more or less separate with limited interaction. And I think part of the design of this conference was, can we bring these together and then create an atmosphere where they can actually engage and listen and discuss these things, rather than forming two different camps that sort of throw grenades over a wall on each other?

[C]

And it was really quite moving to see people who both thought that the other one was crazy when they just sat next to each other over lunch and had some wine, how they both updated to think, oh, wow, this other person is actually much more reasonable than I thought.

[B]

And so for three days by the beach, without any reporters around, with nothing being recorded, all these people got the chance to actually sit down and discuss and hash out what was the world with AI they all wanted to see. They talked about things like, how do we ensure that AI might lead to an economic boom without triggering the biggest unemployment crisis in human history. They talked about, how do we build AI systems that we can actually control, even if these things are way smarter than we are? And how do we take this technology that we still don't understand and make it a serious object of study for universities and other institutions?

[C]

And then Elon stood up at the end and also promised to give $10 million to fund the first ever grant program on not just making AI more powerful, but specifically on nerd research on how to make it safe.

[B]

And this conference had an immediate impact.

[C]

That went a very long way to mainstreaming AI safety and academia. You know, nowadays, if you go down to Neurips or any AI conference, there's going to bunch of technical papers with matrices and integral signs and all the nerdy stuff, you know, which is actually safety research. Once people realized AI safety doesn't just mean shouting from rooftops, stop, stop. But it actually means often doing concrete hands on work. Much of the taboo kind of melted away.

[B]

It led to something that rarely happens in emerging industries. A focus on safety became not only part of the conversation, but, but an early priority in most of the major AI labs and where those most worried about AI and those most excited about it agree to work together.

[E]

You might think they would sort of close ranks and say, oh, there are no risks here, because that would be inconvenient for them to acknowledge. And then the AI safety people would be on the outside and maybe they would have some ideas of safety things, but they couldn't. Like ultimately it needs to actually be implemented by people building the AI. Right. And so that that was an obvious sociological risk that you would get this polarization into two separate communities.

[A]

And am I right that one of the things that they agreed on, that they committed to was working together to try and avoid an AI race?

[C]

For sure, for sure.

[A]

And what did that commitment look like? Do you remember what was said?

[C]

Well, they even signed something. Let me give me a second, I'll give you the right quote. Okay.

[B]

Tegmark ended up pulling out this list of principles that were signed after the conference by many of the people who attended, even some new folks that couldn't make it down to Puerto Rico, like Sam Altman and Dario Amade.

[C]

One of the Asilomar AI principles says principle number five, race avoidance. Teams developing AI systems should actively cooperate to avoid corner cutting on safety standards.

[A]

So essentially this thing is too important for us to treat it like just some kind of product that we're all racing to build as fast as we can.

[C]

Yeah, it's very depressing to look at how some of these have aged. There's also another one that's saying an arms race and lethal autonomous weapons should be avoided. Well, welcome to 2025. There is also Principle 22, recursive self improvement.

[B]

Tegmark says that while industry leaders in AI will still claim that they are profoundly concerned about the risks of AGI, pretty much all of these principles, these commitments have been compromised by the current race to be the company that makes it first.

[C]

Principle 23, the last one, the common good principle that says superintelligence should only be developed in the service of widely shared ethical ideals. And for the benefit of all humanity rather than one state or organization. And welcome to 2025 when you have Dario Amodei from Anthropic very openly saying, for example that the US should crushed China, basically race China to get this first. So it's really fascinating how the ideals, starry eyed ideals that these people had back then have gradually fallen to competitive pressures.

[B]

However, there are still those that believe that we can return to the dream that and the promise of what happened in Puerto Rico. And this time they want even more of us, all of us really. Part of the conversation about how do we get ready? How do we get prepared for superintelligence. These are the AI Scouts. After a short break, Andy interviews two Scouts who make their case. Stay with us.

[F]

The last invention is sponsored by Ground News. Ground News is one of the most helpful tools that I use to avoid the echo chambers and media bias online, especially when it comes to shining a light on our blind spots. So whether you're politically on the left or the right or somewhere in the center, the blind spot feature from Ground News highlights the stories that tend to be disproportionately covered by one side or the other. As an example, take these two stories about President Donald Trump, one which had low coverage among left leaning outlets.

[A]

It's a very important relationship we're going to get along good with.

[F]

China reported that Trump says US will accept 600,000 Chinese students as part of a trade deal.

[A]

I hear so many stories about we're not going to allow their students, we're going to allow. It's very important 600,000 students.

[F]

And another largely uncovered by right leaning outlets.

[G]

Trump's social media company is using crypto.com's.

[F]

Trump family crypto empire expands with crypto.com.

[G]

Partnership that's our transactional Trump Family.

[B]

Make some money when you can by.

[F]

Seeing which stories are amplified or ignored depending on the outlet. Ground News helps you step outside the filter bubbles that shape most people's news diets, giving you a fuller picture of what's actually happening. I really think that if you like this podcast, you're going to like their mission. Go to groundnews.com invent to get 40% off the same unlimited access vantage plan that we use. You can even sign up to stay up to date with the biases in your coverage proactively with the weekly Blind Spot report delivered directly to your inbox. This is a great way to support them and the work that they do. Because Ground News is a subscriber supported platform, we appreciate what they're up to. We appreciate their support for this podcast, so go check them out and make sure to use our link groundnews.com invent so they know we sent you this episode of the Last Invention is brought to you by fire, the foundation for Individual Rights and Expression. There's a pattern that you can trace throughout history. In ancient Athens, Socrates was put to death for asking tough questions of the powerful. Centuries later, monarchs banned and burned books they considered dangerous. And in the last century, authoritarian governments shut down newspapers, censored broadcasts, even jailed their critics. The struggle was always the who gets to decide what people can know. Today, that struggle is playing out in a new arena, and the risk now is subtler search results that quietly vanish, recommendation engines that steer us towards safe and comfortable answers, and AI filters that can suppress ideas before we ever even see them. That's where FIRE comes in. FIRE has spent decades defending free inquiry on our campuses, in the courts, and in our culture. And now, through a $1 million grant program in collaboration with the Cosmos Institute, they are supporting projects that keep free thought alive in the era of AI. Join us [e-mail] TheLastInvention by supporting FIRE, you're protecting the future of free inquiry in America and ensuring that tomorrow's most important questions can still be asked. Once again, visit thefire.org thelastinvention and thanks. The Last Invention is sponsored by Cozy Earth. Around the holidays, I always look forward to the same moment when everything slows down, the lights are twinkling, there's a blanket, a movie, and some time where no one is rushing anywhere. It's my favorite part of the season. And this year Cozy Earth has basically turned that moment into a gift. Their Bubble Cuddle blanket is exactly what it sounds like. Incredibly soft, textured and luxurious without being fussy. It'll become the go to blanket for all your cozy moments. For fireside nights, for lounging on the couch, or for curling up and watching the Grinch with your kids for the 10,000th time. They also make these bamboo pajama sets that are lightweight and sleep much cooler than cotton. It's one of those gifts that people actually end up using every day long after Christmas morning has passed. For the people in your life who are impossible to shop for that don't really need anything, you're giving them an excuse to slow down and enjoy the moment. And one of the best parts. It's risk free. Cozy earth gives you 100 nights to try their products and a 10 year warranty, although once you feel them, you won't Be sending them back right now. Black Friday has come early at Cozy Earth and you can stack our code invention on top of their site wide sales for up to 40% off. These deals won't last, so start your holiday shopping today and if you get a post purchase survey, tell them you heard about Cozy Earth right here on the last invention. Wrap the ones you love in luxury with Cozy Earth.

[G]

My personal philosophy is like, how do we find the win win outcome here?

[A]

Right? So the first of our two scouts is Liv Baree.

[G]

I would love to live in this techno utopian, awesome like freedom maximizing world where humanity and whatever fun new species also emerge alongside it. Get to go and do amazing things together and everybody wins. I would love that future to happen, but I try not to be a naive optimist and thinking that that's just going to magically happen if we just carry on with the status quo. I'm actually extremely concerned that the current trajectory we are on is actually on a lose lose path.

[A]

Liv is actually a famous poker champion, but she's also a game theorist. She has a background in astrophysics and she has spent a lot of the past several years trying to persuade people of what she sees as both the opportunities and the serious risks posed by AI. And our other spokesman for the Scouts today is the philosopher William McCaskill.

[D]

The attitude is one of taking really seriously the potential benefits of highly advanced AI thinking that catastrophic outcomes are not at all the ordained and appreciating though that if AI really does drive rapid tech progress, there will be this enormous number of challenges and yeah, we should be preparing now.

[A]

William is probably best known as being one of the co founders of the effective altruist movement, which gained a lot of influence especially in Silicon Valley over the past decade or so. He's also the bestselling author of the book what We Owe the Future. And as you'll hear in both of these conversations, Liv and William are making this case that the urgency and the opportunity of the very moment that we're living in right now is unique. And they believe that it demands of us, all of us, all around the world, that we join in, in doing whatever we can do to try to get ready for the radical transformation that that AGI is about to bring.

[G]

Our job now, right now is whether you know you are someone building it or someone who is observing people build it, or just a person living on this planet, because this affects you too, is to collectively figure out how we unlock this win win path, this narrow path, because it is a narrow path. We need to navigate. But I do think this win win future is in principle possible.

[A]

Okay, so I want to start off by getting your view, broadly speaking, on the risks versus the rewards of building an AGI that eventually becomes a superintelligence. We can get into some of this in more detail later, but it's just like a very basic introduction. How do you think about the parts of our AI future that could be amazing versus the threats that AI poses that could be catastrophic?

[D]

Well, there's lots of ways that things could go badly. There are risks of enormous catastrophe like global pandemic from man made viruses, or loss of control to AI systems themselves, or the catastrophe of intense concentration of power. Perhaps a single country becoming utterly globally dominant and that single country falling into some sort of authoritarian regime or even dictatorial regime.

[A]

When you say that you think we're on a lose lose trajectory, say more about that. What is this lose lose scenario as you envision it?

[G]

Well, so in terms of the current trajectory, we run up against some kind of planetary boundary, or in this case maybe multiple planetary boundaries at the same time. And it creates these like cascading effects of essentially like institutional collapse, environmental collapse, mental health collapse, all the conflicts that then come sort of downstream of those where we're still living under the nuclear shadow. So there's all those sort of like crises that could happen that might lead to our either permanent curtailment, like some massive catastrophe or complete extinction.

[A]

So you take it that far, you think that it's possible that this thing could be our demise, could be the end?

[G]

Yes, absolutely.

[D]

I think there are many ways in which things could go really quite badly wrong. But there are so many positives as well. So one way in which AI is very different from some of the technologies that people sometimes point to, like atomic weapons, is that it comes along with these enormous upsides too. So one is just the ability to make better decisions, to think better, to have more knowledge. If we have super intelligence, then we can get super intelligent advice. We can make better decisions. You can have AI that helps us reason much better, or does the reasoning itself even helps us reflect better from an ethical perspective too. You can also have AI that helps you coordinate much better, such that if I'm the United States and you're China, well, we're really quite limited in our bandwidth at the moment. There's only so much diplomacy that can happen. But with enormous amounts of AI diplomats, I think you might be able to have many more kind of mutually beneficial agreements, such that the irrational things that seem to have happened in the past, like wars, other sorts of enormous destruction of value. Maybe we don't need to have them anymore.

[A]

So it sounds like you agree with this idea that's out there, that there could be a future unlocked by AGI, where we literally live with world peace. The idea is that we have a more peaceful coexistence with all the help that's brought about by this AGI.

[D]

Absolutely. And then the second aspect is just abundance that AI could bring too, because it's been technological development in the past. That is the primary reason why we are so much richer today than at any time in history. And so if we're facing the prospect of AI and then a rapid transition to superintelligence, well, that is a world with enormous abundance, such that everyone in the world, if that abundance was allocated at all equally, everyone in the world could be millionaires many times over and such that, in principle at least, everyone could get basically all they want right now. And that should be a cause of optimism, because if the pie is going to get much, much bigger, you know, 100 times bigger, thousand times bigger, then it really shouldn't matter what slice of the pie does everyone get. We should instead be much more focused on ensuring that we actually get that big pie, we get to enjoy it, and that it's at least somewhat kind of equally distributed, because everyone can be extremely well off.

[G]

I do think that we need to enhance our intelligence in order to get out of some of these really wicked collective action problems, like climate change, for example. Like, we've understood the mechanism behind this for decades now, and yet for the same reason, like, we can't unplug the Internet if we wanted to. We can't unplug climate change if we wanted to.

[A]

Right.

[G]

So the question to me is, how do we allow all the goodness of competition to create the race to the top of the cool stuff that we want, like solutions to cancer or novel drug discovery, better coordination mechanisms to fix climate change and all of these other huge collective action problems without also accelerating all of the dangerous things, without giving terrorists the ability to synthesize novel pathogens, without creating a ubiquitous surveillance capitalism, which is also a path we seem to be accelerating. So basically, how do we get all the best bits of AI without all of the downsides? And it's a really, really wicked problem.

[A]

I know that sometimes you get labeled as an AI doomer because of the fact that you and the AI doomers agree about a lot of the things that you're worried about, but one of the things that I find so fascinating about you and about this whole camp is the view that it would be bad for us to stop in our attempts to make superintelligence and even be bad for us to pause for too long. Could you unpack that for me? What do you mean by that?

[D]

So if it were the case that we would never develop superintelligence, that would be very bad. So some people have this attitude that, oh, we should just never build it. It's like the science fiction novel Dune in which civilization just decided, no, we're not going to have computers, we're not going to have AI. I think that would be very bad because AI could help us solve many of the other problems in the world. Every year, something like 100 million people die. There's enormous amounts of suffering. Much of that is because we lack the medical technology or the scientific understanding to improve those lies or prevent the early and unnecessary death. Similarly, there's enormous poverty in the world and that could be alleviated significantly because if we had morally distribution, but could also be alleviated similarly if. If the world was just much, much richer than it was today. I also think that this is a good argument against delaying the development of superintelligence unduly. I think we should, as a first, best try to have solutions that mean we get there safely, that don't go via delaying it for years or decades because there's such a loss from all the problems that we could have solved that we're currently not solving.

[A]

I think it would be helpful for us to try and get on the same page about where things stand right now with the AI race, because as far as I can see it, you have the race happening here in the US between OpenAI and Google and DeepMind and anthropic and all these other companies. But then the race that seems to be much more urgent on the minds of lawmakers especially is the race between the US and China. And right now there's a ton of money and there's a ton of support and there's a ton of excitement fueling the American side in that race. The idea that the US has to win this race, first off, is that how you see our current situation and where do you think things stand right now?

[G]

It's true that there is this larger sort of geopolitical race going on largely between America and in some ways the west and China. Like it seems like all trendlines point to, those are going to be the two major players here, especially towards the on the cutting edge race to superintelligence. And that, frankly, Terrifies me because in such a race, certainly under current conditions, where everyone is cutting corners and going as breakneck speed as possible, it's just a race to who can go off the cliff the fastest. No one wins such a thing. But at the same time, there's also the risk of value lock in. If somehow we do manage to safely navigate building superintelligence where it does what we want it to, that means that one person might end up with all the power. And I would personally rather that be a Western values than from what I can tell, the CCP values, because the west is more aligned with my core tenets, which is of personal freedom, self determination, et cetera. If there was absolutely no other option, I would rather the US win that race. But I'm also extremely concerned that it is not possible under current conditions for anybody to win this race.

[D]

So one thing I'm very worried about in the context of AI is intense concentration of power. Because if a single company is developing technology much, much faster in a way that gets faster, in fact with every iteration, so it's not just exponential, it's super exponential, then you could quite soon get to a stage where that company has just greater technological capability than the rest of the world combined. Or if there's even just a single country, then again that country would quite soon, if it was leading ahead of all others, quite soon would just become completely dominant.

[A]

Yeah, this is something that I've heard Tyler Cowen, the economist, who I'm a big fan of, talk about a lot, is this possible future where the US and China, because they are so invested in creating AGI and they're so far ahead of everybody else, that we may end up a few decades from now, or maybe 100 years from now in a situation where they aren't just the two superpowers in the world, but where they are essentially the two powers in the world that the whole planet is divided up between the US and its AI and China and its AI. And this isn't just like a philosophical like, oh, that's an interesting idea. But this is actually something that serious people are already thinking about and trying to come up with different models of the future based around this.

[D]

Yeah, and there are good reasons for that. Based on this idea of just very rapid growth and technological progress. I think I would go further than Tyler Cohen and say that actually I think it's quite likely that really there's just one country that wins out. So in 100 years time, essentially the United States is the world government, or essentially China is the World government, where I think that follows as a thought quite naturally from the dynamics that AI introduces into technological advancement.

[A]

I want to come back to China in just a little bit, but while we're on the concentration of power, I'd love for you to just tackle the risk, as you understand it, of AGI to personal freedoms. No matter what government ends up winning the AI race, what is it you believe that future might look like?

[D]

Yeah, so the world used to be more inegalitarian prior to the Industrial Revolution. You had the nobles and you had farmers and the nobles had a reasonable amount of power and most of the populace didn't. And we've had this move towards democracy and egalitarianism over the last few hundred years. And I think at least part of the story for that is just because human beings are very useful, we think we can contribute very productively to society. But in a post AGI world, that is a world where AI can do all the tasks, at least all the economically relevant tasks that human beings can do, you don't have any way of economically contributing to the world, so you can't sell your labor for wages. Instead, any income you have would have to be either because you own land, you own capital, or via government redistribution. But then that also just gives you a lot less bargaining power too. And so one of the structural reasons why I think we've had a proliferation of democracy and egalitarianism over the last couple of centuries really falls by the wayside. And to take a really extreme example of this, imagine we get to a world which again wouldn't be very far after the development of artificial general intelligence. Imagine you have an army that consists of AI and robots rather than human beings. Well, then we're in this very different circumstance where that whole army can be trained to be loyal to just one person. So if the President were to order a coup, then if the AIs were trained that way, they would loyally obey that there would be no question of disobedience. Unlike in the human case and in the limit, there's no reason at all why a single human being couldn't control essentially the whole economy and or military force if AI systems had been trained to do that. And that's a scenario that I think is really quite likely and extremely worrying.

[A]

All right, so what does being prepared for that risk look like? How do we mitigate that risk in the event that we do create a super intelligent AI?

[D]

Yeah, I mean, so the first thing is just to ensure that, especially in the early stages, individual actors aren't able to stage what is literally a coup, that could be what's called a self coup if the President decides to stay in power unlawfully, where if you've already automated, as in replaced with AIs, large parts of the military, or even a small kind of special guard that kind of protects the President, or if you've automated and replaced with AI, large fractions of the bureaucracy, it would just become, as a practical matter, much, much harder to unseat someone who wanted to become a dictator and stay in power unconstitutionally because they would have this small AI and robotic army able to protect them, or they would have perhaps some large fraction, perhaps even most of the government administration that are supporting them, depending on how exactly the AIs have been trained. In fact, there could be similar worries coming not from the President either. So leaders of AI companies themselves, if we're at this point of time, that AI capabilities and tech progress is going extremely quickly, that in fact, I think there are mechanisms by which the leaders of AI companies could themselves stage a coup if they wanted to as well. And so that's quite a lot more extreme than merely an erosion of democracy, though we should be worried about that too.

[A]

What does it look like for the US to get prepared to take seriously the threats that are posed by the state of the race between the US and China right now? What is it you think we should be arguing for, what should be done?

[G]

It's a really difficult problem. And my advice would be, if I could wave a magic wand, is if at all possible for people to put much more energy into diplomacy? I mean, again, it may. Who knows what's going on behind closed doors, but it feels like right now, at least publicly, no one is trying to do the diplomacy route in the US and China.

[A]

And are you imagining something here that looks like a superintelligence version of what we did with the nuclear arms race?

[G]

Yes.

[A]

Saying essentially, hey, what we're doing is not just dangerous to our adversaries, it's a danger to the whole planet. And so we need to come up with some kind of arrangement here where we can begin to disarm. And maybe the world isn't totally safe, but at least it's a much safer place than where it was at, say, like the height of the Cold War.

[G]

Yes, absolutely. And it's actually quite astonishing if you look back at how nuclear disarmament went so successfully after the fall of the Berlin Wall, because it was right around the end of the 80s that we had the peak number of nuclear weapons on Earth. I Think it was over 60,000. And through the Nuclear Arms Reduction Treaty, there were some really clever incentives set up of checks and balances of like different security teams sort of showing, okay, this is how we're disarming a little bit of tit for tat. They managed to break the sort of game theoretic stalemate which was such a magical thing. And it shows that such thing is possible in principle. One of the main ways that happened though was through diplomacy first. And I don't like the way the current narrative is of like these sort of China hawks and this saber rattling that is going on because it is just adding fuel to an already completely out of control fire. But the thing is, it is possible in principle. So yeah, that would really be my advice is just like, please can we exhaust all diplomatic paths first? And I think one of the big paths to that as well is through education is making people realize that actually our common enemy is not one another or even a difference in views. It's this idea of like game theory gone wrong. It's these game theoretic dilemmas that I often call Moloch essentially of like, well, if I don't do it then the other guy will, so I have to do it too. Essentially this sort of incentive trap that we get caught in that takes us into these arms race spirals. That's humanity's common enemy. And that's the thing we all need to sort of collectively look at and be like, oh, that's the asteroid coming towards us.

[A]

Well, I'm glad that you brought up Moloch, because I keep seeing it in all these different AI forums that I like snooping around inside of and people are just tossing around like, oh yeah, that's Moloch, that's Moloch. And I don't exactly know what it is, so explain it for me. What is it? Or maybe who is Moloch?

[G]

So Moloch is basically the personification of game theory gone wrong. It actually comes from an old Bible story about apparently in sort of the Canaanite times, there was this war obsessed cult that was so desperate to sort of accumulate military power and money that they were willing to sacrifice anything up to and including their literal children, allegedly by burning them in a bonfire in a ceremony to this deity they called Moloch, that they believed would then reward them for this ultimate sacrifice by giving them more military power and money. And so it's obviously an incredibly powerful and dark image. But really what it's a lesson in is like be careful in being so fixated on winning a narrow game, whatever game is right in front of you, or optimizing for this narrow metric of money or whatever it is that you're trying to win, that you don't sacrifice too much of the other things that you care about. And if you dig into sort of, it's often called the generator function. But the sort of driving force behind so many of our biggest problems is it is this process of like, well, I need to win at this game, so I didn't want to talk bad about my neighbor or backstab that person, but if I don't do it, I know that everyone else is going to be doing it anyway, so I have to do it too. It's this act of like sacrificing other important values to win a quick thing to get ahead of your opponents that when everybody does it, is what creates these race to the bottom dynamics. And unfortunately that's what I see going on in the AI world now.

[A]

I feel like this is a perfect encapsulation of what has happened to my own industry, to what I've seen as a reporter over the past 16, 17 years. This idea that you chase after the short term rewards that you get when you publish clickbait or hyperbole or when you tell everybody who is quote unquote on your side that they're totally right and look how awful and dangerous the other side is. And eventually you get to a situation where journalists and media outlets, they're just chasing that attention and the investment in careful journalism that has to fall to the wayside or even the idea that you might publish what's really happening with all the nuance that it demands, well, that becomes this huge risk because that's not going to do very well online. And before you know it, the whole industry has lost its core values and I believe in doing so bled out its trust.

[G]

It's one of the perfect examples of it. Because of the way the Internet works with virality, it happens that generally speaking, more negative stories, certainly more anger inducing stories, especially with very click baity headlines, tend to go viral more easily. And so those who adopt that strategy get a short term leg up over everyone else who doesn't. And over time that pushed even the most respectable news outlets into having to adopt more and more of those tactics. And I think that is basically the main driver of why we are in this information crisis now where no one really knows who to trust. And with good reason.

[A]

Yeah, people don't like to mention that part, but the with good reason is.

[G]

Important to know and that doesn't mean to say that there still aren't many high integrity journalists out there. But if you lean into this stuff once, people don't forget. And I view it kind of like a tragedy of the commons. You know, we talk about, you know, people throwing trash on the ground. Again, one person doing it, oh well, it doesn't matter. But when everybody does it now this beautiful park has turned into a trash heap. Well, that's kind of what's happened with our information commons because people have been polluting it more and more because it's a quick way of getting some eyeballs. And now the entire information ecosystem is just covered in trash and it's dying. That's Moloch in action. I call it the media Moloch.

[A]

All right, so what does that trap look like as you see it happening right now to the AI industry?

[G]

The Moloch trap that AI is caught in right now is the one of a lot of the AI leaders. I even know a couple of leaders at some of the labs. And they don't necessarily want to be releasing products as fast as they are doing. They'd like to spend more time on testing them. As we've seen so many times with new LLM releases, there are some really crazy, unexpected outcomes that these things were doing. There was like the whole Sydney thing, this weird Persona of on the Microsoft chatbot when they launched with GPT4 for the first time. It was threatening a journalist and it ended up on the front page of the New York Times.

[A]

There's my friend Kevin, who found himself talking to a chatbot that was actively trying to get him to leave his wife so that they could run away together.

[G]

Right. Google had their debacle with making the Black Nazis, basically woke image generation. And then this sycophancy of ChatGPT, which was incredibly shocking. These are all unexpected unintended consequences of clearly releasing models that just weren't ready. And okay, the damage on these was fairly limited. Okay, a few people either probably got a bit misled, but I mean we've already seen some people like there was that kid that killed himself because the chatbot he was talking to basically convinced him that he should. And these are all downstream results of products being released before they'd clearly been sufficiently tested. Now do this in two years time when these models are much more capable, they're much better at persuading people. They are also agentic in that they can actually take actions by themselves on the Internet without supervision. So they can actually do stuff that influences the real world. Everyone is in this rat race of who can release the biggest and best models the furthest to keep drumming up, you know, keep the hype machine going. It's an absolute recipe for disaster.

[A]

And the case that you're making is that just like what happened in the media, even if you do not want to be in this race, the incentives are pushing you into it, whether you like it or not.

[G]

Exactly. People I know who work at some of these labs, they're embarrassed by these mistakes. They would love to take more time before releasing their products on the general public, but if they don't release them, then they run the risk of losing their engineers who want to be associated with the latest and best products. It's an incredibly tricky situation that they're in. And so while it's ultimately like pressure should be placed upon those with the most power, I think some degree of sympathy needs to be given to them as well, because they're trapped in this dilemma. And if we aren't honest about the situation, then we don't stand a chance of fixing it.

[A]

I'd love it if you could walk me through some of the ideas that you lay out in your book, what We Owe the Future, where you're essentially trying to motivate people to change the way that they are looking at the world, to change the way that they're looking at AI. And I'd love if you could start off with this concept called longtermism. What is longtermism? And why do you think it's something that's going to help us get prepared in the long run?

[D]

So long termism is the view that we should be doing much more than we are currently to improve the lives of future generations, where the core reasons for thinking that are simply that the future could be very big. So there really could be enormous numbers of people to come.

[A]

And I've read your book, so I know that you mean very big, like billions and billions of people big. So say more about that.

[D]

Yeah. So if we take just our scientific understanding of the world seriously, humanity could last for an extremely long time. There are hundreds of millions of years left remaining on Earth. Billions of years. If we think that society could get to a level of technological sophistication such that beings could live off world, which, again, given our scientific understanding, seems extremely likely. And so that means that when we look to the future of civilization, we're used to thinking like, well, nowadays, five minutes ahead. But maybe if we think long term, we're used to thinking a decade ahead or even a century ahead. But really this place that we're in in terms of history is very early on indeed. If we don't suffer some huge calamity. Right.

[A]

In your book you say that we need to start to think of ourselves as the ancient ancestors to billions and billions of people to come. More people to come than people who have ever yet lived.

[D]

Yeah. So I mean, it's a really striking fact that most people just don't pay attention to or think about is just how early we are in civilization's history. A typical member of human or human originating civilization will be far in the distant future. And they will look back, maybe they'll listen to this conversation with maybe a sense of awe and wonder. But they will think of us as people from the distant past. And in particular they'll think of us as people who had enormous responsibility. Because decisions that we will be influencing and making in our lifetimes will affect actually that long run trajectory. It will affect what sort of lives they have.

[A]

And the case that you make is that we should care about those people, that we should be thinking actively today, that we should be making decisions today, thinking about the well being of those people in the future.

[D]

That's exactly right. So at this point I'm trying to argue for the idea that future people count, their interests matter morally, maybe just as much as interests of the people alive today. And so if there are some actions that impact not just the next few decades or few centuries, but will really impact the whole trajectory of future civilization, those problems that just utterly derail civilization, such that we don't come back from them, the world is worse in a way for this very long time into the future, that those at least become distinctively important and we as a society should distinctively care about them.

[A]

And you believe that one of the reasons that this is so important for us right now is because you think that we are living through a distinctly unique moment in the history of the human race. Make that case.

[D]

The thing that's unique about this point in time is how rapidly we're developing technologically and growing economically. Where for almost all of human history, when we were hunter gatherers and then when we were agriculturalists, there was very little change. The world that your children or even great grandchildren would be born into would generally look very similar to the world that you were born into. And it was only since the Industrial revolution that rates of technological development and economic growth picked up such that we have the kind of 2 to 3% annual growth rates and the rates of tech development that we're currently used to. So Things are changing much faster than they did in the past. And then we have all of these new risks, new dangers and challenges that will only happen once the development of what's called artificial general intelligence. There's only one moment at which that first gets developed, and I think it's really quite likely that happens within our lifetimes. And that is something that is unique about our current situation, something that's different from our ancestors, different from our distant descendants.

[A]

So just to put a fine point on this, you're advocating that we adopt this mindset of seeing ourselves as right now living in this unique moment in the course of human history, and that our actions, our decisions, are going to affect billions and billions of lives to come, in part because it's going to be us that bears witness to the creation of an AGI and what we end up doing with that creation. And we need to bring this sort of mindset to the decisions that we make about what we do next.

[D]

Exactly.

[A]

All right, so how long do you think we've got? Like, what if you had to put a number on it? Looking at the state of AI innovation and investment, where do you think things stand right now? How far away do you think we are from that AGI?

[D]

My best guess is that we'll get it in the early 2000 and 30s. So within the next 10 years, and where I think it's more likely than not, that leads to the sort of very rapid improvements in AI capabilities and very rapid technological progress. But I'm not extremely confident in that. There could be a slower move from where we are today to much greater technological capability.

[A]

When you look at the AI industry right now and you think about where we're at in terms of getting prepared or ensuring that we experience the win outcomes over the lose outcomes, what do you see? Because one of the things that I find really remarkable is this idea that the leaders in the industry are the very people who have been some of the most vocal about the dangers and of the negative consequences of the technology that they're making. And even without any government regulation forcing them to, many of them are spending billions of dollars on AI safety. And a lot of them are sharing their AI safety research and their findings with the public openly. That feels like a really remarkable thing. And so I wonder, does that give you a sense of optimism? Does that make you hopeful that the project you're engaged in right now might work?

[D]

I do think that gives me enormous optimism and hope that I wouldn't have had otherwise. And in fact, since we published what we owe the Future. That was just before the ChatGPT moment. Since then, I've seen this huge surge again in interest in AI safety and the seriousness with which people are taking it. And I think it's a very striking fact, but ultimately reassuring one, that the leaders of all three of the major labs and the whole top three of the most cited computer scientists of all time signed on to the statement that mitigating the risk of extinction from AI is a global priority, on a par with the risks of nuclear war or pandemics. That shows that, at least to some extent, people in power are really taking this quite seriously in a way that, for example, was really not true for the leaders of Exxon and chevron in the 70s when our understanding of climate change was just developing. So that all gives me optimism, but it's very, very far from sufficient. And I think the international political order is disappointing and getting worse on this front, where over the last 10 years there's just been a greater and greater emphasis on China as the enemy, especially with respect to AI. There is intense hawkishness such that, okay, yeah, the very strong default is that there will be an arms race over who can get to AI supremacy. And in fact, we're already in the middle of that. And that's a cause for concern, too, in some ways.

[A]

I get it that your camp gets locked in with the doomers a lot because you are also going around talking about the dangers and trying to ring this alarm. But I also see a lot of overlap between you and the accelerationists, especially in the way that you're trying to inspire people. You're trying to get people involved and really rally them to bring about an amazing future and not a catastrophe. And one of the things I've been thinking about is how much of our society right now is lacking in core beliefs about how religious participation is down, about how there are all these people who are out there trying to find their tribe and trying to find their purpose in things like politics, which is not panning out very well. And when I look at what you're doing, when I look at what the accelerationists are doing, it feels like you're saying, hey, guys, look around at all these problems that we face as a civilization. Look at how our institutions are letting us down, look at how nihilism is spreading throughout the world. This AGI thing, it might be the thing that ends up saving the day, that ends up changing everything, that brings about a freer, less hungry, less competitive, less violent world. Like this thing may lead to us living in whole new ways and curing all diseases. It might even mean that, like a generation or two from now, human beings will be traveling the galaxy. And I feel like trying to bring that technology, that hinge moment in history to fruition. That's an amazing thing to devote your life to. Like, that is something to believe in, to strive for. And it's basically the accelerationist's point. What you're saying, I know, is different. You're saying, yes, let's get there, it's great, but let's do it safely, let's do it smart. But do you feel a little bit like an accelerationist? Do you feel a kinship, maybe with them? Or do you feel like your projects are just totally apart? And I'm wrong here?

[G]

I mean, I want the awesome solarpunk future, you know, I want all of this. Does that make me an accelerationist? I think it just depends on your definition of accelerationism. I 100% think we have to build new institutions. I think most of our institutions are grossly outdated. They're crumbling. They're either non functional or actively making society worse. So I want to accelerate the building of new social structures that manage these incredibly powerful technologies that are emerging in our world. So the question is, it's like, what are we accelerating? I worry that so much of our innovation is going into technological solutions while we're not sufficiently building up the other supportive structures that are required for a lasting civilization, which are the social structures, the actual social institutions of how to manage these incredibly powerful technologies and tools and the state structures to manage the social structures.

[A]

So the idea is that it's not just about building the tech the right way. It's not just about making quote, unquote safe AGI. It's also about having universities and lawmakers and the media and society as a whole be robust and healthy and trustworthy and focused on these issues when the time for the technology arrives.

[G]

Yes, if you let the technology drive the social structure which drives the memes, that's where you end up in the race to the bottom. That's where you get Moloch. But if you flip that stack, if you come up with the good memes of like, what are the high philosophies that we want to to instill as a flourishing future that we give to our grandchildren, that we give to their descendants. If you come up with those and put the effort into those, then you build the social structures built upon those principles and that those social structures be the things that drive the technologies. That's when you get the inverse. That's when you get the win win outcomes. So I am an accelerator in that direction, but I do not want to accelerate the other direction. And my concern is that the current acceleration movement is doing the Mallocian version. They think that just build more powerful technologies, just more, more, more more of that and that will be sufficient. But we need to build the wisdom alongside the power. So I'm a wisdom accelerator carving out my niche there. That's what I want to accelerate. How do we accelerate the wisdom and the social structures that support that wisdom? I agree with your larger point that people need a North Star and we need some kind of religion as a sort of a motivator. And again, and what makes a good religion? What makes a good Shared story a common enemy. And to me that common enemy is this Malachian process.

[F]

Next time on the Last Invention, the accelerationists make their case. The Last Invention is produced by Longview Home for the curious and open minded. A special thanks this episode to Sam Harris, Scott Aronson and Tim Urban for links to William McCaskill's book and Liv Baris podcast as well as how to support Just look at the show notes to today's episode and if you like this show, please share it with your friends and your community and leave us a review on Apple or Spotify. It really helps others discover the show. Thanks for listening. We'll see you soon. This episode is sponsored by Ground News, the app that helps you spot media bias and see a broader picture of the news shaping our world world get 40% off their vantage plan at Ground News Invent this episode is sponsored by Fire Defending Free Thought in the age of AI. You can learn [e-mail] TheLastInvention.