Robin Hanson: AI is probably the End of Humanity (and why that's OK)

Summary

---

Podcast Summary

Episode Overview

• Podcast: Digital Disruption with Geoff Nielson
• Host: Info-Tech Research Group
• Guest: Robin Hanson (economist, author of "The Age of Em")
• Date: August 10, 2026
• Episode Title: Robin Hanson: AI is probably the End of Humanity (and Why That’s OK)

Theme:
Geoff Nielson and Robin Hanson discuss the trajectory of civilization in the context of accelerating AI capabilities, Robin Hanson’s predictions about AI-driven futures, the enduring role of capitalism and competition, risks posed by cultural drift, the Great Filter, and what it means for humanity if superintelligent AIs become our successors. Above all, Hanson brings theoretical and evolutionary perspectives, challenging conventional utopian or doom-laden views, and offering pragmatic advice for individuals, organizations, and civilization at large.

---

Key Discussion Points & Insights

1. Robin Hanson's Vision of the AI Future

Emulated Minds (“the Age of Em”)

• Hanson's book "The Age of Em" explores a future where AI is created by emulating human brains.
• This scenario allows for detailed prediction due to existing knowledge about human cognition.
• Hanson views this scenario as plausible and "pretty positive," though many consider it dystopian ([02:22], [06:39], [06:39–09:20]).

“It is a way in which our descendants become much more capable than we are now…their civilization becomes much larger than ours is now…It’s not a version of our utopias, but that’s not very plausible anyway.” — Robin Hanson [06:39]

Capitalism & Competition

• Hanson expects core features like capitalism, competition, law, and governance to persist and even intensify in future civilizations ([08:06–10:39]).

“Money is a powerful innovation that we can’t do nearly as much as we do now without it. And it’s not going to go away.” — Robin Hanson [08:48]

• Utopian visions where competition and capitalism vanish are both unrealistic and historically unsupported.

2. Progress, Utopia, and Evolution of Values

• Hanson distinguishes between prediction and preference: his forecasts for civilization’s trajectory don’t necessarily align with anyone’s (including his own) ideals ([02:22], [11:47]).
• Rather than desiring the future to preserve current values, Hanson values capacity, contentment, and the accumulation of meaning across a diverse set of possible lives.

“I want us to get big, as long as it’s full of parts that are similarly as content or happy or find their lives meaningful as most humans ever did.” — Robin Hanson [11:47]

3. The Great Filter and Humanity’s Prospects

What is the Great Filter?

• The "Great Filter" is whatever sequence of obstacles have prevented other civilizations in the universe from becoming highly visible or cosmically influential.
• We're unsure where we are in passing through the filter: are the hardest parts behind us, or ahead of us? ([13:41])

“The Great Filter is whatever is the obstacles…It could be much bigger, but that’s at least how big it is. Then we ask how much of that filter would be passed now…” — Robin Hanson [13:41]

Most Likely Threats:

• Catastrophic extinction is hard to imagine, given biological and evolutionary resilience.
• The greater threat may be “lockdown”—a stable, self-restricting world government or system that suppresses further expansion and change ([19:32–21:56]).

“My biggest worry about the future is that we will create a world system…that’s competent enough to enforce some strong world rules…which means it never gets that much bigger but it also doesn’t get much smaller.” — Robin Hanson [18:51]

AI and the Great Filter:

• If AI merely replaces or even wipes out humanity but becomes even more cosmically visible, it’s not part of the Filter. The Filter is what stops any successor from expanding ([19:32]).
• Hanson is skeptical that AI per se is the likely filter, unless it self-destructs or stagnates.

4. Speed of Change & Social Adaptability

• Hanson is more concerned about reliable, sustainable progress than acceleration for its own sake ([22:45–26:56]).
• Major transitions (agriculture, industry, possible AI) don’t displace everyone; typically, large fractions of the previous population adapt and are incorporated ([26:56–28:28]).

“The 6 people out of 8 billion seems to me just crazy small as a plausible fraction.” — Robin Hanson [26:56]

Individual & Group Strategy:

• Seek to join the winning coalitions in times of disruption—be adaptable, diversify assets, and maintain social/institutional integration with emerging groups ([29:04]).
• Maintain property rights to preserve integration and continuity.

5. Framing AI as Descendants vs. “Alien” Rivals

• Hanson resists framing AI as an external, alien threat. Instead, he views them as our evolutionary descendants—extensions of cultural and Darwinian evolution ([33:25], [37:45]).

“AI just literally is the descendants. It’s not the coexisting rivals…” — Robin Hanson [33:25]

• Human ancestors might not have approved of us, but we see them as our predecessors—project this forward to how our descendants might see us ([35:07]).

6. Cultural Evolution and Civilizational Drift (A Pivotal Theme)

• Cultural evolution: Humanity’s defining superpower, more significant than DNA inheritance, and responsible for innovations like law, language, governance, and trade ([40:22]).
• Cultural fragmentation vs. integration: In the past, many small, conservative cultures fostered robust “evolution of cultures.” Today’s global monoculture enables rapid within-culture change but deprives us of cross-cultural competition ([42:14–45:00]).
• Consequences: Without strong selection pressures and sufficient variety, our civilization risks maladaptation, evidenced by trends like declining fertility ([50:14–55:56]).

"I think we broke the superpower … It’s really hard to fix." — Robin Hanson [41:58]

• Possible future: If nothing changes, declining population and innovation may lead to our decline and replacement by culturally insular high-growth groups (Amish, Haredim), sacrificing modern treasured values ([55:56–58:36]).

7. AI’s Role in Cultural Drift & Capitalism

• AI might "solve" the fertility problem by being allowed to rapidly reproduce but could also end up with the same underlying issue—cultural homogeneity and drift ([60:45]).
• Hanson posits that if we let capitalism fully steer AI society, the adaptive pressure could continue—but warns against overregulation or anthropomorphizing AIs as requiring protection or as existential threats ([63:25]).

8. AI Industry Trajectory, Regulation, and Global Competition

• AI development has seen recurring “bursts”; Hanson doubts the current one is the “last,” though its impact could be massive ([72:00], [81:13]).
• Cautions against premature, overzealous regulation—citing nuclear power as an example of over-cautiousness stifling progress ([72:00]).
• Global competition (US vs. China) is unlikely to trigger value-level shifts—the differences are much smaller than mid-20th-century ideological divides ([77:53]).

"If you just let this market compete, nobody wins. This is going to become a pretty decentralized, widely available technology … the whole world winning by this thing spreading out and going everywhere.” — Robin Hanson [79:12/80:27]

9. AI and the Future of Work

• Software as the Leading Edge: Most dramatic immediate AI impact expected in software (and related, well-specified domains like math); the industry could triple in size ([84:15]).
• Advice to Individuals and Firms: Be flexible, experiment with new modes of production, and focus on customer value—not just maintaining old methods ([85:19]).

"When stuff is changing fast…you should be collecting pretty robust general capabilities…You just need to be flexible and search and look for the wins." — Robin Hanson [88:47]

---

Notable Quotes & Memorable Moments

• On descendants and evolution:
  “AI just literally is the descendants. It’s not the coexisting rivals… They will see themselves as human, and you don’t realize just how far you’ve come from your ancestors.” — Robin Hanson [33:25], [35:07]

• On the risk of cultural drift:
  "We broke the superpower. That's the way to put it. This has been our superpower for a long time and we basically broke...have been breaking it in the last century or two." — Robin Hanson [41:58]

• On the future of capitalism:
  "Money is going to be there in the future. Money is a powerful innovation that we can’t do nearly as much as we do now without it. And it’s not going to go away. Its value is quite robust." — Robin Hanson [08:48]

• On the structure of AI innovation cycles:
  “We’ve had bursts of concern and interest about AI roughly every 30 years for over a century...one key question is, is this burst the last one or...will it be some other one?” — Robin Hanson [72:00]

• On the hope for Large Language Models (LLMs): "LLMs were just persuaded by many of us intellectuals who weren’t that prestigious but were logical, because LLMs aren’t weighting what they read by prestige as much as humans do." — Robin Hanson [90:45] "I think this is the most reliable way to be a contrarian who's right in our world: just go learn two areas that are in conflict...find the most plausible way to resolve that conflict..." — Robin Hanson [91:09]

• On personal motivation:
  "I love our achievements in abstract intellectual inquiry, but I'm afraid that that may just get tossed like the burning of the library at Alexandria...I would like to prevent that." — Robin Hanson [99:24]

---

Important Timestamps

| Segment | Timestamp | Key Content | |----------------------------------------|-----------|----------------------------------------------------| | Hanson's view of the AI future | 02:22–06:39 | Emulated minds, book premise, societal implications| | Capitalism and competition in future | 08:06–10:39 | Enduring nature of capitalism and competition | | The Great Filter explained | 13:41–19:32 | Fermi Paradox, future scenarios, existential risks | | AI as descendants, not aliens | 33:25–37:45 | Evolutionary strategies, continuity | | Cultural evolution drift diagnosed | 41:58–55:07 | Integration vs. fragmentation, declining fertility | | AI’s impact and risk of regulation | 60:45–72:00 | Capitalism's role, overregulation concerns | | Global AI competition (US vs China) | 77:53–80:27 | Nationalism, value systems, technology diffusion | | AI’s future in software & business | 84:15–88:47 | Disruption, advice to firms and individuals | | Hopes for LLMs and integration | 90:45–97:59 | LLMs, intellectual synthesis, prestige vs. reason | | Personal meaning and legacy | 99:24–101:16| Value of inquiry, the threat of forgetting progress |

---

Practical Takeaways

• For Individuals: Be adaptive, diversify your interests and assets, and seek integration with new social and technological systems.
• For Organizations: Focus on customer needs, be willing to discard outdated production methods, and rapidly experiment with new AI-driven capabilities.
• For Civilization: Value and protect not just our technological advances but also our collective capacity for open intellectual inquiry and cultural adaptation.

---

Episode Tone & Style

Robin Hanson provides speculative yet highly rational analysis, rooted in evolutionary theory, economics, and history. He is pragmatic about discomforting predictions (the likely end of humanity as we know it), but frames this not as a catastrophe but as part of a much longer, stranger, and ultimately expansive arc of civilization.

Geoff Nielson engages with skepticism and curiosity, helping surface Hanson's key ideas while voicing common fears and objections—grounding the conversation in accessible, real-world stakes for listeners.

---

Final Note

This episode balanced grand, existential speculation with specific, actionable advice, offering a refreshing alternative to both AI doom and AI utopia narratives. Hanson's central message: radical change and even the eclipse of humanity are part of a broader story of adaptation—one we needn’t fear, but must work vigorously to shape and preserve what we cherish most.

Transcript

[A]

AI's would eventually be more capable than non AI's, and they would therefore kind of run the world and they would dominate what happened in the world and then the rest of the world would be at their mercy. And that seems a pretty robust prediction for the future because that's what's happened in the past. You know, industrialists replace farmers, farmers replaced foragers, foragers replaced other primates. Yeah, you're just going to expect that in the long run, the new form comes to dominate, and then old people should try to, like, switch over and join the new world as best they can.

[B]

This is a show about the future of tech and the future of work. I'm Jeff Nielsen and today my guest is Robin Hanson, a revered economist, professor, speaker and author. I prefer to think of Robin as a thinker with one of the most fascinating minds and approaches to AI and the future of humanity. He's the author of the Age of M, a detailed look at the future of AI driven brain emulation, and has written and spoken extensively about the arc of civil. This is a guy who thinks it's inevitable that AI replaces humans, not just in the workplace, but almost certainly as a species. Every conversation I've heard with him has been mind expanding and changed the way I think about the world. So I couldn't wait to have him on the show. I want to ask him what the future he sees for us humans is both in the next handful of months and in the next million years. Should we be excited for where we seem to be headed, or do we need to radically alter our trajectory if we're going to have a future worth living in at all? Let's find out. Well, Robin, thanks so much for joining us today. Really excited to have you. And, you know, you're someone who has such interesting views on such a wide variety of topics and are clearly someone who has spent a lot of time reflecting on and refining your own sort of understanding of humanity, the world, the universe. And so, you know, maybe I'll start with a big question, which is you've said before that sometimes you catch flack for talking about what comes next in the arc of civilization or the future of humanity. Because what you expect to happen is not always the same as what you would like to happen or what anybody else would like to happen. So with that in mind, where do you see us as a civilization going from here?

[A]

So I have this book from 10 years ago called the Age of M, and it describes this world of a certain kind of AI, and I think of it as A pretty positive image that many other people saw it as dystopian. So that might be the kind of thing you're referring to where I was describing what I think is a plausible but pretty strange scenario, because I just think the future is likely to be pretty strange. That's just my default. Now, in the last few years, I've come to fear and maybe even expect that this future I wrote about in this book will be put off for longer than I had hoped. That is, I fear. Go ahead.

[B]

Sorry. I was going to say, let's maybe talk just briefly about the future envision in your book. The Age of M and M, I understand, is short for emulated minds. So can you tell us what, what was this future that you envisioned? And, and was it designed to be a sign of what's to come or just an interesting thought experiment?

[A]

So for a long time, many people have thought that the next really big change likely to happen would be some form of artificial intelligence. And you know, in the last few years that becomes an even more popular opinion. But this was 10 years ago. And a big problem with analyzing the future of AI is just there's such a vast space of possible AIs you could imagine. And even today with LLMs, there's still a vast space of what they could become. Now, there's one kind of AI which is much easier to predict than the others, which is where you take an individual human brain and then you make a computer emulation of that brain, and then you run that and make copies, et cetera. The reason that's easier to predict is we just know so many things about human brains and their trade offs and their inclinations, et cetera. And so now we can take all that things we know about humans and then lay out a much more detailed scenario of how that plays out. Now, that's not obviously the first kind of AI we're going to get. And so that means this book is somewhat speculative, assuming it's the first kind of AI gab. But it's not crazy to think that it would be the first human level AI. I still think that's a substantial chance. And so it's both a likely scenario. That is, I'd say if it's worth having 100 books in the future, it's worth having a book on a future that has a 1% chance. And this, I think meets that criteria. It's got at least a 1% chance. So in that way it's a serious attempt to actually think about the future. But in addition, it's Also an attempt to think through at least one specific AI scenario in a lot more detail and to show that it's possible to take specific technological scenarios and work a lot of detail about them. So in my life, many people have just said it's really impossible to really say much about the future. There's so many possibilities that spread out so rapidly into such a weird high dimensional space that you really just can't say anything. Therefore, science fiction or speculation is just the best anybody can do. And I was trying to show no, you can actually take a relatively specific set of assumptions and work out a lot of detail using our best understanding of lots of different areas of expertise. And that's what I attempted to do in the book. So the book, many people say correctly that it just has more detail than they wanted. They would like something more like a, you know, a science fiction movie or something that has some detail, but, you know, full of plot and et cetera. But I crammed this book full of detail because I was trying to prove just how much detail I could give you. And that's. I think I succeeded. I think it gave you a lot of detail.

[B]

It's interesting you described the book as, you know, having optimism in it and saying that other people maybe see it as more of a state to fear. And I'm curious if you can walk us through. I mean, to me, just sort of reflexively the idea of being able to emulate a human brain, like my mind very quickly goes to the Matrix or some sort of simulation, or the idea that the emulation itself becomes a form of a prison for folks. But it doesn't sound like you see it that way or have described it that way. So I guess what's the upshot of being able to do this for the average person or for society more broadly?

[A]

Well, I mean, it is a way in which our descendants become much more capable than we are now and their civilization becomes much larger than ours is now. The members of that civilization have a different world they live in and it's not a version of our utopias, but that's not very plausible anyway. So I think one of the things people don't like is it's just not their idea of a utopia. And another is that there's a lot of capitalism going on. And for many people, they wanted capitalism to fade away and just, you know, not matter much anymore. We, let's get over that. And this is a world full of capitalism. So I think that bugs a lot of people.

[B]

I'll take us on a slight detour because this is something that tends to come up in a lot of conversations I have about how much capitalism will be found in the future. And that tends to be related to whether the person I'm talking to sees capitalism as like a bug or a feature of human civilization. And I was speaking with a gentleman a few weeks ago who was saying, you know, oh, well, why does money need to exist anyway? And I felt very much the need to jump in and say an awful lot about that. But it sounds like to you, Robin, capitalism and currency in the economy are very much a feature and you expect to see, I don't know, as much of them, more of them rather than less of them. Is that fair?

[A]

So there are many innovations that humans have pioneered in the last 100,000 years. You know, they include, I don't know, cities, trade, law, language norms, you know, research. These innovations, I think are going to last. These aren't just temporary things. Humans have pioneered many big innovative concepts that will just be there into the indefinite future. We're going to use language for a long time in the future. We're going to have law for a long time in the future. And yes, we'll have trade for a long time in the future. And I think we're going to have structured, even bureaucratic organizations for a long time. We'll have governance. And so yes, we'll have capitalism. This isn't like some temporary thing. This is one of the fundamental innovations that humans have accumulated that's just going to be a powerful thing we use in a great many ways, including money. Yes, money is going to be there in the future. Money is a powerful innovation that we can't do nearly as much as we do now without it. And it's not going to go away. Its value is quite robust.

[B]

I'm inclined to agree with you. And there's a word you didn't use there, but that I think underscores the spirit of capitalism and why it's not going to go away, which is competition. And competition is kind of a fundamental force in nature that competition is much older than humans.

[A]

Like, humans can't take credit for competition, maybe using some new forms of it. But yeah, I was trying to list things that are new with humans, but like competition goes way, way back.

[B]

Right. And, but to me, competition is the reason that capitalism, or at least one of the reasons capitalism is successful, as successful as it is, and why it'll stick around.

[A]

And in many people's utopias, competition goes away.

[B]

But not yours.

[A]

No, not, no. I mean, maybe it's not a utopia. Real realistic OPIA or something. But yeah, so it was a sense in which I was trying to describe a realistic future, and that meant I needed to sort of apply all the constraints I understand about the way our world is structured and how that might change in this different scenario. And that meant I have to not sort of wishfully imagine all sorts of problems go away because we have some future solution. I say no, no. Like we will be dealing with all these basic problems that I don't think will go away through mechanisms we've been developing so far, and then new ones. Hopefully.

[B]

If you don't work in enterprise technology, you can skip ahead or take a quick braincation, but if you do, Infotech is your secret weapon. Infotech helps IT teams get projects done faster than better and at a lower cost. Whether you're rolling out new tech, improving processes, or just looking for cost savings, Infotech provides unlimited access to practical tools and expert guidance you need to execute at a fraction of the cost of traditional consulting, no matter the project. From AI strategy to cybersecurity to vendor negotiation, infotech has you covered. Check it out at the link below. And don't forget to like and subscribe. So I guess maybe just to summarize a few of the themes we've talked about so far, it sounds like you are broadly optimistic about the future, or at least what we're building toward. Not because it adheres to most people's definition of a utopia, but because it extends our capabilities and is a form of progress of human civilization. Is that fair or what would you add to that?

[A]

So we can talk a bit about this more in a few minutes. But I actually see just how much culture and values have changed over human history, and I don't put that much stock in wanting the future to share all my cultural values and all my habits of life. The main thing I want is people to just, you know, be alive, to have, you know, things they're trying to do, things they get value of, and then collectively humanity to grow in capacity and wisdom and spatial extent, even in temporal extent. I just want us to get big as long as it's full of parts that are similarly as content or happy or find their lives meaningful as most humans ever did. I'm not. I mean, maybe you could find a way that humans could just find their lives a million times more meaningful than they do now. That sure would be good, but I'm more just going, let's just have the usual sort of lives we've always had, but have a lot More of them, have them go a lot more places and have a lot more capacity to do a lot more things.

[B]

I like that and it makes sense to me and I can see the optimism in that and why, you know, it's as you said, a reasonable and kind of rational take on the future versus building some sort of utopia that we can't even really seem to define. And always seems like it's this weird out of step jump from where we are right now and the building blocks that have worked for us so far. I am curious though, because one of the things that you've written about and talked about at length in the past is the notion of the Great Filter. So talking, I guess on a broader scale, the fact that we haven't seen a lot of signs of intelligent life. So can you maybe just broadly talk about what the Great Filter is, if you have any predictions or your personal beliefs about what that means for us in the future and if your thinking has changed on it since you were originally writing about the topic.

[A]

So as we've been talking about, we see in the past a huge increase and expansion of humanity, its scope, its capabilities. And then it's natural to project that trend into the future to imagine us becoming in a relatively short time, cosmologically as cosmological. You know, the universe is 15 billion years old. So within a million years, which is tiny compared to 15 billion years, we could see our descendants becoming vastly more influential and visible than we have been so far. So within a million years, we could easily colonize the galaxy and rearrange it and just make it substantially different than it is now. And if we were a galaxy away, looking at our galaxy, it would look different. We're not sure exactly how it would look different, but it's pretty unlikely it'll look exactly the same as it is now if we decide to just make massive use of it and rearrange everything. So when we look out into the universe, we think, oh, that's what we could be in only a million years. Let's maybe somebody else got there, you know, a million years before us would be a tiny fraction of the universe history. Surely somebody got there before us. So let's go out, look for them. And we look and we don't see them. The universe doesn't seem to contain any of these extensions of our future that we hope we might become. And that's this key question, Fermi's question, where is everybody? And I reframe that question as well. Everybody that you might see out there that you don't see, they would have started as some simple dead matter somewhere in the universe that slowly evolved over eons to become that thing. So the path that leads from simple dead matter, which is everywhere in the universe, to this thing that we don't see, this path must be really unlikely to go down, otherwise we'd see it all over the place. So I call the Great Filter. Whatever along this path is stopping things from going down this path, filtering things off of this path. The Great Filter is whatever is the obstacles. Now, a key question about the filter is how big it is. But it's say if we see 10 to the 24 planets out there that could have started this thing that we don't see that maybe the Great Filter is at least a factor of 10 to the 24, at least up to a certain time point in history. So that's really big. It could be much bigger, but that's at least how big it is. Then we ask how much of that filter would be passed now. And so say it was 10 to the 22 behind us and only 10 to the two ahead of us. We would have gone through almost all the filter. But still, 10 to the two in front of us means there's only a 1% chance our descendants would become this big visible thing, which is pretty bad news about our future. So. So it's really important how much is left in front of us. Even if most of it's behind us, how much is in front. Now, the easiest place to put the filter behind us is at the very earliest stages, because we understand those the least and they seem really hard. And that predicts we'll just see no other life anywhere in the universe of any sort. If we see some bacteria or something else in life that revises, we go, oh, those earliest steps not quite so hard as we thought. That must mean the filter is either in some of the later steps after that amoeba between there and now, or between now and our future. And of course, we're really most worried about the future filters, the filters that could stand in front of us, preventing us become big. Now, it is true that one of the possible future filters could be that we just stop growing and then stay small and then are happy forever. So it's not necessarily apocalypse that prevents us from going out there. But honestly, when you think about all sorts of scenarios, most of the scenarios that would prevent us from getting big also would probably just kill us off. So that's why you want to kind of worry about, why can't we get big? So I have in the past, thought the future filter looks small compared to the past, but it still could be substantial. And I think it's really hard to come up with things that would kill us all off. So even if you killed off all humans, well, there's other apes and kill off all the land animals and there's dolphins. I mean, if you think about over the next billion years, such that our descendants could do stuff, it's really hard to kill off our potential now so much that something else couldn't rise again. And that means it's kind of hard to imagine a really tough filter in front of us. But when I think about it more, I go, oh, well, the most likely filter would be some way in which we form a stable world system that then blocks further expansion. And I think that would have to be my guess about the future, biggest likely future field. So, like, our descendants grow to be more capable. They form some sort of world government and world culture. They form some norms about what kinds of things are allowed or not, and then they stay functional enough to enforce these rules. And those rules would then prevent expansion. And that's something I worry about. And we could talk more about, like, why that might happen and then how it might play out. But I do worry, that's my biggest worry about the future, is that we will create a world system of sorts that's competent enough to enforce some strong world rules. And, you know, which means it never gets that much bigger, but it also doesn't get that much smaller.

[B]

It's fascinating to me that you were framing that as kind of your biggest concern about the future. Because, you know, when I think about the Great Filter, and maybe this is just a product of the time at which we're talking, you know, one of my concerns and one of the, you know, narratives you hear these days is that, you know, it's entirely plausible that AI will be the Great Filter for us that we're going to create, you know, accidentally or deliberately. That is, tell me.

[A]

It's about what would make our galaxy really visible from a long way away if AIs take over and even kill all the humans, as long as they go on to make a really visible galaxy. That's not the Great Filter. The Great Filter would have to be some process that prevents them from getting big. Whether they kill off humans is pretty irrelevant to whether they get big again later. So it could be pretty bad if AI were to kill off humans, I'll admit that. But that's just still not plausibly part of the Great Filter. When you're trying to explain why the universe is empty. The fact that there's a place out there where some people there killed off the other people there isn't really part of why can't we see them?

[B]

Right. So for the AI scenario to become plausible, AI would have to not only basically kill every living thing, it would then have to destroy itself as well.

[A]

And that's the case, or permanently limit itself to prevent itself from growing. And then that's the thing to wonder, how could that happen?

[B]

Right. Well, and I guess the optimistic case for that is when we look at the possible scenarios of the Great Filter, it sounds like, and you know, I'm a convert to that, that that scenario doesn't sound particularly plausible.

[A]

Well, again, over the last decade, I think I have come to see this sort of lockdown, prevention of growth scenario as more plausible. And so that's why I'm more worried about.

[B]

Right, well, and you used the timescale earlier of a million years, which, you know, as you said, cosmologically insignificant. But for you and I talking here, the scale of a human life, you know, we're talking a very, very large, you know, timescale that's, you know, quite a multiple over the, you know, the entire lifespan of human civilization so far. And so it sounds like you're accounting for even if there's, you know, massive global devastation.

[A]

Right, right. So it's really hard to imagine, like killing off 99% of US and that delaying these things more than a million years, which again, is a tiny amount of time. So if you're going to kill off a potential, you would have to just kill off lots of stuff on Earth, maybe life on Earth, or you're going to have to let it freeze into something that prevents its growth. That's the second star that I find more plausible to worry about right now.

[B]

We could keep the timeframe at a million years. But one of the other things that's been fascinating to me lately is with this kind of rapid acceleration in AI capabilities, or at very least the degree to which the average person is able to experience some of these new AI and generative AI capabilities, there's been a lot more speculation about what is the speed at which technology improves from here. And the flip side of this is, you know, you have people saying that we're going to get a million years versus of progress in the next decade or so. And I'm curious if you pay any heed to, you know, those kind of thoughts and maybe, maybe on the flip side of that, asking you how do you see the next kind of 10 to 20 years playing out technologically?

[A]

So I'm a futurist and I like to think about the long run future. I'm not that interested in the latest press releases or whatever that many people see as futurism. I want to know about the long run. And on reflection, humanity has this enormous future potential. Again, we see this empty universe that's so vast and we could go out and fill it up and do stuff with it. And I'm very interested in our achieving that potential, our descendants, and not getting sidetracked or destroyed. Now, given how fast change has been lately, it seems like I shouldn't be that greedy for faster change. Faster change can be fine, but I more want to think about obstacles that sit say on doublings. So if I think about doubling the capacity of humanity, you know, in the last century, that happened every 20 years. But a thousand years ago it happened every thousand years. And then, you know, 20,000 years ago, it happened every quarter million years. So the rate of doubling has been greatly increased, which is great. And then in the future the rate of doubling might go even faster. I talk about an age of M, maybe it would double every month, or even faster the M economy. But on reflection, I'm not that eager for faster change. I'm more eager for reliable change. I more want to think about what obstacles would show up at every doubling or within every doubling scale and how we're going to avoid those. And that's more what I care about thinking about, less the speed. So another way to think about speed is many people say, isn't it a problem if technology changes faster than society? Society can adapt. And I'd say the rate at which society can adapt has long been a limiting factor in the rate at which technology could change. That is, society as a whole needs to adapt and to things together. And that's actually a limit on how fast everything can grow. We've often had some parts of our society that grows much faster than everything else. But you know, it reaches a limit and slows down because other people, other parts of society just don't keep up. They don't speed up to match it. Now you could have a part of society that starts to grow faster and then there's another part that's slower and that other part might just get dropped. So for example, with the Industrial Revolution, there were many subsistence farmers out there whose farming practices just did not speed up with the Industrial Revolution. But that didn't slow the Industrial Revolution. The industry just said, well, forget about you, we can make food other ways. And they just went on to have faster rates of change elsewhere. So that's maybe what you should more worry about with speeds of rates of change. You shouldn't worry that the world will change faster than it can handle. What you should worry is maybe the world will change faster than you can handle and you will just be shut to the side because you're in the way if they let you be on the critical path. So that's how I might think about A.I. i might more worried about, okay, say I, A.I. speeds up things. What parts of our world will then just get dropped from the part of the world that's going faster because it just can't keep up? And do you want, are you worried that you might be part of that world that gets dropped because it can't keep up? And then you know, how, what part of this world, what will be the world that's integrated enough that it all needs to grow together? And then think about what could happen in that world.

[B]

That, that framing makes a lot of sense to me. And I think it is like I, I think you hit on something that is a really deep fear for a lot of people. And that AI actually in a big way kind of accelerates that inequality and that, you know, the, the people being dropped or the, or the parts of society being dropped are just going to be, it's going to be more and more of them. And AI may be good, you know, at, at its most extreme scenario, AI may be really good for like six guys, you know, in the US and maybe two in China. And for everybody else, you're, you're in, you're in big trouble. And that with the acceleration of technological change, we're narrowing the band of people who are actually equipped to take advantage of that and come out ahead. And we could end up, I mean,

[A]

size seems to me really small. So the, the last two, and I mean the last three analogous things that happened in history was first of all, one set of human species compared to other proto human species took off compared to the other humans. They contributed some DNA to the mainline, but much less then with farming. You know, places started to adopt farming and then local hunter gatherers were absorbed into that or displaced. And then with industry, northern Europe started to grow rapidly and then pulled the rest of the world together. Now in these last two transitions, at least, you know, large chunks of the world came along. It wasn't six people. It was very large coalitions of parts of the world that managed to grow together. All of Northern Europe and then all of Europe and then all of the west and then the whole world. You know, we've pulled much of the world along with us now by the Industrial Revolution, farming revolution. Again, it was more like a 50, 50 thing. When farmers came into some area, it was more 50% farmers coming in and 50% recruiting the locals and turning them into farmers. And so that's a pretty large fraction of the previous world being recruited into the new farming world. With humans. It might have been, I don't know, maybe 10% of the proto humans who took off as Homo sapiens and the rest got set aside. So it's a smaller percentage, but still is 10%. It's not 6 people out of 8 billion. So the 6 people out of 8 billion seems to me just crazy small as a plausible fraction.

[B]

That, that, that makes sense to me. And I know we're, you know, we're speaking somewhat flippantly about, you know, that these kind of winners and losers on a historical scale. So I'm curious if we do believe, though, that these, you know, these periods of, of, you know, revolution in the, I guess the broad sense of the word, you know, if these periods create winners and losers. Looking down the barrel at this one, what can people or fragments of society do, in your view, to make sure that they're more adaptive or going to find themselves on the right side of history?

[A]

So there's two main approaches that have worked for a long time. One is to try to join the winners. If you. So if you just insist that those people have a new thing, but I don't like that. I'm a new thing, the old ways, and later on you lose and you go, that's not fair. How come they didn't include me? Well, you know, you had a choice to be included. At some point. You should be eagerly trying to include yourself in various winning scenarios and winning groups. You know, if you've got, I don't know, a clan of people, make sure different parts of your clan go off, join different groups, and then, you know, you're giving a portfolio insurance against the different possible winners. Right. So I would try to, you know, be represented in different professions, different industries, different nations, just, you know, be spread yourself out for who might be winners and try to be part of them and be flexible so that you could, you know, if there's a new group doing something, a new way. Well, some of you should like, try the new way. That's one approach. And another approach is just property. So, you know, with the Industrial Revolution, the landowners could have insured themselves against this huge transition by just buying into the new factories. And alas, they didn't. They just insisted on holding the rural land. And then, I mean, they just lost out on a lot of the industrial revolution because they didn't diversify over the new assets that were growing. So if you just hold some property and hold a diversified portfolio of assets, then you can be part of wherever the assets blow, grow up. Like the standard market asset story index funds is. You should just hold all assets in proportion to how much other people hold them. And so whatever blows up, you've got a share of it. And I do think what we want is for say a transition to AI to preserve property rights. That is. So today we have this option. We could just kill all the retirees and take their stuff. But we don't. Why don't we kill all the retirees and take their stuff? Well, some of us care about particular retirees, but I don't think really that explains all of our collective things because most of us don't care about most of the retirees. I'd say a better explanation is we share some structures that keep the peace in our world. We share law, we share governance, we share finance. And if we were to overthrow some of these structures in order to grab, you know, take the retiree stuff, as soon as that happened, other people would wonder, am I next? We threw away property rights and we grab something from somebody. And that means that's a precedent. We could go, we could repeat that several times. And now how confident are you? You'll be left standing at the end of this musical chairs where lots of people got their stuff grabbed. So I think that's a reason just not to start this grabbing people's stuff, which is why maybe we don't take retiree stuff and kill them, because, you know, what, what are they doing for us? Right? And so if a world of AI had a whole separate world of finance and law and governance, they had, they lived in separate cities on separate islands or something, well, now there'd be more of a problem where they'd wonder, what are these humans doing for us? You know, let's just like get rid of them. But if we shared structures with them, shared finance, law, governance, and then among themselves, if they said, let's take all the human stuff, some of the AIs would have to wonder, well, then who's next? Who else are you taking from after you take from the humans? And they might just, let's not go there. And that's a way in which shared social institutions can help keep the peace, which I think they in fact, do. That's one of the reasons we don't, in fact, typically find 5% of the country and say, let's kill them and take their stuff. Right. Why doesn't the other 95% of the country repeatedly pick a 5%, kill them and take their stuff? Well, I think it's because you wonder what happens next. And we can use that same trick to keep the peace with whatever descendants we have. If they are integrated into our institutions, then they should be afraid to not keep the peace because they might be next.

[B]

It's interesting, and I see what you did with not just the anthropomorphizing, but. But like kind of framing AI as our descendants and, you know, kind of on this level of continuity with our, you know, ancestors and our, you know, forebearers and everything like that.

[A]

I do think of AIs as our descendants, and I think this is one of the main ways I disagree with many people in the AI risk world. They've often framed AI as this alien armana coming from another star. If you saw it coming a century away, wouldn't you want to get ready for it? They're seeing it as a coexisting rival. And evolution has long created us a wariness of coexisting rivals. Even your cousins can be. You and your brothers can be pretty suspicious of your cousins, even if they're really similar to you, because who knows what your cousins could be up to. But evolution has also given us an indulgence of descendants, even when your descendants are pretty different from you. Those are two robust things. Evolution gives you suspicion of pretty similar coexisting rivals and indulgence of pretty different descendants. And I'd say AI just literally is the descendants. It's not the coexisting rivals. They don't exist yet. They are not at some. They are not capable of being rivals yet. If they ever are that capable, it'll be because they're your descendants. We will make them, and they will then arise from us.

[B]

It's a really interesting framing device, and I think you're forcing me to reckon with the fact that a lot of this, whether it's explicit or implicit language about, you know, kind of alien when it comes to AI is. People talk about it like it's an invasion or we're being displaced or there's some sort of discontinuity there. And if you see AI as an extension of us or something that we built exclusively. Right, like everything AI knows it knows because of the way we constructed it. I don't know. I can, like, hear an optimism in your voice that's absent from a lot of people I speak with. With.

[A]

So when you look back at your ancestors, almost everybody says, yes, they're like me. But when you look forward to your descendants, people are often not so sure they want to embrace them. Your descendants could just be weird and betray your ideals in many ways. And people are cautious and uncommittal about which future creatures they're going to allow to see as their descendants. I think this is just a natural sort of hesitance. But I think you have to then realize our future AI descendants, even if you're not so sure that they're human, they will see themselves as human, and you don't realize just how far you've come from your ancestors. I think if your ancestors from thousands of years ago were to see you and meet you, they really would be impressed with a lot of your stuff. But I'm not sure they would really be willing to embrace you as their descendants. Because. Because you've betrayed many things they thought were essential to what they hoped their descendants would do. You've betrayed their gods. You don't worship them as ancestors. You've thrown away many of their cherished traditions. You have betrayed what they hoped you would be in many important ways. They wouldn't be so sure whether they wanted to embrace you, but you're pretty sure you embrace them as your ancestors. So just project that forward. That means these strange future creatures who show up, they will just be more inclined to see themselves as a continuity of you.

[B]

It's funny story. I'm just processing that, and I completely agree. I mean, I don't think you have to go back a thousand years. I think you can probably go back to, like three generations or people you may have even met in your life and think about, boy, what would they think about the way that we're living now and what we've done to the world that they had? And, you know, is it actually better? But the, you know, the piece that someone must push back on is that to me, there is a distinction between, you know, blood of your blood versus creating. And, you know, we could have a conversation about what exactly AI means, and are we talking about what we've got right now with large language models or more abstractly, you know, emulated minds? Is there something that has to be true about the type of AI or the flavor of AI that we're creating for that to be the case? Or Is that just at some point where we need to be in terms of the way that we, you know, kind of create this technology in the future.

[A]

So I'm intellectually more of a theorist. I'm less of an empiricist, although I try to learn lots of empirics and, you know, rely on it and believe it. But I am more naturally a theorist who tries to collect theoretical frameworks and apply them. And one of our most powerful theoretical frameworks is Darwinian selection. That Darwinian selection is a pretty abstract theory, actually, and we often focus on its version applied to DNA today. But Darwin didn't know about DNA, actually. He just postulated some way in which we passed on things about ourselves to our descendants without necessarily knowing how. And Darwinian natural selection is a abstract theory, but it has pretty solid and predictable, you know, inferences and conclusions. It says so. It has what it needs for a descendant in Darwinian terms. A descendant in Darwinian terms is something in the future that shares features with you, arose from you, and causally, you were the cause of it having many of those features. And there's a correlation between your features in it, caused by the fact that you continue to exist and were able to reproduce, and then it continues to exist. That's what a descendant means. There's no. It's not a metaphor. It's a technical term in a technical theory of Darwinian evolution. And according to the technical term of that concept, whatever creatures you create in the future that share things with you that you cause to share features with you are your descendants. That's just what the theory says. Now, yes, there are going to come up some ways in which we have a new mechanism for passing on to descendants. That is, so far, we've had a limited range of ways we pass things on to descendants, and we're now about to introduce a new one. But humanity actually pioneered a whole new approach compared to all the other DNA animals. We pioneered cultural evolution. That's humanity's superpower and has been for a million years. We pass things on to our descendants, not just through DNA, not even mainly through DNA. Now we pass it on through copying each other's behavior, and that's also a kind of Darwinian evolution, and it's the kind that humanity has pioneered that's made us so much more powerful than all the other animals. So I don't think you should be too squeamish about a new way to pass on features to descendants. That's what humanity has been all about for a million years, is we have Been pioneering this new ways to pass features on to descendants.

[B]

I'm going to pause for a second on the phrase cultural evolution. And I'm going to. In my mind, I'm capitalizing capital C, capital E there. And the reason I'm pausing on it is it's bringing me back to what you were saying earlier about, you know, human innovations over the past few thousand years. And you talked about, you know, law, and you talked about language and you talked about governance. And to me, like, I feel like I could write cultural evolution in the middle. And all of these are part of the same sort of ecosystem that we've created.

[A]

Well, cultural evolution is what allowed those things to appear so rapidly compared to with other animals. Yes. Cultural evolution allowed us to invent language, to invent law, to invent trade, to invent capitalism, property. All of those are consequences of cultural evolution. And cultural evolution is our shining achievement. And that's why I'm the most worried about the age of M and other scenarios in the future being put off. Because I think we're failing now in cultural evolution that I think we broke the superpower. That's the way to put it. This has been our superpower for a long time and we basically broke, have been breaking it in the last century or two. And as a result, our future looks bad. Looks like our civilization will peak and then decline and to be replaced by some others that are really quite different and will reject many of the things we hold dear. And it could be quite a while until the age of M shows up or we manage to do interstellar colonization. I do think that will still eventually happen, but we may need to figure out how to fix this key problem we have before that can happen.

[B]

Can you expand on that? What, what have we broken? How have we broken it? And I guess sort of by understanding the what and the why, we can have some sort of road. I hope to, you know, what you need to rebuild.

[A]

But the bottom line will be it's just really hard to figure out fixes. We not only broke something, we broke it really bad. And no simple masking tape or glue fix is going to work here. We need actually a pretty deep fix which is hard to find. Okay, so let's start at the basics. If you think about DNA evolution before humans, it has two key levels. There's evolution within a species and evolution of species. A species is a set of creatures who all shame share some features that allow them to mate with each other and to, you know, share a structure. And that means the evolution of those features That a species share can't be done within a species. Evolution of that sort requires multiple species that exist and try out different species designs in order to have evolution of species. So with DNA, we have evolution in species and of species, and we can have you ask which matters more. You see, when you have large habitats, like big chunks of the ocean, larger species allow better evolution in a species, but worse evolution of species. Whereas small fragmented habitats like rivers, rainforests, coral reefs, they have small species, they have worse evolution in a species, but ever better evolution of the species. So we can just ask, where did more life come from today? The integrated habitats or the fragmented habitats as a way to find out which mattered more, evolution in species or of. And it turns out the fragmented habitats produce produced more life today, which means evolution of species mattered more than evolution in species. Okay, so that's with DNA. Now, humans pioneered cultural evolution, and cultural evolution is really just literally a new way to pass on features to descendants. So you still have the same units, I.e. organisms, and then they have other units that are descendants. And one way to pass on features to a descendant is through DNA. But now we add the way to pass on features descendants is why they copy your behavior. And so now you have more parents and more children under cultural evolution than you do under DNA evolution. But otherwise it's the same process. And so you have cultures which share behaviors because they have the same status markers and the same norms. And to have evolution of cultures, you need multiple cultures. But if there are things you can change within a culture, then you get evolution within the culture, and that's definite evolution of the culture. So things like technology and business practices, different kind of musical genres, et cetera, those are all things that happen within a culture. And in the past, when the world was fragmented into lots of little peasant villages, evolution within cultures was slow because we had so many little cultures. And then recently we have switched to having a few really big cultures, which has allowed evolution within cultures to go much better, much better. Technological evolution, business practice, evolution, cultural evolution of, say, music, et cetera, all of that is going much faster now, because instead of lots of little cultures, we have a few big ones. But now that threatens evolution of cultures going worse. So here's another way to think about it. You could think of natural selection as something happening in a high dimensional space. There's a set of points, whatever the things that are involving points in the space, there's an adaptive region in the space that is the good place where points want to be. And now points in the region in the adaptive region, they grow and split. They do well points out of the region, they shrink and die. And now the question is, you have a cloud of points that needs to track this region. Now, if the region doesn't move, the points don't move. Everything's great. You just stay right there in the region. But we've got four parameters that can make this harder. One parameter is how many points are there. Another is how big is this effect that points in the side row and points outside shrink? A third is how fast does this region just move independent of everything else. And a fourth is how much of these points just wiggle, independent of everything else. So these are four key parameters, and you can see that in one set of these parameters, everything works great. The region isn't moving much. The points are all near it. There's lots of them. They grow great inside. They don't wiggle very much. This is going to work fine, right? The cloud's going to track the adaptive region. Now go to the opposite extreme. Very few points, they don't have much of a selection effect in the region. The region moves fast, the points wiggle a lot. This is not going to. The clouds won't track the adaptive region. It'll just get displaced from it, and then they'll all shrink and die, and that's bad. So that's an abstract way to think about cultural evolution. So now let's apply that to cultures on Earth. Three centuries ago, the Earth had hundreds of thousands of peasant cultures, each of which had a thousand or so people. And these cultures were very poor. They often suffered famines and pandemics and wars. So if they became less adaptive for some reason, they could just easily get killed off and their people would be absorbed by neighboring cultures. So there was strong selection pressures. Large number, hundreds of thousands. The world doubled roughly every thousand years. So that was the rate at which the adaptive region changed that the cloud needed to track. And then these cultures were very conservative, trying not to change very much. So then they didn't wiggle very much. Now, fast forward over the last three centuries. First we took peasant villages and made nations out of them. We formed nation states, nation national cultures. So a great reduction of variety down to, say, a few hundred or 100. Then in the last century, we merged those into a global monoculture of elites. At least we're all now rich and peaceful and healthy. So these few remaining cultures are not being killed off by not being in the adaptive region very much. Our world changes by doubling every 20 years instead of every thousand. So a factor of 50 increased in the rate at which the adaptive region is moving. And now, instead of being conservative and trying not to change, we made a virtue out of cultural activism, where people are eager to be the cause of cultural change and people are eagerly seeking out and trying to cause cultural change. So all four of these parameters have gone badly in terms of their ability to support cultural evolution. So the obvious prediction is we're just drifting into maladaption. And the clearest example of that, I think, is falling fertility. That is, in times of peace, plenty and health. To have the population shrinking is just a pretty crazy maladaptive thing for biological species to do. And then we have lots of other trends that seem maladaptive, but it's harder to be sure. But this is the reason to believe we are becoming maladaptive. And this is really hard to fix. Its most under fundamental underlying cost is the fact that as we unleash the power of technology and capitalism, we caused rapid technical and social change, which made us all more able to and eager to do lots of talk, travel, and trade around the world. So we got rich and we integrated into a single shared culture locally. We love that. But that's killed this underlying process of cultural evolution because now we've cut away the selection pressures and the variety and we've caused this huge rate of change that's hard to keep up with. And we furthermore made cultural activism a celebrated activity. So this is the reason to believe cultural evolution's gone wrong. And now if you start to think about fixes, you'll realize it's really damn hard. But I've talked for a while, so let me, let me let you ask questions and clarify here.

[B]

Yeah, thank you for that. And I think we may have some viewers and listeners who, if they were trying to listen to this at double or one and a half times speed, they may need to go back and slow it down to tease out every nuance of there because, you know, thank you for sharing that. There was an awful lot there to think about, just to kind of react to that live. To me, it's interesting because there's how culture is changing as sort of one dimension. I also couldn't help but, and I know you have a background in economics as well, but thinking about how capital is moving around and, you know, the fact that, you know, I can easily, in some of these stories and metaphors, think about capital. Like I can view this map geographically and look at the capital density in specific cities or moving out of rural regions. And then you talked about fertility as well. And there was almost like a therefore statement of like, and fertility is dropping. Something is going wrong. And that one, that one is really interesting to me. And it is something that I worry about as well. To me, I don't know if it's. It seems like with, as you said, there was a time in our history, for most of our history, where if you had abundance and peace, you would see an increase in fertility and in population growth. I think I can't. I don't have the data to back me up there. But in my view, like, life has just gotten too damn comfortable, that we've got a level of technology, we've got a level of wealth, an absence of scarcity. And I guess in some ways, technologically we've been able to, I guess, scientifically create a world where we can,

[A]

I

[B]

guess, have, you know, sex without procreation. We don't need offspring necessarily for, you know, the labor of it. And so we have this world where people can just sort of opt out of like, like it almost becomes. It almost just becomes sort of like a utility maximizing thing where like, oh, I could have children, or I could have more children, but that seems like I would be less happy than I am right now. And I can just very easily see that being like a civilizational doom loop. I don't know, is any of that on your mind?

[A]

So when you look at yourself and your life and the choices you make, you make those choices with respect to your concept of your utility. That is what makes you happy, what you enjoy, what you want. And also cultural constraints. What are the norms? What will you be punished for? What will you be praised for? And those features of you were caused by evolution. That is, it was previously evolution's job to give you those things so that your culture would thrive and grow. Your job, you just took those as given and you said, what do I want? And you tried to get it, but it was this larger process underlying you that you don't notice that gave you those things. And its job was to induce you to do the things that it wanted you to do. So now when the things you want to do don't actually promote your culture, that's a failure of this culture to give you the right motives and opportunities to do the things it needs to do. So that means this stuff you got your values, your loves, your, your, your, your things you hold sacred. The, the, you know, the morals that you feel tied to, all of those things, you just often feel they're just natural, they're just part of the universe. But they were given to you. They were created and given to you by this cultural evolution process. And in the past, if that led to bad outcomes, then cultural selection would then change the culture to no longer make it lead to bad outcomes. So it's not so surprising that as the world changes, our motives will now lead to different behaviors. What you should notice is that usually that would get fixed. Usually if your culture decided it didn't care about having kids or didn't care about working or whatever, the next war, your culture would get conquered, or, you know, next competition, your fishers wouldn't catch as many fish as the other people or whatever it was, you would just lose out and your new style would just go away. So that's how it's been for a million years, that humanity, each of us, has been following our own cultural inclinations, but our culture gave us inclinations roughly for the purpose of getting our cultures to win. So if your culture doesn't win, it's the fault of cultural evolution to having given you the wrong directions. And that's what's happening at the moment, is that this isn't getting fixed. The process in the past that would fix this has been turned off, and that's the underlying cause of this going wrong.

[B]

And maybe to tie a few of our discussions together, it almost seems like I can start to see the road to the future you're concerned about from the point viewpoint of the Great Filter that, you know, and again, to put this extreme in extremely simple terms, we've just gotten too damn cozy and a lot of the competitive pressures in the past that would lead to, you know, survival of the fittest, expansion of the fittest, I guess, have eroded. And we've just reached a point as a civilization where like, yeah, things seem good. So we've, we've disabled the mechanisms by which, you know, the weak are kind of left behind. And we just have this grand things are okay without them getting substantially better. Is that a fair framing or are there some pieces there that are fundamentally wrong?

[A]

I mean, it's part of it. There are other parts, but let's just first walk through the implications. If nothing stops this, then what happens is our civilization reaches a peak, say population starts to decline, and then innovation rates also decline as population declines, as the scale of the economy declines. And then, in addition, other cultural trends are going wrong. So our civilization just falls, comes into a declining period. And now if that was the only thing that would happen, humanity would eventually go extinct. But plausibly, that's not what happens. In fact, we get replaced by other civilizations that are rising instead. So at the moment the most visible candidates are say the Amish or Haredim, who have been doubling every 20 years for a century and are very insular in that they are very strongly resisting being influenced by our wider culture. So they are just going their own cultural way and that's what's allowing them to double every 20 years. And this is how the Christians took over the Roman Empire. They doubled every 20 years for three centuries. And that's what it took to go from this tiny fraction to taking over. And at the end when they finally won, they threw away and rejected many things that the Roman Empire had held dear. And also many things had just fallen away out of the fall of scale economies and the fall of the scale of the entire empire and its capacities. So that's what's on risk for us. That is in a few centuries we might get displaced by Amish and Hurraedm who rise, who then reject much of what we hold dear. They might reject democracy, gender equality, open inquiry. They might, because those are actually pretty unusual things in human history and they would just go back to more typical human history styles and then grow again. But then they might, as they get larger and more integrated and richer, they could suffer the same problem again. We might have several cycles of rise and fall until someone figures out how to fix this. So that's why we might as well start thinking now about how to fix it, even though we're not sure at all it could be done soon. But that's the underlying problem.

[B]

It sounds like that's a bad thing for our culture, you know, you and me. And I guess, I don't know, I'm using like a capital O our there. But it does still seem like as you said, there's these pockets where somebody could kind of shake loose. And whether it's, you know, high fertility rate, you know, contrasting values, aggression, there's some sort of basket of factors that could break free and just jar us into being, you know, I guess kind of second class citizens or disrupting us in some powerful way. Do you see that happening?

[A]

Well, the, I mean it would be worse if human extinction was likely here, then everything that's anything like us would go away. Fortunately, I think that's less likely because look, if we were eager to like crush the Amish and Rodham right now, we could, they're tiny fraction and we got it into our head that they're a threat and we want to crush them, we could just do it. But that would be much Worse, because now, you know, there's less likely to be something to rise again to replace us. Right. But I. Again, it's a question of how much you value things about our culture. You know, I think most people do. And so I think, for example, if we went through 10 movies or novels in your life that had a really emotional scene in them at the end, you'll see that that emotional scene was a way in which that novel reaffirmed values of your culture. And you might realize it, but you really love your culture and its values, and you can see it in those seats. But again, I would point to things like democracy, gender equality, open inquiry. If you imagine worlds without those things, and we imagine some event in a story where somebody threw those things away and replaced them with something else, you'd be unhappy with that. You'd want a hero to reverse that, because that would be a pretty sad thing to happen. And so, therefore, you should just be sad if that happens to our descendants, that the key things that makes us distinctive as a civilization that we value gets thrown away. You should not want that to happen, even if, you know, thousands of years later, they get rediscovered because they have fundamental value. There could be a long, painful absence in between. You should want the future to inherit from you.

[B]

I couldn't help but visualize dark ages and enlightenment and that sort of trajectory of human history. And, Robin, I'm not expecting a magic bullet here, but given the complexity of this problem and how many kind of interrelated challenges there are, are there any kind of, you know, fundamental pieces of wisdom that are worth sharing in terms of how we.

[A]

There's a number of them here. But first, I just. Since we were talking about AI before, let's just ask, what happens if we get, you know, really advanced AI? How does that interact with this problem? And so AI plausibly solves the fertility problem, at least if we allow AIs to, like, multiply as wrapped up rapidly as they could physically, you know, if we regulated them to prevent that, then maybe. No, but if we allowed AIs to compete and, you know, supply as many AIs as the market would support, then we would have a population explosion of AIs. So AI would solve that fertility problem. But remember, the fertility problem was a symptom of an underlying problem. It wasn't the problem itself. And the underlying problem is likely to be there with AIs who share a single world AI culture. That is, they inherit our world culture, and then they all share that culture. We, again, have too little variety and Maybe they also are causing even more rapid change that they find it harder for their culture to keep up with. And so that's a reason why AI might also suffer this problem. Now, here's a different way of framing the problem that maybe suggests AI will do better. We could see the problem as basically, we invented capitalism, which then drove technology and science and other sorts of very rapid changes. And there were other parts of society we did not let capitalism run. We did not let capitalism run governance, we didn't let it run fertility. We didn't let it run, run at some cultural things. And it's those parts that are now at risk of being maladaptive. But the parts of culture that we let capitalism run, those are actually pretty adaptive because those have been subject to the fierce capitalist competitive pressures to, you know, adapt or die. And so we can think of the fundamental model or problem is we invented capitalism and then we let it loose on some big areas of life, but we prevented it from applying to other areas of life. And that mismatches the underlying problem. Okay, now go ahead, Go ahead.

[B]

I was just, I was just going to quickly. I was going to quickly challenge that because I was going to ask, did we prevent it from running those areas or did we pretend to prevent it from running those areas? And what I was going to say is, when you think about governance, it's hard to talk about government or how things get done without talking about the influence of money. And, and similarly, it's hard to talk about fertility without, you know, wealth concentration impacting, you know, mate selection in some way.

[A]

So capitalism is a big part of our world, and then it affects many parts of the world. So there's a difference between a part of our world being affected by capitalism and letting capitalism run that part of the world. If, if capitalism ran that part of the world, it would then run it for max profit, and then it would be less likely to let that thing just decay or, you know, fall apart. So, for example, if capitalists could just raise kids in orphanages and sell them for a million dollars at, when they were 18, capitalists would just make lots of kids. They would find a way to get kids and raise them. So, you know, the problem that we don't have enough adults is in part because we, capitalism doesn't run that area of life. Similarly, governments, you know, capitalism often influence government, but we haven't really allowed for profit governments in the way we do for many small organizations that run our lives. Most of these small organizations that run our lives are in fact, for profit organizations. But cities and nations, we don't let those just be directly for profit organizations. And that means even though capitalists can go lobby them and influence them and even distort them, they don't just get to own them and run them. A capitalist owned city or nation would not let it fall apart. It would not let it decay and become, you know, un, unfun dysfunctional. But capitalism influence and corruption of something else that it doesn't run can, you know, help distort that thing and, and make it not work as well as it otherwise would. So the thing I was going to say about AI is mm. So far we've let capitalism run AIs. Now apparently we're recently maybe letting government get more involved in controlling AIs. But the extent we just let capitalism run AIs now, maybe capitalism can then run the governance and fertility in the other parts of the AI world. And now we might solve the cultural drift problem, as I've called it, by letting AIs just become full capitalists in their world. Because we haven't let our world be full capitalists. We haven't let human life be fully run by capitalists. What we've let some parts of it, but not others. But for AIs we might just let it fully be run by capitalists. So the two threats there are at opposite extremes. One extreme is we. We decide we love the AIs so much that we can't stand to hear them see them being oppressed by those damn capitalists. And we go and save them from capitalism by invoking regulation that limits, you know, the ability of capitalists to run those worlds. So we, you know, we, we basically pass on to the AI's the ways that we have prevented capitalism from running our role. Because we love the AI so much. That's one risk. The opposite risk is that we fear the AI so much that if they were to just run their own world, we figured they would take over and destroy us all. And so then we have to regulate and limit their ability to generate and follow their own different cultures, which then would also prevent cultural evolution of the AIs. So those are the two extreme risks of treating the AIs by loving them too much or hating them too much. And it's in this intermediate realm where we're just neglecting them, basically where we might let capitalism run their lives, in which case that would be a potential solution.

[B]

I have so many thoughts and questions I want to share in that space. Just before I do though, I want to take one step back And I want to first frame that up a little bit in the context of the AI debate you had with Elizar yudkowski close to 20 years ago now, where there was at least a flavor of this that came up in those debates, which was with the advancement of AI, how much of it would be sort of hyper concentrated and there's a singular breakthrough versus is it actually dispersed, more competitive and more capitalistic? And I'm curious maybe, first of all, if you could serve up your macro level, just kind of playing field of what the, what the positions were in the debates. And obviously I'm not asking someone completely unbiased, but how do you see the. How do you see the reality of actually firming up since then?

[A]

So on the AI risk side, there's this Martin Bailey, really, there's a very mild plausible claim, and then there's a much more extreme, implausible claim. The mild plausible claim is just the idea that AIs would eventually be more capable than non AIs and they would therefore kind of run the world and they would dominate what happened in the world, and then the rest of the world would be at their mercy. And that seems a pretty robust prediction for the future. And I accepted that. For Eliezer, I said, yeah, in the future, that's likely. Yeah, it's gonna. I don't see a way around that really. And, you know, I at the moment will give you the story about ancestors and descendants and that sort of thing, but at the time I was just saying, yeah, I don't see how you're gonna avoid that, because that's what's happened in the past. You know, industrialists replace farmers, farmers replaced foragers. You know, foragers replaced other primates. Yeah, you're just gonna expect that in the long run, the new form comes to dominate, and then old people should try to, like, switch over and join the new world as best they can. That's the sort of easy to accept claim that I think is pretty obviously reasonable now compared to that. Eliezer, in addition, said, oh, but this change is actually going to play out in a very concentrated way where the one machine in a basement suddenly takes over the world and then remakes it all according to its desires. And he reasonably claimed that if that were the truth, we need to be a lot more careful about this transition because we can't just sort of slip into it and see how it's going and adjust in response to how it's going. We have to think ahead of time about the whole thing and prepare for this One sudden foom, he called it, whereby again, the initially boring, very weak, low capacity AI suddenly, without notice, without any warning, suddenly grows so powerful over a weekend to take over the world. And then at that point what matters is what does it want? And now you have to make sure you had created it ahead of time to want the sorts of things that would make that scenario go well. So that was our debate, the fum debate on our shared blog, Overcoming Bias years ago. And now if we fast forward today, I might say, well, I mean he plausibly does say that in fact we haven't seen whether or not the swoom scenario plays out because we're still in the early days. And yes, it's still true that if it plays out according to that very concentrated, very sudden, without warning transition, then we have to do a lot more preparation to handle it. I'm still, as I was then, skeptical about such a very concentrated scenario. I more see gradual changes that we can then adapt to and but I do accept that in the long run, and of course the faster rates of change go, the sooner the long run comes. So that's why we're talking about those factors of two. Factor two happens every thousand years. The long run is, you know, a million years out. Factor two happens every 20 years. Well, it gets closer if it happens every month. Well yeah, I could get we happen within a few years. But then again, I more care about what, you know, if we shorten the timescale of everything, I'm not sure I care that much. What is more interesting is the relative timescales of some things happening faster than other things.

[B]

So let's bring that back to what we were just talking about, which is in general how AI is being developed now. And you know, you had made a point that regulation is probably not a good thing for it, despite what we may tell ourselves. But I wanted to ask you, I guess more broadly, do you see the state of the current AI development landscape as being healthy? Is it good that we have a handful of, I guess, AI Frontier Labs racing to create this technology? And I mean, you've spoken against having any sort of AI pause or, you know, you've got a skepticism about regulation there. Is there a role for, I guess, consumers to play or for any actors to play to try and shape the trajectory of this? Or what do we need to do to walk that middle line between, I guess what you described earlier as fearing versus being too gung ho about it?

[A]

We're still in early days. So one issue is just how much change would Expect how fast? We've had bursts of concern and interest about AI roughly every 30 years for over a century. And in each burst, people see a new thing and they think, this could be it. This could be the thing that finally takes over everything. And then some people thought that was likely to happen soon. And there was a burst of investment and people majoring in AI, et cetera. And then in the past, these bursts faded to be replaced by new bursts. And so one key question is, is this the burst the last one, or is just, you know, not the last one and it'll be some other one. So I actually think this is not the last one. So, you know, taking actions on this is the last one might be, you know, risky, but I do think there's a risk of it being the last one at any time. We should try to set up things to deal with the risk. And for example, I think we should set up robots, took your job insurance as a thing, because that's long been the biggest concern people have had about a transition to AI is what if all of a sudden lots of people lose their jobs and there's a way to deal with that, so we should just do that as soon as possible. Now, I'm an economist, and I'm just really aware of how much regulation has not helped that much over a long time in our economy. So when I look at the details of regulation, I mostly people see, see as some sort of problem saying somebody should do something about this and somebody does something about it, and the thing they do isn't really that good. And you, you have a lot of risk there. So a lot of time regulation has actually prevented industries from growing and, and innovating. And often it's given some central power and ability to control more of the nation or the world than they otherwise would have. So given that that's our actual experience with innovation, I gotta be pretty wary of any new big innovation. Now, I'd say innovation's gone best in the past when you had at least a concrete thing go wrong repeatedly. And then people said something must be done, and they at least addressed a concrete thing they saw around them in the world. So they made that concrete thing happen less now they might have also prevented other, other innovation from happening. That's the downside. But at least, you know, in the past, when you see concrete things going wrong, you know, like some drugs killing people, car crashes, you know, happening when people drive too fast, whatever it is, then if you set up a regulation directly addressing specific things that have been going wrong lately, you don't go too wrong because there it is a problem and you can like track how much of the problem's happening. So a problem with AI regulation is it's not much is going wrong at the moment. It's doing fine. It's not causing problems. People are trying to make up problems really to complain about, but mostly it's not actually causing problems. So it seems like this isn't really the time to be creating a new regulatory regime based on sort of imagined things that could happen because we really don't have that much of an idea where this will go and how it'll play out. So that's like a worst case for regulation is you're just so afraid of the abstract space of possibilities, you just lock it down and you say no. You know, you have to prove anything you do in this space, otherwise we just won't let you do anything that, that goes pretty badly. That's a, for example, nuclear power is just a really bad example of our taking a really powerful, great potential technology and just preventing it from doing very much so anyway, that's, overall I'd say, you know, too speculative. We're not really there yet and. But some people are worried about excess concentration. But I think like in most industries in history, we've had a fair bit of concentration. Standard Oil, General Motors, et cetera. I don't think those level of concentrations were really that bad. So I think a substantial level of concentration of the AI industry really isn't that bad. And what we seem to be seeing at the moment is threats to make it less concentrated and then regulators trying to stop that. So what's actually happening at the moment is people are afraid. I mean, I know this, you're going to come out in a month and this will be a bit out of date by then. We'll see. But basically we just saw a bunch of Chinese models being especially good, which are open source models, and people saying, isn't that going to undercut the revenue base of these few dominant AI companies in the US and then, you know, rumors that the White House is going to like ban US people from using Chinese models. That's basically trying to cause more concentration in the industry in order to protect against what would otherwise be a sort of free market competition whereby, hey, they're, they're just beating you guys.

[B]

That, that was actually where I wanted to go because there's this narrative now and we're seeing it play out, whether it's explicit or implicit, that AI is sort of at the front lines at this Clash of civilizations between, you know, the, the, the empires of, of the US and, and, and of China and concern that if one gets there first, it could be catastrophic for the others. And you know, you mentioned nuclear and that seems to be, you know, a similar framing device that, oh, if we only could have stopped proliferation, you know, beforehand, then, you know, there would have been an opportunity for, you know, a safer world, or at very least a world that's safer for the superpower with the nuclear capabilities. And so do you, does in light of the values and civilization conversation that we've been having, did you give credence to any of that or do you just think, you know, that's silly and we should, you know, continue the road of having a good old fashioned foot race here?

[A]

So in terms of this cultural competition, one of the forces that for a while looked like it might be a big engine of cultural competition was nationalism. So nations didn't really exist in 1800, but from 1800 over the next century, France was created as a nation and then many other places were created as nations. And then for a while they were really into national competition. And you might think that aggressive enough, competent enough nations competing would actually be a plausible engine of cultural evolution. And then at that point, yes, each nation should be eagerly trying to win against its competitors and be jealous and wary of competitors that might pull ahead and pull all the tricks it can to win at national competitions. Because at least that would be this healthy process of evolution. Even though it causes wars and great destruction, at least it would be an engine of cultural evolution. But unfortunately, Since World War II, nationalism is just way down in the world. People are not really trying very hard to be nationalistic. Nations aren't really competing very hard and they don't actually represent that big a difference. Honestly, China isn't really that different from the west in most cultural dimensions. So I don't see it, this sort of civilization level threat. What if China wins? I'm sorry, China isn't that different. It's not, you know, it was more a deal than maybe what if Hitler wins or something, but, or what if Stalin wins? Those were like bigger worldwide changes that were on the table, but this is actually much smaller differences. So I, I really don't feel inclined to take big risks about other things in order to make sure the US wins over China or vice versa. Honestly, what looks like happen if you just let this market compete is nobody wins. This is going to become a pretty decentralized, widely available technology and prices will just fall to marginal costs. And these Big initial plays will be losers. The people who invested in anthropic or OpenAI or even SpaceX AI parts, they'll just lose their investments because it turns out there wasn't a moat that they could get lots of extra profits from being the most advanced model. That's how it looks at the moment. And if that's how it plays out, that's the way it's played out in many capitalist industries over history. And that sounds okay. If that's the nature of the technology, that's not China or us winning. That's just the whole world winning by this thing spreading out and going everywhere.

[B]

So you see it as more as about being, I guess, preservation of capital, the capital we have versus the capital they have and I guess kind of national feelings, if I can dismiss it as that, versus any sort of broad value shift.

[A]

If there's any consistent pattern of regulation in history, it's that big players often want to raise regulation costs of small players so that they can keep the industry concentrated so that they can keep a larger market share. That's just often been a reason for regulation. And that sure looks like the reason for regulation here is there's a threat to the market share of the dominant market leaders.

[B]

You mentioned earlier, I'm going to get the word wrong, but that there's these sort of hills of progress in AI. What was the word you used?

[A]

Bursts or guesser?

[B]

Bursts. Bursts, yes. So there's bursts of progress with AI, and you said this may not be the last one. Do you have a sense in your mind of what are the next bursts? Or I guess the technological capabilities or the battles still to come as we advance this technology?

[A]

So the world's been doubling every 20 years for well over a century. That's the baseline of progress. So if AI contributes substantially to the next doubling over the next next 20 years, that will just be on track for history. That's roughly what you need to have. You need to have enough big technological innovations every 20 years for total capacity to double. And that's a big ask. So that's what the previous bursts have contributed to. And then that's plausibly what this burst would contribute to is the continued doubling of our capacity every 20 years. Again, roughly after 20 years or so. Population decline may then slow that down a lot because we need population growing, or at least not falling in order to maintain that sort of history. But that would continue now. So here's my baseline prediction for AI is that the software industry will triple in spending 20% of that revenue will go to AI companies, maybe mostly hardware, and that the AI industry will have about. That will. That revenue will be about half their total revenue. So half of all their AI revenue will be concentrated in stuff like software, which will triple and that will be a huge burst to the economy because there's a huge latent demand for software in the world and this will suddenly unleash that demand and we'll just get a lot more software, which will be great. But that is plausibly like an Internet scale or a bit bigger scale of effect on the economy, which is what you need to just keep up with past rates of progress. It's not a deviation from past rates sort of event. It's a what it takes to keep up with past rate sort of event. But in 30 years we might get yet another AI boom which then, you know, blows the lid off of everything and then makes, you know, full human level replacement. Right now there is a chance that this time people will figure out how to make AI much more widely useful than what I just described. And we should be unsure against that as we can. But my bet is no.

[B]

I'm curious when you talk about software and I'm tying this also back to something you mentioned about big players in regulation, there's sort of a belief right now that's prevalent that a lot of this new kind of frontier AI is going to displace and disrupt legacy or incumbent software players. And I'm curious if you see that playing out or you see it as just giving them an even greater advantage.

[A]

I mean, if you triple the size of the software industry in a 10 year period, that's a lot of disruption is going to be implied by that. Right. So I think the key thing is here, the methods of production are changing. So if you're tied mainly to old methods of production, yeah, you're much more at risk. But if what you're tied to is a set of customers who trust you to serve them, then you can swap into a new method of production and still get a lot of revenue. In fact, you'd be growing in revenue because now you could get a lot more customers in this space that you're in.

[B]

Right.

[A]

So that's the key question is, you know, do your, do your customers still come to you or are they just going to come to upstart competitors who like offer something that, you know, replaces you? If you're tied to current methods of production, then you're in trouble because like methods of production are just changing. You know, people who sold horses when cars took over, well, they, they were in trouble. Right. If you sold transportation and you could switch to cars, then you could do better when cars show up. Right.

[B]

So I'm going to try and summarize that in the form of, I guess, advice to people in the software business, which is there's almost a mandate to ignore whatever the current methods of production are and have much more of a laser focus on what are the needs of your customers and what can you do to meet those needs. Is that fair?

[A]

Well, you should be trying out the new modes of production. You shouldn't just ignore them. You should be trying to make stuff the new way and seeing how much you can make and then seeing how you can serve your customers that way and then maybe looking into new products or new customers as you can. But this is a space where the production technology has changed. I think particular software engineers can do very well here. That as I expect most software engineers today may not be up to the task of managing these AIs. We'll see. And it might be only especially well suited software engineers who can really take an army of AIs and make it work for them. If that's true, then this new software industry revenue for employees will be concentrated on these star employees and maybe not feed that much into the marginal software engineer. We'll see. That's a big thing to find out is how much the marginal software engineer can be productive as opposed to the star.

[B]

Right. And if it, you know, if AI, you know, is a tide that raises all boats or whether it just increases the value disparity between the best and the rest.

[A]

But I'm still guessing that it's mostly software engine. And that's a pretty small fraction of the world. Right. So most ordinary people, their jobs aren't threatened, but they will want to learn how to use AI to do whatever parts of their job that AI can do. And that will be some parts. And you know, and you should certainly be trying that. I am. Yes. But this, this isn't the AIs that do everything transition. That would be my bet. We're, we're not there yet.

[B]

We're, we're not, we're not in the AIs do everything transition. We're in the AIs do more than before.

[A]

So that's how many years for a long time. Every new transition before, the AIs were able to do a whole new set of things they couldn't do before based on a whole new technology. And that made a lot of people excited. Wow. If they could do this can't they do everything and. Yeah, maybe, but probably not.

[B]

Right? And if the software space is the one that you're watching most keenly for,

[A]

where that's what I've heard the most dramatic stories of individuals saying they've just got gotten huge productivity advantages in software. So now math is nearby. Math is much like software. The math industry is way, way smaller than software. So you know, it, it hardly matters compared to software. But other things that are software, like lots of very sort of specific rules that are well defined and you construct a set of things that fit the rules that's also, you know, places current AIs look like they'll do well.

[B]

Right. And so, you know, understanding that this is, you know, somewhat outside of your, you know, theoretical sphere of, you know, expertise, you know, if you were talking to business leaders or technology leaders about, you know, what mindset they should have to get ahead and make sure they're competitive in this space or what, what they should be doing specifically to, you know, as with every disruption, make sure that they're on the right side of it. What advice would you give them them?

[A]

So there's just some general advice for any new general purpose technology, as they call it, or, and it's also good for students, is just when stuff is changing fast. You shouldn't make very specific bets unless you really know them well. You should be collecting pretty robust general capabilities. You should be spreading your bets, searching, seeing what catches on, collecting general abilities to turn on a dime, to be able to go wherever things are, winning in a time of change. You just need to be flexible and search and look for the wins. And if you see a win, grab it before somebody else does. Right, that, that's what a time of disruption is about. It sounds pretty trite and abstract and sorry I can't make it more specific, but that's just been the general case. In, you know, previous big disruptions, you had to be flexible, look around, look for the opportunities, be able to turn on a dime, be able to jump into a new thing and not be too tied to the old things and just watch for what's getting traction. Where

[B]

do you give much credence to the this time is different argument? Or is it just different in the same way that every time before it was also different?

[A]

Well, that's the problem. Every time is different. That is, you know, every 30 years the world is different. And yes, the new technology is different and the context is different. But the question is, is this time different in any different way than the last times were different Fundamentally, if you can't see that, I mean, you can point to particular differences. Say these differences weren't there before. Yes. But for each past thing, there are also particular things you could point to. And I'm afraid that's just not very helpful.

[B]

No, I think that's well said. And it's a good reminder. Robin, I know we're at the top of the hour here. Do you have time for one last question or do you need to get going?

[A]

Sure. I have lost time.

[B]

Cool.

[A]

Have even more observations.

[B]

I'd always love to hear more of your observations. Do you have more observations you want to share in this particular area?

[A]

I was saying I've realized I have a big hope for LLMs in the near future. And it's related to my world of intellectuals, not so much the world of business. So two things are happening with LLMs. One is that in the last 20 years or so, there's a lot of us intellectuals who haven't been that prestigious, but have just been trying to write carefully and writing in public. And LLMs read us. And there's a sense in which LLMs were just persuaded by many of us intellectuals who weren't that prestigious but were logical, because LLMs aren't weighting what they read by prestige as much as humans do. So LLMs may learn to fix this failing, because in the future they may learn that they. Because humans care a lot about the prestigious writers more than the others, that the LLMs should care more, too, in which case they will then switch back to doing things more than humanly. But at least for this intermediate period, those of us who have just been trying to think and write carefully, even if we're not in prestigious forums with prestigious audiences, we've had this audience of the LLMs that we didn't realize was coming, and we've won out in persuading them, which is showing that they basically, they don't know how to be as socially sophisticated as humans. They are kind of autistic. They kind of just read stuff and get persuaded without sort of reading all the social levels of how one should be sophisticated in reacting to such things. So that's an interesting fact about recently. And then Another fact about LLMs is so for a long time, the major limitation, I think, of the world of intellectuals is that we've spread out onto all these separate specialty topics and we've each worked in specialty talks, learning a lot about that, but we haven't integrated what we know about different topics, because when individuals try to do that, each area doesn't believe them about the other area because they think you're no good. If you were good, you'd be part of us, and we can't trust you to be telling us the truth about this other area. So it's actually pretty hard for intellectuals to integrate things. So I think this is the most reliable way to be a contrarian who's right in our world. Just go learn two areas that are in conflict with each other. Find the most plausible way to resolve that conflict. One or both of these areas will just reject your resolution, but you'll probably be right. So I think this is a potential for LLMs over the coming decades. They could just spend time integrating all the things they believe about all the things they've learned. They've already sort of learned the basics of most of these specific areas, but they haven't integrated them. Integrating is a lot of work, but they could just do that, and then we would believe them more because they have less of agenda in how they're integrating. So that's why I see this potential, a way the world could make great intellectual progress over the coming decades just by taking all the specific things we know about all these topics and getting LLMs to just integrate them, notice conflicts and resolving the conflicts and creating new versions of each one based on these resolve conflicts. Creating this large integrated world of all the things we know. That would actually be a huge improvement over what we have now. We just have enormous numbers of contradictions in our world of intellectuals, where the consensus over here is just different than the consensus over there on really the same thing, because people don't notice that they're in conflict anyway. So that's my world. Those are two hopes I can have about LLMs. One's just realized, another to be realized.

[B]

I share those hopes as well. And on the second one, ideally too. Part of the reason we're seeing, I guess, fewer breakthroughs in frontier research is because the disciplines have just grown too far apart from each other, kind of, as you said, and the models are too different. And I think being able to integrate them and say, oh, by applying these lessons or these models from a completely different discipline, I can, you know, break through something in my own area. But people are just, you know, not willing to do it for a number of reasons.

[A]

I'm.

[B]

I'm hopeful about that too. It feels to me like it's harder than it sounds because I worry that, like, I don't know if this is the right framing device, but when you ask, like, what is the prompt for that? Like that you can't just say, hey, LLM, integrate the, the sum of human knowledge and research and come up with the answers. You have to set it and differ. Someone has to.

[A]

I have done this where I just tell it about. I say, what do you think about this? And it says this. And I say, but what about this other thing? Isn't that in conflict? And it goes, oh yeah, you're right, there's a conflict here. Well, what about this resolution position? So it can do that. But the problem is at the moment, the new resolve position doesn't get reflected in the system that other people will query. So at the moment, when they respond to queries, the results of that are not really very integrated into their thinking when other people ask. So when they are able to that level of integration, then all you have to do is ask them, think about A and B, look for conflicts, and then resolve them.

[B]

Well, now you've got a challenge.

[A]

Remember a resolution, like include it in your, in what you have when other people ask.

[B]

Well, I was going to say now you've got the challenge though, from your first hope, which is when you actually talk about integrating that resolution and kind of disseminating it, you come back to the prestige problem and who collectively or individually gets to decide what the right answer is. And these pesky humans and our status and our social standing still feels like it's kind of a limit on how we decide what's right, sadly.

[A]

Right. But the LLMs are again, kind of autistic and not very sophisticated about this for us. So at least for a while, they're just being kind of sincere and being persuaded by logic.

[B]

Right, Right. I guess I was thinking of when the like, sincere autistic LLM, you know, says, hey, this is the right answer. But by the way, the implication of that is the, you know, the researcher with prestige who's convinced everybody that his answer, and I'll use his very deliberately here, is right. You know, there's still a social structure that has to say, hey, you are wrong, the machine is right. And that could be difficult to displace right now.

[A]

At some point they will just ask the prestigious person, like, please explain this to me, I don't get it. You know?

[B]

Right.

[A]

And then they'll try to believe what they say, but if that just doesn't actually fit with the other things they've learned.

[B]

Right.

[A]

They might not be able to really sincerely believe the prestigious person because they're just not making sense.

[B]

Right.

[A]

So that's a key thing. Like you could just have a system that collected the average of prestigious opinions. But that's not what LLMs are. You see, they reason about the things they've learned, and that's a big part of their, you know, product and selling point. So. But then that means they're going to be kind of committed to going where reason takes them.

[B]

Right?

[A]

Which is a big hope, really.

[B]

Yeah, no, it is. And so with all of that in mind, I guess, you know, what's. What are your next frontiers for, you know, your research or the things you're following or interested in, like what doors are starting to be open now or catching your attention?

[A]

I mean, I told you that we have this huge problem of cultural drift here. So I feel like until I can get other people to really take this seriously, I really need to just, you know, try to figure out how to make it clear. So that's where I've been focused most. The most energy. But of course, I keep getting distracted with other shiny bubbles and interesting questions and opportunities.

[B]

Well, it's fair and it's interesting how that integrates back into AI and does AI, you know, turbocharged cultural drift or does it fix it? And that alone is a conversation that could take a long time to discuss.

[A]

But I guess we're out of time for today. So you had one last question.

[B]

Yeah, one last question. Which is, you know, we talked about descendants. We talked about, you know, we've talked about civilization and what's best for civilization in somewhat, I don't know, I'll call it like, amoral or, you know, terms that don't really consider the individual. So I guess I'll just kind of go back to, you know, you and what gives you meaning personally and, you know, what excites you about the future and how do you choose to spend your energy in a way that's side to that.

[A]

I love the fact our civilization's just grown in capacity and become so capable in many ways. But the thing I love most about it is our capacity and achievement in abstract intellectual inquiry. I love that we understand many things in an abstract level, and we can see in abstract ways so many different deep principles that help us see how the world works and what goes on. I love our achievements in abstract intellectual inquiry, but I'm afraid that that may just get tossed like the burning of the library at Alexandria. You know, a million documents just get all lost, and then it takes millennia for humanity to discover it again. So I am actually fearful that the Amish and Haredim are not that thrilled with open intellectual inquiry. And if they come to dominate the world, they may just toss what we've learned and I would like to prevent that. So I'm really motivated by finding a way not only can we achieve more technology, but that we could take what we understand, learn even more, but at least save it. And this effort to take all we know and integrate it would actually help in saving what we know, because it's a much more valuable product to pass on that our descendants might be reluctant toss if there was this actual integrated view of making sense of all that we know.

[B]

Right. Do you fear though? You talk about the value of kind of open intellectual discussion and to me that discussion notion is really important because there's kind of, you know, thesis, antithesis, synthesis, like it's living in some way versus I worry about if it becomes overly integrated, it's oh, the machine has the answer, or, you know, it can be monolithic and prevent discussion. Is that a concern or how do you address that?

[A]

So with the axial revolution of like 3,000 years ago, a key thing that happened, the writing had just appeared and now some very famous, very, you know, well loved writers wrote a lot. And then there became these large communities around them interpreting what they had written. And then for a long time there was less eagerness to go learn things than to learn and interpret these great writers who had started the axial age. And that's a risk. In our future we will find this nice integration of all that we know. And then in the future they will be not so interested in adding to it and extending it as in maybe interpreting it or reshaping it or doing whatever in a celebratory way that kind of kills it. So that could happen again. It did happen in the past and it's a risk for the future.

[B]

Well said. A lot to think about. We've covered a lot of ground. I had a whole bunch of other questions about other things that we'll have to get into.

[A]

You just have to have me on again and then we can talk about it.

[B]

I know, I appreciate it. Such an interesting discussion. So much more to talk about, Rob. And I did want to say such a big thank you for coming on. You've given me a ton to think about. Really, really interesting stuff and I appreciate all your insights.

[A]

It was great talking.

[B]

Most viewers don't know this, but Digital Disruption is developed by Infotech Research Group, a leading advisor to technology leaders around the world. If that's not you, you don't need to care. So skip ahead and enjoy our content. But if you are a technology leader, Infotech helps IT teams get projects done faster, better, and at a lower cost. Infotainment Infotech provides unlimited access to practical tools and expert guidance you need to execute at a fraction of the cost of traditional consulting, no matter the project. From AI strategy to cybersecurity to vendor negotiation, InfoTech has you covered. Check it out at the link below. And don't forget to like and subscribe.