A (4:15)

Yeah, you're quite right. I mean, it was stunning to me that if you think about any of the early labs DeepMind Founded in 2010, the two scientific founders, Shane Legg and Demis Asabis, meet each other at a safety lecture. Then you look at 2015. The early discussions between Elon Musk and Sam Altman about founding OpenAI were all about safety and about kind of, it ought to be responsible. We'll be more safe than DeepMind. It was a competitive safety thing. And then you go forward, and the next lab that gets started is Anthropic, which is started as a splinter group that thinks that OpenAI is not safe enough. So they kind of repeat the story that had happened earlier in the rivalry between OpenAI and DeepMind. So each of these labs began with this idea that they were going to be safer. And even Elon Musk, when he does Grok or Xai, he comes in with a record of having proclaimed his terror of existential risk from AI going back to the 2012 zone when he first met Demis Hassabis. So you're right. They're all schizophrenic. And. And it's a pattern. And how do we think about that? Well, in the end, my feeling is that it's an enlarged version of all of us. You and I, Yasho, we also are excited by technology and also scared by it. Yet we take the trade, we move ahead. Why do we do this? Because we're human. If we didn't do it, humans would still be living in caves. We do accept technological risk, and that's what these guys are doing.

B (5:50)

Yeah. One of the kind of leitmotifs of your book is this line that Geoffrey Hinton, who's a past guest of this podcast, sort of says, alluding to Oppenheimer, that the thrill of discovery in Oppenheimer's case, of course, of the atomic bomb is so big that even if you're very worried about its implications, it's impossible to resist. I guess I wonder from your conversations with Dennis and with others in this space, whether there is an ability to govern this technology. I was really interested when I had Nick Soares on the podcast, who's the co author with Eliezer Yudkowski, of his book if Anybody Builds It, Everybody Dies. That he seemed to me to be too pessimistic about the prospects of technological annihilation. He basically thought, definitely AI is going to try and kill us. And then I thought he was really optimistic about our ability to stand up to that through public policy, that we'll get just the right incentives in place that nobody builds this machine that definitely would kill us if we did build it. And I was sort of struck by how pessimistic he was on the first point and how optimistic he was on the second point. Is there something in the story where OpenAI gets built to be really safe on these things? Entropic is a spinoff of OpenAI in a sense. I mean, a hostile spinoff. But people who get worried about OpenAI and now Entropic is in some ways the leader of the pack. Again and again we see that it's impossible to resist these sirens. Does it just mean that our fate depends on what the natural tendency of this technology is going to be?

A (7:38)

No, I think that's too fatalistic. I agree with you that there is, in the Yudkowski view, a sort of extreme caricaturing of both the level of the risk, 100% probability of doom, which to my mind is just ridiculous. Way too high, and at the same time, too much optimism, as you say, about the ability of our policies to do something about it. It reminds me of Jeffrey Sachs arguing about development aid, you know, 20 years ago, where he would always say, he would. He would stress how deeply poor and dysfunctional developing countries were, and then he would say, but if you give them a lot of aid, we can fix all of it. And both sides were wrong.

B (8:19)

Right? Right.

A (8:21)

And I think, you know, that's a classic posture of somebody who's arguing for radical government action is. Is to. Is to exaggerate the problem and then under. Exaggerate the underplay the, you know, the. The policy challenges of getting it right. But I don't think it's correct to say we are hostage to some technology over which we have zero control, because there are ways of controlling it. You can control both the coding of it the algorithmic design of it. And there's a whole field of alignment research to make large language models align with human priorities. And I think if more investment was going into that, we would have a better shot of aligning them better. But that is something that can be done and worked on. An example of this would be the UK AI Security Institute, which does some of this alignment research and came up once with. They discovered a way that you could hack any of the large language models, the leading ones, with a kind of specific phrase that would then sort of unlock it to do things that were supposed to be off limits. And once you discover that, you tell the labs and they fix the loophole, they fix the security vulnerability.

B (9:38)

What was the phrase?

A (9:39)

I don't know, I'm not sure. Not sure that was ever revealed. But hocus pocus, something. Yeah, no, abracadabra, exactly. But the point is you can do something on that algorithmic front and you can also do things in policy terms. To my mind, you know, open source, open weight AI models are ridiculously dangerous. Why would you ever allow this kind of technology to circulate without any ability to call it back if somebody starts to use it for a massive great cyber attack on infrastructure? It's crazy.

B (10:15)

Explain to people a little bit about open source because I think people may not be so aware both about what is in general and what is in relation to AI. I mean, in general, sort of Open source has always been the kind of, you know, idealistic, do goodry approach to software, right? You can customize it yourself. You're not in the hands of some corporate conglomerate. Open source has always been the idealistic thing. We've seen a little bit a tendency towards the AI labs that are perhaps not at very cutting edge, that can create very powerful models, but models that are just a little bit less powerful than the most cutting edge ones going open source because it is a way to attract people to those models and make them use them. Of course, as you're saying, it means that basically you can download a model to your own machine and then run it off of your own machine in a way that then is no longer subject to control by its original creator. In lots of contexts that's going to have positive elements, right? It makes it cheaper to use the technology. It means that for, for example, if I wanted to handle some really sensitive information, if I wanted to create an index for a publisher, and they don't want to have any risk of somebody being able to train their AI models on that information, I can do that with an Open source machine makes me closed loop. I know that the information is not being communicated back to anybody because I'm not running it on the Internet. So lots of positive things. But you were saying, interestingly, that in terms of risk mitigation, it's really bad because the same exact features also mean that if somebody starts using this thing to develop a really potent bioweapon, there's no way that anybody can track that that's what they're using it for, or to stop them from doing it.

A (11:58)

Yeah. There was a cyber attack in Mexico recently on a mass scale, and pretty much everybody's electoral records were hacked. And this was done with the help of Anthropic and I think I've heard accounts that it was more than just the help. Basically a group used Claude to carry out this attack and Claude did most of the hacking for them. And then there was a bit of OpenAI being used as well, of the ChatGPT model. And once the labs discovered that this attack was going on, they just closed it down because they had the ability to do that. It was not open weight, so it was not in the machine of the bad guys. It was being used through a server controlled by the labs. And so that's a real life case of how you could shut down an attack. I mean, I've been to, you know, war gaming exercises organized by the RAND Institute, and you know, the classic nightmare scenario is that you have rolling waves of attacks on Western infrastructure by some, you know, unidentified attacker. It could be the Chinese government, it could be a terrorist group. You just don't know. All you know is that all your infrastructure is not working. Nobody has clean water, nobody has electricity. Everybody's panicking and you don't know how to stop it because you don't know who the assailant is. Why would we risk this? Right? We have war games telling us how this would work. It's like crazy. It's kind of like 20 years of scenario planning on the Strait of Hormuz and you go and do it anyway. And it's on that level of idiocy. So we're staring this in the face and we should be doing something about open weight models. It's not easy because they circulate already, but at least stop the more strong ones which are going to be created starting now.

B (13:46)

It's very interesting who's released them so far, right? So I think a lot of alarm models by meta Facebook have been open source. The Chinese have released a lot of open source models. And again, I think that's basically because companies like Deepseek, as well as the many other companies in China that now have pretty powerful AI models, you know, have created models that are very capable and very powerful, but they're not as capable as the latest versions of ChatGPT and Claude and Gemini. And so there is a kind of competitive reason to, you know, release this open source model if you're a little bit behind because nobody is going to pay premium dollar to have access to your model since then you're going to use the most high performing one. But these models are powerful enough that if you can make it available to people much cheaper, then they're going to get a lot of use and that's good for visibility and all kinds of other things. I imagine it is kind of striking that the Chinese government in general is obviously quite conscious about its ability to control what's going on in its country and in some ways around the world has allowed that to go forward. And that speaks to a kind of broader conventional wisdom that obviously one of the dynamics here is a competitive race between the United States and China and that supposedly China is less interested in AI safety than the United States is. I think you're not so certain about that conventional wisdom.

A (15:10)

Yeah, that's right. So I mean, a standard view, I think in kind of US government circles is, look, the Soviet Union and the United States lived through the Cuban Missile Crisis. They understood the existential risk from nuclear technology. They get it. They get that some weapons can be existential. The Chinese, on the other hand, their idea of catastrophic 20th century risk is sort of the great famine. I mean, it's the Cultural Revolution, it's politically originated disaster. It's not technologically, you know, near misses. And so on the, and to the contrary, in China, technology is associated with a miraculous growth the last, you know, 25 years. And so they want technology, they love technology. I mean, that's the kind of classic story. And if you try and talk to them about limiting technology, you know, forget it. And I just spent eight days there and I was really, really struck by how both top research academics and also leaders of the industrial AI companies were talking about safety. And when I was there, there was this Ferrari of open claw, this agent you can download into your computer and you can let it do agentic things, you know, whatever, manage your email or do your shopping or something. And it's pretty good technically, but it's also dangerous because you have to make your computer naked to this agent and you don't know what it's going to do. With all your data and it's some sort of piece of open source code, so who knows what it's going to do? And all of these both kind of Tsinghua professors, other AI researchers I was talking to were saying people shouldn't be downloading this. And yet there were lines of mom and pop Chinese outside the Tencent headquarters lining up to get the engineers from Tencent helping to install it on their laptops. And in the end, the government weighed in and said, no, no, people should not be installing this. So I think that, you know, the debate is tipping in China and as you say, I mean, the Chinese state is not averse to regulating stuff and notably the Internet, it's regulated that a lot. And so why wouldn't they want to control open source, which could become dangerous? It seems to me too defeatists to just assume they won't.

B (17:30)

Part of the point here may simply be that the word safety is a very broad and capacious term. And I've heard the argument made that people just mean very different things by it. Right. So part of it is that the United States are very influenced by sci fi and so on. And so the kind of safety we imagine is will the rising of the robots and will the robots try to killers. So even at a kind of slightly lower level, things like the risk of engineered bioweapons and so on and so forth. In China, when people talk about AI safety, part of what they mean is political safety, that these machines need to be aligned with a particular set of views and not give too much information about particular historical events and portray a positive image of, of the Chinese Communist Party. Now, of course, a lot of what happens in, in post training in the United States is to make sure that, you know, Gemini and ChatGPT and Claude and so on don't step on various social and political taboos as well. Right. A lot of the work and post training goes into making sure that, you know, chatbots know not to, you know, use certain kinds of slurs and not to wade into different kinds of territory. That might be politically sensitive here. But have you found in your conversations in China that there is just a kind of, you know, when you say the word AI safety, people just mean something different by it. Or do you think that difference has been overstated?

A (18:58)

Look, I think it's true that there are lots of definitions of safety, but I was talking really about specifically alignment risk. You know, the idea that the robots will not be aligned with what humans want and it will actually attack humans. And that's what they were Talking to me about, and that was the point about this open claw thing. We had people installing an agent which might just take it unto itself to do something which is not good for the user. And so it's not about political debates on what the large language model should say or shouldn't say. It's actually about this more existential thing and quite closely related to existential is that it's not just the machine that can be bad actor, a human bad actor can get it and use it to make a bioweapon or something.

B (19:50)

Yeah, and those two risks are obviously interlinked but important to be distinguished. What kind of levers does public policy have to govern the kind of existential risks of AI safety or the more kind of day to day bread and butter risks? One problem is simply one of competition that, you know, I've had conversations with people in the European Union who I found to be slightly naive about this, who believe that the Brussels effect that allows them to set a standard for, you know, how a car works, you know, then gives this big incentive for car manufacturers to apply the same standards across different manufacturing stand points. And so therefore essentially you as a significant market can force a regulation that is by and large going to be listened to in the United States and other places as well. I think in the context of AI where both there just aren't very significant European AI companies so far, certainly not on a global scale. DeepMind is an interesting story because it started in the UK, but it's now owned by a US conglomerate in Britain of course is somewhere between the continent and the United States in general. But certainly when we talk about the members of the European Union, there's not many significant AI players.

A (21:12)

There's Mistral.

B (21:13)

There's Mistral, right. There's Mistrial and there's the new thing that Yann Lecun is founding that in relation to the US and China entities, or to DeepMind, if you want to continue counting as a separate entity, there's very small players, you know, and then it's just, you know, if somebody designs a bioweapon on one of those open source machines, whether they are, you know, that's not going to stop at the frontier of the European Union. So that's one problem about just kind of geography and how do you align it. But there's a broader problem of how do you actually make technical rules, like what kind of rules would allow us to have alignment. Do you have a view on how to even begin thinking about this policy space?

A (21:56)

Well, okay, these are two different Very different points. Let me just focus on this open source thing. I mean, the first thing to say about the Europeans is that they're happy to tolerate open source because Mistral is producing open source. Because Mistral produces is in precisely the position you mentioned earlier, which is that if you're not at the frontier and you can't compete on quality, you compete on availability and you make it open source. That is what the French strategy is. And so they are, you know, they haven't even got to the threshold of being serious about open source before they start to proclaim the, you know, extraterritoriality of their regulation. You know, trying to stop open source is non trivial. There's a huge lobby of, you know, companies like Facebook Meta that create it. And in Europe, you know, obviously the French government wants Mistral to succeed and so it wants to support its open source tactics. But if there was a policy shift and governments decided they wanted to control it, you would just say to labs that were in your jurisdiction or wanted to do business in your jurisdiction, you can't be open source. And at least if it's a frontier model, because there's going to be academic models that are experimental, that are much smaller and those will be open source and that's probably fine. But I'm talking about the big frontier models. And if you're Mistral, you want to sell to U.S. consumers, you probably want to raise capital in the U.S. you have lots of touch points with the U.S. so they're going to obey an American regulatory decision. The big problem is you have to get China on board because they are an ecosystem unto themselves. And the assumption in the US is no point even trying that conversation. And I'm saying based on just having been there, no, there is a point. I don't think Trump is going to do that because he's not interested in anything other than AI acceleration. But I think.

B (23:55)

And he also fundamentally just believes in a zero sum world, right? I think he just fundamentally thinks that most deals have a very clear winner and a clear loser. And so that doesn't make it very appealing to try and strike a deal. The point of which would be that both sides can win from it.

A (24:12)

I think there's an opportunity here for a Mark Carney style middle powers initiative. He made this famous speech at Davos where he said we can't rely on the US if we're Canada, if we're the Europeans, we need to get together and do stuff together and not wait for America because it's just not going to be useful. For the next three years. And I think on AI, that is something where Europe and maybe Britain in particular, could start a discussion with the Chinese about how to think about open source. The UK has some credibility on this, both because DeepMind is based there and because their government security institute, AI Security Institute, is extremely good. And you begin the discussion. You can't consummate the discussion until you have a new American president, but that will come at some point, and I think it's worth getting that conversation started.

B (25:04)

Open source is one element of this. What about more broadly? Right. I mean, if you imagined a real deal between a new US President and perhaps a new leader in China that are serious about these things and that recognize that this is a genuine risk to humanity across borders, what could actually govern these AI technologies? Because, again, we probably don't want to shut AI down. It's not feasible. And there's also a lot of good things that can come from artificial intelligence, tremendous progress in medicine and other areas in science more broadly that we don't want to foreclose. And at the same time, the people who have created these technologies are themselves extremely worried about how incredibly harmful the technology could be to humanity. Let's put aside for the moment all political constraints and imagine that we can write the deal and it's easily going to be approved. You and I, the creators of this framework. Now, do we know what that framework would entail? Because it feels to me like there's very, very naughty intellectual questions, even about what kind of rules and regulation would A, effectively control those existential risks, and B, do that without forestalling all of the potential benefits from a technology.

A (26:24)

Well, you know, Demisisabis, the central character in my book about AI, has for quite a long time advocated what he calls a cern, as in the center for European Nuclear Research, a kind of governing body which would oversee AI and on a multilateral basis would propose policies and enforce them, or at least set the policies, and then maybe national governments have to do the enforcing. And the elements of that policy would be, we don't want open source, at least not for big models. We do want more investment in alignment. So this is a branch of the science and the engineering that needs resources, and probably you might need tax incentives to force the private labs that whenever they spend a billion dollars on a training run to make it more powerful, they're going to have to also set Aside, let's say, 20% of that specifically for safety research. And one could discuss the details, but I think the point is that there is a Kind safety is both a private good provided by a model creator to its customers. So if you're Google, you don't want your customers to feel the thing is unsafe because they don't like that. So you have some private incentives to create safety. But of course there are lots of spillovers into broader societal risk infrastructure collapsing, terrorist groups being empowered, which are public goods, not private ones. And therefore the public authority the government needs to ensure that the level of investment in safety rises to the socially optimal amount. And that implies either government spending on research and engineering into alignment, or it implies taxing the labs and nudging them to do it. So I think those are two important things. First, don't do open source. Second, do more research on alignment. And I think the third thing is just like the Food and Drug Administration looks at drugs and determines if they're safe to be released into the market, so too you should have a FDA for AI models so that their models should be looked at and assessed and are they safe, they should be red teamed and then they can be released to consumers. That doesn't exist anywhere at the moment in the world, which is to me crazy. But I think we're going to get there in the end because either governments will change their minds and do it, or they will be a Three Mile island disaster type of thing and the public will freak out and then we'll do it. So it's only a question of whether we do it before or after we suffer some AI catastrophe.

B (29:24)

You mentioned p doom earlier. The probability that this incredibly capable new technology will lead, however you want to define doom, at the most extreme level of the death of humanity. At the somewhat less extreme level, the enslavement of humans to our new AI overlords, which ironically we yourself as a species have created. Created. Where do you put pdum after thinking about these topics for the last few years?

A (29:54)

Well, I'll tell you a little bit about the journey. So I began researching my book on DeepMind right around the time ChatGPT came out in late 2022. And I already thought that machines were obviously going to be more intelligent than humans. So normally more intelligent beings or agents will dominate the less intelligent ones. So I could see that there was a theoretical risk. But I consoled myself with the idea that although the machines are more intelligent than us or will be soon, they don't have an incentive to dominate us. We are evolved over centuries and centuries to want to survive, pass on our DNA and to fight viciously to be able to do that. Machines don't Reproduce in that way. They're not evolved in that way. They don't have a survival instinct. Therefore, even if they're cleverer, they won't dominate us. And the moment I lost my faith in that argument was when I went to see Geoffrey Hinton, who was on your podcast, and I sat in his kitchen in Toronto for two hours and we debated this. And what he pointed out to me is, okay, you're going to have a powerful AI in the future, Sebastian, and you're going to be worried that your enemy is going to mount a cyber attack on it. So how do you defend against the cyber attack? Well, as a human, you're way too slow to respond to an attacking cyber attack. So you're going to empower your own AI to defend itself against the cyber attack. And once you've empowered your own AI to defend itself, you. You've necessarily given it a sense of self preservation, a sense of pain, a sense of fear, a sense of proactive defense, and then you've erased this distinction on which I had based my confidence. Right now. It does have a survival instinct, and it's cleverer than us. So it's not that I get up every morning. I mean, when you. The bottom line on the P Doom thing is a Rorschach test. People give a high P Doom if they are temperamentally pessimistic about life and the world and everything. And Jeff Hinton, as one of his students said to me, a student who'd been a PhD student 15 years ago, before it was really powerful, was saying, Jeff was always worried about bioweapons finishing off humanity before he thought that AI would do it. So he always had doom about something.

B (32:27)

And clearly there's something in the human psyche.

A (32:30)

And in my case, yeah, in my case, analytically, I see the case for being very worried. Temperamentally, emotionally, I just can't. And this, by the way, this comes up, I mean, I wanted to write my book partly because it's like, what does it feel like to be creating an existential technology? I mean, what is the tingling sensation that you might be destroying humanity with what you're doing? And what was really at first kind of crazy, but when you think about it, not surprising is that people would give some safety lecture and they would describe the possibility of human annihilation. And as they would be doing this, they would be smiling, and they might even be slightly laughing at some points. And you think, why is that? Well, basically contemplating the annihilation of humans feels absurd. And the absurd is a close cousin of humor. And so this is one of the things I observed as I, you know, there is a sort of fascination that, you know, people are drawn like moths to the fire by catastrophe scenarios. And it's something deep in, you know, I mean, I guess all the sort of second coming predictions of apocalypse, all that kind of religious iconography. I mean, this is deep in the human tradition. And yeah, so I felt the first sort of revelation of doing my book research was not only Demis, Hassabis and DeepMind, but all the other labs began, as we said earlier on, with a strong perception that thinkers this could be disastrous. At the same time, they processed this feeling of disaster, sometimes with giggles and sometimes it was so difficult to internalize that they ended up just laughing. And thirdly, even when they weren't laughing and they were being really serious and trying to think about how to fix it, they would go through experiment after experiment or hypothesis after hypothesis about how you would safeguard it, and none actually worked. So with Demis, he wanted to negotiate with Google. When Google was the parent company, it bought DeepMind in 2014. He wanted to oblige them to have a sort of independent safety oversight board that would have the final say on the rollout of AI so that it wouldn't be in the hands of a corporate board that wanted to make profits. And initially Google sort of agreed. They held the first meeting then that was a disaster because it was chaired by Elon Musk who set up OpenAI to compete with DeepMind. So then they began the secret negotiation, this whole thing called Project Mario. And I discovered all about this in my book and was given all kinds of leaked documents from inside the company. And what you see there is three years of negotiations between Mountain View, Google and DeepMind in London on how to put in place AI governance and graft that onto a for profit company. And at the end of the day it didn't work because the for profit company couldn't accept the idea of empowering these outside independent characters who would have a say over their proprietary technology. They just couldn't bring themselves to do it. And then there are other iterations of these experiments that through the story of DeepMind, one can tell about how do you make AI good, how do you make it beneficial for the world? Super difficult. And I think that's why in the end we talk correctly about governments intervening. It needs to be policy and it needs to be policy on an international level with the US and China talking to each other.

B (36:13)

Yeah, I mean, one of the things that just strikes me as interesting is that Humanity has always thought that doom is upon us soon. I'm from the story of a flood in the Bible to basically every juncture in history had some kind of millenarian cult that said, the destruction of humanity is upon us tomorrow and prepare for it. And often probably with a smile on their face. There's probably something in the human psyche which makes that thought both scary and exhilarating. And part of what's exhilarating, of course, is that you stand at the cusp of history, that it's somehow your generation and you're going to be involved in the final battle, and so therefore your life matters. So there's a kind of strange set of assumptions there. Now, of course, it would be easy to use that historical context to ridicule any of these concerns, right? I mean, humanity has been around for however many million years, and human history is at least 10,000 years old. And why should it so happen that you and me are alive and the people you're writing about in your book are literally creating the technology that is in fact going to bring about doom? Isn't that just our ridiculous need for significance in the world seducing us into making those ridiculous assumptions? At the same time, you look at how quickly this technology is evolving, how powerful it is, the fact that it is the first time that there is or is going to be, depending on your exact interpretation, a technology that is just objectively more capable than humans on Earth and before that could all go horribly wrong, is not exactly far fetched. And so it's hard to sort of reconcile this fear about chronocentricity, that we always tend to center our own time as the most important in the world, and the recognition that we've toyed with scenarios of doom at every juncture of humanity, and therefore likely this one is a little silly and exaggerated too, with very cold, rational reasons to think that, well, in fact, humanity now, from nuclear weapons to some of our biotechnological capabilities to AI, has created tools and machines that are so vastly more powerful than anything that was there for 99% or 99.9999% of human history, that there is reason to worry if this time may really be different.

A (38:55)

I think I just learned a new word. Chronocentricity.

B (38:59)

Yes,

A (39:01)

congratulations. That really won respect. That was good.

B (39:07)

It's not my term to be clear, but yes.

A (39:11)

Yeah, so we have chronocentricity, so we kind of exaggerate the importance and uniqueness of our own time. It's kind of this time is Differentism, Correct? Yeah, but I think you're right.

B (39:22)

That's exactly it.

A (39:23)

Yeah, I think you're right that the reason why this time could be different is precisely because this technology is different. It's a new form of cognition. We haven't had that before. And a machine that can invent more machines is something we haven't had before. But even if it wasn't totally new, I mean, let's say we downgrade our estimate. I mean, in my book, I suggest, you know, this could be the biggest thing since the arrival of the human ability to do abstract thought, which is thought to be 70,000 years ago. Now we have a second form of cognition that can do abstract thought. But even if you think that's exaggerated and you say it's more like the Industrial Revolution, that's still pretty big. And the Industrial Revolution brought about all these social and political convulsions that, you know, led to Marxism and communist manifesto in 1848 and a slew of revolutions across Europe. And, you know this history better than anyone. I mean, it's very disruptive. I mean, you know, I'm willing to be Marxian in this sense. Technological change drives social and political change, and it can be fairly revolutionary and bloody right. So we shouldn't just be sitting here in the 21st century and forgetting the lesson of the 19th century that the Industrial Revolution was highly disruptive.

B (40:49)

So speaking of the Industrial Revolution, I want to ask you not about P doom, but our society gets really screwed up, which is the other obvious analogy to the Industrial Revolution and previous technological transformations. So the first thing to say about many of these previous technological transformations, that they were in fact terrible for many people at the time. Obviously, in retrospect, the Industrial Revolution is a very positive thing. Humanity is thriving across a huge number of dimensions to a vastly larger extent than we were before the Industrial Revolution. But for about 50 years, the living standards of average people did not go up, and there's huge economic disruptions, and many people whose skills were displaced and so on and so forth. The other difference is that at that time, certain kinds of craft skills and so on were automated away, and people who had invested real energy into learning those, often to a very impressive standard, no longer had a use for their skills. But there was always a kind of reservoir of demand for the kinds of things that humans could do that remained. If you lost your job as a peasant or somebody who weaved things by hand, you know, you could attain a higher level of schooling than your parents did, or your kids could attain a higher level of schooling than you had. And you could go into, for example, a rapidly growing number of jobs in offices or more broadly involving cognitive skills, including in factories and so on. Right now we're at a point where that kind of escape route may no longer be available. And there's obviously a huge debate from some people in Silicon Valley who think that every job is going to be gone in three years, which I find to be very naive to certain economists who basically think, oh, it's not going to have any impact on the job market at all. What do you think the economic impact of all of this is going to be, including on the job market? And how has talk to Denis and other people in the space shaped and perhaps changed your view on this?

A (42:59)

Yeah, I mean, one thing to start with is just that this debate has been going on inside AI circles for a long time. So I mentioned that when DeepMind was acquired by Google, they did have one safety oversight meeting and that took place in 2015. And at that meeting Mustafa Suleiman, the co founder of DeepMind, made the argument that the pitchforks were going to be coming, people were going to be displaced from their jobs, and they would come after the makers of AI waving their pitchforks, forks, and it was going to be another revolution. And Eric Schmidt, who was the CEO of Google at the time, said, no, you don't understand economic history. When you display some jobs, new ones get created and all that familiar argument. Right. So this is not a new debate. And in some ways it's worth recording that because we're still having this debate and we need to move to actually doing something about it. Right. And I mean, my view is, look, humans can retain a role in some areas of the economy and also just in some areas of living. So what can we do? Well, human to human interactions are probably something that we're going to be better at. I know that AIs act as therapists, but I think that probably humans provide better quality companionship for other humans. And that bleeds into things like selling, enterprise sales. I suspect that humans will retain an edge on some of those things where EQ and just being a biological intelligence who sits down in a chair next to somebody else and looks them in the eye, I think that's going to remain powerful. Then there's goal setting. We don't want machines to be setting goals for us. And so whether that's goal setting in the sense of being a political leader or just a volunteer organizer at a local level, there's all kinds of goal setting up and down Society, and that will remain a human thing. Entrepreneurship, clearly a good example. And then there's going to be a whole range of changes in how we spend our time, which have the character of chess. And by this I mean, you know, a computer defeated Garry Kasparov in 1997. And so we've had, we've run the experiment almost three decades of computers being better than humans at chess. But the number of chess players has gone up. The number of people who watch human chess players has gone up. No human fans watch machines play against machines. Instead, the human champions train with the help of machine champions and get better that way and discover new strategies thanks to the AI. But the people's passion for chess and the time they spend on chess has gone up, not down. And so I think we're going to be quite good at discovering new hobbies, activities, ways of competing with each other, ways of getting fulfillment, ways of finding meaning in life, which are not sort of in the traditional category of paid for activity. It's not a job, but it is a passion and it can keep us going.

B (46:22)

But we do need jobs, at least in our current economic system, in order to have a livelihood and so on. Right. And there's a concern that, you know, I really don't believe the line that all jobs are going to go away and so on. I think that's terribly naive about both what actual jobs consist of and of all the kinds of regulatory and other obstacles to a lot of those kind of substitutions. But it would be enough for two things to happen, which is A, that a lot of people who now are in very well remunerated employment lose their job, and they're terribly upset about that. And B, as a result, there is now an oversupply of highly skilled human cognitive labor, which then depresses the wages even of those people who do retain a job, because the number of people who could plausibly substitute for them and has gone up a lot. And if you imagine that playing out even for a period of 50 years, which would roughly equal that of the Industrial revolution, let alone forever, that would have profoundly troubling consequences for our political system, for our ability to sustain social peace, for people's sense that the institutions are working, and so on and so forth. So I guess how should we think about that?

A (47:39)

Yeah, look, I agree. I mean, I think a key point you just made, which more people need to understand, is that you don't need to posit that all people in some category lose their jobs. If only 20% lose their jobs, they will compete down the wages of the other ones, and that will create mass unhappiness. So a small displacement. Well, not small, but 20% if you think about it. Covid unemployment spiked to something like 10, 12% at some point. And that triggered just a ginormous fiscal response from the government. It was a total crisis and so forth. We're talking even if it just was a 12% of the workforce lose their jobs. That is politically and socially unacceptable if we judge what the government's response was during COVID stimulus checks being mailed out to every single American, all that kind of thing. So, yeah, I think it's very troubling in a funny way. This is actually the reason why I got to write my book. Because I went to Demis Hassabis at the beginning of my project and I said, look, you may not particularly want to spend 30 hours speaking to an author so that they can get deep access and write a book about you, but you don't have a choice. And the reason is, first of all, if you're right, as you say in all your speeches, that AI is going to be the most important technological invention in human history, it follows that you, as the creator, Demis, are one of the most important people in human history. And if that's the case, then there will be a book about you. Get used to that. And furthermore, you should welcome a book, because if you're going to disrupt people's lives with your technology and it changes the way they bring out their children, it changes the way they conceive of themselves as being human, because there are now machines that can think you better explain your motives for doing this to the world because otherwise it won't be accepted. And I feel like, actually, if I go down the list of the leading AI lab leaders, right, you've got Sam Altman at OpenAI, who, especially following this debacle with the Pentagon, is viewed as a slippery opportunist by lots of people perfectly willing to undercut an attempt to stand up for safety principles in order to snag another contract with the Pentagon. You've got Elon Musk, who has his own vainglory, and I don't think most people would trust him. And you've got Dario Amade, who does stand up for principle, but is very dialed into the kind of science lens on how we live. I mean, he's a very deep scientist, he's very smart, but I think he underweights sometimes.

B (50:40)

The

A (50:42)

difficulty of turning what he calls a database full of geniuses. That's his expression for AI into actually using that intelligence for positive effects on economic productivity or whatever else you want to do with it. There are so many social institutional frictions. I think perhaps that is not what he thinks about because he's such a pure scientist. And then there is Demisisabis, who I wrote about, who I think is by far the the most relatable, normal, and kind of reassuring figure in the field. And it's good, I think, that people should get a chance to understand what he's like because it maybe makes it easier to accept what's going to happen.

B (51:32)

Thank you so much for listening to this episode of A Good Fight. In the rest of this conversation, we talk about the relationship between artificial artificial intelligence and venture capital and why this new technology may actually make venture capital less important than it was before. We also talk about the prospect of an AI bubble. Sebastian argues that there's about a 50% chance, up to 50% chance of OpenAI going bust in the next 18 months, and I asked him whether that would be the beginning of another great recession and economic catastrophe to rival 2008, or whether it will turn out to be a relatively minor event that can easily be absorbed. To listen to that part of the conversation, find the answer to this question. To get full access to all episodes of the Good Fight, please go to writing.jasamunk.com listen become a paying subscriber. Click the Setup Podcast button to make sure that you get the full unpayworld version of these conversations directly pushed to you from your favorite podcast app every week. Writing.yashamonk.com Listen click on set Up Podcast.

A (52:55)

Sam.