B (4:11)

So you wrote a book called Deep Utopia. It's a very interesting exploration of what if AI goes right. When I encounter the ideas though, I do start to ask whether I would want to live in that kind of utopia and whether utopia would be a positive thing at all. Can you define utopia before we get to deep utopia? Can you define utopia so we can contrast that to this idea of a deep utopia?

A (4:44)

Yeah, generally in sort of utopian literature, it's usually people coming up with like a blueprint for the ideal society. They have some vision of what, like, what would a perfectly fair society be like, where everybody has enough and everything is nice. And so this kind of utopian literature has kind of fallen a little bit into disrepute, partly for good reasons, in that oftentimes people with these social visions, if they actually gained power to implement them, have created a trail of havoc and misery. And so people became a little bit skeptical in the second half of the last century in particular after sort of the Soviet experiment and the Nazi experiment and some other regimes that ran with crazy ideologies, resulted in massive human tragedies. People became, well now maybe this grand vision for society thing is, is actually really dangerous. Well, so that's kind of what utopia traditionally mean, a deep utopia. I really just used the word because I am interested in the philosophical question of what would the ideal. What would a great human life be like if you abstract away from a lot of the constraints that currently Limit what we can do. So imagine you had super advanced technology. You have AIs and robots who can do all the work. You have super advanced biotechnology that give us unprecedented control over our bodies and minds and psychological states in this kind of condition. And let's suppose also that somehow governance work really well as well. So we don't have wars and oppression, but just wave the magic wand and imagine a really great society. So then under those conditions, what would a really great human life be look like? And so that actually brings us into. And I'm glad you sort of started thinking about this question, would I actually want to live in this kind of world? Because once you think about a lot of what we base our sense of dignity and worth and fill our lives with are because our efforts are needed currently. So you might pride yourself of being a breadwinner or of making a positive contribution to the world at large, or maybe just within your family, like you're a value person who contributes something. And to the extent that we define ourselves by our ability to make some instrumentally useful contribution right then in this world where AIs can do everything better than we can do, that is a kind of threat to our sense of self worth. We would certainly need to rethink a lot of the fundamentals of our values if we are moving closer to this world of human redundancy.

B (7:30)

Do you think those values are malleable? Or are they an echo of what I'll call an evolutionary algorithm running in our brain?

A (7:38)

Our values are what they are, but I think you can maybe distinguish superficial values and the deeper values underneath that justify or underpin them. And so my hope is that although a lot of the superficial values we have will need to maybe give up because they no longer make sense in this kind of solved world, that there are deeper values that could be more fully realized than is possible today if we remove some of these constraints. So you can kind of go through a bunch of different candidate values that people might have and see whether they would be instantiable in this kind of soul world. So like, if you take the simplest value first, like purely hedonistic, subjective, well being pleasure, let's say, like having fun in the sense of being in a certain subjective state. So that would be trivially easy to realize in a solved world in that you would have very advanced neurotechnologies that if, if all you really wanted was pleasure, you have like a super drug without side effects that could give you as much pleasure as you want, or maybe more direct ways of interfacing with the human brain. So you could check that box. If that's actually enjoying life and feeling good about it, that's a check mark right off the bat. You can then go through a bunch of others. Where it starts to get more problematic is when we get to values like meaning, purposefulness, significance, where if all problems can be better solved by machine, then what would give us purpose in our lives if there's nothing we need to do? So then to the extent that you think your life goes better, if it has meaning in it or purpose, then maybe to that extent it would actually be a worse life in this old world because it wouldn't be useful. Now, note that you have to distinguish it between the subjective and the objective sense of, say, purpose. So the subjective sense of feeling imbued with a sense of motivation and drive, and there is something you're striving towards that you really want and that energizes you. So that, again, would be trivial if you have these very advanced neurotechnologies. But some people think there is like an additional element, like what you might call objective purpose, that not just is there something you feel you want to do, but there is something that actually needs doing. And that is a lot less clear whether you could have that in a solid world. A, most problems would already be solved, right? If it's really a utopia. B, the problems that remain would be better solved by machines than they would by you in a technologically mature society.

B (10:40)

All of my anxiety around AI hinges on one idea that I believe. There's no way for humans to get around unless we rewire our biology. And that is it isn't about the pursuit of happiness. Happiness derives from pursuit. So I think from an evolutionary standpoint, that we have been designed over God knows how many hundreds of thousands of years, generations, to have to do very hard things in order to survive. And so evolution only has pleasure and pain to get us to do the things. And so I think when you work hard in pursuit of something that's valuable to not only you, but to other people, and you feel like you're about to be successful even before you're actually successful, that moment is like the greatest thing that life has to offer you. And I think that when people are in pursuit of something and it gives the meaning that you were talking about man, that's when it feels good. And the second you're either not working hard, you're just being given things, or you are working hard, but it doesn't matter because everybody already has everything that they need, and there's literally nothing that you could contribute to the group that would make the group better off. Then you ask a fundamentally corrosive existential question, which is, why do I matter? Why exist at all? And I think if we end up with a social structure that leaves people asking that question, we are really in trouble. Do you see a flaw in that fundamental base assumption?

A (12:19)

I think there is some truth to that as a psychological observation about our current minds, that some of the malaise in modern society might be from the absence of certain kind of survival pressures or opportunities that were always there in our evolutionary past. Just as kind of obesity is probably a function of refrigerators and plentiful food and stuff and fast food. It wasn't really a problem when you were a hunter gatherer. And so there is a kind of mismatch between our physiology and our current circumstances. And I think that's also true mentally to some extent, although it's amazing just how adaptive humans have been that we can still thrive as much as we can in these very different environments than we kind of. But I think a key question here is whether we value this purposeful striving you describe because it creates mental health and good feelings, or whether we value it for its own sake. So right now it's not important to differentiate these, because the only way you can get maybe the mental health and well being is by actually doing these hard things that you describe and then feeling the satisfaction. But in this hypothetical that I explore in Deep Utopia, these two elements could come apart. You would have these kind of the perfect drug that could induce exactly the same sense of satisfaction and fulfillment and energized relaxedness or whatever it is that hard effort produces now, but without actually having to make any effort. Another way to get at it maybe is to what extent do you think our artificial purpose would be a good enough substitute for the natural thing? So real purpose might be like you're being chased by a tiger and you really have to run as fast as you can, otherwise you get eaten. And there are very real stakes there, right? It's not something you just randomly make up that you happen to like to run away from the tiger. Once the tiger is there chasing you, you know what you have to do. And it's a given and it's a very real purpose. But contrast that to somebody who is playing a game, like maybe they really want to win, but in some sense the game itself is like an artificial purpose. There's no reason, if you're playing golf, there's no reason why the ball has to Go into these sequence of holes. Other than that, we just decided, let's try to do this. We make up this random goal. And then once you accept the goal, then you have the purpose of trying hard to achieve it. And people can work decades to try to perfect their golf game and find a lot of satisfaction. Right. But in some sense, the over the whole thing kind of made up. It's artificial. Right.

B (15:14)

It is, however, run the thought experiment of imagine if you could play golf, but no one would ever know about it. No one would ever see, no one would ever know that you were better than they are. It, I think, is a proxy for social status. So you are trying to rise within a hierarchy. And so the question becomes, this is really. There's. There's two things here that we have to put on the table because there's a huge hurdle before what I'm about to get to. But let's say that everything's taken care of. Utopia is here. As you defined it earlier. We don't want for anything. Everything's equal. And now will people be interested enough in status games that that will be fun or will they be like, but this is all just a status game. So there's a deep emptiness. I think we have clues right now that answer that question. So right now you have video games that are unbelievable. As somebody that discovered Minecraft in my 40s, I will just tell you that game is unbelievable. I cannot believe that kids get to grow up in a world where that game just exists. But nonetheless, if I'm playing it and it doesn't feel like it's going anywhere other than I'm playing it, it does have an emptiness, which I think is part of part. It is a small part. This is a huge problem, but a small part of the sort of male sense of meaninglessness that certainly is sweeping across the West. Okay, that. That's the question you're putting forth. I think we have enough of an inkling to say it probably doesn't pan out as well as we would want it to. But I think there's probably a more important question that has to be asked before we even get to that.

A (16:53)

Which is.

B (16:53)

Which is humans with their current value set now, their current brain wiring now, how are they going to. Are they going to accept or radically push back against AI when they see that it will lead us to a world where they are irrelevant by today's standards. Let's talk about money you're leaving on the table every time you pay full price for a health product out of pocket. You could be missing out on 30% savings. There are over 40 million HSA accounts in the US holding $159 billion in pre tax dollars. That's where Trumed comes in. Trumed helps qualified customers use pre tax HSA dollars on health products that can qualify as medical expenses under IRS guidelines. You complete a health survey. Trumed handles the documentation and the purchase the and you can save an average of 30% on products like 8 Sleep Peloton, AG1 and others that studies show can treat or reverse many common chronic conditions. Stop paying full price for products your HSA was billed for. Go to trumed.com impact and check out what things you want that may qualify. It takes just a couple of minutes. Trumed is for qualified customers only. HSAFSA tax savings vary let's talk about the most powerful force in wealth building. Compounding. There is a reason that people have talked about this forever. You put an asset to work, it generates more of itself over time. The curve bends way in your favor. Every serious wealth builder knows how compounding works. Most gold owners, though, have completely opted out of it. Monetary Metals changes that equation. They've built a way for your gold to earn yield paid not in dollars but in more physical gold, up to 4% per year. More ounces stacking every single month. Compounding in one of the only assets that can't be printed, inflated or debased. And storage and insurance are included by the way, so hidden fees don't eat your yield. The average person is passively absorbing inflation. This is how you compound your way out of it. Click the link in the show notes or visit monetary-metals.comimpact to learn more. This is a paid advertisement. Let's talk about the worst investment most guys make on repeat cheap clothes. You buy them, they look fine, but six months later they're pilling, shrinking or just falling apart. So you replace them. You do it again and again. You're spending more over time and you never actually have anything worth keeping or wearing for that matter. That's the whole model behind Quince. I've got one of their 100% Pima Cotton Tees and the quality is immediately obvious from the second you pick them up. They're soft, well constructed, the kind of thing that holds up over time. And that is the point. Refresh your everyday with luxury you'll actually use. Head to quint.com impactpod for free shipping on your order and 365 day returns. Now available in Canada too. That's Quince Q u I n c-e.com impact pod for free shipping and 365 day returns quince.com sl impact pod if

A (20:27)

I had to guess, I think more likely it be we'd be seduced by it and, And the displacement. I mean in part this is economic people losing their jobs maybe right? Or downwards pressure on wages that will arise as automation advances. Interestingly in this case it might initially hit more certain kind of white collar work that like traditionally automation mostly affected lower skilled workers. But this kind of current language model technology seems to hit right at the sort of mid level white collar work like people who summarize documents for a living and stuff like that. So interestingly exactly, if the AI succeeds at automating a wide range of jobs, there will also be massive economic growth which to some set extent might offset the kind of economic impacts of unemployment. Like if you have a booming economy, there are more tax revenues, it's easier to. There's more demand in other sectors that haven't yet been automated. There's more people can spend more money on, you know, hiring gardeners or nurses for their grandparents or whatnot. But it still leaves this question about meaning and purpose and social status. I'm wondering about social companions in this context, like AI social companion bots. This might become another kind of.

B (22:06)

Is that a very nice way of saying sexbot?

A (22:09)

Well, it would encompass that, but it could also just be friends and fans and all, all the different elements of social interaction. Like maybe even some sort of fake status. Like you were saying that if people want to feel that they are high status in the real world, maybe they aren't and it's just a frustrating experience. And rather than work for years and decades to get like one notch up on the status hierarchy by I don't know, like stressing yourself out in the gym to get a slightly better body or like educating like all these achievements, they are hard and take a long time, right? Imagine if you could instead tap into this virtual world where you have perfectly realistic virtual characters and where you're playing, you're the king or something like that and you have these admiring digital characters that maybe that if that becomes good enough would be extremely compelling to people for the same reason. I mean drugs are really compelling. And often in case the worse your real life is, the more attractive. Like a kind of the alternative of some opiate or something is right, that can sort of. So I think that yeah, this whole AI social companion technology will advance very rapidly over the next couple of years.

B (23:34)

Do you see that being like online dating where at first people are weird about it, and then it just becomes the norm.

A (23:40)

Yeah, I wonder. So on the one hand, there is like something slightly dystopian seeming about. Like, if you imagine a world where more and more of our time is spent not with real people, but interacting with these, like, bots. On the other hand, it might be one of those things where there's like a generational thing where, like, I'm the old fuzzy daddy, like, grandpa, you don't get it. And then people who grew up with this is like, yeah, of course. It's just much more interesting. This AI bot is much wittier and really pays attention. And these humans are kind of a drag and we are all just using these AI bots now. And like, if that happens, is that because this whole generation will have made, like, a big error, or is it just that they would have more familiarity with this and on its merits, chosen to spend their life in that way rather than by hanging out as much with other fellow humans? It's hard. So it's easy to have opinions about these things, but it's hard for those opinions to actually be grounded in some kind of objective truth as opposed to just merely reflecting your own personality or upbringing.

B (24:57)

Agreed. I think there's certainly value in both. But one of the more interesting things about you and the way that you've approached these problems is with the anthropic principle and finding ways of to at least ground things in probabilities. I'd like to talk about your probabilistic look on what does AI do in the next three to five years, and then what does AI do in the long run? Obviously we're talking probables here, but as you've used that to great effect, I'd love to see how you think through that. Using anthropic principles.

A (25:35)

Well, yeah, I don't know about using anthropic principles. I do feel AI timelines appear relatively short from this point on. I mean, we are really far along the path towards AGI already. The things that are now possible. If you had asked people 20 years ago, I think many people would have assumed, well, if you can do these things, if you actually can have, like an AI that can have a conversation with you in ordinary language and like, you can't even tell whether it's an AI or a human. Like, unless you're really an expert, you know exactly how to pro. That. That. That seems like AGI computers that can write code, like, at the level of like, maybe an entry programmer. So we have this tendency, I think, with Each advance in artificial intelligence to move the goalpost and to immediately discount and take for granted what's like, the same thing happened like when Compute Deep Blue beat Garry Kasparov in chess. Like before that, people saw chess as this great game of the human mind. Like, the most complex thing the human mind could do is to learn to play chess as the high level, really deep logic. And then after computers could do it, we said, ah, it's just a game of chess. Like, there are simple rules. And then the same happened with Go. And then when you could have AIs look at pictures and actually visually understand what's in the picture. And now with natural language, I feel there are not that many of these steps left before you have AI that can do all kinds of AI research better than humans can do, at which case you have, I think, an intelligence explosion, because then you have the AI research being done by digital minds at digital timescales, and then you get a very rapid feedback loop. Right. With each subsequent improvement, the force that is doing the improving gets stronger, and you might then have some kind of singularity. Now, exactly how many years that is away is hard to tell, but I think we are no longer in a position where we can be confident that it couldn't happen even within some very short period of time, like a year or two. I'm not saying it will, but we are not in a position where we can be really sure that it won't. It's like it might just be somebody makes some other breakthrough of the level of that transformer architecture and applying that to the already really large models we have. You know, maybe that will be enough to sort of unlock a lot of latent potential there. Or maybe there will be like, need two or three more of these advances or more scaling up of the size of the data centers. We just don't know exactly, but we are sort of, I think, close enough that we can't be confident that it couldn't happen anytime.

B (28:35)

What would you advise to somebody that is a junior in high school? Now, these are American terms, but junior in high school. Now they've got to get really serious about where they're going to go to college or whether they're going to go to college, what they're going to study, how can, if we are, and I heard you, this is not a guarantee, but if we are potentially within a year of AGI, how can somebody even plan for the future? It just seems like such a big question mark.

A (29:01)

Yeah, I mean, but to be clear, it could also be 10 years or 15 years or like it's just. Yeah, I think, I mean, I'm always wary of giving a general advice to everybody. I feel that's like giving advice on what's the best shoe size. Like what's good for one is like, you know, so some people maybe are too hard on themselves and good advice to them might be to, you know, ease up a little bit, like go easy on yourself. And then for other people, that might be exactly the wrong advice. They might actually need a stern message. You really need to pull yourself together here. Like you're just wasting your time. Discipline. So the same message might be completely right for one person and wrong for the other, depending on how they are currently going wrong. And similarly with career advice, you know, it depends a lot on what your talents are and what your passions are in life.

B (29:49)

I think more than I'm looking for something specific, I'm looking for a guiding philosophical principle. So I know that you used to run the Institute for the Future of Humanity. So you, I'm sure, have thought a lot about where we go, how we deal with it. Well, and so yes, we're not going to say you should be a dentist, but I'm guessing you have a framework that people would benefit from in terms of facing such a rapidly changing environment.

A (30:18)

Yeah, I mean, so it depends like, so there's like a small fraction of people might actually, you know, be looking to directly contribute by researchers or AI scientists and stuff like that. That's like one avenue I think in general probably useful to familiarize oneself with the current tools and the next generation so that you kind of know roughly where things are and what they can and cannot do to be adaptable. But then for other people who are not really technically minded, I mean it might be that going in the opposite direction, you know, being really developing your skill with people. I think there's like enormous needs for various care professionals. I think like, say with elder care, like if we just had more resources in theory, every old person should have like their own full time personal would be great, right? Some younger person that could live with them full time and just help them if they fall, help them up. We can't afford that. But in principle the need is kind of unlimited there. I would also say that don't forget to actually enjoy life right now. I wouldn't sort of plan on a 40 year career and make big sacrifices now for 10 or 20 years in the hope of then it paying off like when you're in your 50s and 60s because, you know, maybe the future doesn't exist at that point, it would risk being a kind of. Yeah, I would maybe focus a little bit more on short term strategies.

B (31:53)

When you say the future may not exist, what do you mean?

A (31:57)

Well, I mean several different things actually at the same time. But in one, one thing I meant was that if this AI revolution happens within the next five or ten years or something, then these long term investments in human capital that we might make now with a payback time of 20, 30 years might not pay off. Because by that time maybe human capital will have depreciated as a result of AIs supplanting us across the board. So that's one sense in which the future would not exist. There are other senses in which it might not exist as well, related to the simulation hypothesis, which we don't need to get into. But yeah. So enjoying things now also, like with college education, I think if you would like really enjoy your time at college, that's one thing, then maybe do it. But if it's just something you have to drag yourself through for the sake of getting a diploma, I would maybe seriously consider if there are not ways to sort of cut out those three or four years to get straight to what you want to do. And similarly with PhD programs that can take in the US five or six years, that's a long time. I think in many cases it may be too long for it to be worth it these days.

B (33:11)

Just because the rate of change is so accelerated.

A (33:14)

Yeah, because the timelines might be shorter. And I mean if you imagine if you had like, so suppose you had the view that there was like a 10% chance every year that the world will blow up and be destroyed. So then like you wouldn't make 20 year investments really. Right. Like you, you'd kind of focus on things that have a shorter payback time. So like having a de facto higher interest rate or hurdle rate for your, your own long term investments maybe would make sense in this picture. Now I would like hedge a little bit because this could all be wrong. And if the AI thing doesn't happen or if it's banned or it stalls out, you don't want to end up completely dry either where you have nothing. You're 30 years old, you lived for the day planning that the AI revolution would happen. Somehow it fizzled out. That was like a global ban. And then you're now a 30 year old with nothing. No skills, no job, no nothing. So you might want to, depending on like what your sort of social safety net is, you might, might want to hedge Your bets a little bit there.

B (34:22)

Now I know that a lot of people have what they call a P doom number. So how likely you think we are to basically blow up the world, whether with AI or something else. As I'm listening to you, it does beg the question what is your P doom number?

A (34:40)

Yeah, I don't actually have a specific number, but maybe one way to think about it is if you could divide it into. So there's like ways in which things could go really badly, right, where you kind of blow up the world or some dystopia. Then there are sort of the more utopian scenarios. Everything is clearly we cure a lot of diseases like wonderful prosperity, so each of those would have some probability. But I think there's like a third bucket in the middle which is actually perhaps maybe the most probable, which is that the world is such that even if we could actually see what would happen, even if you had like a little binoculars so you could look at the future and study it, you wouldn't really know whether to count that as a success or a failure. It would maybe be very different from the current condition. Better in some ways, worse in some ways. Strange. There's some kind of minds doing stuff there. They are not exactly human minds, but they're sort of a little bit doing the same thing. So do you count that as. As there being humans around or is like are we all dead and just replaced or did we sort of grow into this new life form? So I think it's not obvious that the future would be such that if we could see it, we would necessarily know even what to think of it. You can think of it in an individual life. So right now we have children like say a 4 or 5 year old who eventually becomes a 25 year old. And the 25 year old is quite different from the 5 year old in many ways. Like mentally they have different interests. They're no longer interested in the toy train. They are interested in their romantic partner or their job prospects or US politics or whatever. It is like the Roman Empire in many ways. What was there at age five is all gone. And yet we don't think it's bad for the child to grow up. I mean, in fact, most of us would probably think it would kind of be something sad and unfortunate if a 5 year old never grew up to become a 25 year old if they remained at the level of a five. So I wonder if there is like a similar thing where we now are basically children in the skip. None of us ever get the chance to truly Grow because we just like biologically developed for 20 years and then stagnate and then we sort of rot away and die after a few more decades. Just biologically, we can't live for 500 years continuously growing and expanding and learning new things. We are kind of cut short. And maybe 80 years is just not enough to really fully realize our inherent potential. We are kind of zapped by our rotting biology. There might be different kinds of lives that would become possible. If you could live for a million years and if you could gradually upgrade your capabilities, that might be really wonderful. But that would maybe change us as much as like the 5 year old is changed when he or she grows into a 25 year old or more. What that perhaps suggests is that especially when we're zooming out and thinking about these more radical scenarios, we should not really focus so much on comparing two states like the current state and some later state, but maybe thinking more in terms of trajectories leading out from the current state and then evaluating how desirable those are. Like maybe it's fine if ultimately we end up in a very different weird post human condition, like in 10,000 years from now. Like, but if we went there slowly and we sort of had a chance to grow into it properly, that kind of trajectory might I think, be more attractive than one where we just remain humans and keep doing the human like things for 500,000 more years. I don't know, 5 million more years. Like at what point is enough? At some point you'd want to maybe sort of unlock the next level, right? It's like playing the same level of a computer game. At some point you need to move on and maybe similarly the kinds of values and lives that can be lived with our current human physiology is like a limited set of all the possible values. Maybe we haven't yet exhausted it. We might want to spend some more time and go slowly through the level rather than just skip to the sort of the final level. That might be another mistake. But still thinking in terms of a trajectory that eventually leads to greater forms of development, including ones that ultimately take us out of the human realm. Study and play come together on a Windows 11 PC. And for a limited time, college students get the best of both worlds. Get the unreal college deal. Everything you need to study and play with select Windows 11 PCs. Eligible students get a year of Microsoft 365 Premium and a year of Xbox game. Pass ultimate with a custom color Xbox wireless controller. Learn more@windows.com studentoffer while supplies last ends June 30th terms at aka mscollegepc. Experience a membership that backs what you're building with American Express Business platinum. Unlock over $3,500 in business and travel value annually with statement credits on select purchases from brands like Dell, Hilton and Adobe and other benefits. American Express Business Platinum. There's nothing like it. Based on total potential value of statement credits on select purchases and other benefits, enrollments required monthly and other limits and terms apply. Learn more@americanexpress.com Business Platinum Starting a business can seem like a daunting task unless you have a partner like Shopify. They have the tools you need to start and grow your business. From designing a website to marketing, to selling and beyond, Shopify can help with everything you need. There's a reason millions of companies like Mattel, Heinz and Allbirds continue to trust and use them. With Shopify on your side, turn your big business idea into Sign up for your $1 per month trial@shopify.com specialoffer okay,

B (40:39)

you're playing with a lot of ideas here, and I want to start pinning some of them down. So one is the idea of trajectories. And I think people right now today are going to care a lot about that. So through regulation, through what people end up pursuing as entrepreneurs, we're going to have a tremendous amount of influence over what gets developed, what doesn't get developed. And so I think that's the big question of today is what trajectory do we want to see this go on? And so I'm very curious to hear your take on how much can we control the trajectory and do you see an ideal trajectory?

A (41:19)

Most of the uncertainty about how AI pans out is uncertainty about how hard the challenge is that we will confront rather than uncertainty about the degree to which we will be getting our act together and make a good effort. Like, we don't know how hard this is. We've never had a machine intelligence transition before, right? We haven't studied a million other planets where some human, like species developed AI. And we can, like, study the statistics we have, you know, coming to this afresh, we have no idea whether it's like, relatively easy or like fiendishly hard. So I think that's where most of the uncertainty is coming from. And then a little bit uncertainty, like, obviously we can at least nudge the odds in a better direction if we really make a good effort. We work on this collaboratively. You know, we are really smart about it. We studied hard, like, be careful. Then like, we can improve the odds a bit. But, but most of it is just, I think, baked in. So in that sense, I'm kind of fatalistic. Like you could say I'm a moderate fatalist in. In the sense the moderate coming from. Well, we can still affect the odds at least a little bit on the margin by getting our act together. But fatalist in the sense of, for the most part, it's probably just baked into our situation and the technology itself.

B (42:43)

When you say it's baked in, what do you mean that it's going to happen?

A (42:46)

The outcome for humanity, for example, whether we end up destroying AI kills us all, or we achieve alignment and manage to align it to human values. I think some of those things might be baked in in the current situation.

B (43:07)

Elon Musk has said that he thinks of AI as a demon summoning circle and that we should be very careful about what we wish for. I'm hearing tones of that in what you're saying now. And he said his life got a lot better when he became more fatalistic about AI. What do you think about his take, his level of anxiety about AI Warranted? Not warranted?

A (43:32)

Yeah, it seems warranted. And he is also the founder of xai, which is an AI startup and as well as Tesla, which has major AI operations, and one of the original investors and founders of OpenAI. So I guess his attitude is complex. I think he recognizes that there will be big dangers, but it doesn't necessarily follow from that that the conclusion is that each person should unilaterally remove themselves from the race.

B (44:06)

Okay, so when you look at this, you have a similar this is going to happen. This stuff is baked in. I see a world where we end up bifurcating as a species. So I consider myself wildly technologically optimistic. I have a natural bent towards somehow things will just work out. But I also look at what I can feel brewing in culture right now, which is a massive resurgence of religious fervor, people reconnecting to that, refinding faith. Accounts that are focused on faith on podcasts and YouTube's. YouTube are starting to dramatically increase in popularity. And I think in many ways this is a response to a hyper technological world where even just us humans are using technology a lot, whether it's ozympic and losing weight, whether it's antidepressant medication, whether it's AI that they see this influx of what I think many will read as anti human things. And there's this feeling, a desire to connect with something traditional and certainly divine. And I see that creating a bifurcation in society and what I predict timeline gets a Little bit fuzzy because it's all going to be predicated on the rapidity with which AI disrupts our normal life. So on whatever timescale that is, I think what you will see is a group of people that spring up that I'll call puritans that will not want to use AI. They won't engage with art that's created by AI. They won't support companies that use AI to create their product or their marketing. And then other people that will sign up for neuralink when it becomes available and they will literally augment themselves. They will use AI whenever and wherever humanly possible. They will fantasize about free energy and the utopia that AI is going to bring. And I think over time those guys will end up pulling apart. Especially if AI helps some people augment themselves. And you could be augmenting yourself directly. You could be augmenting your children just through gene selection, let alone gene editing. Do you think that's plausible? Likely delusional?

A (46:31)

Yeah, I think the debate is likely to become polarized. If we're talking about the sort of public debate about what should be done about this. We're already seeing a little bit, you know, on the one hand the sort of the doomers, right. And then the ex that like kind of go forward with all maximal speed on everything crew that are sort of dividing themselves up into two different tribes that can now start to hit on one another. And I think maybe broader segments of society will be recruited into this debate as the impacts start to be more widely felt. I could see, yeah, like it's interesting to think about how the speed of development might impact the degree to which this happens. So I think maybe there are actually three different regimes. So like extremely sudden, fast, like super intelligence is invented, like next week just comes out of the blue, right Then, okay, there won't be much more polarization than there currently is because people didn't see it coming in us. And so now I think also maybe if it's extremely slow and happens over many decades, then it might be kind of the boiling of the frog phenomenon where people are using this technology. And of course every little increment makes it better. If you're going to have a medical diagnosis spot, surely you want it to make slightly fewer errors rather than more errors. And so every little step along the way will just be better. If you have self driving car, you want it to be slightly smarter so it's crashes less often for every application. It's clear that more capable means better. And so if you just follow that long Enough you eventually end up with super intelligence. But at no point is there a clear jumping off an alarm signal. Now in the intermediate scenario where you have kind of turbulence where people feel dislocated because every other month there is a new thing and now a big sector of workers were laid off and now there is this other thing that has created these propaganda bots that are running around and then there's like deep fakes or like then and then some big disaster happening because the AIs were running the power grid and it all sort of malfunctioned. And like in that kind of world, or drone swarms coming in and killing a bunch of people in war, you can imagine maybe more that's that turbulence creating a kind of increased resistance. Now I think you were also asking not just about like the conversation around this, but also whether different communities will form that sort of like the Amish decide to only use certain technologies and that whether many people will opt out of this AI technology.

B (49:25)

Very much so, yeah.

A (49:27)

I don't know it, it's kind of. So I mean unless you go full like, unless you're like really hardcore about it, like some of these communities are where you don't even want cars and stuff. Otherwise it's pretty sort of integrated into the modern economy. Like if you are using Google, like you're using AI, right? In the future every car will have AI. The electricity grid will be optimized with AI algorithms. You know, all these different systems that you interact with, like the doctor, they will have like probably use some AI bot and you have some weird mole, they'll take a picture of it and scan it and then some like skin cancer diagnosis system will look at it and flag it. Like they will just be everywhere. So it might not be easy to opt out unless you're really willing to sort of completely rip. Tear yourself off. That's kind of the fabric of modern society.

B (50:30)

Do you see that happening? That seems self evident to me that that's going to happen.

A (50:36)

Well, the question is on what scale, right? So I mean there are people who live off the grid or who are Amish and stuff, but they are still a small fraction of the world population, you know, with a higher growth rate because fertility rates are larger. So if you imagine rolling the tape forward hundreds of years, then eventually those groups would expand and others would kind of dwindle into non existence unless they change their ways. But I'm just thinking the timescales for that kind of population dynamic level stuff to play out is multi generational. Whereas the technology is moving forward year by year. And so I'm thinking like, yeah, it will not have. There will not be enough time for these slower processes to really would be my guess, to really have a big impact.

B (51:27)

So here's how I see it, and maybe you can pull me back off the brink, which I would love. But it ultimately, when humans feel either emotionally distressed or financially distressed, which usually the two are intertwined, they will go all the way to killing their fellow humans with just absolutely no problem whatsoever. If you take the French Revolution, right, Things got bad enough economically, they just pulled people into the streets and started beheading them. I don't think we are fundamentally different than that version of humanity. And I think if AI begins to disrupt enough jobs and creates enough turmoil, that it's not like the Industrial revolution where, yeah, you had a generation that had it kind of rough because they weren't able to rapidly change, but there was just such an economic boom that everybody, the people that were winning from it far outweighed the people that were losing from it. And so it ended up being fine. But I think what you're going to see is a disruption that happens so quickly and touches on the one thing that if you break, you're going to have a real problem, which is meaning and purpose. And the only hope we have, and this is ultra dystopian, is that we have enough entertaining things that people are numbed to the fact that they're not no longer climbing, that life isn't going to be better for them than it was for their parents, that they've lost their job or whatever, and so they drink, do drugs online porn, play video games, and that just becomes a sort of get by existence and they just sort of give up on it. That that's the hopeful outcome, but I think the more likely outcome is that this becomes a political divide where the battle ends up being drawn along the lines I was saying before, where people that utterly reject it and just want to absolutely shut down AI, put it back in a bottle, and then people that wanted to develop it. And just another terrifying twist, if AI comes out slowly enough that we see, let's just say China makes a major advance, but it's not a big enough advance where we would automatically lose in a war. I could see a preemptive escalation of violence to shut it down, to make sure that either we're able to hit parity with them or we're by, by elevating ourselves or by tearing them down.

A (53:54)

It's very hard to predict these kind of Social, political dynamics and cultural dynamics. We don't have the kind of scientific theory that that can tell us how social sentiment will change over the course of five or ten years when you start to, I mean, I think like in the past, a lot of revolutions, I guess is like for lack of food to eat. I think that hopefully would be relatively easy to supply with some degree of political mobilization, especially in these rapid growth scenarios. So you could have bread and circus. The purpose issue might be harder to remedy. But then maybe the line is, well, let's be honest here. Most people's lives today, just how grandiosely purposeful are there really? You go in, you make a paycheck, and then you spend the rest of your hours relaxing or having fun or playing with the kids. Like most people are not really trying to change the world or imagining that they are like some historical figure bestriding humanity to shape its destiny. It's just not reality. And so if you didn't have to go in and work for eight hours every day doing some pretty boring stuff, maybe that you don't really want to do and you'd rather sleep in and have fun and like, you know, would that really be such a tragedy? You got the same paycheck, let's say, but without doing these chores, that seemed like a win potentially. Hopefully a lot of the energy that people put into work could then be put into instead building up leisure activities. Like to have clubs and hobby organizations that create sort of activities for people who now have more free time to do things. And so that would have to be this cultural reset that seems like maybe a better outlet for the surplus time and energy than trying to tear everything down.

B (56:05)

It's interesting. So have you read A Brave New World? What do you think about that? Because it feels like you have sort of dueling dystopias. You've got on the one hand, 1984, massive suppression. You could think of this as an AI tool that's watching you all the time. Like if you reread 1984 with the thinking of AI doing the surveillance, this suddenly becomes super real. So you have that version of dystopia where even wrong think gets you punished. And then you have over here the other version, which is just keep taking, taking your drugs, feeling good, being blissed out all the time. Both read as dystopias. Did you take A Brave New World to be dystopic? Is there something that I'm missing in that interpretation?

A (56:55)

Yeah, I think it's missing some elements that if they were added, would make a World lot better. So there is like, no, it was a long time since I read it, but I think like no real romantic love, for example, in Brave New World. No appreciation for true art and beauty at a higher level, as opposed to sort of easy distraction and shallow kind of flim flam. And so if you imagine a Brave New World like scenario, but where people actually had a lot of free time that they spent being with people they loved and cultivating hobbies and like appreciating great literature and learn, like cultivating the art of conversation, you know, maybe taking arts classes to deepen their appreciation of great art. Like all kinds of things. Like also less cerebral things. Like some people might be doing more sporty things or being into nature or whatever it is. But that a society where people kind of were. Yeah, focusing on sort of developing a high culture of living well, I think could be pretty utopic. The other thing with Brave New World that I think cast a kind of dystopian poll over it is the really stratification of their society where people are sort of destined from birth to be a particular class. Most people in Brave New World have various degrees of engineered mental retardation. I think they add alcohol to the fetus to sort of deliberately brain damage certain people so that they would be suitable then to work as elevator operators or menial jobs without having some kind of. So that obviously makes it pretty horrific. If you imagine a society where everybody were more allowed to encourage to sort of grow to their full potential and to be full participants in a kind of democratic politic, then I think that would also brighten up that kind of future. The main thing that might be then missing is kind of certain forms of heroics that there would be no need for sort of heroic self sacrifice, like, you know, some great warrior who sacrifices himself for the sake of the comrades by rushing the enemy with a spear, like that kind of thing. Okay, so that might be a certain value in that. That would no longer make any sense in Brave New World, but we can probably have a pretty good life while just kind of reducing or redefining a few of those particular values really fast.

B (59:46)

So I want to move on now to how this could actually be amazing. I think there are some interesting, what some people will read as religious implications, but there's certainly value system implications. But before we do that, or as a ladder into that walk me through, what will AI actually do? Boots on the ground. That could lead to a utopia where we're able to pursue more leisure things, our passions, whatever. What is AI Actually going to do well.

A (60:22)

So it could start with a lot of work that could be automated. So instead of having some guys who have to drive the garbage truck around the city every morning to collect the garbage, you could have a self driving garbage truck with an optimus robot that tops off and picks up your garbage can and does all of that automatically. But then you can go through job by job. And if you really have artificial general intelligence like AI with the same learning ability as a human mind, then basically every intellectual human job would every, every while, every job that could be done over like a video conference link could be automated. Right? And then if you have robotics to go along with that, then also all the manual work could be automated with very few exceptions. Like for example, that might be job for athletes. If consumers just happen to prefer to watch humans runners compete than to watch robot runners compete. Right. Maybe that's just a root preference that people just prefer to watch to humans, then there might still be jobs for humans in those areas. Or people just want the human priest to officiate their wedding. Even if the robot priest could intone the same words just as well as the human. You might just care about a certain task be done by human. Then those would be carve outs, but for the rest you could automate. So this would create a potential enormously high rate of economic growth. Because it's like you can just build more and more robots and run more and more instances of these AI minds. So everybody could have a bunch of these that would be on the lookout for their interests and helping not just to clean up the home and to cook their food and but AI bots who were like scanning for really great movies that you would watch that knows all about your preferences, that could check in on your health and make sure you take your medicines or spend hours just researching your particular health conditions to optimize everything. So everybody could in that sense, as step number one, live like extremely rich people in terms of their, their access to human labor and the cost of products would go down if you take human labor out of the equation and it's all automated. But then beyond that and more exciting I think is you could actually move the technology envelope outwards by doing like accelerating the pace of new research. Like for example, in medicine there are a lot of conditions now that no matter how much money you have, you can't cure them. Because we don't have cures for all cancers. And for hard. So there you could imagine accelerating and have like a thousand years of medical research progress in just a couple of years when you have these digital minds working on this, like maybe unlocking cures for to reverse the aging process, et cetera, and then forestalling a huge amount of human misery and death that is currently pretty unavoidable. And in other areas, like much better entertainment, where you could have AIs making these movies and artworks and virtual reality computer games designing experiences and just generally organizing things in a more delightful manner. And then beyond that, I think also these ways of then improving the human organism itself, our own brain, is kind of ultimately limiting how good our lives can be. So then you could imagine these different forms of upgrading opportunities. Maybe step one, extending your healthy lifespan. I think that's perhaps an obvious one. It's just no fun to kind of have terminal diseases and just have your body decaying and all into all the pain. Like you're just like making sure you remain healthy. And then on top of that, you could then start to maybe add boosts to your well being, to your ability to understand your musicality, your sense of humor, your ability to form deep emotional connections to other people. And then kind of having a trajectory where people can continue to grow and develop and achieve ever greater levels of flourishing. And I think like, yeah, and then freeing up space for more spiritual practices and aesthetic experiences as opposed to kind of low quality, mundane, work related experiences that currently occupy much of our waking hours.

B (65:17)

In a solved world, which I think you define as all things that we know to be technologically possible have actually been done so in that world. Do you think that we'll be able to perfectly manipulate our nervous system so we could see and feel and hear and taste and basically completely manipulate our senses to orchestrate experiences?

A (65:42)

Yeah, I think so. I think if we have our organic brains still, there might be some limits to that. I think it could get pretty close by combining like perfect virtual reality. So you could have like whatever sensory input that you choose, right? Sound and smell and vision and stuff combined with imagine some kind of super drugs that allow you to fine tune each emotion precisely the way you want it, without side effects and addiction potential that already I think gets you some way there. Then if you imagine other kinds of neurotechnology, like kind of neuralink interfaces or other more kind of invasive forms, gene editing, various brain circuits and stuff, I think you could get even farther along that path to sort of having fine grained real time control over the content of your experiences. If you go to more radical scenarios where maybe you upload into a computer and sort of digitize digital minds, then you might have even more fine grained control because those data structures would be completely accessible. You could sort of edit every little neuron in real time potentially. But I think even with biology, you could get pretty close to that with mature technology.

B (67:01)

Okay, so that gets into what I think if, if that comes true, I. That is where I think we inevitably end up, when I put my utopian hat on that. Are you familiar at all with the Japanese genre of storytelling known as isekai?

A (67:19)

Not really, no.

B (67:21)

It's kind of like, ready player one. You, you wake up inside of a video game, but you're actually in the video game and so you can have relationships with the characters. If you get killed, you actually die. There's a whole thing about that that feels to me minus the you actually die part, that feels like if we could completely manipulate our nervous system or upload ourselves, that the sort of ultimate expression of life, the maximally cool way to live would be to have like, you go to Netflix and you're like, ah, today I want to be a master chef from the 1800s France, go. And you then go live as that person for however much time you want. You could speed time up, you could slow your perception of time down, whatever. So you could play for an hour, but it felt like you played for 50 years. You could do something I want to do, like I'm traveling the cosmos and now you don't actually have to violate the laws of physics to travel somewhere billions of light years away. It would just be an imagined simulation and. But you're actually. You feel like you're there, you smell everything. And I don't know, with our current way of responding to things, I don't know that people could avoid that, that you would end up with an. A universe that is tailored to you, that you may change frequently, or you may live in one for an extended period of time. But that feels like, again, with, with the way we are now, because obviously we could change our own biology so fundamentally through AI that you can throw this out the window. But that feels like the end game.

A (68:59)

Yeah. I wonder though, whether it might not be possible to do even slightly better than that. I mean, you might say if it's either that or the current world. I mean, that does seem pretty good because the current world has a lot of. I mean, it's not just like, you know, your baby smiling at you or like a sunset. Right. Most of the real world is pretty much a horror show for many people, for many animals. And so even just getting rid of the negative might be just, well Worth. But if you have all of that, I think one of the sort of values that seem possibly missing from the picture you painted there is this value of, again, coming back to purpose or meaning or significance. There is like, some sense, I think, when people consider that, yeah, these would be like, you know, a lot of fantastic experiences that would be a lot of fun, but it seems a little sort of shallow or like a little arbitrary or random or atomistic. And so I think one might want to think about whether there would be the ability to have a lot of that, what you just said, but with some constraints that create more of a structure around it, that within which we also can have some forms of purpose. I think there are these constraints particularly rising from our commitments social, culturally, to other people and to traditions. So think of a very simple form of purpose in a technologically mature world. Suppose that person A happens to really want that. Person B gets what they want. So person A cares about B. They really want person B to get what they want. They just happen to have this desire. And then person B now is in a position to give a purpose. If B wants that A should do something on their own steam, not just outsource this to some robot, but that A should put in their own effort to do a certain thing. Now, A has purpose in the sense that the only way they can achieve their goal, that B gets what they want, is by A themselves actually putting in the effort, because nothing else would actually satisfy B's preference. And so now A would actually have a real purpose. The only way they can achieve their goal is by putting in effort themselves. No matter how advanced the technology is that they have access to, there would be no other way of actually achieving B's preferences. So I think in this very reductionistic way, it seems a little hokey with like, person A wanting this and person, But I think more complex and subtle ways of that could actually be a plausible framework of constraint for these future deep utopians where they kind of care about various traditions, various social norms and activities in some ways that give them reason for doing things on their own steam. Even if they're technologically, they would be able to outsource a bunch of stuff. Like, for example, if you value upholding a tradition, there might be no other way of upholding it than actually by continuing for humans to do various things on their own. Like setting up a population of robots that kind of enacted these past ceremonies or something might not count as continuing the tradition. And so maybe there would be a kind of more ceremonial and social aspect to These future utopian lives that would still give a lot of room for this kind of unbridled fun, but maybe there would still be sort of more, I don't know, serious contours around it that gives it a little bit more shape and structure. That could also allow the utopians to realize various forms of purpose, including natural purpose, and not just this artificial purpose that we see in game playing, where you make up some random goal just for the sake of having a goal, but where they care about each other's opinion about things. And maybe, like what you were saying, wanting higher social status. If the thing that gets you social status is doing things on your own steam, like, then people have to do things on their own steam if they actually want to achieve the social status. So something like that, that kind of, on the upside, gives maybe some scope for human effort, but then on the downside, has this amazing safety net where the AIs make sure that anybody who falls seriously sick is kind of cured and that everybody has all the food they need and all the material conditions for a great comfortable existence, that everybody's brain kind of have the opportunity to experience great levels of joy. Some people are just born depressed and their whole life sucks just because they had too little, too few neurotransmitter receptors in some area of their brain. Like, their whole life is just basically ruined. You could fix that, lift up the floor, and then at the very top, you could still have these kind of little games that people are playing, perhaps to sort of make small differentials through your own effort. You could change your life. You could make it, instead of being fantastically good, you could make it super fantastically good. Something like that. That could be sort of the span, the stakes that, that, that we fight for.

B (74:19)

If we could create AI, whether in a robot or whether in a simulation that was indistinguishable from a human, do you think people would care whether they're interacting with a human or they're interacting with a simulated human that probably start

A (74:40)

caring less about it as they get more experience with these interactions would be my guess. I think one important question for me would be whether this robot is conscious, whether it's not just externally hard to distinguish from a human, but if it has the same inner psychological life as a human has, in which case I think it would be a moral patient. Like, it wouldn't just be a thing that you can throw in the trash if you no longer want, would be a thing who has welfare interests of its own. It matters how life goes for it, not Just like whether it makes humans happy, but it would be like a member of this population that we would want to have a good future. And I think, in fact, this maybe is a little bit of a tangent now, but like the general ethics of digital minds is going to, I think, become increasingly important. It is like even the AIs we currently have, we are very soon, if not already at the point where it's not really. We can't be very confident that they don't have some rudimentary forms of moral status or awareness or subjective capacity for feeling pleasure or pain and stuff. So it will increasingly also become important how well the future goes for these AI minds themselves.

B (76:01)

If you're saying it's already hard for us to tell for sure one way or the other, what metric will we use to know when this is something that we can just turn on or off, make it do what we want? How we know when we cross a line into something that has moral rights?

A (76:17)

Well, it's a difficult question. I think there needs to be more philosophical and scientific work on that. I'm involved with some research groups trying to do this. But at least we can say something like, if at one extreme you had an AI that was exactly functionally identical to a human that had, that lived for 80 years, that had like a human like body, human like memories, that had this artificial brain structured very much like a biological brain, I think in that case it would be a very strong moral case, that we should treat it as a moral subject as well, that it would be wrong to mistreat it and be cruel to it, et cetera. But I think something far short of that. I think just as most of us would agree that at least some animals have various degrees of moral status, it's wrong to be cruel to a dog or to mistreat a chimpanzee if you're a medical researcher. Even just in a mouse, if you want to perform an experiment, you have to anesthetize it before you do surgery on a mouse, because we think it is capable of experiencing suffering. So if even a mouse has plausible claim to sentience and at least some simple form of moral status, then AI systems that are roughly equivalent to a mouse in their sort of behavioral repertoire, I think would also be prima facie candidates for moral status. But it does get more complicated because in some cases AIs can achieve similar behavioral output using very different internal mechanisms. And so then you need to think more carefully about what exactly is it that we think matters morally. These things can kind of come apart More with humans and animals, you have a cluster of things that all come together like animals. They can sort of. Yes, they have a brain and then they can squeak and they have eyes and they have like a body that. They're like a whole bunch of things that you don't need to have all of those attributes together with AI so you need to think more carefully about which ones of them are actually the weight carrying elements.

B (78:35)

Do you think it would be morally acceptable to create a simulation where, let's say the AI is conscious? So is it morally acceptable to create a simulation where a young girl dies of leukemia?

A (78:53)

I think we should avoid doing that. I don't really feel in a position to sort of offer some overall verdict on human history. All of the good things and the bad things and how to weigh that up. It feels more like a judgment that we are too sort of limited to really even understand what, what, what is on the scales there.

B (79:25)

It feels like. I'm not entirely sure I understood that answer in the context of the question. So what does it have to do with human history?

A (79:37)

Well, I was imagining that where you would go with that would be to say, well, if we look at all of human history, it has not just one girl dying from leukemia, but a whole bunch of other horrible things as well.

B (79:52)

Yeah, there's two paths before us that I'm, I'm interested in. So path number one is where you're going, which is, does that make you think any differently about whether this is a simulation? Which is something that we haven't talked a lot about, but you're obviously very famous for putting forward the simulation hypothesis, which is that what you're living in right now could be a simulation and in fact mathematically probably is, but if it is, it seems pretty immoral by our own standards. If all of the. I mean, I guess it comes down to the question of can we say that the suffering is intentional and is that where the moral break is? Or do we just say a set of rules, let's call them physics, are put in place and then what, what comes, comes. And maybe it couldn't be anticipated. And yes, unfortunately, suffering is going to be one of the things that may or may not have been anticipated.

A (80:43)

We don't know. First of all, supposing we are in the simulation, what exactly the motives are for having created the simulation or what the alternatives would be if these hypothetical simulators, we know so little about their world or the choices that they face or about them, that it's hard to really come in strong with a Clear. We know exactly how you should do things. We've never even met them. We have never heard. I think like. So I feel a little. It would be kind of hubris to sort of come in there loudly and bang our chest and sort of tell everybody what's what and how they have like we are these like idiots running around here having no clue about what's going on in the universe. So that's the first thing. Also, I guess, at least theoretically there seems to be some possibilities in a simulation of avoiding or ameliorating or remedying various kinds of ills that might occur in a simulation. So there might be partial simulations, there might be characters in a simulation that get continued after the simulation. And you could imagine models in which people opt in to being in a simulation. There's like a whole big space of possibilities, some, some of which I'm sure would be like morally very bad, but also many others that it's harder to evaluate the morality of.

B (82:17)

Let's take a morality question that's closer to home. How would you feel if an AI could write a Nick Bostrom book as well as you can?

A (82:24)

Well, it would I guess save me a lot of time and effort then.

B (82:28)

But imagine that somebody puts it under a different name, Tim Brostrom and wow, it's shockingly like your last book.

A (82:39)

If they had that level of AI, I'm not sure why they would constrain themselves to writing my kind of flawed human outputs. They presumably would be able to do something better and more. It seems harder to write a book that's exactly the kind of book somebody else would write than to write something better. Because like I would have to. Yeah, so it seems like an AI that could for each person write a book in their. That person's style that would be as good as what that person could write. I think an AI that could do that could probably also write a better book than any of these people could write.

B (83:29)

If you could request the AI not be trained on your works, would you ask it to not be trained on your works?

A (83:35)

Yeah, it's a hard, hard, hard to, to, to really to know these. I mean I think if it's. They are. They're out there on the Internet. So I don't know. I mean it would be nice maybe to. I mean one is always aware of like giving up rights in perpetuity forever for any purpose, including purposes we have not yet imagined. Like it just seems like a big license to grant. So like on just being conservative with not kind of giving up big chunks of possibility Space without knowing what's in there. I think one might not want to do that. Like if it were one specific AI for one specific case, it would be easier to evaluate it. Like limited downside, it's going to be used for this thing. Okay, that. But like if you give it up for any use, for whatever purpose, in perpetuity, what they have on these contracts, like throughout the universe, using known and unknown new media of communication. I don't know if you have signed one of these for these firms. They, they. Sometimes your mind boggles at what kind of thing you're actually putting your signature to when you're signing one of these.

B (84:58)

Yeah, they are, needless to say, very expensive.

A (85:01)

Like you scroll through all of these updated terms of services. Like you click yes, I'm sure in one of these updated there's like something and you hereby sell your soul to the devil. Or you're like if somebody threw that in there, like this is like, yeah, I feel a little bit like it's weird that we currently are responsible. Each one of us have this kind of dictatorial power over an entire human life. So like each of us is like our own life, right? We have complete like. Which is kind of weird if you think about it being. Having complete dictatorial power over an entire human being for their entire life. That's like, like a big responsibility. Are we really capable of shouldering that responsibility? I feel sometimes maybe not. It's just that there is no alternative. Like there is no kind of other set of people you could trust to do a better job than each person themselves. But you do feel if somebody screws up their life, they are like 20 years old, they get into bad company, they make some bad decision and then either they're in prison for the rest of their life or they're in a car accident and are maimed for the rest of their life, or they're some little dumb decision they make early on have these decades long consequences. I feel sometimes that there is a disproportion. Maybe our choices should have consequences, but a little bad decision that you suffer for decades for. It feels. So I worry if all of us are slightly. It's a little bit like we were maybe 14 year olds whose parents died and we had to fend for ourselves in the world like people had to do before social welfare services existed. So yeah, maybe they can kind of do it, but it seems a little like they're not really fully ready to take full responsibility. And I feel maybe we're all a little bit like that

B (86:58)

currently Given that you made me think of Ilya Sutskever, what do you think he saw that? I read it as scared him enough to leave OpenAI and start a new company called Safe Super Intelligence. What do you think he's worried about that makes safety the number one thing in his mind.

A (87:20)

Fortunately, they haven't released any details. And it's probably better if AI Labs have some form of infosec information security so that not all advances are immediately distributed all over the world. Because if at some point there is like some actually dangerous capability advance, it might be better to have the option at that time, you know, to decide whether it should be disseminated rather than sort of forfeit that opportunity to steer, I don't know, like, probably just like some possible path, technological path forward that seemed to, you know, maybe be able to scale more than the other path, and that seemed harder maybe from a alignment point of view, to get to work or something like that. I don't think it was like any currently existing system, but more maybe a research path or something like that.

B (88:13)

And what do you think about alignment? I've heard you say that we need to give AI our values. My pitch would be that we need to give AI a completely different set of values where it does not value progress, for instance, because if it's constantly trying to progress and ever sees us as problematic in that we've got trouble. And because humans kill so many humans, I definitely don't want them to have our values.

A (88:39)

Depends a little bit on whether you are willing to sort of differentiate between more superficial values and deeper values. So we have many things we value because we think they are associated with various kinds of consequences. So maybe you value exercise because you think it will make you healthy and strong and, you know, successful. But if. If you learned that you were actually wrong about these things, like, in fact, for you, the way your particular body works, exercise actually will harm you and, like, make you die early because some valve will break when you put too much pressure on it. Like some. That, like, if you learned that, then suddenly you would no longer value exercise. Like, in fact, you would value not doing exercise in that scenario, right? Because it would kill you. And so, so there you discover exercise is not really your basic value. It's like what you actually value is perhaps more like health, happiness, strength, or something like that. So then maybe you could ask the same question for each of those. And then it's only when you dig down to your sort of ultimate values that you would have at least a candidate. If you think we should Align AI to human values. Maybe that's what you would align it to, rather than some of these kind of instrumental values. So that's one observation. Another is with human conflict, which is the source of a lot of misery. There's a kind of. Obviously, if you align AI with one set of humans and they are at war with another set of humans, then it might not be good for the set of humans that they are at war with if they just get more powerful tools to defeat their enemy. So if you care about all of humanity, presumably you would want the AI to be aligned either with all of humanity or with some entity that actually cared altruistically about all of humanity, as opposed to just some random little bit of humanity. But then it is also possible to try not to align it in that way and to have AIs with more limited goals that deliberately diverge from human goals, because maybe you just feel more confident that you could actually instill those goals. And with something like not desiring growth, I think was your example, there are problems that crop up there that even if you don't sort of explicitly instill a value of growth, it might just emerge as one of these instrumental sub goals. So it wants something else. This is kind of the one of the points of this paperclip maximizing AI example. So you have an AI that doesn't want growth. It wants to make paperclips, as many paperclips as possible. This is like the goal you give it. But then it turns out that by, say, growing, by getting more resources, it can make more paper clips. So then you have an AI that actually, for instrumental reasons, want to grow, wants to avoid being shot off, wants to increase its intelligence, accumulate more power, not because it cares about or values those things in their own right, but just because it kind of calculates that I can actually make more paperclips if I have more power and more resources. So some of these goals that you might want an AI not to have would sort of, unless you're really careful, just be a natural side effect of almost any other goal that you put into the AI. It's these kind of convergent, instrumental reasons. And that's one of the reasons why you might worry about scaling up these AI systems developing very powerful agents. The catastrophe scenario doesn't depend on somebody deliberately putting in a really evil goal in there. It could just have some random arbitrary goal, and then bad things are done in the name of that goal that emerge as sub goals.

B (92:39)

Nick Bostrom this is an incredibly pivotal moment in human history. And I have thoroughly enjoyed spending this time with you. Where can people follow you and hear more about your philosophies?

A (92:50)

The easiest way is just to go to nickbostrom.com it's my homepage. There's like videos linked and a bunch of writings. I'm not really active on social media, so yeah, just go to my homepage.

B (93:03)

We'll have to get a bot out there publishing in your voice. We'd all be better for it. Thank you man so much for joining me today and everybody at home. If you haven't already, be sure to subscribe. And until next time my friends, subscribe. Be legendary. Take care.

A (93:16)

Peace.

B (93:18)

Let's talk about a pattern that is guaranteed to be killing your progress. You know what you need to do? You need consistent nutrition. We all do. You need vitamins, probiotics, greens. We all know that we should be doing more of it. When your morning gets chaotic, you skip it. When you travel, you skip it. When your routine breaks, everything tends to break. And that inconsistency compounds against you every single day. AG1 is designed to solve the execution problem. One scoop 8 ounces of water and you're done. You're getting 75 plus ingredients, vitamins and minerals, pre and probiotics, nutrient dense superfoods. Everything that used to require six, seven different supplements and perfect planning now happens in one drink that takes about 30 seconds to make. Right now, AG1 is giving you $87 worth of free gifts with your first subscription. You get a welcome kit, travel packs, vitamin D3 plus K2 and flavor samples. Click the link in the show notes or visit drinkag1.comimpact to claim this offer.

A (94:22)

Some Follow the Noise Bloomberg follows the money. Whether it's the funds fueling AI or crypto's trillion dollar swings, there's a money

B (94:31)

side to every story.

A (94:32)

Get the money side of the story. Subscribe now@bloomberg.com.