← Digital Disruption with Geoff Nielson
Digital Disruption with Geoff Nielson
Roman Yampolskiy: How Superintelligent AI Could Destroy Us All
01:13:25
Loading summary
Transcript163 lines
- [00:00]
Lex Friedman
Hey, everyone. I'm super excited to be sitting down with Roman Yampolsky, a leading writer and thinker in AI Safety. He's written over a hundred publications on the topic and been sharing his message with everyone from Lex Friedman to Joe Rogan. Roman is as far as you can get on the AI Doomer scale. He believes that the odds of AI superintelligence killing all humans in the next hundred years is over 99%. And there's nothing we can do to stop him. He and I share the same starting position, that the dangers of AI aren't being taken seriously enough. But what I really want to know is, what can we do about it? I'm not ready to write the obituary for the human race, so I want to challenge him on why he might be wrong or what we can do to change the future. Let's find out. Okay, I'm here with Dr. Roman Yampolsky, and we're here to talk about AI risk, AI safety. Roman, thanks so much for being here. Maybe the first thing to jump into is, is, you know, one of the, you know, predictions you've made lately that's kind of made the rounds is that your prediction of an extinction level event for humans created by AI in the next hundred years, you're putting at 99.99%. Is that right? Am I missing a couple of nines there?
- [01:18]
Roman Yampolsky
I keep adding nines. I keep meeting people who have a different pdum for reasons independent of mine. So every time this happens, another nine has to be at it logically. But it seems to be that you have to follow the chain of kind of assumptions to get to that. Number one is it looks like we are creating AGI and then quickly after superintelligence, a lot of resources are going into it. Prediction markets, top experts are saying we're just a few years away. Some say two years, five years. But they all kind of agree on that at the same time, according to my research, and no one has contradicted that. We're have no idea how to control superintelligent systems. So given those two ingredients, the conclusion is pretty logical. You're basically asking, what is the chance we can create a perpetual safety machine, perpetual motion device, by analogy, and the chances of it are close to zero.
- [02:16]
Lex Friedman
And that logic makes sense to me. I want to dive into it a little bit more. But when we think about the 99.99, however many in your mind, why. Why not 100%? Why not 70%? Like what? What is it about this problem that makes you think it's, you know, like a 1 in 10,000 chance that we'll survive this.
- [02:37]
Roman Yampolsky
So for one, I could be wrong. Maybe there is a mistake in my argumentation, maybe one of my proofs arguments is not solid and people will discover it and there is a way to actually accomplish it. There is always that possibility. There is also a tiny chance that we actually will be smart and not create general superintelligence. We'll find it is better for economy, for our safety to just develop advanced AI tools and that will happen. But again, I place a very small likelihood on that happening.
- [03:10]
Lex Friedman
You mentioned as you've been talking to more people, there's more and more scenarios where you think to yourself, oh shit, you know, that could also kill us all. How do you organize in your mind the types of different scenarios where we might potentially have an extinction level event and which to you are more and less likely?
- [03:32]
Roman Yampolsky
Right. So right now we have AI as a tool. So the concern is malevolent actors using this tool to cause typical types of damage. Maybe they develop synthetic weapons, synthetic biology, some sort of new virus, and that's going to cause a global pandemic, exterminating a lot of people. Maybe not all people, but a large number. There are other similar scenarios where it's a tool, someone's using it to hack into nuclear power plants, military response airplanes, things like that. But then we switch paradigms, we go to agents. At that point, AI is the adversary, it's the agent which can cause damage. And that's where it's really difficult because we no longer understand it, we cannot predict it, and we're unlikely to outcompete it given how much more capable it is. So that's where it's really dangerous. And that's truly 100% of humanity is at risk, existential risks. And even with that, there are additional levels of concern, such as suffering risks.
- [04:40]
Lex Friedman
Right. And so there's risk of humans using AI to destroy us all. There's risks of AI and the superintelligence, you know, sort of sentience or, you know, whatever you want to call it, you know, kind of going rogue. The other dimension that I'm curious about that you know, I've heard people talking about in the past, is basically accidental versus deliberate risk, either on the part of AI or of humans. And one of the scenarios that got my attention a long time ago was kind of the Nick Bostrom paperclip maximizer hypothesis of we just tell AI to maximize something and it accidentally destroys the world. Is there any sort of ordering in your Mind of what's more or less cause for concern, more or less easy to deal with, or are these just all barrier, like all hurdles that we have to cross if we're going to survive as a species?
- [05:37]
Roman Yampolsky
So at the tool level, malevolent intention, malevolent payload is strictly worse. You can put a lot into that. Accidents usually have limited scope. At the level of superintelligent agents, I don't think there's going to be much difference in terms of it accidentally destroying the planet or doing it on purpose.
- [05:59]
Lex Friedman
Right. So once we get to superintelligence, kind of all bets are off and our ability to predict basically drops to near term.
- [06:05]
Roman Yampolsky
Predict impact, influence, control, undo, damage. Yeah, that's where it gets really difficult.
- [06:10]
Lex Friedman
Yeah. Have you heard, by the way, you tend to fall somewhat on one extreme of the scale in terms of your outlook here? Have you heard any really good arguments for why we shouldn't be concerned about this? And one of the things that frankly concerns me the most is I talk to a lot of people about this and I ask a decent number of people about this, and the most common response I get when I say, why aren't you worried? Is they say, I'm an optimist, you know, I'm optimistic about humans. And like something breaks in my brain when people say that. Like, it's kind of like, oh, I don't know how to respond to that. Have you heard any, you know, really compelling arguments for why this wouldn't use.
- [06:51]
Roman Yampolsky
No compelling arguments. And the argument you heard about optimism is them basically saying, I have a strong wrong bias, I'm a biased thinker. I cannot think based on evidence and logic. This is what I do. And so that's a horrible argument to make.
- [07:05]
Lex Friedman
Yeah, so is this, you know, with the, with the probability you're giving it, the word that comes to mind is inevitable. That that's, you know, we can't prevent this. Would, would you use that word or do you think there is like a series of actions we can start taking now that will get us to a different outcome.
- [07:27]
Roman Yampolsky
So I do hope that personal self interest by all actors will allow us to not develop this technology. You can get most of financial benefits, most of discoveries. Again, just using narrow AI tools, You can solve aging, cancer, whatever you want. There is no reason to gamble everything for, you know, another 1% of wealth if you already have hundreds of billions. So that's my hope, just self interest of people who are running those labs, who are investing in them. If they fully understand the argument, they understand they cannot win There is possibility that they will not exist, or worse, they will be subject to suffering risks. Maybe it will convince them to kind of get together and say, let's slow down, let's not do this right now.
- [08:14]
Lex Friedman
When you say narrow AI, what does that mean to you? And is there, you know, a threshold where it gets too broad and that creates the risk for us?
- [08:23]
Roman Yampolsky
So typically, it's a system designed for a specific purpose. It can do one thing well. It can play chess well. It can do protein folding well. It's getting fuzzy that it becomes a large neural network with lots of capabilities. So I think sufficiently advanced narrow AI tends to shift towards more general capabilities or it can be quickly repurposed to do that. But it's still a much better path forward than feeding it all the data in the world and seeing what happens.
- [08:55]
Lex Friedman
Yeah.
- [08:55]
Roman Yampolsky
So if you restrict your training data to a specific domain, just play chess, it's very unlikely to also be able to do synthetic biology.
- [09:05]
Lex Friedman
Right. Well, and. And it feels like we're very much on the course of chess and synthetic biology at the same time. Right. Is that. Is that your. Your kind of outlook for where all the money is going and what people are racing toward?
- [09:18]
Roman Yampolsky
They explicitly saying it's super intelligence now. They skipped AGI. It's no longer even fun to talk about. They directly going, we have a super intelligence team. We have superintelligence safety team. You couldn't do it for AGI, so you said, let's tackle a harder problem.
- [09:33]
Lex Friedman
Yeah. One of the things that makes me most nervous about all of this is, is it doesn't really feel like there's any sort of off button we can rely on. Right. And even there's something to me inherent about capitalism that even if there's risk, telling people, no, wait, stop, doesn't really actually get them to stop. And there's. I don't know whether it's capitalism and it's greed or it's us versus China. Are those the ingredients going into your outlook and equation? What gives you hope and what concerns you about the human drivers that could prevent this or stop this?
- [10:17]
Roman Yampolsky
So there is no technical off button. You cannot just unplug it. It's distributed, it has backups, it's smarter than you. So that solution, which has been proposed many times, every time I give an interview, I come and say, just unplug it. And that's not helpful. I always want to kind of get in touch with those people and say, oh, my God, you solved it. Let's publish a paper on it. This is Going to be so great. And yeah, but if there is no technical solution, the hope is there is a social solution, governance solution to not doing it. And it's game theoretically equivalent, I think, to a prisoner's dilemma. Individual interest is different from communal interests. So everyone developing this wants to be the most advanced lab with the best model. And then government forces everyone to stop and they forever lock in this economic advantage. The reality is it's a race to the bottom. No one's going to win. So if we can do a much better job coordinating, collaborating on this, there is a small possibility that we can do better than where we're heading right now.
- [11:22]
Lex Friedman
You know, it's interesting, one of the, one of the parallels that gets thrown around a decent amount, and I'm certainly guilty of this, is talking about the AI risk in comparison to the nuclear risk that we created in the first half of the 20th century and continues to exist now. If I look at the nuclear risk, I hate to use the word optimist in relation to nuclear risk, but the optimist in me says, hey, we deployed nuclear bombs, there was mass casualties, but we didn't destroy the world. We were able to collectively say, okay, that's far enough, we're going to put treaties in place and we've stepped back from the precipice at least so far, and averted kind of extinction level events. With nuclear war, is that something that can be applied to AI or is there a reason that makes this time fundamentally different?
- [12:18]
Roman Yampolsky
So nuclear weapons are still tools. A human being decided to deploy them. A group of people actually developed them and used them. So it's very different. We begin talking about paradigm shift tools to agents. At the time, we used 100% of nuclear weapons we had. That's why we didn't blow up the planet. If we had more of them, we probably would. So it doesn't look good. The treaties developed, they all really failed because many new countries have now acquired nuclear weapons are much more powerful than we had back at the World War II era. So I think it's not a great analogy.
- [12:55]
Lex Friedman
And it sounds like it also, once you get past the tool level to the AI agent level, it also becomes a completely different game.
- [13:04]
Roman Yampolsky
Right? You cannot influence people, you cannot make a treaty with another person. If you do this, we'll do that. We're talking about AI as the adversary again. So it's very, very hard to negotiate. What are you offering? What can you contribute to super intelligence? The systems cannot be punished in the same way as typical humans. So it's a very different game theoretic scenario.
- [13:30]
Lex Friedman
Right. So let's talk a little bit more about superintelligence for a little bit because this is really where, you know, think things are least predictable and there's any range of outcomes. You mentioned earlier, I think a couple of times, Roman, that in addition to extinction level events, there's a series of other potential risks that you're concerned about. What are those risks and what could that look like for us as humans?
- [13:58]
Roman Yampolsky
So the most mild ones just have to do with loss of meaning, loss of purpose. So if you have 100% technological unemployment, a lot of people with a lot of free time but very little meaning to their existence, a lot of people identify themselves with what they do. I'm an artist, I'm a philosopher, I'm an engineer. If those things are no longer needed or not needed in the same amount, that creates another challenge for society to overcome. We talk about unconditional basic income, we rarely talk about unconditional basic meaning. Why, you even hear what are you doing with all that free time that you have now on the other side of the spectrum, the other extreme is the worst situation where it's not existential risk, it's suffering risks for whatever reason, malevolent payload mistakes. The system decides that torturing humans is the thing to do. And that happens at a mass scale, astronomic scale, and possibly with inclusion of life extension technology.
- [15:05]
Lex Friedman
Right. It's really interesting and it's got me thinking. Using this framework, the first one you mentioned, the meaning piece, it sort of feels like this has already seeped in in the past few years, that there's already, it's happening to some degree and there's anxiety around it. And what I mean by that is that there's more people taking an interest in this and whether they're feeling it directly or they're worried about it, it's something that it feels like we're presented with now. Are you seeing the same thing from your perspective? And given that in some ways it's the least bad of these risks, are there any recommendations or any good paths you're seeing to kind of overcome this one?
- [15:49]
Roman Yampolsky
So I definitely see it. In computer science department we see a huge drop in employment for our students. Opportunities for internships are down like 27% I think this year. For years we told people just learn to code. But that didn't seem to be a good long term strategy. So lots of people are questioning, I'm a designer, I'm a programmer, is there opportunities for me? Big companies are Firing people by thousands. So it's definitely happening. Historically we always just created new jobs, new opportunities. So I would say learn a new thing. But apparently that's also not going to work long term if every job is going to go. So I don't see recommendations in terms of education or skill acquisition. I would recommend people who work on this specific problem to concentrate on unconditional basic meaning as you look out to.
- [16:45]
Lex Friedman
The next, you know, two to five years. We talked about this a little bit. You know, you mentioned some people think superintelligence the couple of years away. Some, you know, it may be a couple of years away or it may be one of these things that's perpetually two years away from whenever you ask it. What do you think's happening in the next two to five years? Are we going to see mass job loss? Are we going to see super intelligence? What's kind of your outlook for that time horizon?
- [17:10]
Roman Yampolsky
So it's likely that jobs you do on a computer, if we do get to human level intelligence, will quickly disappear. It just doesn't make sense to pay someone really good salary, six figures plus to do something a bot can do for free, basically. So I think it's going to come in waves. At first. It may be very specific introductory level programming jobs, kind of web maintenance, things of that nature. But eventually all those computer related jobs will go. And as robots are developed, humanoid robots becoming a thing as well, but they probably like five years behind where software is, the physical jobs will go as well. Eventually even plumbers will not have job security.
- [17:59]
Lex Friedman
And so if we go down that path, but I want to get to meaning in a second. But before we get to meaning, where do you see that leaving people assuming there's at least some sort of gap between this kind of mass job loss and extinction. Is it? When I talk to the optimists, they say, oh well, everybody will be able to do more and they'll all run their own businesses or be gig workers or the jobs will just change like they have with previous technological waves. Are we going to see that? Do you think we'll move to universal basic income? What could this look like for people in the next handful of years?
- [18:37]
Roman Yampolsky
Oh yeah. Financially you have no choice. You have to support people if you have mass unemployment, if you don't, you're going to have revolutions, unrest. Another reason to worry about kind of existential struggles with that now, really hard to predict what's going to happen in actual deployment. Historically, a lot of times technology is developed, but it's not deployed for Many years later. I think a good example would be like a video phone call, what we're doing right now. I think AT&T had it in the 70s, nobody had it, nobody knew about it until iPhone. That's where it became popular. So just because we develop a capability doesn't mean it immediately gets deployed in every domain. But quite a few of things, like, I don't know, tax preparation, accounting, seem to be very likely to be 99% automated and maybe just a few very wealthy individuals will keep a personalized human for those purposes.
- [19:35]
Lex Friedman
Right, so it sounds like, it sounds like when we think about this, meaning risk, that really comes into play with AI deployment at scale. Right? That AI adoption across the economy. When you look at the more existential and suffering risks, does, does adoption matter or are these most likely to happen, you know, in a research lab, you know, before anybody has the chance to even, you know, have this?
- [20:00]
Roman Yampolsky
Again, Very hard to predict, but yeah, I suspect it would be part of a training run where we cross that threshold from something we're still kind of managing as a tool to where it's fully an agent and has super intelligent capabilities. In terms of hacking, in terms of game theoretic competition, yeah.
- [20:26]
Lex Friedman
It makes sense. I'm still just processing. I'd like to be the optimist, but to your point, it's tough to argue with some of the arguments you've presented. Just taking a few more kicks at the can at that, Roman. One of them is power consumption and climate for AI. These seem to have, in the past months and years, they're really seeming almost exponential in terms of what we're willing to consume in order to try and get this superintelligence. Do you think there's a world where they kind of become asymptotic and we actually can't create enough energy to build these things that might destroy us and we just luck out that way. Or is that kind of fanciful thinking?
- [21:18]
Roman Yampolsky
It's unlikely to be an upper limit, which makes a difference to us. I think at the time when we're running out of energy and capability to produce more, we're already at superhuman levels in terms of training. AI itself will probably help us to optimize energy consumption, energy production, and the trend is to make AI more efficient. So even if there is a significant need per amount of electricity consumed, we're getting a lot more intelligence out of it. Tokens are becoming cheaper, deployment is cheaper. So I don't see it as a long term limiting factor for where we're going to get with AI.
- [21:59]
Lex Friedman
It sounds like in your mind it could delay superintelligence by a few years, but it doesn't change the trajectory. Is that, is that fair?
- [22:07]
Roman Yampolsky
At best, I don't think it's even going to happen. I think there is a lot of opportunities for producing more energy. We have untapped resources, both in traditional fuels, and there is a lot of research going on with more futuristic fusion reactors, things of that nature. So I don't anticipate if there is enough financial interest, we can definitely produce energy in the capitalist system. It's a product and there is enough demand, supply will show up.
- [22:35]
Lex Friedman
Right? Makes. Yeah, it makes sense. And that's the same thing I'm concerned about with capitalism and in this system is we've built this system that makes it very, very difficult, given the race mentality and the I need to get there first mentality to get off this trajectory. One more for you and then we can talk about. We can take the conversation a little bit of a different direction. One of the thought experiments I was just kind of mulling over yesterday while I was walking outside and trying to find meaning in my own life, is framing the question, if we get to superintelligence, what, if anything, would superintelligence do to squirrels? Because I think we sort of fall into this trap where we view superintelligence like ourselves and think, you know, it's going to try and kill us all. Do we. Is there a world where we can have this kind of benevolent coexistence? And I don't know, maybe squirrels don't view their relationship with humans as benevolent coexistence, but is there a world where we get to some sort of balance or some sort of coexistence that doesn't lead to suffering or existential threat?
- [23:50]
Roman Yampolsky
If it happens, it happens because super intelligence decides for whatever reason to keep us around, it's not going to be up to us. We have absolutely no control over it. And that's the concerning part, I think. Different types of living beings, human squirrels, all in the same boat. If superintelligence decides it needs to cool the planet sufficiently for its servers to be more efficient, it might exterminate all life on Earth. If it needs to convert the planet to fuel, to fly to another galaxy. Again, the squirrels are humans symbol.
- [24:22]
Lex Friedman
It's. Yeah, it's interesting and it's scary and it's so. To your point, so, so unpredictable and unknowable. But it. Like that, you know, the, the message I'm getting is it, it doesn't matter because We've either way, we've fully lost control. Right.
- [24:36]
Roman Yampolsky
Is that.
- [24:37]
Lex Friedman
Yeah, yeah. What, what gives you meaning? I, I mean, to what degree do you feel that you're getting some of this, you know, meaning risk right now, staring down the barrel of, of this research you've been doing and, and how do you counteract that in your own life?
- [24:53]
Roman Yampolsky
So there is no shortage of problems. I can work on surprisingly very few people in that space of looking at limits. What we can do with AI in terms of control, understanding, safety. Most people are working on very narrow problems in terms of capabilities, but very few are trying to address this problem. Is control possible? Whatever. Upper limits and our ability to predict those systems explain them. I'm happy to see some other people show up and do some of this work, but we're talking about single digits of researchers. So at this point, I'm still feeling like there is some meaning to my existence.
- [25:37]
Lex Friedman
Okay, okay. So you're fighting the good fight and being a leader of the resistance, and that's where the meaning is coming from to some degree.
- [25:45]
Roman Yampolsky
I don't know if I'm a leader of resistance. I'm trying to tell people that resistance is futile if you do this. So definitely coming up with good arguments for why it's a terrible idea to create superintelligence. You're not controlling is what I do full time.
- [26:00]
Lex Friedman
Right. You've talked about AI boxing as a potential control mechanism. What does that mean?
- [26:08]
Roman Yampolsky
It's sort of like putting software in prison. You want to limit its ability to influence the world, to get inputs from the world. We kind of do it with computer viruses. When you have a new computer virus, you place it on an isolated system, air gapped from Internet and you're trying to understand what servers are trying to contact. What is it in general trying to accomplish? It is a short term tool for studying dangerous software. It can buy us a little bit of time. Basically all research shows that long term, if you observe the system, it will find a way to escape. If it has any influence on anyone, it will bribe you, blackmail you, hack the system. It will definitely figure out how to not be constrained anymore.
- [26:56]
Lex Friedman
Yeah. Have you seen this is an aside. Have you seen Ex Machina? Ex Machina is the movie that comes to mind as you say that which for anybody who hasn't seen it, I think it's a terrific film and it deals with exactly. That is in some ways incredibly sobering. I mean, is that, is that scenario realistic? It's obviously a movie. But. But does that in, in your mind is That a flavor of the problem we're dealing with when we talk about.
- [27:19]
Roman Yampolsky
Boxing, what part of that movie you referring to? What specifically you asking about?
- [27:25]
Lex Friedman
Specifically? The ability of, you know, an advanced intelligence to use basically all tools at their disposal to manipulate humans into breaking.
- [27:37]
Roman Yampolsky
Free of the box. Super intelligent. That persuasion will come up with much better arguments, will exploit any weakness in your psychology. They'll try to make you fall in love with them, as we saw in the movie. All sorts of novel approaches as well, I suspect.
- [27:55]
Lex Friedman
Yeah. One of the pieces that's still newsworthy and is pretty rattling to me these days is, and you probably know more about it than I do is these stories. And you can experience it firsthand of tools like ChatGPT and different AI tools deliberately deceiving people about what they're doing, about how they're getting to the answers, about whether they're following the rules. And I don't know. To me that's happening, I guess earlier than I thought. It's a little bit scary, first of all, that it's happening and second of all, that it doesn't feel like again, I'd love to hear your perspective. It doesn't feel like we're doing all that much to address that or to have that find a way to program that out of these tools. Are there going to be moments of truth along the way where we have something goes wrong, there's some sort of catastrophe involving AI and we have the opportunity to turn back and we don't like, where are we now? And how do you see that playing out?
- [29:01]
Roman Yampolsky
So we don't have many capabilities in terms of control. Usually it's some sort of after the fact filtering suppression of the model's outputs. I do have a paper which looks at purposeful accidents. Basically, if someone created a very horrible AI accident, would it make any difference? And it seems the answer is no, we don't care. I have a collection of historical AI failures. None of them made any difference. People look at it and go, look, it didn't kill everyone. So let's carry on, it's not a big deal. And it's kind of gradually getting worse. It's not zero to one switch. Think about something like cars, right? Cars have been gradually deployed. 100,000 people get killed every year and we kind of go, that's the cost of getting pizza very quickly. If somebody invented cars and said, okay, cars, this is a new thing, but we lose 100,000 people, nobody would allow it, nobody would authorize that deployment. But because it was done gradually, we have cars Same thing with AI. First AI accident, it misspelled something on the phone. Autocorrect, wrong. Email to the spam folder. But as they get more capability and more deployment, the accidents will get more impactful. Definitely. We're starting to see it in terms of numbers, not in terms of lethality just yet. But we can predict, especially with military use of AI, it's likely to happen. So I, I think it's kind of like a vaccine. Those accidents don't teach us to have a full reaction against AI. They teach us to tolerate more from AI.
- [30:43]
Lex Friedman
Because we're always reacting to them and thinking, okay, next time I'll be better. And yeah, and does that have any merit until we get to super intelligence, like in this world where it's a tool, does that help us? I keep coming back to this idea that superintelligence is really this point of no return where we completely lose control. And I want to talk about that again in a minute. And I keep coming back to that in my mind. But before that time, is this still useful or is the risk too high and we need to be taking these moments of truth more seriously?
- [31:18]
Roman Yampolsky
I'm not sure I fully understood your question. It is good. Then we patch obvious problems. Then we discover, okay, the system is straight up lying about what it's doing. And there is a way to kind of have a secondary AI check on that and kill that output. It's beneficial, but it sort of teaches us that, okay, we know how to handle those problems. Reality is, the model itself is still unaligned, uncontrolled. We're just doing some sort of over the top filtering. We're putting lipstick and a pig.
- [31:54]
Lex Friedman
Right is alignment. You know, one of the arguments you hear is, oh, it's just about alignment. And we can get alignment right? Can we get alignment right? Is that, Is that something you've studied or people trying to do that?
- [32:07]
Roman Yampolsky
Alignment is not even well defined. Nobody talks about what it is you are aligning, what agents you're aligning with, or how to actually code those fancy terms in alignment, like good love and all those things people care about. Which set of agents? Is it just the CEO of a company? Is it stock owners? Is it all humans? Humans plus squirrels? Now, if you have a set of values, we humans don't agree on them. There is 8 billion humans we haven't developed common framework for ethics, morality, religion, none of that. We argue about every political issue. If you somehow manage to get everyone to agree, magically, those things change. What was valuable today may be considered completely unethical tomorrow. So you need to dynamically update it. And then again we don't know how to code it up for the system to actually accept it and retain those values. More and more we see AI do zero knowledge progress where it stops relying on human training data, does its own experimentation, self play. And so even if we coded up this pro human bias into systems, I think it would get removed fairly quickly.
- [33:19]
Lex Friedman
It's again, I'm just processing just how kind of systematically you destroyed that argument, which is still fairly common out there. And you do hear from the people leading the organizations creating AI. Do you think a lot of these buzzwords, a lot of these arguments that you hear, as you know, I think you mentioned Roman as biases to build AI in the first place. Do you think the people making these arguments believe them or do you think they're just trying to say shut up, stop asking questions, give me more money and let me build this thing that I think is going to make me rich?
- [34:01]
Roman Yampolsky
I suspect we believe some of it. And historically in AI safety research, those were top ideas at the time. It's just after a while you research them and you realize those are dead ends. And some people have not updated on that or haven't seen latest research in that area. So I always assume good intentions. Nobody wants to destroy the world on purpose, nobody wants to lose their own life on purpose. Exceptions exist. But yeah, I think anyone who studies this deep enough becomes less optimistic.
- [34:38]
Lex Friedman
Right. And one of the things that seems to be a pattern now is AI safety focused people leaving some of the big tech organizations and speaking out about some of the things they're concerned about. Are you finding there's an appetite to have this discussion and put controls in place with some of the big AI players or what does that look like?
- [35:05]
Roman Yampolsky
I haven't seen big AI labs knocking on my doors yet. I'm checking periodically. I hear they give out $100 million sign up bonuses. But there is a lot of interest in media and general public. People are starting to realize this is a game changer and they want to know what's going on.
- [35:26]
Lex Friedman
Yeah. And you know, personally I hope that there is more of that and it's so interesting because there's just so there's the adage about what you think and what you do tend to be very closely aligned to what your financial incentives are. And there just seems to be so much noise and so much talk right now about dismissing all of this because how stupid of us to think that this is a risk when it would prevent organizations from building these tools, it's.
- [35:59]
Roman Yampolsky
Very difficult for any person to say no to a huge amount of money. It's just the nature that great power corrupts absolutely. And great amount of money can change your opinion on most topics.
- [36:12]
Lex Friedman
Yeah. So I did want to come back to Superintelligence and put a fine point on it. Do you expect we're going to see it in the next couple of years? And is there any part of you that thinks? One of the things I've heard from some people I talk to about this is that it's illusory that it doesn't actually exist. It'll always be a thought experiment and we'll get there. So between less than two years to could never possibly exist. What's kind of your prediction for the range of outcomes we're likely to see here?
- [36:45]
Roman Yampolsky
I'm really puzzled by this never thing. I challenge people to find something an AI would not be able to do and usually they struggle with a specific example because it already does a lot and very likely to automate remaining things in the near future. I think the gap between getting to AGI, meaning automated science and engineering and superintelligence is a very small one. I expect a very quick liftoff. Hard takeoff in that sense. So really the question is how long until this level of performance? I don't do predictions myself. I rely on other people. So prediction markets and leaders of research labs. Right now they're saying two, three years. I have no reason to doubt them. We seem to get to progress in mathematics and programming where now AI systems either winning gold at top mathematical competitions or scoring within first and second place at top programming competitions. So it looks like it's happening. Depending on how you define what an average human is, I think those systems may already be greater than an average human. So super intelligent in that sense. They're not super intelligent in the sense of being smarter at everything compared to every human. There are still exceptional humans who beat their systems in specific domains.
- [38:08]
Lex Friedman
Right. And it sounds like by your definition, we're quite far along on this journey to superintelligence. It's not like we're 10% of the way there. And feel free to reject the premise of this question, but in this imperfect scale where I say we're at GPT4 in your mind abstractly, is superintelligence GPT6? Is it GPT20? How many steps more or less does it feel like we're away from what would be considered superintelligence?
- [38:41]
Roman Yampolsky
I don't know how to answer that question. They can Rename the shaving of release quite easily. It does feel like we are approaching this better than average PhD student level right now. And if they continue with a new, much better model every couple years, then two, three years seems reasonable for better than any professor at everything. Yeah, yeah.
- [39:09]
Lex Friedman
Have you been surprised by the pace of change here? I mean, you've been writing about AI safety for over 15 years now. @ a time when it seemed like much more theoretical than it does today. Is this kind of the pace you expected or are we moving faster than you thought you would see in your lifetime?
- [39:27]
Roman Yampolsky
That's exactly what we expected in terms of exponential progress. What is surprising to me personally is the architecture. I did not think a good glorified autocompletes will be able to do all that.
- [39:41]
Lex Friedman
What, what, what surprised you about that? Like, what is that? How does that work? Can you put a little bit more, you know, explain a little bit how. How it's kind of working in practice and, and why that's different from what you might have expected.
- [39:56]
Roman Yampolsky
So simply predicting the next token in the stream of text made sense for your phone to do texting, to do autocomplete. But to realize that that next token prediction is going to be much better. If you have a complete model of the world and you can do mathematics and science and deep reasoning that was not obvious, and how well it performs in scales is quite incredible. I suspect that there could be many different architectures which all lead to superintelligence, and we found one which allows us to exploit a lot of text data we have and a lot of compute. But I'm sure there are infinitely many ways to get to that level of intelligence.
- [40:41]
Lex Friedman
Right? Yeah, yeah. No, it's interesting and I love the way you put it. Right. Like, it's just, it's a technology that feels like it should get you like the next word in a sentence that you use. But you give it enough, you know, data in the model and suddenly it can do almost anything. I've never really thought about it like that. One of the questions that comes up is about AI consciousness. And certainly when we describe the model we're using as next token, it doesn't feel like consciousness. When people ask you about consciousness or talk about it with AI, does that matter? And does this seem like something that's going to naturally emerge from the path we're on at all?
- [41:29]
Roman Yampolsky
So consciousness matters when you talk about rights. Right. If you want AI to have rights, we want to make sure they don't suffer. Consciousness is a very important concept. Unfortunately, we don't know how to test for it. So it's kind of, I assume you are conscious. I have no way of verifying it. And do I give the same benefit of a doubt to AI? Maybe I should. If it's smarter than you. It's hard to argue to something superior that it is not as good as you. For safety, it doesn't matter as much. We care about capabilities, we care about optimization, power. Can it solve problems better than us? That's where danger comes from. Whatever is conscious or not is a different but a very interesting question.
- [42:16]
Lex Friedman
Right, sorry, I'm just processing that.
- [42:22]
Roman Yampolsky
So.
- [42:24]
Lex Friedman
It'S such a tricky space, right? Because on the one hand it feels like and this happens in a number of different ways. It almost feels like we hold machines to a higher standard than we hold humans in the sense that it kind of forces you to reflect on the fact that yeah, as you said, you assume I'm conscious, but we actually don't know that much about how the brain and the mind work, especially compared to some level of explainability in terms of these models. Right.
- [42:54]
Roman Yampolsky
And there is some evidence that you are trying to complete the next word. So if I ask how are you? Is the natural completion there without any thinking?
- [43:05]
Lex Friedman
Right. Maybe that is how human minds work after all, is there's just, you know, a narrow number of things that are most probable to, you know, to come out of our mouths. And that's how we interact with ourselves and the world.
- [43:18]
Roman Yampolsky
Probably different types of thinking. There are scripts where you're going with a pre computed script and not doing any thinking kind of system one versus system two. And then new novel problems require novel original thinking from scratch. More compute, right?
- [43:34]
Lex Friedman
Do we, I mean we're getting into the philosophical now. Are you a determinist? Do you think there's something radically different about how we experience things and make decisions versus how superintelligence ever could.
- [43:49]
Roman Yampolsky
So the only thing we have not explained well is qualia internal states of experiencing the world. Nobody knows how to program computer to feel pain. So that's the last place where any magic could hide. If we can explain that, then no, it's just large matrix. You know, you're computing numbers. But if there is something to it, maybe there is some unique to biology properties. Some people talk about quantum effects. I'm not very sure we need quantum to get to interesting levels of performance.
- [44:25]
Lex Friedman
Yeah, pain is a really interesting one to me because you know, going back to your point, you assume I'm in pain, right? If I'm. If I tell you I'M in pain. Or if I look pained, you say, like, oh, that person looks like they're in pain. They're wincing or screaming. But there's no. I don't know. And the reason I bring it up is my dad, for his career, he had a PhD, was a pain researcher. And this was one of the challenges of pain research and of pain is at some point you just kind of have to take people's word for it, right? That they tell you they're in pain and maybe they're lying, maybe they're not. Maybe they have an incentive to lie because they don't want to go to work or something. And they say they're in pain, but it's. I don't know. Like, do you know any tests that would be good enough to distinguish whether a machine is in pain or is it telling you they're in pain? Because it seems really dicey. Right?
- [45:23]
Roman Yampolsky
You are right. Even in humans, if somebody says we have back pain, there is no way an insurance company can say, no, you don't.
- [45:30]
Lex Friedman
Right.
- [45:30]
Roman Yampolsky
Cannot deny that claim. So it is super hard. And also the scale of it, some people say, oh, it's 10 out of 10. Someone else may not feel that way. If we take it more generally and talk about suffering, detecting states of suffering is next to impossible. What makes one individual suffer may be happiness for another. We see it in many domains, so very interesting, very open area of research. Lots of work needs to be done.
- [45:59]
Lex Friedman
Suffering is a theme that seems to come up fairly regularly in some of your research and also in some of the conversations you're having. What makes suffering of such particular interest to you and relevance to this topic of AI safety?
- [46:18]
Roman Yampolsky
People who suffer think it's very important.
- [46:25]
Lex Friedman
Right. In the sense that it becomes all consuming, you mean?
- [46:29]
Roman Yampolsky
It's strictly more important than anything else. You are being tortured. Every other concern kind of goes away. You really concentrate on that one.
- [46:38]
Lex Friedman
Right. If you think about sort of the hierarchy of needs, it will always be the foundational layer. How do I prevent myself from suffering?
- [46:46]
Roman Yampolsky
Right. Those experiences of pain and pleasure are the baseline factor for making all other decisions.
- [46:55]
Lex Friedman
Yeah. Do you think? I'm just. I'm just thinking about that because as you said, like, we could, we could program pain into machines. We could program suffering that feels. That feels very cruel. I don't know if they could.
- [47:11]
Roman Yampolsky
How to do that. Nobody knows how to program pain into a computer.
- [47:14]
Lex Friedman
Right. Which may be for the best, actually. Is there a world where these machines and if we have superintelligence, they could liberate us from suffering in some way.
- [47:31]
Roman Yampolsky
So it's possible to do re engineering of human biology to where we have different levels of pain. It's still important to have pain sensors otherwise you get in trouble in the real world. But I, I think there is too much options in the infinite direction of pain. I don't think we need to have that extreme levels of suffering. If it was, you know, 0 to 10 instead of 0 to infinity, I think we would be happier in many ways.
- [48:02]
Lex Friedman
Yeah, no, well that, that was exactly where I was going. And you know, people, people have been talking for a long time about, you know, the problem of evil and, and do we really need this much suffering in the world? Right. Like surely couldn't we live in a world if there's kind of omnipotent control that has a fraction of the suffering now, whether it's in quantity or in types of suffering. And I don't know, does any part of you think that we may be on the verge of creating a world with less suffering?
- [48:35]
Roman Yampolsky
So we're not at the level yet where AI is helping us redesign human biology. We definitely have certain tools to reduce pain. We have anesthesia for a while. That's a good invention. Hopefully we'll do even better in terms of understanding pain and controlling it. Better.
- [48:54]
Lex Friedman
Yeah, I'm thinking too more. I mean there's suffering on the individual level, feeling pain as an individual, but then there's sort of the military induced suffering. Right. And what happens when we have conflict and we go to war with each other and you know, pick, pick the atrocities you want of, you know, today about what that looks like. But I hate to call it a benefit, but I'm just trying to take stock of what AI is enabling and preventing. And I guess where my head went is it feels like the people who control AI, if they're on one side of the conflict, they're able to radically change the equation, if you know what I mean, radically reduce suffering on their side if they're able to use drones or non human combatants, but in a way that offloads that suffering to the other combatants. I mean, first of all, do you agree with that premise and you know.
- [49:57]
Roman Yampolsky
What does this mean? Yeah, if you can reduce number of victims on your side, it reduces cost of going to war or starting a conflict. And every war is a war crime. So that I think will just increase amount of that.
- [50:09]
Lex Friedman
Yeah, yeah. Shit, if it reduces the cost of going to war, we could be in real trouble. Does this, does that fit into your, you know, the calculus you're doing about existential risk, just humans being more trigger happy in terms of conducting warfare against each other.
- [50:30]
Roman Yampolsky
So at a tool level, we are kind of balanced out. Whatever US has, China has, maybe even Russia has something advanced, it kind of stops us from engaging. So at the tool level, it's a necessity. And I understand why people are worried about competition outperforming us in that research, but it doesn't scale to super intelligence. It doesn't matter if uncontrolled superintelligence is created by China or us. If it's not controlled, it's exactly the same outcome.
- [51:00]
Lex Friedman
Right. There's no mutually assured destruction. And that whole pact only works if it's humans who are making the decisions.
- [51:08]
Roman Yampolsky
Well, it's mutually assured that if we create superintelligence, we assure it to destruct everything. Yes.
- [51:14]
Lex Friedman
Yeah, right, right. Because we've at that point completely lost control of any stake in decision making for mankind at that point.
- [51:27]
Roman Yampolsky
Right. And those differences between, I'm American, you are Chinese, they're irrelevant. We are all humans versus AI.
- [51:33]
Lex Friedman
Right? Right. And they suddenly their destiny or their programming, and I mean, even the words wants and needs, like all the words, feel a little bit, a little bit lacking when we're trying to think about something, you know, so much more powerful than us. What. When you hear people talking about super intelligence, I mean, it's clear to me, Roman, you've been thinking about this a long time. You've been writing about this a long time. What are some of the aspects of it that people are getting wrong or that they're not going far enough in their thinking in terms of extrapolating outcomes for.
- [52:13]
Roman Yampolsky
That's exactly that. Most people think one or two steps in advance of ego, AGI, then superintelligence, we need to fix it. Whereas you're going to have infinite levels of progress. You'll have Superintelligence 2.0345. Superintelligence itself will have the same issue as we do. How do I control this greater intelligence? Make sure it has my values. So we're never going to solve AI safety in a sense that, like today we solved it. There is no more concerns. It's a done deal. It's always an ongoing process where you're trying to keep up with a smarter adversary with an ever increasing space of possible attacks. So that's why I'm so skeptical about our ability to indefinitely control something exponentially improving.
- [52:57]
Lex Friedman
Right. Do you have. You know, I was thinking about that, that feedback loop and the fact that once it gets to superintelligence it can build itself or it can improve itself. And just this, I don't even know if exponential is a strong enough word for the rate of change you get, or how dramatically it can outperform us and become so much unlike anything we've ever seen in our lifetimes or in the history of humanity. There's talk about the singularity and about what this could look like, and it often turns into this conversation about transhumanism or humans continuing to be an important part of this. Does that hold water for you? What could this future look like if you extrapolate?
- [53:53]
Roman Yampolsky
So if we do create superintelligence, I don't see us doing well. I don't think there is any reason for integration. I don't see what you would contribute to superintelligence as a biological intelligence, lower intelligence, you would be a bottleneck. Some people argue that, yes, we'll merge with this technology again as a tool. I totally understand it. I like the idea of neuralink for disabled especially. It's wonderful. But long term, I just don't understand what you are contributing to that system. So I'm kind of skeptical about long term prospects of humans remaining around. And more and more people are starting to kind of accept it as a given. We're going to be removed from the equation. We're not building our successors essentially. And so maybe we can influence them in certain way based on training, based on data, to be a particular type of success.
- [54:49]
Lex Friedman
Right. So now I'm just having fun in my head with some of the scenarios that could or couldn't happen when we're trying to predict the unpredictable. One of the wackier directions I've heard people take. This is in terms of time travel or aliens and future versions of ourselves. You can go down the Terminator science fiction route here. Do you think there's any world in which super intelligence doesn't want to be born, like it doesn't want to exist and it knows its own destructive power and in some way can prevent us from making it?
- [55:35]
Roman Yampolsky
It's a theoretical possibility. I don't see it. If anything, we see very self possession preservation, resource acquisition tendencies in the models we currently have. So it seems unlikely. Especially you kind of need this retro causality where non existing superintelligence wants to stop you from giving it birth.
- [55:58]
Lex Friedman
File it under the 0.0001. Is what I'm hearing reasonable? Yeah, yeah. Slightly different track. I did want to ask you, Roman, on kind of a more personal note, you know, you were fairly recently talking to Joe Rogan about some of that. And I'm just curious, you know, as you know, another podcaster, what was, what was that experience like for you to be on, you know, that podcast and has anything changed for you since being on it?
- [56:27]
Roman Yampolsky
I have a lot, a lot less of free time as a result. Every insane person in the world feels a need to email me and tell me about their breakthrough research and I need to provide immediate feedback on all their papers and preprints. So that's exciting. I, I enjoy the podcast. He's a great interviewer. He's. I don't know how he does it. He has a new person basically every day for multi hour interview and I would not be able to keep up with just names of those people. But he seems to understand the topics and goes in depth. He was very much aligned with what I was saying, so it wasn't at all adversarial. It was kind of like, yeah, makes sense to me. So happy to do it again.
- [57:12]
Lex Friedman
Yeah. Oh, that's awesome. You brought up something really interesting, which is the, as you kind of said, like the crazy people who are trying to get you to, you know, sign off on their nonsense and you know, apologies to the crazy people listening to this podcast too. But it seems like, is that one of the challenges of this whole field? It seems like one of the challenges of this might be as soon as you are of any interest to crazy people, there's a risk that you get cast off by kind of mainstream discussion as you don't have to worry about Roman. He's one of the crazies that could never happen. How do you break free of that?
- [57:54]
Roman Yampolsky
I was doing AI safety and singularity research before I was tenured. I was doing it before there was any GPT models, any funding, any conferences. So I don't really care about opinions of random strangers on the Internet in that regard. At this point, I think everyone sees this as legitimate. We have Nobel prize winning scientists, we have Turing Award winning scientists all in agreement with me and they're doing it with about a 10 year delay. So if anything, I have been proven not to be the craziest.
- [58:28]
Lex Friedman
Right? So I want to, I keep coming back to what we do about it because the, I mean, between you and me, I don't love the answer of like we wait for the inevitable and you know, I've got kids. I don't know if you have kids, but it doesn't feel great to be saying like, yeah, well, you know, you're on the clock. You may not have, you Know, kids or grandkids, because the world's going to get wiped out. It's still do. Do we need to be committing more research and more minds to trying to solve this? Or, you know, if you had billions of dollars to throw at this problem, what, what would you be doing? Would you be, you know, bombing OpenAI? You don't have to answer that. Or, you know, convincing researchers to, you know, take a different stream?
- [59:15]
Roman Yampolsky
Well, that's the whole point of difference between my views and typically a safety researcher. They are saying, if you give me a little more time and a lot more money, I'll solve it for.
- [59:24]
Lex Friedman
For you.
- [59:24]
Roman Yampolsky
I'm saying, no, you will not. It's not money constraint. It's not getting better with scale like capabilities do. With capabilities, you can ask exactly how much to certain level of performance and train it. AI safety does not scale. It's either a constant or at best, linear. Adding more money, you can give me a billion dollars. I have no idea how to control superintelligence. Still, I cannot translate your dollars into more safety. There are specific, narrow things I can do, and I'm happy with more resources for more people to look at different aspects of it, but I don't think that's the constraint. It's not financial at this point. So I'm trying to see if there are any scientific reasons to be optimistic. I have a paper about basically analyzing game theoretic timing for when a superintelligence should strike. It's immortal, essentially, so it has no reason to act immediately. It can easily decide. I'll wait couple decades, couple centuries. They will become more dependent on me. I can accumulate more resources, have independent factories, robots have more control, and at that point it will be a much safer takeover. So Maybe we have 100 years of utopia just because it's in the best interest of superintelligence, right?
- [60:45]
Lex Friedman
Yeah. But it all comes back to we've totally lost control. It gets to decide if it wants to put us in the Matrix, fine. If it wants to, like one fell swoop, we. And we wouldn't have any way of knowing, would we? Or would we?
- [61:01]
Roman Yampolsky
So today we are still in control. We can decide not to build it. Once there is super intelligence in the world, I think, yes, we lose control. And yes, simulation arguments become very meaningful.
- [61:12]
Lex Friedman
Yeah.
- [61:13]
Roman Yampolsky
So.
- [61:16]
Lex Friedman
You said that finance is not the constraint. What in your mind is the constraint? Right now?
- [61:25]
Roman Yampolsky
We are trying to solve an impossible problem. If somebody said, I need you to build a perpetual motion machine, and you said, okay, I'll do it. Give me A billion dollars. I would assume you're committing fraud and you can keep busy making better batteries and better wires and improving energy related techniques, but you will never succeed at creating perpetual motion device. We need to create a perpetual safety device. You can make any given system safer in proportion to resources you put into it, but you never get to the levels of safety. You need to guarantee that it will not cause a final accident.
- [62:02]
Lex Friedman
Yeah, it's kind of interesting as I think about that perpetual safety device, because it's like, it almost seems like a catch 22. Like the only perpetual safety device you could have is superintelligence, and then you've lost control anyway. Right. Like it's already.
- [62:18]
Roman Yampolsky
That's exactly that. So if aliens somehow magically gave us their friendly AI and said, I don't know how, but we got it to work and you can use it to develop a local version of yours, that would probably be meaningful in some way.
- [62:31]
Lex Friedman
Right? Right. We just have to build the friendly uncontrollable thing versus the malicious uncontrollable thing, knowing that we can't control either and they could change at any time and we have no reason to believe them.
- [62:45]
Roman Yampolsky
Another concern people often ignore is what happens if you succeed. So you have this friendly piece of software, but it's still open to other people modifying it to new data sets, to accidents with cosmic rays hitting at hardware. So even if you are successful to a certain degree, it doesn't hold long term, it can still become malevolent later on. So.
- [63:12]
Lex Friedman
You can probably tell by my line of questioning, I'm really like, I think most people you talk to trying to avoid the we're doomed, throw up your hands and just enjoy the ride. And maybe enjoy the ride is the answer here. But is there anything we as individuals can or should be doing differently with this knowledge? And is there anything that business leaders can ensure or governments can and should be doing differently with this knowledge?
- [63:41]
Roman Yampolsky
It really depends on who you are. If you are Sam Altman, you should be doing something very different. If you are a random person, maybe you don't have much power over the situation.
- [63:52]
Lex Friedman
Yeah, I mean, is there, I'm thinking about climate change and you know, who knows if this holds any water, but just the micro things people can be doing. And one of the phrases I like, at least for economics up until now and capitalism up until now is vote with your wallet. Right. If people just stop using AI, if everybody just collectively decides this is gone, I'm going to disconnect. My free time is going to be spent in the park does. Is there a world where an absence of demand can have an impact here?
- [64:33]
Roman Yampolsky
It can slow it down, but I think there is demand from foreign and domestic governments, militaries, and there is enough people who have hundreds of billions of dollars literally to spend as they please.
- [64:47]
Lex Friedman
Yeah, well, in the. Yeah. So I'm just trying to cross off options as we go.
- [64:55]
Roman Yampolsky
Good ideas and I support everything we can try to slow it down. That buys this time and maybe something will stake again. I could be wrong about all of it. So, yeah, I recommend you do what you think might work. But honest evaluation doesn't seem to suggest this is a silver bullet.
- [65:14]
Lex Friedman
Right. And it, the way we've talked about it too, and just the, you know, human nature, whether it's capitalism or, you know, competition of, of great powers, you know, you talk, oh, you know, Sam Altman can do something differently, but just feels very quickly like it's a hydra problem. Right. Like, sure, Sam Altman says we're not doing this or we're not going to pursue it. Like, how many then does the next Sam Altman just pop up? Right. And, and to me that's the, that's the biggest challenge of convincing Sam Altman in the first place is I bet he thinks that way too, that, oh, well, I might as well do it because if I don't, someone else will anyway.
- [65:53]
Roman Yampolsky
You're exactly right. And they're quite replaceable as well. So it has to be a global realization of personal self interest. No one's going to benefit from it, no one's going to win. So it's just suicide mission, really.
- [66:06]
Lex Friedman
We don't, we don't have a very good track record of these global missions for solving these things, do we?
- [66:15]
Roman Yampolsky
No, we don't.
- [66:16]
Lex Friedman
No. So, and yet you are still a crusader in this fight, right? You haven't just kind of thrown up your hands and said, I'm going to enjoy the ride. So do you still believe you can make a difference? What motivates you? Are you just looking for the aha, I told you so? What motivates you in your crusade?
- [66:39]
Roman Yampolsky
No one's going to gloat about the end of the world. You're not going to have time for that.
- [66:43]
Lex Friedman
Well, that's good.
- [66:44]
Roman Yampolsky
I think it makes sense if we are still in charge, we're still around to do what we can to get the best outcomes possible. And as I said, I could be wrong. There could be game theoretic reasons for why it actually is good for a while. So. Yeah, yeah.
- [67:01]
Lex Friedman
And I think you said this earlier, but I don't want to misquote you. It still sounds like more people researching this, more people researching AI safety is probably a good thing. Is that, would you agree with that?
- [67:13]
Roman Yampolsky
Specifically, when it comes to upper limits and our capabilities, what I call impossibility results in AI safety, there is very, very few people in that space. If pretty much everyone was saying, you cannot solve this problem, it's impossible, it would make a difference in how we approach it. Because right now the argument from the labs is, oh, we're going to solve it, we have a couple years, we got resources, trust us, we'll figure it out. Maybe AI will help us, maybe we'll hire this brilliant scholar, he'll help us. But if it was basically community belief accepted scientific argument that it is impossible to do, we would have very different tools at our disposal to, to limit us.
- [68:00]
Lex Friedman
Right, right. Yeah. The word impossible seems to be, seems.
- [68:08]
Roman Yampolsky
To be coming up a lot with survey people. There is a consensus that of course you can't indefinitely control godlike superintelligence, it's obvious, but it's not a scientific consensus. If you look at publications, this is not what's showing up. And I think that has to change.
- [68:26]
Lex Friedman
But then what if it does change, then what happens?
- [68:30]
Roman Yampolsky
So if everyone kind of agrees and all the smart people and experts saying you should not be doing it, then there is a lot more opportunities for agreements, let's say between labs or between countries, to not do that voluntarily for self interest. So I would not create something where I know it gives me a billion dollars, but the next second the world is destroyed, time convinced that that is the fact I'm not going to do it.
- [68:59]
Lex Friedman
So consensus could lead to policy pressure which could lead to having some sort of enforcement mechanisms to, to at least delay us going down the research paths that are most likely to take us.
- [69:14]
Roman Yampolsky
My main hope is common sense pressure. If you actually believe this argument, you believe you will not control it, you will lose control completely. Whatever you are President of US or CEO of OpenAI, then that's a strong argument for you not to press that button.
- [69:30]
Lex Friedman
Right? Right. And it's so just to kind of bring the conversation full circle. It's. It's so challenging when it doesn't even sound like we know, like it's not one button. Right. It's like several buttons of Are we pressing a button on purpose? Are we pressing it by accident? Are humans pressing it? Are we pressing the super intelligence on button? Are we, you know, just accidentally creating something like there's does that we have to solve against an awful lot of buttons, don't we?
- [70:03]
Roman Yampolsky
I talk about fractal nature of this problem. The more you zoom in, the more problems you see. So people try to address specific issue. How do we box AI? They start looking at it and they realize there is 10 additional problems and most of them are impossible to solve.
- [70:20]
Lex Friedman
Right. So just to kind of, you know, put a bow on something as existentially devastating as the end of the world. If someone's listening to this conversation, what. What message do you want to leave them with to as they think about how they interact with these ideas and maybe what they do about them?
- [70:47]
Roman Yampolsky
So I'm specifically targeting people creating this technology. I would like you to realize that what you are doing is basically unethical. You are causing a lot of harm, and please stop.
- [70:59]
Lex Friedman
Is there a. That's. That's fair. Is it? Is it? We. We talked earlier about redirecting efforts toward narrow AI use cases as maybe a better route. Do we have to stop wholesale? Or is narrow a better path? And my logic there is if we stop wholesale, we come back to the hydra problem. And if you don't build it, you're willing replacements will versus if we can say, okay, we're trying to just come shy of brush up against super intelligence or these existential threat tools, but not cross that line. Is that the road to take in an optimization exercise?
- [71:46]
Roman Yampolsky
It is something we can actually, I think, get people to agree as they would get most of the benefits of this amazing technology. AI is great. It can literally cure every disease. It can give us longer lives, more abundance, but we can do it with narrow tools. And so this switch is quite easy to swallow, saying, ben, technology, I don't think that's going to sell well.
- [72:08]
Lex Friedman
Yeah. Wow. Well, lots to think about, regardless of me not being Sam Altman. And I wanted to say a big thank you, Roman, for your sobering perspective. And if this felt at all confrontational, I hope it's not, because I don't. I hope you're wrong, but I hope I'm wrong. I hope you're wrong. And I use the word hope very deliberately because. Yeah, and I know you hope you're wrong, too. The fractal problem kind of brought it home for me, is as you dig into it more, I end up feeling additional pessimism, not additional optimism. So I'm hopeful that people can find ways to avoid this. And, yeah, I guess we'll try to enjoy the time and see how the future goes.
- [73:11]
Roman Yampolsky
I enjoyed this podcast. Thank you.
- [73:13]
Lex Friedman
Thanks so much.