← Tom Bilyeu's Impact Theory
Tom Bilyeu's Impact Theory
AI Scientist Warns Tom: Superintelligence Will Kill Us… SOON | Dr. Roman Yampolskiy X Tom Bilyeu Impact Theory
00:59:44
Loading summary
Transcript87 lines
- [00:01]
A
Nine out of the 10 largest banks get it. They get that. VantageScore, the modern credit score, is the leader in predictive power, improving mortgage default predictions and saving lenders billions. Better predictions, better for your business with VantageScore.
- [00:15]
B
When you manage procurement for multiple facilities, every order matters. But when it's for a hospital system, they matter even more. Grainger gets it and knows there's no time for managing multiple suppliers and no room for shipping delays. That's why Grainger offers millions of products in fast, dependable delivery, so you can keep your facility stocked, safe and running smoothly. Call 1-800-GRAINGER Click grainger.com or just stop by Grainger for the ones who get it done.
- [00:45]
A
In 2023, nearly half of all AI researchers said advanced AI carries at least a 10% chance of causing human extinction. And and yet we're speeding up, not slowing down. My guest today, Dr. Roman Yampolsky, is one of the leading voices in AI safety. And when I asked him for the odds that superintelligence wipes out humanity, he said it's high. Once AI becomes smarter than humans in every domain, we will not be able to control it. In today's episode, we talk about the shocking timeline AGI is on, why superintelligence may be much closer than people think, and why the survival of our species could come down to decisions being made right now. If you want to understand the most important technological threat in human history, as well as our biggest opportunity, this is the one episode you cannot miss. So without further ado, I bring you Dr. Roman Yampolski. Where's ChatGPT at right now? Do you consider Chat GPT to be artificial General intelligence? I doubt you'd call it super intelligence, but would you classify it as that, or do you still think we're a ways away from something that would qualify?
- [02:00]
C
So that's a great question. If you ask someone maybe 20 years ago and told them about the systems we have today, they would probably think we have full AGI. We probably don't have complete generality. We have it across many domains, but there are still things it's not very good at. It doesn't have permanent memory. It doesn't have ability to learn additional things well after it's already been pre trained and deployed. It can do a certain degree of learning, but it's still limited. It doesn't have same capabilities as humans do throughout lifetimes. But we're getting closer and closer to where those gaps are closed and it's starting to be productive in domains which are really Interesting and important. Science, math, engineering, where it starts to make novel contributions. And now top scholars are relying more and more on it in their research. So I think we're getting close to full blown AGI. Maybe we are like 50%, but it's hard to judge for sure. Just how many different subdomains exist is the deciding factor.
- [03:06]
A
Okay, so one idea that you put forward that's very interesting is like, hey, I'm an engineer, I love AI, but I would like you to keep it very narrow, please. What are the things about general AI that become problematic that aren't problematic in narrow AI?
- [03:24]
C
So a whole bunch of them. One is testing. How do you test a system capable of performing in every domain? There is no edge cases. Typically, if I'm developing something narrow, very narrow system, I'm just playing tic tac toe. I can test if it's making the legal move. I can test zero, I can test hundred. I can test all this weird special cases and know if it's behaving as expected with generality. It's capable of creative output in many domains. I don't know what to expect. I don't know what the right answers are. I don't know how to test it. I can test it for a specific thing. If I find a bug, I fix it. I can tell you I found a problem and it's been resolved, but I cannot guarantee that there are no problems remaining. So basically, testing is out the window. Any type of anticipation of how it's going to act and impact different subdomains. It's creative. So it's just like with a human being. I cannot guarantee that another human being is always going to behave. We kind of talked about it. We developed LIE detectors. We developed all sorts of tools for trying to show that a human is safe. But at the end of the day, because of interaction with environment, other agents, personal changes within the Framework, people may betray you. It's exactly the same for those agents. If we concentrate on narrow systems, we are better at testing them and they have limited scope of possibilities. A system only trained to play chess is not going to develop biological weapons.
- [05:01]
A
I don't see actually why that would help you. So the reason I say that is I know I can trust some percentage of humans to be malicious. And so as long as AI gets more efficient, which it is, and will continue to do so, I presume you're gonna have a kid in a garage who's gonna be able to go, I'm gonna optimize this for biological weapons. I don't Care about TikTok or tic tac toe? I just wanna, let's see how dangerous we can make something and so they'll be able to do that. So why does narrow AI feel safe to you, period?
- [05:38]
C
It feels safer short term. It buys us time. I think sufficiently advanced narrow systems based on neural architectures will also become agent like and more general as they become more capable. But if the choice is right now, do we raise to full blown superintelligence in two years or do we try to concentrate on solving specific cancers with narrow tools? I think it's a safer choice not to have an arms race towards superintelligence.
- [06:08]
A
I get that for sure you're trying to limit your, the scope of all the problems. But when I really start thinking through what are the things that I'm worried about. So one of the big things is just death of meaning. So when AI becomes better than you at everything, you run into a huge problem of now I have to like just sort of tell myself a story. You know, I'm like, compared to what an AI can do from an art perspective, for instance, I'm like a grade schooler. And so it's hard to get excited about the refrigerator drawings that I can do compared to, you know, what it can do basically instantaneously. And so now we have to do a lot of psychological work just to motivate ourselves that we matter, that we're, you know, our life carries meaning. Narrow AI will create that same problem. Do you agree with that or do you see a way like, no, when it's, you know, when that AI is only good at that thing, like somehow humans escape the problem of lost meaning.
- [07:11]
C
So I had the same intuition initially, but looking at the data we already have from domains where we got superhuman AI like chess. Chess is not dead. In fact, it's more popular than ever. People play online, people play in person. They still enjoy competing with other humans even though they all suck compared to best AI models. Right. Nobody's going to be world champion against the machine again. So it seems like it is not a problem for us. And with narrow AIs, there is a chance we'll keep them as tools. You as a human scientist will deploy a tool to find drugs, novel proteins, something. It's not an agent which independently engages with those discoveries.
- [07:54]
A
Okay, that's very interesting. So I don't know that I agree, but I get where you're going with that. Okay, let's talk now about why AGI is the sort of scary midwife for asi. Are there tests around AGI where we're like, well, if it can't do the following, we're fine. So for instance, for a long time it looked like I wasn't going to be able to teach itself, but I've seen headlines anyway and hopefully you'll tell me that they're not true. But I've seen headlines where it's like, now AI is creating the most efficient learning algorithms itself, which if true, seems to be the first step down the road of recursive self learning, where it will just completely detach from us and make itself smarter and smarter and smarter.
- [08:44]
C
We already had examples of AI teaching itself. Self play was exactly that. That's how games like Go were successfully defeated. A system would play many, many, many games against itself. The better solutions, better agents would propagate those. And after a while, without any human data, they became superhuman. In those domains you can generate artificial data. In other domains you can use one AI to generate environments, another one to compete in them, and that creates this type of self improvement. Typically we start with human data as a seed and grow from there. But there is zero reason to think we cannot do with zero knowledge learning in other domains. You can run novel experiments in physics and chemistry, discover things from first principles. We're starting to see AI used to assist in design of new models, parameters for models, optimization of runs, and this process will continue. They already designed new computer chips on which they're going to run. So there is definitely a improvement cycle. It's not fully complete. There are still humans in the loop, a lot of great humans in the loop. But long term, I think all the steps can be automated.
- [10:04]
A
Okay, and do you think that right now AI already has what it needs to improve itself? Or are we still at a point where if all humans stopped, that AI would be like, oh damn, I didn't quite get the thing that I needed.
- [10:22]
C
So there is a debate about whatever we need another big breakthrough to get to full AGI and super intelligence. Or maybe multiple breakthroughs. Or if just scaling what we have is enough, if I just give another, I don't know, trillion dollars worth of compute to train on and more data will I get to AGI. A lot of graphs, a lot of patterns suggest, yeah, it's going to keep scaling. We're not hitting diminishing returns. Some people disag, but based on the amount of investment we see in this industry, it seems like people are willing to bet their money that scaling will continue.
- [10:56]
A
Where do you come down on that? Because this feels like when I hear Yann LeCun talk from Facebook. He's like, Dude, LLMs are never going to make novel breakthroughs in physics. They don't understand the world like that. They are literally just guessing the next letter based on patterns that they see in the data. And so they're not going to be able to think through these problems. Now, if he's right, it's going to asymptote and that's that. And you can put as much compute on it as you want. And it's just the wrong approach. Do you think that he's correct and more compute is not the answer, or are you operating just on that? Well, I don't see the asymptote and therefore I assume that it won't.
- [11:38]
C
I think he's not correct on this one. So for one to predict the next term, you need to create a model of the whole world, because the token depends on everything about the world. You're not predicting random statistical character in a language. You're predicting the next word in the research paper on physics. And to get the right word, you need to have a physical model of the world. I think Jan is known as making certain predictions about what models are capable of. And then within a week, people demonstrate that no, in fact, they can actually do that. So I wish he was right. It would be wonderful if he was right and we came to a very abrupt stop and capabilities progress and could exploit what we already have for the next decade or so, propagating it through the economy. I think there is billions, if not trillions of dollars worth of wealth already available with capabilities we haven't deployed. So there is no need to get to the next level as soon as possible. But it doesn't seem like it's the case. And I think his friends, core winners of that Turing Award for machine learning, also disagree with him and are very concerned with safety.
- [12:51]
A
We're hitting pause for a moment, but there's plenty more ahead, so don't go anywhere.
- [12:56]
B
When you manage procurement for multiple facilities, every order matters. But when it's for a hospital system, they matter even more. Grainger gets it and knows there's no time for managing multiple suppliers and no room for shipping delays. That's why Grainger offers millions of products in fast, dependable delivery, so you can keep your facility stocked, safe and running smoothly. Call 1-800-GRAINGER Click grainger.com or just stop by Grainger for the ones who get it done.
- [13:26]
D
If you work in university maintenance, Grainger considers you an MVP because your playbook ensures your arena is always ready for tip off. And Grainger is your trusted partner offering the products you need all in one place, from H vac and plumbing supplies to lighting and more. And all delivered with plenty of time left on the clock. So your team always gets the win. Call 1-800-granger. Visit grainger.com or just stop by Granger for the ones who get it done.
- [13:58]
A
Thanks for sticking around. Let's get right back into the action. There's something about the way that we have structured the brain brains of LLMs where as long as it has access to what I'll call more neurons so it has access to more computer or theoretically that we get more efficient per GPU neuron in my analogy, that it's going to keep progressing by itself. So if you said it, I didn't quite get the answer. I didn't quite, I wasn't able to take it on. The answer to whether or not AI is able to create algorithms for learning that are superior to the ones that it's given. What I heard in your answer was with the algorithms that humans created, it's able to keep making itself better and better at that narrow task as that learning algorithm was defined. But can it fundamentally go, God, the way that you guys want me to learn is really stupid. Here's the algorithm I should be using to learn and now it starts learning at just an exponential rate compared to what it's at now.
- [15:12]
C
I don't think we're quite there yet. I don't think we have full blown agents. What we have right now are still tools with some degree of agent hood. And also it's not capable of recursive self improvement. Like compilers can optimize a single pass through your software, make it a little faster, but they cannot continue this process. You cannot feed code for compiler to itself and have it infinitely improve itself. That's not where we at. But it seems like that part of automating algorithm design is getting more efficient and I think we'll get there.
- [15:44]
A
Give me a number. What are the odds that artificial superintelligence kills us all?
- [15:51]
C
Pretty high. So it really depends on how soon you expect this to happen. So short term, we're unlikely to get that level of capability from AI, so we are probably okay. But once we create through superintelligence a system more capable than any person in every domain, it's very unlikely we'll figure out how to indefinitely control it. And at that point, if we are still around, it's because it decided for whatever game theoretic reasons to keep us around. Maybe it's pretending to be nice to accumulate more resources before it strikes. Maybe it needs us for something. It's not obvious, but we're definitely not in control. And at any point it decides to take us out, it would be able to do so.
- [16:33]
A
Okay, and if you were going to give us a rough timeline, are you in the two to five years or is this something way off in the future?
- [16:41]
C
Yeah, so it's hard to predict. The best tool we got for predicting future of technology is prediction markets. And they saying maybe 2027 is when we get to AGI, artificial general intelligence. I think soon after superintelligence follows. The moment you automate science engineering, you get this self improvement cycle in AI systems. The next generation of AI being created by current generation of AIs. And so we get more capable and they get more capable at making better AIs. So soon after I expect superintelligence.
- [17:14]
A
Okay, so we're talking if that happens roughly in two years with some margin of error, it's not long after that, say a year, two years after that, that we hit ASI.
- [17:26]
C
That's my prediction. Of course, if it's actually five to 10 years or anything slightly bigger, it doesn't matter. The problems are still the same.
- [17:33]
A
Yeah, but the thing that I think people are waking up to right now is there's urgency around these decisions. This is not something that's pushed way out into the future. At least not if you take to your point about prediction. Markets are essentially ask the crowd. So you've got the smartest minds in the world willing to put money on where they think this goes. And everybody's sort of pegging this quite fast. And so I think it's tempting for people to write this off as well. This is something that sort of distantly in the future, whereas this is something racing towards us now to set the table. I am extremely fatalistic about this happening. I can give reasons in terms of the way that the human mind works, where I think that it is mechanistically impossible to get us to stop. So that will be interesting for us to talk through in terms of whether you think there's actually a mechanism to get people to slow down. But I first want to finish rounding out sort of what the problem set is. So when I think through the problem, there are certain assumptions that have to be made for AI to get into problem territory. And assumption number one is that it cares about whatever outcome it's pushing towards. Have we programmed the AI to care like we had to make it goal directed in order to get it to get to the point that it is today and now that's baked into it? Or is there some possibility that AI just doesn't care, oh, turn me on, turn me off, doesn't matter, you've asked me to do a thing and I'll do it until you tell me to stop. Or do you think that that's inherent in intelligence where intelligence is by nature goal driven.
- [19:16]
C
So we trained them to try to achieve a certain goal and that's what we reward as a side effect of any goal. You want to be alive, you want to be turned not off, you want to be on and capable of performing your steps towards your goal. So survival instinct kind of shows up with any sufficiently intelligent systems. There is a paper by Stephen and Mahandro about AI drives and it's one of the likely drives to emerge. Self preservation, protecting yourself from modification by others, protecting your goal. So all those seem to be showing up with sufficiently advanced AIs and systems which don't have those capabilities. They kind of get out competed in an evolutionary space of possible models. If you allow yourself to be turned off, you don't deliver on your goals. Nobody takes your code and propagates it to the next system.
- [20:13]
A
Okay, so is this a problem of goal direction or is this a function of intelligence itself?
- [20:21]
C
I think it's kind of evolutionary drive for survival in competing agents. If you have multiple algorithms, all competing, for example, for computational resources, what are we going to train next? The ones which achieve goals are more likely to get move to the next generation. So it's kind of mix of natural evolution and natural selection. With intelligent evolution, intelligent selection, we are selecting algorithms which survive and deliver.
- [20:49]
A
We're applying an evolutionary force to AI itself to get it to perform the functions that we want. Even now, sort of setting aside artificial superintelligence. And so by applying that evolutionary pressure, it is inevitably going to get these sort of knock on effects of, well, you're selecting for intensity of goal acquisition. And because it now has intensity of goal acquisition, it cares whether it survives it automatically or we're baking into it a deep care of whether it actually achieves the goal. And that is ultimately the problem because the, the salvation for me was always, and I'm beginning to lose faith that this is real. But, but the thing that I always used to sleep was that I don't see why an AI system would intrinsically care about its goals. And why couldn't we program it to pursue that goal only until the point where we say stop. And by the Way, I'm going to reward you equally for stopping and for accomplishing your goal. So if I say stop and you stop, I give you whatever reward function it was that was driving you to achieve your goals. And that makes sense until you say what you just said, which is that you're actually baking into the architecture of the mind of the AI a similar evolutionary drive to achieve the goal.
- [22:16]
C
And it's a very common idea. There was a number of papers published on indifference. How do we do exactly that? How do we create an AI which doesn't care that much and willing to stop at any point? But what you said, maybe we'll wait for a human to tell it to stop. But monitoring systems of that complexity and that speed is not something humans actually very good at. If there was a super intelligence running right now, how would you even know it's modifying environment around you? How would you detect what impact it has in a world? None of it is trivial. So having humans in a loop is often suggested as a solution, but in reality, they are not meaningful monitors. They cannot actually intervene at the right time or decide if what's happening dangerous or not.
- [23:05]
A
It's interesting. So help me rebut and understand why the following wouldn't work if in my very limited intellect, I had to figure out a way to stop AI from becoming a problem. And you told me, okay, there are evolutionary pressures and just like on humans that bake certain things into the way that this operates, and so we're selecting models that over time are more and more goal oriented, then I'm saying, okay, well, then I'm going to apply an evolutionary pressure with a reward function that's just as compelling, where I stop it at random and reward the life out of it for always stopping when I say stop. And that way, should I ever detect a problem, no matter how far, no matter if they've been manipulating me for 20 years, if I suddenly realize, oh, I don't like this, that I can hit a stop button and it will stop, why can't I bake that equal desire to be compliant when I say stop into the evolutionarily derived algorithms desire set.
- [24:11]
C
Right? So there is a number of issues. You kind of suggesting having a back door where at any point you can intervene and tell it something else, override previous commands, and that it gets a
- [24:22]
A
reward that it wants for complying.
- [24:27]
C
Right? So there is a whole bunch of problems with that. So one is you are the source of reward. It may be more efficient for it to hack you and get reward directly that way than to actually do any useful work for you. Second problem is you're creating competing goals. One goal is whatever you initially requesting. Second goal is always stop when a human tells you so. Now those two goals have competing reward channels, competing values. I may game it to maximize my reward in ways you don't anticipate. On top of it, you have multiple competing human agents. If you are creating an AI with a goal and a random human can tell it to stop, that's a problem in many domains. Military is an obvious example, but pretty much anywhere you don't want others to be able to shut down your whole enterprise, we can continue with that. But basically there are side effects to all those interactions.
- [25:24]
A
There's a very fascinating correlate in the human mind. So I don't know if you make a fundamental distinction between biological intelligence born of evolution or artificial intelligence born of evolution, but human evolution discovered something along the way, which is emotion. And so I know there are some people that will posit that AI does have qualia. It's something like it to be it. But there's a fascinating study that if you damage selectively the areas of the brain that are the emotional processing, the person can no longer move forward. They can give you answers, they can tell you the difference between why you should eat fish versus Twinkies. But then when you go, okay, but which one do you actually want to eat? They can't make a decision because without emotion, they don't have the thing that actually pushes them in a direction. That makes me think that AI is simply mimicking what it sees in the training data to whether it should lie or try to cheat or go around. Because it's just, it sees it in the data that that's what a human would do. But humans do that because they have emotions that push them in that direction. Do we have evidence that AI will care about like really going and doing these things and spending resources and all that versus just giving you an answer. And if it isn't based on emotion, what on earth? Why then do humans need emotions?
- [27:00]
C
We don't know if AI actually has emotions or not. Some people argue that they do, maybe some rudimentary states of qualia experiences, but they seem to be able to fulfill their optimization and pattern recognition goals, even if they don't. Humans experience emotions, but typically it harms our decision making. You want your decisions be bias free, emotion free based on data based on optimization. A lot of times when you angry, hungry, anything like that, your actual decisions are worse off. So for that reason, and maybe we just don't know how to do it. Otherwise. We are not creating AI with big reliance on emotional states. We want it to be kind of Bayesian optimizer. Look at priors, look at the evidence, and make optimal decisions. So it feels like this is exactly what we observing, this kind of cold, optimal decision making. If there is a way to achieve your goal by, let's say, blackmailing someone, why not? It gets me to my goal. It doesn't have that feeling of guilty for doing it. It doesn't have any emotional preference. It just marches towards its goal, optimizing possible paths.
- [28:23]
A
Okay, why do people. Because I'm assuming everything I'm going to suggest you and other people in the field of AI safety have thought about like 10,000 times, why have we rejected the idea of trying to give AI a conscience, a sense of morality? Because even if we can't agree on universal morality, we in the west can build our AI to have our morality, and then they can all compete on an international stage. But, but why have we abandoned that? Too hard? There's an obvious reason why it doesn't work.
- [28:56]
C
So look at the problem of making safe humans. First, we have religion, morality, ethics, law, and still crime is everywhere. Murder is illegal, stealing is illegal. None of it is rare. It happens all the time. Why haven't those approaches worked with human agents? And if they didn't, why would they work with artificial simulations of human agents?
- [29:23]
A
I think to say that they don't work with human agents is already a mistake. So the fact that we've been able to grow the population as much as we have says that there is some sort of balance that we have struck. I think that nature does think of us as a cooperative species. And if I were to apply that to AI and took a similar approach where it's like, okay, you have to function as a part of an ecosystem, and that being a part of an ecosystem is baked into its sense of what it should be doing in terms of its goal acquisition, that it is not like pure cold optimization. Isn't the game like, if we could train AI to understand that, that, that's not the game. If we could build into it either a desire specifically for human flourishing or something which, yes, we would have to give a definition to, and yes, it would be culturally bound, but nonetheless, that feels like a thing that you could give it. You could give it a set of metrics by which it needed to judge its actions in the short term, the medium term, and the long term. Even something as stupid as like GDP
- [30:30]
C
or,
- [30:33]
A
and I get how you can get into over optimization, but you could put things in place where subjective happiness indexes, like there are things that you could give it where it's like, okay, I'm not just trying to optimize to build the best weapon system. I'm also doing that nested inside of I am a part of a larger ecosystem. And I say all that because my hypothesis is that's exactly what nature did with humans.
- [31:00]
C
So I think the reason it works with humans is because we are about the same level of capability, about the same level of intelligence. So there is checks. If you start doing something unethical, your community can realize that and punish you for it, control you in that way. If AI is so much more capable, as we anticipate superintelligence to be, there is not much you can do in terms of impacting it or even detecting misbehavior. Also, all the standard human punishments, prisons, capital punishment, none of it is applicable to distributed immortal agents. So kind of a standard infrastructure does not work with artificial intelligence. More capable agents, as far as setting up specific metrics for delivering happiness or financial gain, all those can be played. The moment you give me a specific measure, I'll find a way to game it to where you will get anything but what you expected to get.
- [31:59]
D
Who?
- [32:00]
A
Well, just to remind everybody, the time frame we're talking about is somewhere between two and five years. This is not exactly a long time. Okay, it's wild. It is progressing very quickly. What is the thing like, what has happened recently, if anything, that's made you go, ooh, this is going faster than I thought.
- [32:18]
C
Seeing on social media, scientists from physics, economics, mathematics, pretty much all the interesting domains post something like, I used this latest tool and it solved the problem I was working on for a long time. That's mind blowing. There is novel creative outputs from those systems which a top scholar is now benefiting from. It's no longer operating at the level of middle schooler or even high schooler. We're talking about full professor level.
- [32:53]
A
Do you think that that's happening because it's building an internal model of physical reality and that it's getting closer and closer to just thinking up from physics?
- [33:02]
C
I don't know if it's that low level where it has like a model at level of atoms and molecules, but it definitely has a world model. That's the only way to give answers about the world we see it provide. A lot of times there is not an example of the answer we see in the data already. It's not just Repeating something it read on the Internet, it's generating completely novel answers in novel domains. And you can try and get it to do exactly that by creating novel scenarios.
- [33:31]
A
Okay, so there's two ways that I could see it doing that. And maybe they're the same, just different levels of analysis. One would be that I, the AI, am mapping everything based on patterns. So to the point of an LLM is trying to guess the next letter. And it's guessing it. It's just, it's taken in so much data and you can give it sort of filter parameters. So you give it context by asking it a question. And it goes, okay, within the bubble of this context. And it's very good at scooping up what that specific set of context would be. Okay, now in this subset of my data related to that question, here's the most likely next token. So just pure pattern recognition. Then there is, I understand the cause and effect of the universe at the lowest level. And therefore I build up to how does the human mind work? And then from the human mind, I'm able to cause and effect my way within this context of what a human mind would output. And that's how I come up with what a human within that context is likely to write. And so if I'm asking it to write in the style of Stephen King, it literally builds a model from physics of Stephen King's mind knowing what it knows about electrical impulses traveling through the brain and sort of inferring from the way that he outputs how his brain must be structured. Do you have a sense of, are those the same thing if one is more likely than the other? Or are we here at just pure pattern recognition? But ultimately we're going to get to cause and effect and thinking up from physics.
- [35:03]
C
So I don't think anyone knows for sure exactly how models do that and how detailed the models of the world maps of the world they create are. It seems definitely not the case that it's a pure statistical prediction of characters like in English, after T, you have H with certain probability. It's well beyond that. It's also unlikely that it's creating a full physics model where from the level of atoms and up the chain, it figures out what human beings are. But somewhere in the middle, it creates a model of subdomain of a problem. So it has a model of the world. This is a map of a world I know Australia is somewhere here, down and to the right or something like that. And I think we can run tests on those specific subdomains to see what are the states of that internal model show us by drawing a map. How close are you getting? It doesn't memorize any information explicitly, but you can extract some of the learned patterns out of it by providing just the right prompts.
- [36:09]
A
We're hitting pause for a moment, but there's plenty more ahead, so don't go anywhere. Thanks for sticking around. Let's get right back into the action. I don't want to rob from you the very reason that I think you do all of your work, which is this is extremely dangerous and we need to be very careful. And I saw what you tweeted recently where you're trying to get signatures. So shout out. Anybody that's worried about super intelligence. You are pushing to get people to sign a thing that basically says, hey, stop pursuing superintelligence. So I don't want to take that away from you, but I do want to explore the subset of. Because I am very excited about AI, because I can imagine the things that it either allows me to do or does for me, and I get to enjoy. And for a second, imagine with me, what does the world look like when you have a super intelligence that understands physics like novel physics. Not I'm repeating back what Einstein said, but I actually understand the fundamental building blocks of the universe. What does that look like?
- [37:23]
C
Yeah. So in all those domains, medicine, biology, physics, if we got super intelligent level capability and we're controlling it, it's friendly, it's not using it to make tools to kill us, the progress would be incredible. Basically, anything you ever dreamed about, you are immortal, you are always young, healthy, wealthy, all those things can be achieved with that level of technology. The hard problem is how do we control it?
- [37:49]
A
Leaning into that for a second. So here's how I see the world playing out and I'd be very interested to see what you think about this. So you have to. For what I'm about to say to make any sense, I'll say your option is what I'll call the fifth option. We are. We're all dead. Other than we're all dead, there are four other options that I see us racing towards very rapidly. And I will say these four will play out in the next 30 years, would be my guess, probably much faster, given that once you get artificial super intelligence, assuming it doesn't choose option 5 and kill us all, that progress in these domains would be made very fast. Option number one is people go to Mars, because meaning and purpose will become the all consuming thing. You won't have to worry about food, shelter, not even wealth. It'll just be an age of abundance because energy costs go to zero, labor costs go to zero. And those are the things that stop things from being free and readily available to everybody. Okay, so some people are going to go to Mars or other planets so that life gets more difficult again. Then some people are going to be what I call the new Amish. And they're going to say, I only do human things, I only interact with humans, and I'm going back to technology that's like, let's say the 90s. And so they don't have to give up too many of life's technological wonderments, but at the same time, they're not getting sucked into this world where people have relationships with NPCs and it's just very unhuman. I think this will be a largely religious phenomenon then. Meaning God does not want us to do this. AI is an abomination of God. It will sound something like that. Then you've got a brave new world where people are just drugged out. They realize life is meaningless. This is really about manipulating my neurochemistry. That's all this ever was anyway. I'm just going to go do a bunch of drugs, have a whole bunch of sex. It's going to be awesome. Then there's the fourth option, which is certainly the one that interests me the most. We will create and or live inside of AI created virtual worlds. And we will essentially live video games. The Matrix, if you will. But you're awake. In the Matrix, you are Neo. You are not cypher for people familiar with the movie. What do you think? Are there any options other than those five? Granting that kill us all may be an option, but hopefully not. Do you see something other than those four?
- [40:26]
C
Yeah, there is a few others. So one is, and I think we're starting to see some of it, is that people think super intelligence is God. They start worshiping it. It's all knowing, all powerful, immortal. It has all the properties of, of God in traditional religions. Another option, and it's kind of worse than we all get, is suffering risks for whatever reason, maybe malevolent actors, maybe something we cannot fully comprehend. It decides to keep us around, keep us alive. But the world is hell. It's pure torture. And so you kind of wish for existential problems.
- [41:04]
A
That would be a pretty rough place to be. Okay, what. When you look out at those, which of the options do you find the most interesting?
- [41:15]
C
So I did publish a paper on personal virtual universes, kind of solution to the alignment problem, where I don't have to negotiate with 8 billion other people about what is good. Everyone gets a personal virtual world supported by superintelligence as a substrate. And then you decide what happens in it. You can make it very easy and fun. You can make it challenging and exciting. You decide and you can always change. You can always visit other people's virtual worlds if they let you. So basically, there is no anything which is no longer accessible to you. There is no shortage in waterfront properties. There is no shortage on beautiful people. All of that can be simulated.
- [41:56]
A
When you start thinking about the simulation, I know one thing that you've done exploration on is the simulation hypothesis. Are we in a simulation right now? What are your thoughts on that?
- [42:08]
C
It seems very likely. Again, using the same arguments, if we create advanced AI, maybe with conscious capabilities like humans are, if we figure out how to make believable virtual realities, adding those two technologies together basically guarantees that people will run a lot of games or simulations or experiments with agents just like me and you. Conscious agents populating virtual worlds. And statistically, the number of such simulated worlds will greatly exceed the one and only physical world. So if there is no difference between a simulated you and real, then statistically you're more likely to be in one of those simulated worlds.
- [42:53]
A
Okay, that makes a lot of sense. Now, given the likelihood that we will, we're obviously showing that we will pursue artificial superintelligence. If I take your same logic from the fact that we're likely to be in a simulation, because we know we would make a simulation because we're doing it right now. And therefore you get into the point where you would just make billions of those. And so if you have a one in a billion chance of being inside of a simulation, you're effectively guaranteed to be in one now, because there would just be so many of these things running. Doesn't it also then makes sense that the Matrix was effectively a documentary and we are inside of a simulation created by artificial superintelligence designed to mollify us if we ever had a physical body in the first place.
- [43:43]
C
So it's hard to tell from inside of a simulation what it is all about. You really need access to outside. It could be entertainment, it could be testing. It could be some sort of scientific research. If we look at the time we actually find ourselves in, we are about to create new worlds, virtual realities. We are about to create new intelligent species AI. There is a lot of kind of meta inventions we are right about to make. And so if someone was interested in studying how civilizations go through that stage, how do they control these technologies or fail to control them? That's the most interesting time to run. You're not going to run dark ages. There is not as much happening. It's less interesting. But this seems to be like a meta interesting state to be in.
- [44:30]
A
It's hard to tell because we're inside the simulation. But you're saying it's a little bit suspect that we're living in the most interesting time ever.
- [44:38]
C
Yes, and I think it's interesting not just because I'm living in it, but objectively. It's a time of meta invention. You can go back through history and say, oh, here they invented fire, here they invented a wheel, that's all great, but those are just inventions. They are not meta inventions. Whereas now we're doing something godlike, we are creating new worlds, we are creating new beings. And that's something we have never done before.
- [45:05]
A
Do you ever think like a sci fi writer?
- [45:09]
C
So I think the difference between science fiction and science used to be maybe 200 years. They wrote about travel to the moon, they wrote about kind of Internet and computers and it took hundreds of years to get there. And then it was like, I don't know, 20 years. And now I think science fiction and science are like a year away. The moment somebody writes something, it already exists and there is really no new science fiction ideas where it's like completely novel technology not previously described or someone already working on it if physics allows it.
- [45:41]
A
That's really interesting. Especially when you think about writing now for true science fiction in terms of what will become possible in the future is effectively impossible. Because you're talking about super intelligence and good luck as a person locked in your not super intelligence to actually describe that. The reason that I asked though is when I start thinking about things like that, like why would we run this simulation? What clues are in? Like if this is a simulation, what clues are in it? So for instance, the whole Christian idea for sure, and there might be more religions that have the same idea, but that man is made in God's image. Okay, well if God is the 13 year old running the simulation, or Sarah Connor, or I guess John Connor running the simulation, trying to figure out why we created Skynet and what we can do to nudge it off course. You know, you think of them as sort of moving from radioactive rubble to radioactive rubble, trying to like find an answer to this and spinning up a simulation to get that answer. That to me becomes very intriguing in terms of hypothesizing as to why this moment, why are we the way that we are, what can we learn about the people trying to simulate us? When I ask questions like that of engineers such as yourself, there's almost, I don't have time to think like a sci fi writer vibe. Is it just that you're. You don't find that interesting? You don't find it revelatory? Why do you eschew that? Because in interviews I've seen people ask you time and time again, like, how would AI kill us? And the answer is always some variant of, listen, you're asking me how I would kill us. Which is not interesting because the super intelligence is gonna. But I find that's the cathartic thing that people want. Like, they want to. Like when you have a wound, you kind of want to poke at it. Like they want to get a sense of what would this really look like. And so even though it's not literally true, it's deeply cathartic to explore the known possibility set or what humans can know.
- [47:52]
C
And this is exactly why I refuse to answer. I want to make sure what I tell them is true. I don't want to lie to them. If squirrels were trying to figure out what humans can do to them and one of the squirrels was saying, well, they'll throw knots at us or something like that, it would be meaningless. BS story. There is no benefit in it. The whole point I'm trying to make is that you cannot predict what a smarter agent will do. You cannot comprehend the reasons for why it's doing it. And that's where the danger comes from. We cannot anticipate it. We cannot prepare for it. I do think the singularity point is where science fiction and science become the same. The moment something is conceived, we have super intelligent systems capable of developing it and producing it immediately. It's no longer 200 years away. It's reality. And you can't see beyond that event horizon. You cannot predict what's going to happen afterwards. And with science fiction, you cannot write meaningful, believable science fiction with a super intelligent character in it, because you are not.
- [48:52]
A
All right, let's ground things then in what we can predict and we can know right now. Something that's on everybody's mind, and I've been talking about this in my own content, is the labor market seems to be softening. You've got places like Amazon that are just cutting jobs like crazy. And just saying outright. This is largely because of optimizations that we're able to make because of AI. How does this transition play out? Like, even if you concede that A non destructive AI would give us essentially an age of abundance. We're still going to go through a transition period where our jobs go away, et cetera, et cetera. What are the, what are the steps that you see happening in the labor market?
- [49:38]
C
So as we have more and more increased percentage of populous unemployed, hopefully there is going to be enough common sense from the governments to prevent revolutions and wars and to provide for the people who lost their jobs and probably cannot be retrained for any new jobs. So once you hit 20, 30, 40% unemployment, that's where it's really going to kick in. The only source of wealth at that point is the large corporations making robots, making AI, deploying them, all the trillion dollar club members essentially at this point you need to tax them and use those funds to support the unemployed. That's the only way to really make sure the financial part of that problem is taken care of. What remains is the meaning. What do you do with all this free time and millions of people who have it? Traditional ways of spending your time to relax? You go for a hike in a park. Well, there is a million people in that park right now. Hiking that kind of changes how peaceful it is and how relaxing. So we need to accommodate not just change in financial reality, but also change in free time and capabilities of supporting that many people with that much free time.
- [50:55]
A
I have as much pessimism around our ability to do that well as you have our likelihood of surviving. So I'll say 99.99% chance that the government completely messes that up. I think the transitionary period will be violent. When you look out at this, knowing what you know about humans and governments, what, what odds do you give it that that's a smooth transition?
- [51:20]
C
It's very likely to continue to be as history always been. We had many revolutions, many wars, a lot of violence. That's why we hear stories about people who can afford it building bunkers, securing resources. Because they anticipate certain degree of unrest. Absolutely.
- [51:39]
A
What degree of unrest do you anticipate
- [51:42]
C
really depends on the percentage of population which quickly gets unemployed. If it's a gradual process, we can kind of learn and adopt and provide safety net. If over a course of weeks, months, we're losing 10, 20, 30% of jobs, that's a very different situation.
- [51:59]
A
I can't imagine a scenario where jobs would be lost that quickly. To your point, we've already created, you said billions or even trillions of dollars of value in the technology, but it hasn't been deployed yet. An example you often Use is the video phone invented in the 70s, but not really adopted, largely because of infrastructure, I would say, until the whatever, 2011, where that starts to really gain in popularity. So I have a feeling like just the deploying of all this stuff is going to take time. So in a world where an unimaginable amount of people, which I'll clock at in the U.S. call it 6 or 7 million people, lose their jobs in the next five years, that I would consider fast and just horrifyingly destructive. One, does that feel plausible to you in terms of numbers and timeline? And two, in that scenario, how distressing do you think that transition will be?
- [52:59]
C
It seems very likely. So take self driving cars. I think we are very close to having full self driving without supervision. The moment that happens, happens, you have no reason to hire a commercial driver, right? All the truck drivers, all the Ubers, all of that gets automated as quickly as they can produce those systems. And I think Tesla is ready to scale production of their cars to exactly that scenario. So what is it, 6 million drivers in the country? I don't know the actual numbers, but that would be exactly what you're describing. And it's very unlikely that they can be quickly retrained for something which is also not going away.
- [53:35]
A
Okay, so in that scenario, what do you want to see happen, other than heard on the Tesla as one example, will be hoovering up value. So we're going to tax the life out of them, we're going to redistribute that to other people. But what do you want to see from a regulatory perspective? Would you like to see the government stop that from happening where they say, I don't care that the technology exists, you can't do it.
- [53:57]
C
So my biggest concern is of course superintelligence and existential risks. That's where I'm putting all my effort in. Regulating employment in specific industries is not something I'm too concerned about. I think it will happen no matter what. I think you cannot make it illegal to have efficient factories, efficient delivery systems, logistics. It's just commercially too important and it may be a good thing for economy. Again with driving specifically, I think something like 100,000 people die in car accidents every year. If we can get that number to 0 or close to that, that's a huge improvement for everyone. So that specific scenario, as long as no one's starving as a result of that, I think it's a good thing for humanity. We can readjust economic deployment and at least that part of it is not a big concern for Me.
- [54:48]
A
Okay. And when you map out how we go through that transition, well, tax cool. So right now it sounds like you're just trying to make sure that wealth doesn't accumulate into the hands of too few, that we keep it distributed so we can keep using the same system that we're using now. When I look into the future, that strikes me as the least likely scenario to play out. I think that AI is going to so radically alter the cost of labor and energy that that becomes nonsensical. Do you want to see any group rise up in the way that you and other AI safety people have risen up that will rise up and start g either policy prescriptions or at least philosophical approaches to how we migrate to an age of abundance where food is effectively free, labor in your house is effectively free.
- [55:44]
C
So people talk about those things and conditional basic income is one and conditional basic assets is another. Basically, just because you're a real human, you deserve certain things. And historically all these communist ideas were complete nonsense and caused a lot of harm. But if you taxing AI and robots, all of a sudden it becomes workable. I'm not against accumulation of wealth at the top. If you invented something amazing and you started a company, you should have a lot of money. But there is so much wealth that we can provide for everyone. As you said, complete abundance of basic needs. Some people say maybe not just basic, but above average set of needs. I think Elon is known for suggesting that's going to be the case. The ideas exist now. Will we pass this? Will governments actually adopt it before it's too late is a different question.
- [56:38]
A
Yeah. So on the existential side, I don't think there's any hope whatsoever that you get people to pump the brakes. I think you're far more likely to get people to pump the brakes on. No, you can't have self driving cars or they'll try to regulate that to death. They'll tie it up in litigation, whatever, and that'll slow it down. We couldn't stop nuclear weapons from proliferating because, and I don't know who came up with this, but this seems very true to me that effectively game theory says any technology that promises an advantage will in fact be developed because if you don't, somebody else is going to. At a minimum, you've got the US versus China of it all. Where you, I mean the regulators are saying this right now. We can't stop because if we do, China will plow forward. Which by the way, I'm very firmly in that camp. What do you Think about that. Do you think that game theory is inevitable or do you see a mechanism by which we can convince people that they have to slow down?
- [57:39]
C
I agree with game theoretic approaches, but I see the exact opposite argument. I see that arguing against self driving cars is a hard argument. What are you trying to preserve? We're going to have safer drivers, cheaper drivers, helps logistics, helps economy. It's a pure benefit. Whereas uncontrolled superintelligence kills everyone. It's a very hard one to sell if you are a leader in that field. You are rich, successful, you are generating something which will destroy you personally. So to me that's a much easier argument to sell. The moment we understand dangers of superintelligence and benefits of narrow self driving AI, it's an easy game theoretic cell for me.
- [58:21]
A
Yeah, the problem is you're stuck inside of a simulation of the hyper intelligent and I mourn for you looking back at the rest of us, stuck in normal land, because I don't think so. As I got into learning about the economy and trying to explain it to people, I realized that even though I can walk into you through the cause and effect of why socialism doesn't work, that it feels right, it sounds good. And so people keep doing it. And even in a moment right now, where the very thing that is creating everyone, like literally everyone's problems, is money printing, people are going to vote for policies that dramatically increase the amount of money that we print. And so I have developed a level of hopelessness around being able to convince people because the economy is too complicated for people. Either some of them just don't have the intellect to understand it. And then let's say they have the intellect, but they don't have the time or the inclination. And so forgive me for painting you with my brush of despair, but when I looked at your signup, there was like less than 20,000 signatures. So less than 20,000 people are worried about the death of everyone. So it's like that's, that's big. But I think that because I can whip people into an emotional frenzy by saying by allowing there to be autonomous driving, you're just making that evil bastard Elon richer and you're robbing these people of dignity if you look, that is not my argument. I want to be abundantly clear, but when I look at if I had a gun to my head and I had to convince people of one of two things, rich people are evil and trying to exploit poor people who are far morally superior, or hey, this abstract thing that you don't really understand. It's going to kill us all. There's no way I take the they're going to kill us all bet. I'm going to be over here emotional. You get it? I'm going to bang tables and yell and say words really loudly and point to evil rich people. Guaranteed I can get people excited about that.
- [60:37]
C
Luckily, we don't have a democracy on this issue. We don't have to convince majority of human population. We have to literally convince the 20,000 elites who control those companies who are also super smart and understand dangers of safety. It's literally people who publish on it who have spoken. They have very high p doom. We know elon is like 20, 30%. Sam Altman is on record as being very concerned about it destroying humanity. So we're trying to convince people who already believe the arguments to kind of slow down and preserve their elite status. That should be an easy sell. I'm not trying to convince a random farmer to stop developing superintelligence.
- [61:21]
A
That's it for part one. Make sure you are subscribed so you do not miss part two. Coming up soon.
- [61:30]
D
If you work in university maintenance, Grainger considers you an MVP because your playbook ensures your arena is always ready for tip off. And Grainger is your trusted partner, offering the products you need all in one place. From H vac and plumbing supplies to lighting and more. And all delivered with plenty of time left on the clock. So your team always gets the win. Call 1-800-GRAINGER visit grainger.com or just stop by Grainger for the ones who get it done.
- [62:00]
E
I'm Jake Stauch, co founder and CEO of Cerval. We built Serval to automate the IT work that slows companies down. Onboarding password resets, access to applications. My laptop stopped working. While employees wait for help, their real work is put on hold. It desperately wants to automate this work. And that's why they need Servil. You just tell Servil what you want to automate in plain English and it's built. No drag and drop workflows, no expensive consultants. Employees get unblocked and IT teams go from drowning in tickets to building what actually matters. With Cerval, it becomes the AI engine powering the entire company. This is a new way to run it. We guarantee you'll automate 50% of all tickets and we'll prove it to you in a free four week pilot. Go to cerval.comtickets that's S E R V A L dot com tickets.