← Future of Life Institute Podcast
Future of Life Institute Podcast
Reasoning, Robots, and How to Prepare for AGI (with Benjamin Todd)
01:27:01
Benjamin Todd joins the podcast to discuss how reasoning models changed AI, why agents may be next, where progress could stall, and what a self-improvement feedback loop in AI might mean for the economy and society. We explore concrete timelines (through
Loading summary
Transcript136 lines
- [00:00]
A
Welcome to the Future of Life Institute podcast. My name is Gus Dacher and I'm here with Benjamin Todd. Benjamin, welcome to the podcast.
- [00:08]
B
Hi.
- [00:08]
A
Thanks.
- [00:09]
B
Great to be here.
- [00:10]
A
Do you want to start by introducing yourself to our audience, maybe talk a bit about what you're working on at the moment?
- [00:16]
B
Yeah. In the past I founded with Will MacAskill 80,000 Hours and then was the CEO for 10 years. In the last year or so I've been focusing on writing and writing about understanding AGI and how we can respond to it both individually and as a society. And the main thing I'm working on right now is a guide to careers that tackle AGI for 80,000 hours.
- [00:42]
A
So one of your essays is about reasoning models. This is a reasonably new phenomena where you can have an AI model think for longer on certain questions. Maybe you could tell us how does that work? What are the advantages?
- [00:58]
B
The basis is a very simple innovation called chain of thought with a large language model. Instead of when you ask it a problem, instead of asking it to just kind of generate the solution in one shot, instead you ask it to generate a chain of reasoning towards that solution. So it produces, you say like okay, we're going to solve this math problem. How would you reason towards that? And then it produces a token of reasoning. It then reviews that token and then produces another one. And then it produces a long chain towards that solution and then the extra addition. Then you already get a big boost just by using chain of thought. But then where it really gets going is then when you use reinforcement learning on top of that. So if the solution is correct, then you adjust the model, which is called reinforcement, to be more likely to do things like that in the next time. And then you can do that loads and loads of times with loads of examples until the model gets better at, better at generating these chains of reasoning that tend to lead to correct answers.
- [01:58]
A
Are there any kind of deep technical reasons that this has only recently started working? Reinforcement learning is not a new technique. Maybe chain of thought wasn't possible to the extent that it is now before.
- [02:11]
B
Yeah, so you know, chain of thought started working a bit with GPT at least definitely. By GPT4, the reasoning models paradigm has really only started getting going in 2024. Maybe the wider world still has not quite recognized this because these models are best at things like difficult mathematical and scientific reasoning, which just most people aren't doing in their day to day life. They're just using it as a chatbot and they haven't realized how much better it's gotten at these things. But, yeah, in terms of why it just started working in 2024, I'm not actually sure anyone totally knows the answer to that. But at a very high level, I think one way this can happen is if each step of reasoning only has a 90% chance of being right, like 10% chance of being wrong. By the time you've tried to reason through 20 steps, I think you only then have about a 12% chance of being correct. So previously with language models, they kind of couldn't keep it together for long enough to really get to any answers. But what seems to have happened is around early 2024, the models have just about got to the point where now they can reason for quite a while, like maybe the equivalent of an hour or at least minutes, and probably maybe like equivalent to a human thinking for an hour about something. And then the next thing that happens is if you can't even get close to an answer, you can't do reinforcement learning because there's no reinforcement signal. But once you start getting some reasonable fraction of the time, the right answer, then you can get the flywheel going and start using reinforcement learning to make it even better.
- [03:49]
A
Yeah, the underlying model has to be of a certain quality, or it has to produce the right answer at a reasonably high percentage of the time. And for this reasoning to work.
- [04:00]
B
Yeah. And I actually think this phenomenon comes up a lot in different parts of AI. I think we might end up with quite a similar thing happening with agents, where right now they kind of don't really work each step, they just kind of fall apart. You can't really do reinforcement learning, but we might suddenly get to a point where they start to work pretty well, and then you can use reinforcement learning, make it even better, and you get quite dramatic change.
- [04:25]
A
Yeah. And this is really, I think, a common experience kind of looking at how AI is developing. It seems to be that there are these thresholds where AI is bad at something until it's actually pretty good at that thing. Just a couple of years ago, I was discussing with AI experts whether large language models could ever become good at math or programming. And with reasoning models, it now seems that AIs are excellent. Maybe they are the best at exactly math and programming. So maybe we could see something similar with agents. You think, what is the connection between reasoning models and agents?
- [05:06]
B
I mean, one very simple connection is just if you have a really good reasoning model, you can kind of use that as the brain of the agent, the planning module. And so the better reasoning models we have that can do good Planning and can figure out what the right next step should be, then the more likely agents are to work.
- [05:27]
A
Now, one advantage of reasoning models is that they might be able to generate data that can then be used to train the next generation of models or even the same model. How can this possibly work? It seems like something that. It seems like an idea that's too good to be true. Where. Yeah, how can this work?
- [05:48]
B
Yeah, I mean, it works in this case just because the solutions can be easily verified. It can have a large language model, solve a bunch of math problems, and then it's quite quick and cheap to check which solutions are actually correct often. And so then at the end of that process, you actually just have a bunch of new correct solutions to these problems and also a whole chain of reasoning that leads to that solution. And that's super good training data. And there's nothing circular about it. It's just because it rests on them being easily verifiable.
- [06:26]
A
So you would expect domains that are not easily verifiable to be less as useful of domains to use reasoning models in. Like, for example, I'm thinking writing a fiction, writing a novel, for example, it's difficult to get feedback on whether the novel is good. It's difficult to. Is there even something that can be kind of formally verified about the quality of the novel? Something like that? Will we see this divergence between domains that can be easily formalized where we have strong progress, and. And domains that can't be formalized where we perhaps don't have a strong progress?
- [07:05]
B
Yeah, and that's what we've seen in the last year, where there's been a huge divergence in the kind of, like, hard scientific domains. There's been way more progress than in the others. Yeah. And I think looking forward, you could almost see this as the key question of kind of like forecasting AI progress is how many domains will be amenable through reinforcement learning. Will we just be able to ride the current techniques to superhuman levels of performance across most tasks, or will it just be limited to math, science, programming? Yeah, I mean, there's a few things that go into that. One is it at least seems true to a little small extent, at least, that if a model gets really good at maths and science, it does actually get a bit better at everything else. It is learning some type of general general logical reasoning that is useful, but it kind of remains to be seen how big that effect will be. And then the other thing is, like, how good can we make the reinforcement signals in these more nebulous domains. And yeah, that's how that's going is getting a bit out of my expertise. But, you know, I understand with things like, with something like writing, you might be able to use some AI models to rate intermediate outputs. So you could have an evaluation model which checks and then you could use that as a reinforcement signal. You can also use human feedback, though obviously that's much more expensive to gather that type of data. And then there's the kind of final feedback that comes from whether the novel sold a lot of copies, though that's a very long horizon thing. So you can't get a fast iteration cycle with that type of thing.
- [08:43]
A
Yeah, that's an interesting point. Does this mean that when we're looking at something, a question like, does this piece of code compile correctly or does this model do what I want it to do? What's the accuracy of this model? Something like that. Those are questions that can be answered rather quickly in a kind of fast feedback loop. When you're interacting with the world at large, you're interacting with human systems that are moving slowly. And so the question here is whether there will be a wall where reasoning models, perhaps agents, won't be able to interact with the human world as well because the feedback is simply too slow. The feedback cycle isn't fast enough.
- [09:26]
B
Yeah, it's more that, yeah, you won't be able to rapidly train models with those feedback signals. But I mean, there could be. It's possible that you'll be able to kind of break things down again into like, much smaller tasks that. Where you can get quick feedback and chain them together. I think this really remains to be seen how well all this will work and is a really central question about the next couple of years of AI progress.
- [09:50]
A
Yeah, there's another quite central question which you mentioned before, which is something like, how much progress do we get if reasoning models are only good in programming, mathematics and the hard sciences, how much progress would you expect from models being good in only those domains?
- [10:10]
B
I think in terms of economic growth, it's possible it would be quite small because not much of the economy is kind of difficult scientific reasoning. But I mean, if I was going to make a more bull case, it's possible those systems could be very useful for accelerating certain parts of scientific research. And then those scientific discoveries could then cause a lot of economic growth. The kind of strongest case for acceleration would be like, well, actually, yeah, these models are still not very good. They're not good at social skills, they're not good at business strategy. They're not good at physical manipulation. Lots of things you would need to have maybe a very general AGI. But if they're super good at programming and maths research, that could be really useful for doing AI research, in particular doing ML research. And so then, and then that could then unlock the next paradigm or wave of progress after that. So I think that would be the strongest case for rapid progress based on this.
- [11:04]
A
And why is it that AIs are particularly well suited to do AI research?
- [11:10]
B
Well, I mean, the biggest thing is what we've just been saying, because ML and programming are domains where you can get this reinforcement signal. So then the current models are becoming really good at exactly those types of tasks, which is exactly the type of thing you need to do AI research. Well, but yeah, I mean, a few other factors. One is like, it's fully virtual, so you can just do loads of experiments virtually, but without having to say, like, wait for lab results or wait for something to happen in the real world. The other thing you were just saying, and then there's a kind of other big factor which is it's also what the people doing AI research understand the best how to do AI research. So. So it's very natural for them to try and use the things they're developing to help with their own work.
- [11:54]
A
Yeah. Isn't there a big barrier here in terms of training runs being incredibly expensive? So there's probably some kinds of information about machine learning research or results in that field that you can only get by running experiments that are very expensive.
- [12:10]
B
Totally. So the extent to which that's true is in a way, kind of the key way of seeing whether there's going to be something like an algorithmic software feedback loop and intelligence explosion based on that or not. So, yeah, you can kind of think if we get AI virtual AI researchers and so AI AI researchers, you can think of that as really expanding the labor pool of people doing AI research. But there's two main inputs into AI research. There's the labor or the researcher time, and then there's compute, which you need to run all the experiments. And compute will stay the same in the short term because that's just a time determined by how many chips we have in the world. So even if you increase the labor pool a lot because computer's staying the same, it might not. That's a big reason why there might not be that big an acceleration of AI research. But yeah, I mean, the large training runs still only take about three months historically, so you can still, you know, in a year you could still you know, you could train in theory a bunch of much more advanced. You could do three, three whole generations in a year if you were maxing it out, which is still about 10 times faster than we've had in the past. And then I think the bigger thing on that is that because in this reinforcement learning paradigm, you don't actually need to run these massive training runs necessarily. They're using much less compute to do this reinforcement learning on top of the large pre training run. So you can get much faster iteration cycles in this. And apparently this has been a big trend in the AI labs recently is they've been preferring to distill the models into these smaller and cheaper models which are a bit less powerful, but then you can iterate with them way faster. So you could have 10 generations in the time when previously and you only had one generation and then you can actually end up ahead even if your kind of starting position is a bit worse.
- [14:01]
A
Explain that. Is it just because the model is cheaper to run?
- [14:06]
B
Yeah, so you can just do way more experiments with the same amount of compute.
- [14:10]
A
All of the AI companies are still gunning for very expensive, very large training runs. Do you think anything fundamental has changed with reasoning models? And if so, why are we still kind of scaling compute in this very ambitious way?
- [14:26]
B
I want to distinguish between total amount of compute spent on all forms of training, including post training and then like a large pre training run. And I think the large pre training runs like Training GPT5 and GPT6, those have been delayed I think compared to what we would have guessed a year or two ago. But instead that compute is now being used for reinforcement learning or just increasing inference. So more test time computes and then soon I think it'll also be used a lot on these like eight getting agent experiments going and getting agents to generate data as well.
- [15:03]
A
So you think there's actually been a move away from kind of large traditional kind of foundation model training runs to spending that same compute doing more inference time, using it at inference time and using it for experiments instead.
- [15:19]
B
Yeah, definitely in the last year. Previously the Metaculus forecast was for GPT5 to be released in like around now in March. But that's I think when I last checked, they think now it's going to be the summer, so like July or something like that. And then instead, yeah, all the recent models that have been released have been reasoning models. So there's been a clear shift recently. Exactly. What happens going forward is not clear. But like my guess is the returns from improving the reasoning models or working on agency will be bigger going going forward than just doing another 10x or 100x to the pre training run.
- [15:57]
A
Oh, that's interesting. So this actually might mean that we have kind of crossed some level of quality for the foundation model where it's now more efficient or there's more low hanging fruit in running that model in the mode of a reasoning model.
- [16:20]
B
Well, yeah, my thinking was just the reasoning model paradigm is still right at the start. So you're on a relatively sharp curve still. Whereas Most people think GPT4 to GPT 4.5 was kind of not a game changing amount of change. So I mean still I think it's been slightly overstated how bad it was because GPT 4.5 caught up with O1 on a bunch of reasoning things. But it doesn't have to do the reasoning part, which actually seems quite good.
- [16:54]
A
One useful thing we should touch upon is how likely we are to get a positive feedback loop in AI research. So you can lay out the different kinds of feedback loops we might experience.
- [17:07]
B
Yeah, with different types of positive feedback loops. The one that is kind of most concerning and has also had the most attention is a purely algorithmic feedback loop where if you get to the point where you have an AI that can substitute for people doing AI research, you can do a bit of back of the envelope estimate of how many of those would we be able to run in 2027, say or end of the decade if we used all our compute to run those? And those estimates tend to be between say 1 million and a hundred million equivalents in terms of how many tokens of output they can produce according to humans. So if the quality is also similarly good, then it's kind of like expanding the AI research workforce like at least a hundred fold now. But there's the fact that we just said which is the amount of compute wouldn't increase at that time. So then you have this question, if there was a hundred times more AI researchers, how many, how much more, how much faster would AI algorithmic progress actually be? And that's quite a difficult question to model. You can try to estimate historically as inputs into AI research have increased, how much has say algorithmic efficiency increased? And one key factor is each time you double inputs, do you get more than a doubling of algorithmic efficiency or more generally, what we care about is algorithmic kind of quality overall. And the past record is a little bit ambiguous about that. But Epoch has this paper where they look at some estimates and they kind of conclude it's around the threshold. It could Be below, it could be above. So I think that that kind of means as a very high level estimate, I would be kind of like, well, it seems like it's kind of a 50, 50 whether it would actually become a feedback loop or not. And then once the feedback loop starts, that you could also have increasing, diminishing returns. So that can also dampen out the feedback loop quickly.
- [19:01]
A
Yeah. Why would that happen?
- [19:03]
B
The idea is, as you make more discoveries, it becomes harder and harder to make more discoveries because the easiest ones have been taken to some degree that's taken into account in the past estimates, because that's also been happening in the past. As we've doubled each time, it's become harder to do the next doubling. But as you get, say closer to fundamental limits, you might kind of expect that diminishing returns to. You didn't expect them to kind of increase even more than they have in the past. So then, yeah, weighing all of these different factors and figuring out what will happen is, is kind of, is, is difficult. But Tom Davidson has a new paper where he looks through all the dynamics of all these, and I think his bottom line is we would see something. His is something like a 3 to 10x. We'd see like 3 to maybe 10 years of AI progress in one year. So probably not more than 10 years in one year. He thinks that's relatively aggressive. A couple of years of progress in one year seems kind of like a reasonable place to be at, which is.
- [20:04]
A
A Wild thought because AI progress, just say in 2024 is already pretty fast.
- [20:11]
B
Yeah, and you also have to picture this happening. This is at a point when AI can already basically be doing AI research. So it's already very good. And, and then it suddenly goes like three more years of progress in one year. So, yeah, that could be pretty crazy actually.
- [20:25]
A
Tell us how it could be crazy. Maybe paint us a picture of the impact of a feedback loop like that.
- [20:32]
B
So if you just look in the past, algorithmic efficiency has been going up 3x per year. So that means with the same number of chips, you can basically run three times as many of the same model. So if you get three years of progress in one year, then that's 27x increase in algorithmic efficiency in one year. So it means if you have those, say, 10 million automated AI researchers in at the start of the year, by the end of the year, you can then run 300 million, so 30x more on the same chips, so nothing else has changed. And that's an underestimate because that's Just algorithmic efficiency. In reality, you'd also have three years of improvements in post training techniques and so reinforcement learning type stuff or whatever they're doing at that point. And you could also, you could almost train a whole extra generation of, do a whole generation of pre training because, well, it would be like, it would be half a generation, roughly 30x. So you'd also go from like GPT 6 to 6.5 in one year. And so all those would happen at the same time. Yeah.
- [21:34]
A
What about chip design? Because there's, there's algorithmic improvements, there's the improvements to the, to the AI researchers doing AI research themselves, but there's also improvements to the hardware. How would the chip design fit into this picture?
- [21:50]
B
This is a slightly underappreciated aspect of the situation, which is even if you don't get this algorithmic feedback loop, it seems much more likely that we do get a feedback loop and chip design. And there's kind of two levels to that. One is that AIs could help with doing chip design itself. And I mean Nvidia is already using AI a lot to help with its chip designs. So then maybe you get again a similar type of thing where you get several generations of chip design progress in one year type thing. You'd need to kind of do the maths on exactly how fast it would be. But then there's the second level which is just simply producing more chips. Historically, this key parameter, if you've doubled all the inputs into the semiconductor industry, how much more compute do you get out? Historically it's been much more than a doubling. So the kind of empirical case for this feedback loop working out is much stronger than the algorithmic one working out. On the other hand, it's a bit less risky or a bit, it's a bit easier to deal with because it will be slower because each generation you have to produce all the chips and ship them. And that takes some significant time. It's not that you can just have like three generations in one year. It would probably take three years or something, but it would still be super fast compared to normal economic growth.
- [23:13]
A
Yeah, you describe the impact of a feedback loop in AI as an industrial explosion. If you think about kind of AGI level AI plus robotics, perhaps, what does that look like in your mind?
- [23:27]
B
Well, yeah, in a way that's, that's the kind of like the third level of feedback loop, which is. So you have algorithmic feedback loop chip design and production. And then the third level is when you can automate industry in general. So a Kind of a complete loop of production which would require robotics. And that one is almost the one with the strongest empirical support. Because if you double the number of workers and factories, you'll roughly double the amount of output. And in fact it's more than that because as things scale up, they get more efficient. So you actually get more than a doubling. And that would mean you get faster than exponential growth for a while until you hit, until you hit some type of diminishing returns again. Epoch have just released a new economic model trying to look at this and they actually, you know, they see growth accelerating over like a 10 or 20 year period. So it's not even, it's not this like one off. Oh, we get a big loop leap and then it's kind of flat. It's like things could keep accelerating to maybe like very high rates. Like the end question is just like what's the whole, what's the complete doubling time that you could achieve with like if everything was fully optimized, how quickly could things double? And it seems at least that's possible. That could be, you know, more than 100% per year.
- [24:48]
A
What I fear isn't really fully coming through when I have conversations like this is how crazy the world would become if something like this happened. Why isn't this always the kind of front page news, do you think? Why isn't this just even the possibility of thinking about this? And we can discuss how likely it is, but even the possibility should receive a lot of attention. But perhaps it isn't receiving as much attention as I think is warranted.
- [25:19]
B
If robots and AIs could say, produce the solar panels and chip factories to make enough chips to double the number of AIs and robots within a year. So on the Earth we're only using about 1 10,000th of the solar energy that's coming in. If you get that to 1% of solar energy, we're just still maybe not that high, then that's 100x more energy use would be possible. And so this doubling thing could quite quickly go to say kind of like 100x the output of now that would just be getting started because with the sun there's another, I forget the exact figure, but I think it's maybe four or five orders of magnitude more energy. And so say within like I don't know if you can do, how many doublings do you need to get 100x? I don't know if you know your powers of two, eight or something. So within eight years you're like at 100x. But then, like, then after that, we're suddenly like in space constructing solar panels around the sun, which is not on the scale of things, not that technically hard. And so we could literally go from current society to Dyson sphere is being created in a span of say, like 10 to 20 years, which, yeah, I think is a kind of radical change of the economy that people are not really at all taking seriously. Even people who are pretty into AI Humans are really bad at kind of extrapolating forward things that haven't happened before. And yeah, Covid was a great example of this, I think, where you could pretty clearly see an exponential curve of cases in, say, January or February. And basically very few people took any action about it until it was completely hitting them in the face and hospitals were overwhelmed and just had to shut everything down. And this is, in a way, a much more abstract and weird thing to think about than just, oh, people are getting a disease.
- [27:12]
A
I mean, in some sense, the conversation around AI and AGI and superintelligence and all of these terms have become much more mainstream since the ChatGPT moment in 2022. But still, it seems like we as a society, we're not grappling with some very important questions around this. Is this fundamentally a social problem or is it just that people perceive this to be kind of wild speculation? And I'll see it when I believe it. I'm looking out my window. I can't really see anything that's changed. And so you're predicting all of these radical things. There's been a lot of people throughout history that has predicted radical changes. And so, yeah. Do you think there's a concern about seeming weird if you actually believe and act, crucially act on beliefs like this?
- [28:08]
B
As a quick caveat, I'm not predicting this is definitely what will happen. I think all of these feedback loops, there's a chance they don't work or AI doesn't advance to that level in time, that. That type of thing. Yeah, I mean, I think it's interesting because I think even with myself, I believe some of these things intellectually. It still takes me a long time to actually internalize on a more gut level that this could be really happening.
- [28:34]
A
I feel the same way.
- [28:36]
B
Yeah, I still don't internalize a lot of it fully, for sure. I've, like, internalized more over time. I kind of like. But I think feeling the AGI is actually a big spectrum, and I feel it more and more over time, but still I don't fully feel it. And, yeah, I think a Lot of it's just to do with this. Yeah. Until something is completely hitting you in the face, it's pretty hard for humans to get motivated to do anything about something.
- [29:05]
A
There's a question around whether we can actually kind of fully internalize these beliefs and feel the AGI in our guts, so to speak, where we are not evolved to handle questions like this properly. I think we're not used to dealing with things that are moving this quickly and timescales that are this short. So the question is whether we will learn to internalize beliefs that are accurate about our situation before we are severely kind of overwhelmed by the situation.
- [29:42]
B
Yeah, I think that really. I think that really remains to be seen. It could well be that most people wake up after it's already quite a bit too late, though. Yeah. I do think there will be some, whether you want to call it warning shots or just like very powerful demonstrations that, as we've seen already, many more people are taking it seriously than they did in the past as better capabilities have happened. And I think that will keep happening and there'll be more and more waves of people realizing this is a big deal. Which. Yeah, I mean, just as a kind of aside for someone thinking about career planning, I actually think still, in many ways this is quite an early. It's still quite early even. It's. It is a very weird situation because it does feel like everyone is talking about AI a lot. But the number of people really working full time on tackling this, especially the. A lot of the risks, is still probably under 10,000. But I think between if we're on this timeline where the techniques do just keep working and AI keeps improving to a transformative level before the end of the decade, then by that point. So between now and five years from now, AI is going to go from what it is now to being just like the number one economic, political, social issue. It'll be like the front page every day will be to do with AI. And that's a very long way from where we are now, where when the O3 results were released which showed that this new reasoning model paradigm was yielding really impressive results that wasn't reported in any of the newspapers. And in fact, the Wall Street Journal was running an article about how GPT5 was behind and disappointing on that day, which is really missing the point because even if GPT5 is a bit disappointing and behind schedule, it doesn't matter because we've got this even better thing now that's completely taking off and that just is totally missing the mark. Reporting to be focusing on the old paradigm.
- [31:31]
A
Yeah, and you mentioned this before, but there's also a phenomena where if I ask one of the reasoning models an incredibly difficult problem in programming, mathematics or physics, I'm not really in a position where I can accurately evaluate how well it's doing just simply because I don't know the domain well enough and very few people are. I think it's true to say that very few people are good enough at physics, programming and mathematics to accurately evaluate. Is this output genius or. It's difficult to distinguish between outputs at a high level if you don't have a deep understanding of the domain.
- [32:13]
B
Totally, though actually the even bigger thing is people are still using only the free version of ChatGPT, which doesn't even include 01. So they're actually still using a one or two year old model and being like, oh, it's not got better.
- [32:27]
A
Yeah, you always have to account for the kind of. For problems such as that. That's true. All right. I think one of the things that could serve as a form of warning shots or something that could make people much more interested in AI is if they see robots moving around physically in their environment. Where are we with robotics? Do you think we are mostly limited on the hardware side or mostly limited on the software side?
- [32:55]
B
Yeah, I haven't heard a clear answer to that. My super rough read is that algorithms are the bigger bottleneck. Making really good robotics is a much harder challenge in some ways than language models because for one thing, we don't have the data set and it's quite expensive to build the data and build a really large data set. So yeah, I have heard other people saying that there are still some kind of hardware limits around like really, really precise motors. Like, if you think about how complex a hand is, it's not just the extremely subtle manipulation it can do, but it's also, say, all the sensors like that we have in hand so that you can, you know, if you can hold an egg but not crush it, you need to be able to feel the exact pressure in your hand to do that. And so having all of these cheaply in a package, I think is also a bit of a bottleneck. But my main sense is if we just had a big leap in algorithmic progress for robots, a lot more stuff would start working.
- [33:56]
A
How quickly do you think we can scale up our production of robots? One of the things with production that you mentioned is that as you mass manufacture something, it decreases in price, sometimes quite radically. So there's a question of how quickly we can ramp up production to get those decreases in cost really depends on.
- [34:19]
B
How good robot capabilities are at that time. So in my post, I imagined that we just had a sudden transition where humanoid robots start working. And then the question is from there, how quickly could you scale it up? But that's not exactly what will happen in the real world. In reality, it'll be like a more gradual thing, at least for a while, as kind of things get gradually better. Yeah. One thing I looked at to try and answer that question was imagining that car manufacturing capacity was converted to robotics. And yeah, you can do a very rough back of the envelope based on this because a car is about a ton of kind of industrial material all put together, and a robot is about a tenth, a tenth of that. Actually, it's a little bit less. Like you could say a robot is. A humanoid robot is 80kg, maybe. We'll actually have a bunch of smaller robots that are specialized for particular things, so they'll actually be like 40 kg or something. Cars are about a ton and a half. You could say, well, robots will be more complex to make, so we shouldn't kind of convert one to one. But even if you convert, say like half or a third current car manufacturing capacity could produce something like a billion robots a year.
- [35:31]
A
That's a lot. And we should also remind ourselves that modern cars are, in a sense, robots. They're much more complex than they were say 50 years ago. They contain a bunch of chips, a bunch of sensors. Think of like a modern electric car that has all kinds of cameras on it and so on. So, yeah, of course, robots are probably even more complex than that. But modern cars are complex.
- [35:57]
B
Yeah. Though I also heard someone saying that cars are also hard to manufacture because you're dealing with these big heavy parts, whereas with robots you'd be dealing with much smaller and lighter parts. And so that's like one respect in which it's easier. I agree. Probably the kind of sheer complexity of say, making a robot hand would be higher. So, yeah, I think overall it's not a crazy comparison.
- [36:16]
A
You have some kind of estimates of how cheap robots could become in our production costs, how cheap it could be to run them and so on. And those are quite interesting. Maybe you could tell us what world we might be in there.
- [36:31]
B
Yeah, I mean, the main estimate is just based on what you mentioned earlier. A typical industrial scaling curve is roughly every time you double production, it becomes 20% cheaper. And that's what we saw with solar panels. It varies a bit from industry. It could be 40%. It could be 10%. But if you assume a similar cost curve on robotics, if you say roughly now they cost a hundred thousand dollars, some of the most recent ones are actually a bit cheaper than that. If you then imagine a scale up to a billion robots a year, it should cost at least 10x less. So that would be $10,000 per robot. That also. Roughly. Well, yeah, so I think actually it could even go beyond that. One other way of limiting it is to think again just a comparison with a car. So if a car costs about $10,000 but a robot is only a tenth as much material, you might think in the long term it would be more like a tenth the cost of a car, maybe a little bit more because of the complexity. So that would be a couple of thousand dollars per robot. And then yeah, if you imagine those last for a couple, they, they can work for a couple of years, 24, seven, then yeah, $2,000 a robot, then it's, it's under, it's under 10 cents per hour for the hardware. And then, yeah, so actually the, yeah, maintenance could be about the same. Again, maybe electricity. That would actually, electricity prices could go up a lot if we making all these robots and all these AI chips. But at current electricity prices that would be something like $0.03 given like current, current power consumption. So yeah, you end up with a total cost per hour of maybe like 20 cents at full scale.
- [38:08]
A
Yeah. And again, this is a wild conclusion that it's difficult to kind of fully absorbed. But I mean if we imagine having a robot that's able to solve a bunch of tasks in the physical world, that's able to work 24, 7 for 20 cents an hour or so in running costs, just say $1 an hour. That would be revolutionary. Right? There would be an incredible amount of demand for that. I could use 10 of those robots just to do things around the house or, or help me with things. So maybe this question is dumb in some sense, but would there be demand for such robots? Do you think, do you think people would resist buying them out of nostalgia for human labor? Do you think they would be. Maybe they become illegal, maybe they're assisted by unions and so on. Do you think in some sense demand would be there? But do you think in actual kind of in practice that demand would be allowed to be expressed in the market?
- [39:10]
B
It does seem like once we get to the point where there is a lot of automation and people are actually losing jobs on a big scale, both from AI and from robotics, it seems like there's going to be some type of huge backlash at some point against that. And it seems hard to predict exactly what the result of that is. On the other side, there will be these kind of huge economic forces in, in favour of if it costs say like $20 an hour to have a cleaner clean your house now, but a robot could do it for 50 cents an hour. You know, people are going to really, really prefer the robot. I mean also with the robot you don't have to worry about like privacy and they can be available 24 7. And there's like many other advantages potentially. I mean there are also some other disadvantages. Like I, it does seem like cyber attacks become much more dangerous when there's robots everywhere because if someone actually can take over your robots then they could kidnap you in your own house while you sleep.
- [40:08]
A
That sounds absolutely horrifying. So of course they would be useful to have around the house. But a human factor like that, if that's a real worry that you might be murdered by your own household robot, I mean this seems like a scene from Black Mirror or something. Do you think factors like that, worries like that maybe legitimate worries for people could hold back adoption of robots?
- [40:40]
B
I mean definitely it's going to hold. A lot of people probably would be creeped out or worried about that. But again, I think it is very hard to say how it will go because it might just mean that people take cybersecurity way more seriously. And maybe if there's very few instances of this happening or just people, you know, they just get used to the robots being around and take it for granted. And you do also have to consider the other sides because you know, like humans are not perfectly safe either. Like there's a chance that a human cleaner steals from you or like yeah, very, you know, all kinds of other stuff. So eventually people just have to make the, make an overall trade off. It seems like with self driving cars people get used to them pretty fast and it's a bit weird at the start and of course they can malfunction and accidentally kill you but statistically they're already about 10x safer than human drivers and that will just increase over time. And so it seems like at least in that case it's a pretty clear win in favor of self driving I think.
- [41:38]
A
And that's perhaps a case study where adoption of technology, where people prefer to adopt technology because it's just so convenient to ride in a self driving cab and if it's also more safer, that's a win win. I'm thinking I've become more interested in human Factors limiting adoption of technology. Just think of something like augmented reality glasses or kind of early stage glasses that you wear around. One thing that I think has prevented adoption of such glasses is just that it's perhaps it looks weird to wear them, it's not fashionable. Perhaps people are concerned about being recorded. Very kind of down to earth here. Human factors that are not predicted by a model of the kind of pure economics of the thing. So, yeah, I am becoming more interested in whether adoption of technology would be limited by human factors. But I think, as you mentioned, that the economic incentive to adopt robots would be so enormous that these kind of concerns would be swept aside. Especially in manufacturing. Right, especially in robots for manufacturing goods. Yeah.
- [42:51]
B
Or think about an industry like mining or oil wells or something like that. That's quite dangerous. I mean. But I do just to step back a bit, I do agree with the general point that in so many jobs and industries, deployment of AI and robotics will be very slowed down by a lot of these types of concerns. And this is actually why I think, as I think we might be headed for again, quite a weird world where AI actually advances to very capable levels before most of the economy has actually changed at all. Especially if you can get this algorithmic feedback loop like you could have several years where most of the AI is being used to do AI research. And so suddenly you've gone, in a couple of years, you've gone to kind of like super intelligent levels of AI. But most jobs are just continuing as they were before. And this is one reason why people might wake up quite late. To go back to our earlier point, your daily life might seem exactly the same, but over in OpenAI's lab, maybe within coding, maybe a lot of that's automated and that's generating enough revenue to pay for the training runs. But then suddenly OpenAI has basically super intelligent levels of AI, just hasn't been deployed yet. And then deployment could be very fast because you now have extremely capable AIs that can help through the deployment.
- [44:08]
A
It seems like a scary world, right? It seems like we would want to have public information about the quality of the best available AI models so that we have at least some time to react. But if everything is becoming more internal to the AI companies, maybe that's not happening.
- [44:27]
B
I was not even imagining that. It's not that they're not being transparent, it's just that it's so hard to internalize until you see things hitting you in the face. And in this world, because when I walk around the street, everyone's still doing Their jobs, just like before. But superintelligence exists somewhere. I know that intellectually, but many people won't take it seriously still until they actually see the real world impacts.
- [44:50]
A
Yeah. There are also many jobs and you're thinking something like lawyers, doctors, and so on. It might be the case that I can diagnose myself using a, an AI model quite well, but I still need a doctor to prescribe me medicine, or I still need a lawyer to go through the kind of formal steps of having a document delivered to court. And that can only be done by a human and only be judged by human judge and so on. How much do you think factors like that will play into adoption of AI?
- [45:24]
B
I mean, I think a lot. Yeah, I think that will be in. There'll be some significant transition period where people are using AI advisors, but regulation and just people not wanting to have AIs making decisions will mean there's still like humans in the loop from a lot of things. But then, yeah, over time the main pressure on that is, if you say so, you could imagine a world where you say, well, every company still has to have a human set of board of directors who can officially veto things that the AIs do. But then that means that those human decision makers become the key bottleneck in the economy because that's the one bit that can't be sped up by AI. And so then you end up with huge economic pressure to take them out of the loop of more and more things so that you can unblock the whole cycle of production.
- [46:16]
A
Competitors will be thinking about replacing their board. And so maybe you now need to think about whether you need to replace your board and so on. So kind of standard competitive pressures weigh.
- [46:27]
B
In on less and less things. Yeah. And like same if you think about the lawyer case you mentioned, well, that, you know, paying that human will not only slow things down, but that's kind of an extra expense compared to like the AI lawyer will be basically free. But if like a human wants so.
- [46:44]
A
Free but without actual power in the legal system and that's. That might be a key issue. I'm not expecting this to be the case over the long term. And here long term might be 20 years. Right. But I could expect some. As you talked about, I could imagine the AI economy, so to speak, moving at incredible speed and the human economy to be limited by kind of the way we've been doing things by law or by convention for quite some time where it is, you know, in many countries it is simply legal to have an AI hand in documents. To a court and definitely have an AI judge a case in a court. And I just don't see a change to something like that would have to go through Parliament. Then that takes years. And this is just one example. And in many industries there are many examples such as this. So if you agree with that picture. Do you see a world? Yeah. What does a world look like where we still have of the legacy human systems but AI is moving very fast?
- [47:55]
B
Well, yeah, a lot of it was. I think it really remains to be seen how long that type of situation would actually persist because like I was saying, there would be these huge economic incentives to take humans out of the loop from more and more things. If one country is able to do that better than another, that country could quickly get ahead economically. So I don't know whether that would actually be stable for like a 20 year period. It might just be more like a couple of years.
- [48:23]
A
Is that, is that your. The way you expect things to go? That that's a system like that is unstable and it collapses under competitive pressures rather quickly.
- [48:34]
B
I mean, yeah, I do think it is really hard to say because I guess the point on the other side would be there does seem to be quite a homogenous global elite culture in some ways. And so the idea that say, pretty much all countries would just not want to go down this path of letting the AIs make all the decisions that shouldn't be totally off the table. So maybe even though it's like not a kind of competitive, it's not an equilibrium from a kind of strictly game theoretic point of view, it does seem like the world does sometimes manage to just coordinate into situations like that.
- [49:02]
A
Agreed. Okay, I want to talk about how an ordinary person can prepare for AGI. You have an excellent essay about this on your substack. So first of all, let's get some, some issues with this question out of the way because people in my audience will think that, you know, does this even make sense to ask that question? Preparing for a world of AGI is. Is. Is like preparing for the Industrial Revolution. But the Industrial Revolution happens in three years instead of however long it took. The worry is that the world is going to be transformed to such an extent that your actions simply don't matter. Do you think? Yeah. Why isn't that the right frame to think about this question?
- [49:44]
B
Yeah, the main thing I would say is that might be correct. It might be that we're just completely powerless in the face of this. And just to clarify, I'm talking about here from a Kind of a personal perspective, not from a socially. What should we do to tackle this? There's a lot that society could do to better prepare. But yeah, from an individual point of view, one way this sometimes gets summed up is like death or abundance. Like either there's an X risk and we all die and there's nothing I can do to not, not die in that X risk, or it's just some massive abundance utopia where everyone just has more than. More than everything they need. So nothing I do really makes any difference to that. But yeah, the main, kind of my main pushback against that is, yes, there might be nothing we can do. But in terms of what from a personal preparation point of view, what you should focus on is the scenarios where what you do now can make a difference and you can kind of ignore, unless you think they're like 99% of the probability mass, you can kind of ignore the scenarios where you just can't affect the outcomes. And like all your chips should be put into preparing for the scenarios where what you do now can have some effect.
- [50:53]
A
Also, by personally preparing, you might be able to put yourself in a better situation to help the world. So this is not exactly an exclusively kind of egotistical idea. Right. This is also about kind of creating people that are able to adapt to the changes ahead and might be able to help the world adopt too. So, yeah, let's dig into your advice here. Your first piece of advice is to find the people that are in the know. Seek out the people who have some clue what's going on. How do you do that and where are those people? The first problem, of course, is there's disagreement about who's in the know here. How do you go about finding those people? Not exactly, you know, setting aside who exactly those people are.
- [51:42]
B
Yeah. I mean, in some ways there's a very deep question there about, like, who should you trust? But, you know, I do think there are a lot of people who are tracking AI very closely. There's people who've been prescient in the past and it makes sense to, to at least read what those people are saying. But I think, you know, it's even better if you actually have some people, you know, personally who are more in the loop about this type of thing. Many of say, like the past guests on your podcast would be qualified here. But I mean, I guess some of the things I read, I mean, obviously I listen to Dwarkesh, I read Slates code, Astral Codex 10, Zvi's newsletter, the 80,000 Hours podcast. So Yeah, I mean, there's actually a lot of great substacks now that are tracking AI. And then, yeah, knowing some people in the industry, knowing some people, especially people who take the more transformative scenarios seriously, because I think that's still a big thing, kind of lacking from the broader AI discourse.
- [52:41]
A
Even if you can look at these trends and see big changes coming, it might be difficult to act on that information. I wrote to you in preparation for this conversation about the fact that I learned about COVID somewhat earlier than society at large, but I felt like I couldn't really act on the information. Maybe that's just a failure on my part to kind of act with conviction when I have some information. But it seems to me that there are a lot of people, at least I get emails from people like this that think big changes are about to arrive, but feel like there's probably. There's probably nothing to do about it. Right. There's probably nothing to act on here.
- [53:23]
B
Yeah. I mean, it could be with COVID it just was a case where you kind of got unlucky, where you had the information, but it didn't turn out to be useful. But I think we should have a very strong prior that in general, more information is better, even if a particular case doesn't work out. And then I think there was some stuff that people could do in Covid, and I mean, I know I didn't manage to do this myself, but a lot of people managed to hedge their investments and save a lot of money before the downturn. I did manage to move to the countryside, which then made the next year much more pleasant for me than I think if I'd stayed in London and got that done in time. And I actually think if I had been able to act on Covid, say, even a week earlier, it would have been. It would have been valuable. I was running 80,000 hours at that point. And we prepared a lot of material about what's going on with COVID and how you can personally help with it, but it kind of like we didn't quite get it out early enough to really get as much attention and be as useful as it could have been. But if we could have done that a week earlier, I think that would have. It would have been much more useful to people. So I almost kind of wish I'd actually acted a bit sooner in the COVID case, though. That's from a social impact perspective rather than a personal prep one. Yeah.
- [54:32]
A
Another piece of advice is to save as much money as you can. Why is that useful? You often hear the opposite advice. Sorry to break in, but you often hear something like the opposite advice. If we get AGI, if we get perhaps even superintelligence, money will become irrelevant. We'll live in such abundance that money isn't the problem anymore. Maybe talk about why that is not exactly the case.
- [54:58]
B
There's a few things to say about this. One is just if you put some more uncertainty into the equation that pushes you back in favor of saving again. So if you imagine, say if you're like, okay, well it's definitely death or abundance, then yes, obviously spend all your money now, but we're not sure that AGI will arrive with a hundred percent certainty soon. And if you spend all your savings but AGI doesn't happen soon, then you're, you know, you're significantly worse off because now you don't have a pension and so on. But if I've just spent 20% more money per year in the next five years, that's not going to make that much difference to my wellbeing. Like maybe I go on an extra holiday or something. But so there's a kind of that, if you consider that asymmetry it actually means. And you can model this formally, there's, there's this thing called the, the Merton share, which, Merton's portfolio problem, which is about like how much to save given your discount rates and so on. And it, you can model this out and if you, even if you put in quite a big discount rate because of the chance of money not being useful anymore, it doesn't say you should spend all your money, it's just like you should spend a little bit more, like 1% or 2% more compared to what you would have done normally. But then, yeah, I think the even bigger issue is there could be a third scenario where money is still useful in the future. And well, firstly, AI will probably make returns on investment go up hugely because there's going to be this like mother of all investment booms as we build out all the infrastructure to run this AI and robotics. So capital will be really scarce for a while and that means the returns on capital will be really high. So if you save money now, that could turn into like 100 times more money in, you know, post intelligence explosion, maybe, maybe a lot more. So you have to consider you're firstly getting way more money and then on the other hand you'll then be able to buy things that you just can't buy now. So I can't, however much money I have, I can't buy life extension Like, I can't buy life extension technology, but maybe that will be possible to buy in 10 or 20 years. And if you consider. Yeah, a key way of seeing the situation is does your does for all of your goals that you might have in life, do they just completely flatten off with a certain amount of money, or can you keep buying more of what you value with. With additional resources? And right now, there is quite a big difference between being a millionaire and being a billionaire in terms of your lifestyle or your ability to achieve your values more broadly, considering that it's not just about your comfort, but you might have other preferences for things like social goods or. Well, life extension, I think, is maybe the best example where if you can just buy more years of healthy life, I think many people would just want to buy as many of those as they could.
- [57:50]
A
It's interesting. Now, in today's world, there are some goods where I can probably have the same smartphone as the richest people in the world. I can read the same books, I can watch the same TV shows. And of course, they can have much more influence on the world than I can. But I. It is true that something like life extension is a technology that might require more money in the future. And there might be a separation between the rich and the not rich in our ability to afford it, unless it becomes much cheaper over time as it's more widely available.
- [58:30]
B
The hope on the other side would be that things would just be exponentially getting cheaper. And so even if you're poor, you just wait a few more extra years. But yeah, I mean, there's some things that are just scarce. So like land on the Earth, there's a fixed amount. So however much money you have is like how much land you would be able to have in the future. And this could be where land is becoming much more expensive as well, because land could be used for robot factories.
- [58:53]
A
Yeah, I agree. I've heard this line of reasoning before, that land is kind of like very quite simple economic argument, that land is scarce and therefore, you know, it's limited in the way that many other, many other things you can buy aren't. And so it'll become much more expensive. But will it? Is it the case that, for example, something like farmland would be much more valuable? Exactly, because you can use farmland to build robot factories. Whereas in today's world, we have certain cities where land is incredibly expensive because of various social factors, because of regulations and so on, limits on how much you can build. Would you expect real estate in San Francisco to skyrocket? Or is it more something like land in the middle of Arizona where you can get a lot of sun and you can build factories.
- [59:46]
B
I've been thinking about the buying versus renting question in light of AI, but also in general recently. And I think, I mean, yeah, I would treat these as two separate markets. So the kind of the rural land, one that would be ultimately driven by how much solar energy is falling on that land. And then also could that land be converted into some much more productive use in the future? The land in the centre of city, that's essentially a luxury consumption good. And so the question there is, will people with future AI wealth want to be able to have a house in the centre of the cities we have today? And I mean, that seems quite likely to me. I do think there could be a move out to the country, out to be more spaced out when it becomes like much more cheap to get around with transport and people have much more. They can build like really fancy new houses very cheaply out in the countryside. And there's no economic reason to be in the city anymore. Like your work isn't there? But yeah, but there would still be social reasons. And you know, especially if you think of land, say like along the river Seine in Paris, well, you could see why people would still want to visit that and spend time in that type of environment. Even if we were way more wealthy, I mean, maybe even more so than today, because as you get wealthier, you probably value leisure time and social time more than we do. Now.
- [61:11]
A
That makes sense. So land in very desirable cities would become like a luxury good, like luxury clothing or handbags or expensive cars or something like that.
- [61:23]
B
It's also finite. So in a way.
- [61:24]
A
Yeah, exactly. It's finite in a way that. That isn't true. Okay. Which skills would you expect to have most value going forward? Of course, this is like almost an impossible question and this is bound to change radically over time. But do you have a guess as to which skills would be valuable?
- [61:45]
B
Yeah, I mean, this is, yeah, like you say, a very big question. If we are actually heading towards this world of intelligence. Explosion, explosion, then it could be eventually that pretty much everything gets automated from a personal planning point of view. Now the question is more about how do you stay one step ahead of the current wave of automation and then earn a bunch of money while that's happening, which you then save and then you can live off even if all the other skills get. Even if all the skills get automated. So just. Yeah, the question is like, what's the. Over the next, say, transition Period. What will be most valuable. And I think one way to sum it, nice way to sum it up, is like you either want to get as close to AI as possible or as far away from AI as possible and as close to AI as you're working on improving AI or deploying it. And you can see now those skills already extremely well compensated. And yeah, they're the pretty difficult skills to have, but clearly they're going to be very valuable. And that's basically because they're a complement with this AI automation. And then on the other side it's just things that the AI is going to be worse at and those things will become the bottleneck in production and so their value will also go up. I think this is a thing that people often don't appreciate. All the stuff that AI is bad at, those things will increase in value over time as AI gets better because those will be the things that are still needed for humans to do. Figuring out what those are is harder. But we have touched on this already in the conversation. Any task that's amenable to reinforcement learning, we're going to see AI get a lot better at that in the next couple of years. Also, any task where you can take a big data set of examples and then use that to pre train a model, those will be the things that are best covered. And then the things which will be hardest will be the cases that are least like that. So basically these much more vague like long time horizon, undefined type tasks and.
- [63:40]
A
What would be examples there?
- [63:42]
B
I think a lot of entrepreneurship management type, kind of high level planning, coordinating lots of things, figuring out what to build in the first place and then setting up lots of AI systems to actually do all the well defined chunks. But yeah, basically breaking the things into the well defined chunks in the first place. But I mean a lot of kind of social relationship stuff could also be more like this. That's also an area where we have a lot of, we might have a lot of preferences to do it with a person for a while. So any jobs were just like relationships is really a key part of it.
- [64:15]
A
Examples I often hear about include something like jobs that are in the physical world dealing with people and that involves a lot of variety of tasks. So something like physiotherapist or nurse or tour guides, something like that, where it's, where it's a mix of people, skills and moving around physically and you're doing a bunch of, you don't know which moves you're going to do when you come into work that day.
- [64:44]
B
Yeah, like unpredictable environments. I mean, partly. You're also pointing out there that robotics is lagging cognitive knowledge works type stuff. So anything involving physical manipulation would also. It'll be good. The trouble with that one is, as we were discussing, that could change quite fast. But there could be this transition period. I think Carl Schulman talks about this, I think in his episode with Darkesh about. There could be this transition period where loads of people are just employed and they have an AI just telling them what to do, but they're building a factory, so they're kind of being used as it's like for their physical manipulation skills becomes the most valuable thing they. They offer for a while. But that. That only lasts until really good robotics is developed.
- [65:27]
A
And if you had something like that, you would probably also be able to record their movements in specific ways, use that as training data. So again, it doesn't seem like a situation that will hold for a long time.
- [65:42]
B
Yeah, I think that seems right. Whereas, I mean, yeah, something kind of like someone who does like a luxury travel experience where they take you to a private kitchen and you taste lots of food with them in a Moroccan tent in the desert or something. Maybe people would really value that type of experience and they'd really want. The human touch would be a big part of it.
- [66:09]
A
Yeah. And this might also be the case for something like this is not really a career path that's available for many people, but being a famous person. Right. Famous people can often get paid because they are themselves. And that's something that can't really be outsourced to AI. You are beginning to see kind of automated influencers where they will lend their kind of physical appearance and voice to be recreated and then fans can interact with a model of them. But I still think there's probably tremendous value in being a person that's known and people want to. Want to meet the actually famous person.
- [66:48]
B
Yeah. And that's a really interesting kind of general phenomenon, which is if you think of AI as making labor and then eventually robotics makes physical manipulation less valuable and then makes all the kind of other resources that you could have more valuable because those remain like important resources that aren't being cheapened by AI. And yeah, that we've talked about capital as one, because we'll still need the capital to build all the robots and the factories and chips. But then another is these other resources like relationships or fame, which potentially become a bigger part of the economy over time and remain valuable.
- [67:28]
A
Yeah. And another one you mentioned is citizenship, which is also something that you Recommend that people get citizenship of a country that has a lot of AI wealth or that will have a lot of AI wealth. My first question there is, isn't the process of becoming a US citizen, say, extremely slow? I often hear about people who have been living there for 15 years and contributed to the US economy, but aren't actually US citizens yet. And so is this something that matters on the timescales we're talking about?
- [68:01]
B
I think it's quite a bit faster than that. I forget the exact time horizon, but I think if you enter now on a work visa, I thought it's more like a kind of a five, seven year period and then you can apply for citizenship.
- [68:13]
A
Yeah, you're probably right about that. The 15 year is like an extreme example.
- [68:18]
B
I mean, immigration is terrible. So I'm sure there could be things that have knocked someone off that timeline. That's if everything goes well. And then, yeah, it comes down to what you think, what are your timelines, but it would only be 20, 30 by the time you might be able to start applying. So I think there could still be time. I mean, I also think if there's an intelligence explosion happening and you're already a work, you already have work permission in the U.S. the intelligence explosion itself will take several years. So you firstly have to get to AGI that can do AI research and then you have to have the whole intelligence explosion. And would you be thrown out of the country, like hopefully not after you've been there that long. So I think there could still be time. I mean, I'm not doing this personally, but that's partly. I just like I'm too lazy. It's too much of a personal sacrifice to move to the US now. But I don't know, maybe I will regret this.
- [69:14]
A
I mean, the question of citizenship is interesting because your citizenship determines your piece of the cake in kind of like a national economy. And countries that will do well doing a kind of run up to AI or AGI will be able to redistribute more in absolute terms just because they'll be so much more wealthy. And again, this is of course speculative, but do you expect kind of welfare programs to hold during a transition into AGI, or do you expect that they won't be able to honor the obligations that they have to their citizens?
- [69:53]
B
These programs in this world, the economy is growing very fast, so I think it actually becomes easier to honor your obligations. Yeah, I mean, my best guess is there will, there would still be significant welfare. One factor is just there's. There's inertia like the U.S. i forget the exact figure, but you know, I think it taxes something like 30% of GDP and then a lot of that essentially ends up in welfare programs. That's like the biggest federal expense. So if that just carries on as it is, you actually end up with a lot of. A lot of redistribution. But then the even more important point is just there would be enormous political pressure for this. Because if everyone is like having their wages pushed down by AI and then there's like a couple of a small number of tech elites becoming trillionaires, people are really going to want to tax that AI wealth and not just let everyone starve, I think. So. I think that that would only really. You'd only really get the bad scenarios where no one's getting any restoration if there was some very well locked in type of authoritarian government which was able to just ignore the will of its population. But I think with say, a country like the US currently, it would be politically untenable to do that. I mean, I suppose if the change was fast enough, maybe it could be, yeah.
- [71:07]
A
Or if power was concentrated enough in say one or two, perhaps even one company reaching superintelligence first and then, you know, being able to becoming basically masters of the universe before the government is able to respond.
- [71:24]
B
Yeah, then we have a lot of problems.
- [71:27]
A
Of course. Of course you advise that we should make ourselves more resilient to crazy times. This is something that's more easily said than done. I think it's. I mean, we have, we've now lived through the COVID times that were somewhat crazy, but not anywhere near as crazy as I would say expect an intelligence explosion to be. What are the lessons you've taken from. From kind of trying to be resilient during COVID Yeah.
- [71:54]
B
Someone once described to me as. Well, with the intelligence explosion, you could imagine it being a bit like, you know, that in two years Covid is going to start like the first few weeks of COVID and then it will just never stop. Because it's not like a one or two year thing. It's like. No, it actually gets faster and faster maybe until everything is totally unrecognizable. So as a kind of frame for how to spend the next couple of years, that can be quite useful. Yeah. I think I don't have anything super innovative to say about how to be more resilient. I just say do the kind of normal, basic things. So have some kind of like healthy routines that help you be less stressed, like make sure you get lots of time with friends you do exercise. I think finding a good therapist is helpful or some type of coach type person who you can talk to about things. Yeah. Finding things that help relax you, whatever they are. Yeah. Having like an environment, a nice. I mean. Yeah. Personally, I kind of like the idea of being based in the countryside through a lot of this stuff because I feel like I would be less stressed because there would be nature and I would be able to tell myself that if there was like a bio threat or a nuclear threat, I'm a bit safer. Yeah, I think those types of things would be the main ones on my mind.
- [73:07]
A
There's kind of maybe a trade off between how good you feel in your everyday life and how relaxed you're able to be and then your level of engagement with the world. So one way of relaxing is to disengage. Right now I want to walk around in my garden, I want to talk to my friends in real life, I want to take walks. And it's too stressful to follow along what's happening in AI and it's too stressful to follow along the news even. Is there a strategy for kind of strategically engaging with the world, getting the information you want, getting all of the actionable information and then perhaps disengaging also, or having periods of disengagement with the world so that you're not in this loop of scrolling social media and getting, you know, trying to follow along and having this feeling of you're productively kind of kind of getting new information, but really you're just stressing yourself out?
- [74:03]
B
Yeah, I mean, I think how to do that practically will differ from each person, but I think thinking about exactly the things you're saying there seems like very useful to think about. Like, how do you get information efficiently? Like rather than just generally scrolling Twitter, can you find five sources that you think cover the basics and just read those once a week or once at the end of each day? Some type of. Yeah, I think batching is a really big. I mean, it's hard to do in practice because this stuff is so addictive. But yeah, the more you can have periods of true rest where you're actually unplugged and then periods where you engage and how to do that will vary a lot by person. Like, do you want to have a kind of a Sabbath type day where you take one day fully off the phone each week, or do you prefer to say, like, you know, I sometimes go on meditation retreats and then take a whole week off totally unplugged? I think what type of routine works that will vary a lot by person.
- [74:58]
A
Yeah, I guess one issue here is that as I expect things to go, many things will feel like this is the one exception, this is the one emergency that you absolutely need to follow. But there'll always be three of those things happening at the same time. And so there's a question of how do you stick to the systems and how do you have a sense of proportion of how big of a deal many of the issues. Maybe I should be more concrete here. What I'm imagining is something like, you know, OpenAI announces a new breakthrough. You try the model, it's exceptional. Two weeks later, China decides to invade Taiwan. Now there's a new open source model that's perhaps better than the model from OpenAI. And it's just you're not able to sit down and understand what's going on before the next thing is happening. It seems that we're just not equipped to think about the amounts of information we're getting at the speeds that we're getting them in a productive way. So do you have to limit the information you get to an extreme degree in order to be productive?
- [76:07]
B
I mean, I think even you just saying all this out loud is already helpful for people. Just imagine this is the world we're going into, and then think about how you might respond to that at the time and also what could you do now to make yourself better prepared to navigate it? And I think in the thing you were just saying there, I think having some type of good information network would be very helpful. Like, ideally, you want to be able to just ask someone, okay, how good is the OpenAI model really? And then they can basically tell you. And then I think that's one big piece of navigating that type of thing. Yeah, I think the other one would just be, I mean, people do this now where they just follow random crises in the news that they can't do anything about, and then they feel bad, but it's not actually. And I guess this will just become a much bigger issue. But yeah, it's always asking yourself, what am I actually able personally to do for both my own goals and also from a social impact point of view and really trying to focus on figuring out those questions rather than just generally following things.
- [77:11]
A
I think that's very good advice. I had an experience like you just described with Russia's invasion of Ukraine, where I'm following along. I'm unable to do anything, but I feel like I need to follow along, and that's quite unpleasant. And also just not productive for the world. Not actually helping by following along. There's a lot of information out there about everything now, so it's easier to follow along by the minute in these kinds of situations.
- [77:36]
B
One particular thing on that is I find metaculus very useful for these types of things because often there's just like some kind of key parameter that matters. So, like, I think during Ukraine I was trying to figure out, like, what's the chance that London gets nuked? And there were kind of like forecasts that would look at this and I could see, like, if that was spiking up, then maybe I should leave town. But I could kind of like not follow the news besides tracking that forecast. And there's this really cool group called Sentinel, run by Nuno, who basically track a bunch of different potential catastrophes and then do a roughly weekly update on them.
- [78:10]
A
Yeah, that also seems useful to follow. So you get everything you need to know. You don't need to read headlines. You look at this number that, at least in theory, has kind of condensed all of the available information into one actionable number about how big of a deal something is.
- [78:29]
B
That one was actionable to me. But yeah, it would depend on if I was Trump, then I would be tracking very different metrics because I'd have different goals.
- [78:38]
A
Yeah, of course, of course. You also advise us to prioritize things you want to have in place before we get to AGI. As we're speaking, I'm still having trouble understanding what it is exactly. You mean there is it that you want to have certain experiences that are only available before AGI or what. What are the types of things that you would advise us to have in place before AGI?
- [79:04]
B
Yeah, I'm trying to just point at a very high level heuristic, which is if there's something that AI would be able to do much better than you in five years time, then you should delay doing that thing until those five years. I think that's a big thing. So, I mean, an example from my own career planning is I was wondering, should I write more about AI or should I write more about effect altruism is like an example you might have. And I thought, well, clearly I should write about AI now, because if we're about on the brink of intelligence explosion, that'll be super valuable. And if we're not, then I can always write about effect altruism later in the more normal timeline. And so that was like a case where I thought, yeah, it was better to delay the effective altruism case. I think maybe Another personal life example. This one's a little bit controversial, but if you think in a normal world, it's. If you would be kind of indifferent between having a family now and starting a family in five years, so many people aren't in that situation. Like, waiting five years would actually be a big cost. But supposing that you're in one where you're relatively neutral about that, then it does seem quite tempting to me to then delay, delay, make that delay. Because if we're in the AI soon world, there could be all these very urgent things you want to do to prepare, earn more money, or maybe you want to just work on AI safety and help. It could be the most impactful time in history. So it really makes sense to focus on social impact the next five years. And you might also want to see what's going to happen before having a family, because if it's. Try and get a better sense whether it's a good or a bad scenario. So that was one where I thought that type of thinking, you can think, yeah, what stuff is urgent will put me in a better position before AI versus things that theoretically could be done later, I think is an interesting thing to reflect on.
- [80:52]
A
And in that vein, there are also projects that it might make sense to abandon. So, for example, I think you mentioned this to me in preparation for this conversation about whether you should write a book or spend years writing a book. Maybe some of the same reasoning goes for whether it makes sense to start out right now trying to become a mathematician. I don't actually know whether the situation there is so extreme, but I could imagine a world in which AI in a couple of years is just fundamentally better than humans are at mathematics. And so this is also about abandoning projects, correct?
- [81:29]
B
Yes, exactly. Yeah, yeah. I mean, there could also be a role for the thing you just said, the kind of bucket list thing where if you think, well, maybe there is a chance that it does all go badly, and these are the last five years, maybe there's also some things you want to do before that. But yeah, I think there's a bunch of different framings here that are all useful to think about.
- [81:49]
A
Last question here. You write about how the intelligence explosion is likely to begin in the next seven years, and if it doesn't do that, it will take much longer and that we will have much more information about which world we are in in the next three years. Why is it we can say, we can make statements like that with such precision? Which curves or which trends are you looking at?
- [82:15]
B
A key thing is Most fundamentally, AI progress is being driven by there being more compute, because more compute means you can run more AIs, you can do bigger training runs. It also means you can do more experiments to improve the algorithms and then secondly by more labor going into AI research. So more AI researchers, human ones, both of these things are increasing very fast now, and we're getting very fast AI progress. But if you look at projecting these trends forward, basically around 2030, the exact time depends on the bottleneck, but let's say between 2028 and 2032, it just becomes very hard to maintain the current pace of increase of both of those things. So basically, the amount of compute and algorithmic progress we have will start, start to flatten off around that point. It could be quite a gradual speed slowdown, in which case it could last well into the2030s, but at a slower rate. Or it could be a relatively like quick diminishing, say if just profits aren't large enough on the AI models, people might be like, well, we're not gonna buy, not gonna buy the next round of chips to scale, so we're stopping here. That could also happen. But yeah, just. It's kind of, it's a bit of a weird. Bit weird in a way that all of these things seem to. All of the bottlenecks to seem to roughly line up around 2030. I think current rates can be sustained for the next four years relatively confidently and then the kind of four years after that. So 28 to 2032, less clear, probably slowing. And yeah, there's some precision around that. I mean, there is also maybe we just get another paradigmatic breakthrough like deep learning itself, and that maybe that's better thought of as something that could happen at any time. So maybe if we got that in 2030, then maybe everything carries on for another while, but in a new paradigm.
- [84:00]
A
So it's basically either the current paradigm stagnates and we can see that it's not sustainable to keep giving the inputs to the scaling that we're doing now for many, many more years. So either the current paradigm stagnates or we get something like an intelligence explosion rather soon.
- [84:21]
B
The chance of finding a new paradigm depends on how many people are doing AI research. So to some extent, that just fits into this model where if we have exponentially increasing AI research workforce, then the chance of finding a new paradigm is roughly constant per year. But if the workforce stops increasing, then also the chance of finding a new paradigm decreases a lot too. Just to make the point about compute more concrete GPT6 probably costs about maybe 10 or $30 billion to train. It will cost that in 2028. That seems like we're pretty much quite close to having chip clusters that will be able to do that training run, just given what's already in the pipeline. But then, you know, going to GPT7 would then cost another 10x more. So then we're talking about over a hundred billion dollars, which is like still affordable, but is getting much harder. That's kind of like a whole year of profits from Google to fund that.
- [85:17]
A
One training run in the scale of the future of human civilization. It's not that much money. Yeah.
- [85:25]
B
I mean, interestingly, it would be bigger than the Apollo program and as a percentage of gdp, maybe it's kind of getting up to Apollo and Manhattan program levels. The thing that maybe there's a few other things that could stop you. So by that point, pretty much all of TSM's leading Taiwan semiconductor, their leading nodes will be used for AI chips around by then. And then that means we can't create more AI chips unless they actually build new factories, which isn't the case now. The case now they're just replacing mobile phone chips for AI chips. So that can be done very easily. You'd also be going like, we'll be something like 4% of US electricity would be used on data centers say in 2028. But then if you want to go another 10x, you have to go to 40% of US electricity on. So you had to build a lot of power stations, which is totally doable. Like you can just build gas power stations in two or three years. Yeah. And there'll be huge economic incentives to do it if we're on this trajectory. But it's like, it's definitely becoming a lot harder than is now. Each, each order of magnitude of scaling.
- [86:32]
A
It's exciting and it's scary to see what's going to happen here. Do you want to refer listeners to your substack? How can they find out more about what you're thinking about?
- [86:42]
B
Yeah, following my substack or on Twitter is the best place to stay up to date and what you can do about AI Guide I'm writing will be published, but also I'll be writing about a lot of the other topics we've talked about.
- [86:55]
A
Fantastic. Thanks for chatting with me. It's been a lot of fun.
- [86:59]
B
Great. Yeah, thanks for having me.