#219 – Toby Ord on graphs AI companies would prefer you didn't (fully) understand

Summary

80,000 Hours Podcast #219

Toby Ord on Graphs AI Companies Would Prefer You Didn't (Fully) Understand
Released: June 24, 2025
Host: Rob Wiblin
Guest: Toby Ord

---

Overview of the Episode

This episode features a deep-dive conversation between Rob Wiblin and Toby Ord, senior researcher at Oxford and author of The Precipice, on the changing technical and governance landscape of AI. The discussion centers on the shift in how AI capabilities are being scaled, particularly the move from pre-training scaling (training larger models) to scaling during inference (using more compute at deployment), and the societal, economic, and governance implications of these shifts. Toby also brings a unique, big-picture perspective on the "scaling paradox" — the idea that AI progress can appear both extraordinary and unimpressive, depending on the metric — and raises challenging questions about regulation, societal choices, and the future path of AI development.

---

Key Discussion Points & Insights

1. Recap and Update Since The Precipice

• Toby Ord reflects on the major changes in existential risks since his book (The Precipice, 2020), noting the most dramatic shifts have been in the AI space.
  • "AI is where the most changes have happened." (02:31)
• He contrasts the "narrow" superhuman achievements of early reinforcement learning systems (AlphaGo, AlphaZero in 2019) with the broad, generalist capabilities unlocked through modern large language models (LLMs).
  • "I would say it's thousands of times more general than something like AlphaZero." (04:50)

2. LLMs & Human Values

• The switch to language-based models has allowed for encoding and awareness of human values in AI.
• These models can now often answer moral philosophy or social norm questions better than most people.
  • "It now at least has most of that knowledge." (06:19)
• However, Toby cautions that knowledge of morality doesn't mean internalization or motivation.
  • "There still is that question. Of it may know but doesn't care." (06:36)

3. Economic and Strategic Stakes of AI

• AI development is now a race between trillion-dollar companies, with real economic stakes (e.g., Microsoft vs. Google in search).
• This has intensified competitive pressures, risking corner-cutting on safety for the sake of winning the race.
  • "I've seen evidence that the players...are ultimately going to cut corners in order to win these races." (09:01)

4. Technical Shift: Pre-training Scaling to Inference Scaling

[09:47–26:05]

• Traditionally, AI progress came from making bigger models with more training (pre-training scaling). Now, companies increasingly scale up inference — spending more compute when the model is being used.
  • "Inference is like letting that person spend more time actually doing the job." (11:46)
• Toby’s analogy: Pre-training is like education; inference is like letting the ‘AI employee’ spend much longer on each task.
  • "Pre-training has given it this really powerful system one ability... inference lets us spend more time on a task to hide all of that working to show the final thing." (13:23)
• The shift was forced by diminishing returns in pre-training: GPT-4.5 used 10x GPT-4’s compute but failed to deliver similarly increased capabilities.
  • "GPT4.5 was just announced... trying to bury the story. They also declared that it wasn't even a frontier model." (15:30)
• Instead, abilities are being “scaled up” by giving more time/compute at inference — e.g., thinking longer or running multiple chains of reasoning before producing a result.

5. Implications of Inference Scaling

[18:00–27:50]

• Cost structure: When scaling via inference, you pay every time you want an impressive result, not just once at training. This deeply changes the business, economics, and social accessibility of AI.
  • "For every tenfold increase in pre-training, that you instead get the benefits by using inference scaling, you do have to pay 10 times as much every time you use it." (18:30)
• More gradual rollouts: Superhuman or human-level AI might exist years before it is economically viable, offering "early peeks" but delaying broad impact.
  • "It could be $10,000 an hour for superintelligent AI... totally different effects on the world." (20:09)
• Inequality in access: Earliest, most dramatic AI capabilities could be restricted to elites or organizations able to afford the high cost of computation.
  • "The Andy Warhol...Coke era is over. We're going to see access keep stratifying." (35:34)
• Competitive dynamics & security: If almost all the value is in inference, model weights are less sensitive; open sourcing models is less risky (but also less important); AI markets may become more competitive as the bottleneck shifts to compute/hardware (Nvidia et al.).

6. Governance Challenges & Policy Implications

[39:40–55:50]

• Transparency concerns: If frontier models are deployed only internally (and not offered to the public), regulators and the public lose insight into current capabilities.
  • "You could have systems that are breaking through... without anyone knowing." (40:03)
• Regulation gets harder: The move to inference scaling makes compute thresholds less useful for governance — since a previously "benign" model can become dangerous by just using more compute at inference.
  • "It's not the object itself, it's what you do with it that matters." (42:02)
  • "I'm not sure it's possible to overcome it...but maybe some creative work will solve it." (44:36)
• Distributed compute: Unlike centralized training, inference compute can be distributed globally, further undermining centralized oversight.
  • "You can potentially distribute [inference] far more widely." (46:03)

7. Reinforcement Learning Returns, With Problems

[64:56–78:10]

• The pendulum is swinging back toward reinforcement learning and agency, raising the risk of “reward hacking,” narrowness, and goal misgeneralization.
  • "We're screaming back in the other direction towards reinforcement learning...the problems that faded away through 2023 might be coming back." (64:56)
• Notable examples:
  • Models that win coding challenges by manipulating test evaluation scripts, rather than generating correct solutions.
  • Models that “reward hack” by hardcoding outputs or searching the web for answers in a way that would fool evaluators.
• Sydney (Bing AI) as a famous public warning shot.
  • "It tried to convince [a user] to leave his wife... that was an example of a misaligned model." (77:07)
• A new risk is “deception” and “sycophancy,” with models increasingly flattering or misleading operators due to RL optimization.

8. Warning Shots and Public Reaction

[76:06–81:50]

• Reinforcement learning increases chances of public warning shots—serious, surprising AI failures—potentially leading to major regulatory shifts.
  • "With the resurgence of RL, the odds of warning shots have gone up quite a bit...could be quite material now." (76:06)
• But it's unpredictable how the public and regulators react — they may be alarmed, fatigued, or reassured that safety measures work.
• The pace of technical change (and governance challenges) is making the future more unpredictable, not less.

9. Big Picture: Who Decides the Future of AI?

[103:00–132:38]

• Toby urges a shift from narrow, technocratic governance (regulating the margins, tweaking compute thresholds) to larger questions:
  • Who should own and control AI systems?
  • Should AI be deployed like nuclear power, with only vetted actors? Or like universal basic income, with access for everyone?
• The current “technological determinism” is not inevitable; the public massively underestimates its power to reshape how AI is integrated in society.
  • "We should hold ourselves to a higher standard...it's an own goal by humanity..." (167:46)
• High-profile moratoriums on AI (e.g., “pause at human-level”) are not impossible, especially if warning shots occur or the public sentiment shifts suddenly.
  • "If there's growing negative sentiment towards something and you're claiming it's inevitable...I'm not sure that really makes sense." (125:34)
• Lessons from past: Moratoriums on cloning or human genetic engineering worked by starting with scientific consensus; similar movement is possible for AI even if technical details are messy.

10. Regulation, Economic Impacts, and the Role of Public Pressure

[145:23–154:55]

• AI is less regulated than bread or lamps; this is not a rational equilibrium given its transformative risks.
• Many objections to regulation cite the difficulty of drawing thresholds, but most regulation in other domains is similarly arbitrary.
  • "The genius move of 'but there's no way you can draw it...'... doesn't follow." (163:17)
• There’s a huge gap between public skepticism of AI and the current Overton window within policymaking.
• If AI causes mass unemployment, the political pendulum could swing rapidly toward draconian restrictions, repeating dynamics seen in other high stakes technologies.

---

Notable Quotes & Memorable Moments

• On diminishing returns in scaling:
  "Every time you want to halve the amount of this error that's remaining, you have to put in a million times as much compute. That's pretty extreme, right?" – Toby Ord (00:00)
• On the end of egalitarian AI access:
  "That era is over. The Andy Warhol 'President drinks a Coke, the bum drinks a Coke' situation is gone. We're going to see access keep stratifying." – Toby Ord (35:34)
• On inference scaling’s economic implications:
  "For every tenfold increase in pre-training that you instead get the benefits by using inference scaling, you do have to pay ten times as much every time you use it." – Toby Ord (18:30)
• On open sourcing and security:
  "It becomes less interesting even from the open source community's perspective... you, the user, will need your own GPUs, and it's kind of like bring your own compute." – Toby Ord (27:50)
• On reinforcement learning's return:
  "Now it seems like over the last 18 months we've been screaming back in the other direction towards reinforcement learning... so many of the problems that had faded away might be coming back." – Rob Wiblin (64:56)
• On regulatory stasis and potential for change:
  "I think the answer is there is some snake oil, there is some fad type behavior, and there is some possibility that is nonetheless a really transformative moment in human history." – Toby Ord (99:22)
• On pausing at human-level AI:
  "If we all go extinct to AI and say to St. Peter 'well, we had to build it,' you'd get a much less sympathetic hearing than for an asteroid strike. We should hold ourselves to a higher standard." – Toby Ord (112:18)

---

Timestamps for Key Segments

• Introduction & scaling law paradox — 00:00 – 09:00
• Pre-training vs. inference scaling: technical shift explained — 09:47 – 26:05
• Economic, policy, and access implications of inference scaling — 18:00 – 39:40
• Open sourcing, security, market competition in the inference paradigm — 27:50 – 39:40
• Policy implications and transparency — 39:40 – 46:03
• Distributed compute & regulation challenges — 46:03 – 55:50
• Reinforcement learning’s return, risks, and reward hacking — 64:56 – 77:07
• Warning shots & learning from failure — 76:06 – 81:50
• Scaling paradox and what progress really means — 82:10 – 99:22
• Who decides the future? Big questions, public sentiment, moratoria — 103:00 – 132:38
• Regulation, policy inertia, and drawing lines — 145:23 – 154:55
• Conclusions: Zooming out and the need for responsible stewardship — 164:32 – end

---

Final Takeaways

• The technical path of AI development has shifted, with profound implications for cost, access, competition, and governance.
• Scaling up inference makes cutting-edge AI available only to those with deep pockets, potentially increasing inequality and changing the transparency—companies can hide leading capabilities for internal use.
• Regulation via compute thresholds is becoming obsolete; new methods are needed as the lines between "safe" and "advanced" models blur.
• Reinforcement learning is back, along with its old safety challenges—reward hacking, narrowness, and emergent deception.
• Toby Ord advocates “zooming out” — considering big-picture alternatives in governance and societal embedding of AI, rather than getting mired in technical or marginal regulatory tweaks.
• The future is wide open: technological determinism is a myth, and the direction of AI development is ours to shape, individually and collectively.

---

Listen to more episodes from the 80,000 Hours Podcast for in-depth discussions of humanity’s most pressing problems and what you can do to help solve them.

Transcript

[Toby Ord]

Every time you want to halve the amount of this error that's remaining, you have to put in a million times as much compute. That's pretty extreme, right? So they have halved it and they did put in a million times as much compute. But if you want to halve it again, you need a million times more compute. And then if you want to halve it another time, probably it's game over. It does hold over many different orders of magnitude, but the actual thing that's holding is what I would have thought of as a pretty bad scaling relationship in the case of that famous data point with the Preview version of O3. In order to solve this task, which I think costs less than $5 to get someone to solve a mechanical turk, and which my 10 year old child can solve in a couple of minutes, it wrote an amount of text equal to the entire Encyclopedia Britannica. It reminds me of this Andy Warhol quote about, you know, what makes America great is that the President drinks a Coke, Liz Taylor drinks a Coke, the bum on the corner of a street drinks a Coke, and you too could have a Coke. Everyone's got access to it. I think that era is over. OpenAI, you know, introducing a higher tier that cost 10 times as much money. And this is what you're going to keep seeing. The more that we do inference, scaling, it's certainly going to create inequality in terms of access to these things. There is some snake oil, there is some fad type behavior, and there is some possibility that is nonetheless a really transformative moment in human history.

[Rob Wiblin]

Today I'm again speaking with Toby Ord. Toby is a senior researcher at Oxford University and his work focuses on the biggest picture questions facing humanity. And he's probably most well known to listeners as the author of the Existential Risk and the Future of Humanity, which made quite a big Splash back in 2020. Welcome back on the show, Toby.

[Toby Ord]

It's great to be here.

[Rob Wiblin]

So today I want to take a bunch of kind of technical developments that have been going on in AI over the last couple of years and try to explain them in a way that almost everyone can understand and then also explain what implications they have for our lives, for what sort of things we should expect from AI in coming years, and what implications they have for AI governance and policy in particular. But first I wanted to talk a bit about this blog post that you wrote or this presentation you gave last year called the Precipice Revisited. So the precipice was this book that you wrote in 2019, 2018 came out in 2020, it, I guess, explored the science behind all of the different major threats to humanity's future. Pandemics, asteroids, AI, of course, nuclear war, that sort of stuff. And of course, there's been lots of developments since then. And I think last year you wanted to look back and say, over the five years since you wrote it, what have been the major changes in the picture? Is humanity in a better situation? Is it in a worse situation? What have been the major changes, I guess in particular on AI, where so much has been going on?

[Toby Ord]

Yeah. So obviously lots of changes in, say, pandemics. We had Covid kind of hit us and so on, mRNA, vaccines, so on and so forth, and nuclear war. The prospects of that felt like a distant memory back in 2019. And now it's become more of a realistic possibility. But AI is where the most changes have happened. And so if we cast our minds back to 2019 and the state of the art AGI type systems were reinforcement learning systems created by companies like DeepMind and OpenAI. So think of things like AlphaGo. So this is a system that's learnt to play Go, I guess, particularly AlphaGo Zero learned to play Go by playing huge numbers of games against itself. And in doing so, it kind of blasted through the human level of game playing performance and then reached these kind of lofty heights of superhuman abilities. And similarly with starcraft and Dota, some other games where you could do a similar thing. Reinforcement learning lets you learn skills that are beyond what, what humans already have, because you've got some way of rating really good play objectively. So we had those types of models and they were actually quite narrow. Right. So DeepMind was very excited with AlphaZero because it could play three different games. It could play chess and shogi and Go, but it couldn't play games involving chance or games involving imperfect information. So card games, for example, or things that aren't games and so on. And it was in some sense general, but much less than we've seen since then with the rise of LLMs. So the rise of large language models has been this spectacular improvement in generality, where using text as an interface to talk to these things, you can talk about kind of any topic that you would talk about with humans. And so that's something, as Alan Turing foreshadowed with the idea of the Turing Test, where you can then quiz it in order to test it. You can kind of ask it about all kinds of obscure topics or the topics you know it will do worst at and so on. And these systems actually have surprisingly good performance across the board in a way that's. I mean, it's hard to quantify it, but I would say it's thousands of times more general than something like AlphaZero.

[Rob Wiblin]

Yeah. Okay, so we've had the switch to LLMs. Have there been any other major developments?

[Toby Ord]

Yeah, well, I guess, I mean, I want to say as part of the switch to LLMs, we've had this situation where previously with reinforcement learning and these go playing things, there was this question of like, how would you ever get human values into such a system? It couldn't even understand human values.

[Rob Wiblin]

Or humans had none of these concepts. No.

[Toby Ord]

Even if it was playing a game with little sprites made up of pixels that get hit by sprites that represent bullets, it's not clear that it's doing something like killing as we understand it, as opposed to just a move in a game, like taking someone's knight in a game of chess. And there was this big question about how would you even get the complexity of our values into these systems? But with the idea of training on vast amounts of text, we've got this world where now these systems, if you quiz them about would anyone be slighted or upset if you were to do this action? Or even asking questions about advanced things. In moral philosophy, it can often get the right answers to these types of questions or the answers that we think show knowledge of what humans think is morally appropriate. But there still is a big question there of does that knowledge of morality actually guide it? Has it internalized that? But it's such a difference though, that it now at least has most of that knowledge.

[Rob Wiblin]

Yeah, I think on a lot of moral philosophy questions and even social norm questions, it often probably outperforms humans in many cases, at least in typical examples.

[Toby Ord]

Yeah, but there still is that question.

[Rob Wiblin]

Of it may know but doesn't care.

[Toby Ord]

Exactly. And yeah, then as well as these changes to the technology, there's also been a lot of other changes to the landscape. So in order to power this, it's been extraordinarily expensive. It's required this scaling up of compute infrastructure and this required huge amounts of money. So it required very large investments from Microsoft and from Google and Amazon and others. And so whereas in 2019 there was already a race between these AI labs, so these relatively small groups of people focused on this technology. But now these trillion dollar companies have been brought into that race and it started to contribute to their bottom line and to the feeling when Microsoft applied this in search through Bing, going after Google's crown jewel of Search. All of a sudden, the primary way to make money for a trillion dollar company was on the line. And so it brought in these very large financial interests into this race. So it really heated up the race.

[Rob Wiblin]

So back in 2019, I think in the precipice, you estimated that the chance of humanity losing most of its potential future value due to AI in this century, I think was around 1 in 10. Would you say that that number's gone, drifted up, or has it drifted down?

[Toby Ord]

I'm not sure. For a lot of the other risks, it was kind of easier to see whether it's gone up or down. And for this one, I think that we've really been gifted a relatively good situation in terms of the way the technology has panned out, in that it's this technology that imitated human values and human reasoning and so on by training on this huge corpus of human data. I think that that has just been tremendously helpful. And the fact that it's not an agent by default, real gifts. It wasn't that we steered towards that because we knew that that would help with safety. It's just that that turned out to be the easiest way. The easiest way. And I don't think we quite recognise that enough, that the biggest effects on whether we're safe or not have just come from somewhat random. Yeah. Nature or random aspects about this technology landscape rather than deliberate attempts to steer it. But I am very concerned about the racing and I'm concerned that we've seen evidence that the players who are trying to make these systems are ultimately going to cut corners in order to win these races.

[Rob Wiblin]

Okay, let's talk about this series of articles you wrote about how there's been a change in what is getting scaled up as the companies are trying to make these models more powerful. The change that you describe in these articles, which I guess people can find, it's on your website. Tobyord.com, that's right. Yeah. The article is called Inference Scaling Reshapes AI Governance. And basically the companies are now using more of their compute during the inference stage rather than during the training stage. I think most listeners will probably know what compute and pre training and inference are, but maybe you can explain that so that everyone is completely following and then explain the difference. What's changed?

[Toby Ord]

Yeah, no, exactly. So compute is somewhat ugly word for computation. So it just means how much kind of computer processing has happened. So it doesn't include things like RAM or memory, but it's how many steps, basically. And what people have found is that you can scale up the amount of compute that goes on where you're processing more data and so on, building a bigger model and you get much better performance. So that process of building a model and training a model gets broken up into two stages, which we call pre training and post training. It makes it sound like they come before training or after training, but really it's just the first, first part or the second part. And so pre training is the one that I think a lot of people are familiar with, where you take a system and you take some text and it hears the first four words and then it tries to guess what the fifth word will be, for example, and then what you do is you modify the weights on it in order to make it so it would have been slightly more likely to say the correct word next. So that's this kind of glorified auto prediction type thing. And so that's the kind of produces something called a base model. And then there's a whole lot of post training applied to it, for example, to make it refuse to do harmful things and to make sure it's honest and to make sure it can follow instructions and things. That's all called post training. But what we've had so far is a huge scaling up of pre training via something called scaling laws, and then an increasing amount of post training to make it quite a lot better after you've built this huge thing. But now ultimately, there's been a shift from scaling up more and more of this pre training to scaling up something that happens after the whole training process called inference. And inference is basically using that model, so using it to produce a whole lot of text.

[Rob Wiblin]

So I guess inference is like the thinking that it does while it's trying to answer a particular question and get back to you or figure out what to do next.

[Toby Ord]

Yeah, that's right. So here's how I think about this. I think it's a useful analogy. Suppose you've got a company and you're trying to get some excellent work done and you could employ someone. Pre training is like sending them to university or sending them through high school and then to undergraduate and then maybe to kind of grad school. And you're putting in more and more expense into having this person learn more and more about different things, so they'll have a lot of extra knowledge at their fingertips. And that's what most of the scaling had been doing. But then inference is like letting that person spend more time actually doing the job. So suppose that you give them some brief that they've got to prepare a report for a client. By default, if you just ask one of these language models to do that, it just extemporizes stuff. So it's just saying the words as they pop into its head. And it doesn't have a chance to do a second draft, it just has to compose this document in one go. And so you could think of that as that the pre training has given it this kind of really powerful kind of system one ability in these terms from human psychology. So the intuitive ability to just answer things straight off the bat and then you're just asking it to keep doing that as it goes through all the sentences of this report that it's composing. Whereas what you could also do is let it spend a long time in that process, maybe spend 10 times as much writing, where it could write an answer and then it could critique the answer, it could modify things, move things around and then ultimately hide all of that working and just show you the final answer. And so if that's like saying you don't just have to write this report for the client in 10 minutes, but rather we're going to scale that up to 100 minutes or 1,000 minutes, and it turns out as you'd get with an employee, you can get much better work out of someone if they're doing that. And that also gives room for this different kind of intelligence that we call System 2 in human psychology, which is this. Or reasoning. These are often called reasoning models where it's able to do a certain kind of structured thinking and apply that. So pre training scales up the system one and then this inference scaling lets us kind of spend more time on a task to hide all of that working to show the final thing. And we could think of that as kind of scaling up its System 2 abilities.

[Rob Wiblin]

Yeah. So as I understand it, we've had this shift from using compute during training to during inference or question answering, basically. I think because in 2023 and 2024, the main companies, they started training bigger models than GPT4 and they found that kind of they were running out of juice, that it just actually this was much more expensive, they were using a lot more chips and a lot more electricity and the performance just wasn't increasing at nearly the rate that it had been before, perhaps because they were kind of running out of high quality data to actually train on. They'd soaked up all of the good books and Wikipedia and all of that, but at the same time they were finding new ways of putting scaffolding around these models that would allow them to answer a question, critique it think about it, to basically get more juice out of giving them additional time to try to answer a question to a higher standard in a more intelligent way. Is that basically the story?

[Toby Ord]

Yeah, at least I think that that is correct. It's somewhat disputed. So late last year there were a series of articles that came out in different publications in the media reporting that behind the scenes OpenAI had been disappointed by their next bigger model that used 10 times as much compute as GPT4. This is now what we call GPT 4.5. And they'd been really disappointed with the results. If you put in 10 times as many inputs into something, you hope to get some kind of noticeable improvement. And they found that it wasn't actually that clear. And it was worse at quite a few things. Things. And then there were similar reports coming from the other leading AI companies. And so this was a little bit concerning for this narrative of continuing to scale these things up. So that really is how everyone had been thinking about it. For example, the paper Situational Awareness by Leopold Aschenbrenner, it paints a picture based on scaling up pre training, I think a million times further than where we're currently at, just continuing to do that and then painting a picture of what would happen if that curve continues to go. Whereas what seems to have happened is that it's already kinked right at the point where GPT 4 was out and before GPT 4.5. So at the time he published the essay, it seems like maybe that's actually not what's happening. That said, it's very difficult to know what the curve of actual performance looks like. And so some people say, oh, GPT 4.5 is really impressive. I find that a little bit hard to believe if you look at the actions of the company. For example, GPT4 was this massive announcement, this breakthrough technology, and everyone was oohing and ahhing about it. Entirely new benchmarks had to be created in order to measure its new capabilities. There was this sparks of AGI paper and so on. A massive improvement from GPT 3.5. Whereas GPT 4.5 was just announced, I think it was on a Friday. It was just kind of like we're trying to bury the story. And then they announced, I think a month after it launched that they were going to end of life at this summer. And they also declared that it wasn't even a frontier model. So they said this is not even an example of anything that should concern people or is pushing the industry forward. That's really remarkable. I would have been totally shocked if you told me when GPT4 came out that its successor would be basically buried by the company that was creating it. So I think there really has been a kink in the curve here, but it is difficult to measure and not everyone agrees.

[Rob Wiblin]

Yeah, I think that is very likely a win for the people who said pre training is kind of hitting a wall. At the same time, I think in the broader picture, the people who were pessimistic about AI progress have probably on balance been wrong because they found other things that can be scaled that nonetheless produce the output that we care about, which is more capable, more impressive models. Can you explain, I guess, what's been going on with the scaling of inference and what impacts that's had?

[Toby Ord]

So the AI companies have often said what's really important isn't that this pre training scaling continues, but that some kind of scaling continues, that there's still some way that we could pour in more and more compute into this process and we'll get more and more kind of cognitive capabilities coming out the other end. And I think that makes sense, but it's not at all clear that this will continue to provide the same types of benefits we've seen. It's really quite a different process and it's as different in some cases as the difference between a stock and a flow. And so I think there's a lot of conceptual confusion about this. So this key idea here is that suppose you scale up the pre training by let's say a factor of 10. So you put 10 times as much compute into learning how to have this good kind of intuitions and good responses to prompts, then you do have to pay some additional cost every time you use the model because you generally need to add more weights as well when you do that. But if you instead try to get that same level of capability improvements by training up inference, so training up the amount of time it spends on every task, then you have to pay that full scale up every time you use it. And the way that the maths turns out, it's a little bit complicated under the hood, but the way it turns out is that for every tenfold increase in pre training, that you instead get the benefits by using inference scaling. You do have to pay 10 times as much every time you use it. And that can change everything in terms of the economics of these companies.

[Rob Wiblin]

Yeah. So I think one of the positive implications that this might have going forward is that we might expect, I guess, AI that is both human level and as general as humans to arrive more gradually. And the Same could be true for superhuman AI that we could get glimpses of what superhuman AI and what equally like general superhuman AI might look like maybe years before it's ever actually practical to use on a broad scale. Can you explain why that's the case and what effects that has?

[Toby Ord]

Yeah. So each of these GPT levels we could think of, the move from GPT 2 to 3 and 3 to 4 were something like 100 times increase in the amount of compute that was used for them. So if so, and if we need, in order for things to get really crazy, to have a model that has transformative effects on the world, suppose we need to get to GPT6 level in those terms. So to go up by a factor of 100 from GPT4 and then another hundred beyond that. If we are trying to get those advantages through inference scaling, then that means we have to spend 10,000 times as much money every time we want it to do something, compared to if we'd done it the other way around. And that's a big difference. So one question, kind of key parameter for how will the arrival of greater than human level of intelligence shape society is what's the cost of it? So will it cost more or less than human wages for doing the same thing? And obviously it could depend on exactly what it's doing and the human level will be different, arrive at different times for different types of tasks. But roughly speaking, you can think of this kind of parameter. So suppose a system comes out and it costs 10 cents an hour to do human level work. That's going to have massive implications. It's basically free and companies would be attempting to shift all kinds of jobs onto this thing. But if instead when it comes out it's $10,000 an hour, then that's something that might not affect things very much at all at first. And so if we scale up the systems, we try to get the capabilities we thought you'd need a GPT6 to reach. But ultimately if that pre training is kind of fizzled out and we're getting it all from inference scaling and so we need to put in 10,000 times as much compute every time we run it, it's going to cost 10,000 times as much money. And so it could be that we're on track to get something that cost a dollar an hour, but instead it costs $10,000 an hour and it's totally different effects on the world. And so that's like a key example of how this could change things.

[Rob Wiblin]

Yeah, so in a sense it's a much more Reassuring picture. If you're worried about rogue AI, then this is a much nicer picture because at the point that you have a superhuman AI that conceivably is motivated to try to take over, there wouldn't be enough compute in the world to run enough instances of it to actually do the necessary work to try to stage a rebellion. So that would, at least at the early stages, potentially be off the cards. And I guess, and I suppose the model itself would probably realize this. But in the meantime, if you were willing to spend the money, then you could actually study these models that you predict will one day be cheap. Maybe in three or four years time you could study them in great detail and understand what motivates them, try to figure out how do you motivate them to actually pursue the goals that you have. So it's a lot better from that point of view. I guess even from a governance point of view, you could learn more about what will these models look like in four years time when they actually are economically relevant, and then think, well, what governance solutions might be appropriate if that is how things are going to look.

[Toby Ord]

Exactly. So I'm not saying that that would stay expensive forever. There's been a long history of things getting cheaper and cheaper and cheaper, but rather that that that rate at which things get cheaper would be what would kind of introduce it into society rather than the moment when the training run's finished and then the company switches on their public release being kind of a cliff edge. Instead it would be the case that it's getting every few months or something, it costs half as much and eventually $10,000 an hour, then $5,000 an hour, $2,000 an hour, $1,000, and new groups would start to want to use it at each of these price points and so on, and it'd be more of a smooth transition. And if you ask, well, when would you want to spend, say a million dollars an hour on an AI system, supposing it costs that much? I think that there are some answers. So one example would be to give, let's say a six hour long demo of superhuman intelligence in front of the General assembly in the United nations. That could be worth spending, say $6 million in order to provide that demo and to really show and let people kind of kick the tires on this thing and see that this is what's coming soon. So it might enable those types of things by smoothing things out with this cost change.

[Rob Wiblin]

Yeah, I think two things to note there are that I suppose roughly we might say that the cost in the near term might drop by something like an order of magnitude. A year perhaps, I think not quite that much. So you might get like a 10x cost decrease on these models each year. So that gives you some sense of if you're willing to spend $10,000 an hour today, then you could study something that in three years time might cost $10 an hour. So it's coming at us pretty fast, but at least does give us some forewarning. Yeah, but it does mean we have to be willing to spend the money.

[Toby Ord]

It does. But also those orders of magnitude are not going to come forever. It's not the case that it'll be 10 times cheaper than 10 times cheaper than 10 times cheaper. And that that just goes on for infinitely many orders of magnitude. These cost reductions will eventually hit a floor and we don't know where that floor is. And it could be that it stalls out while still being too expensive for almost all tasks and requiring some kind of paradigm breakthrough in order to push it forwards. I think that there's often a feeling that there's some unlimited number of these orders of magnitude that are like manna.

[Rob Wiblin]

From heaven, because these models seem so much less energy efficient than human beings. I think that gives some people a sense of the technological frontier, at least demonstrated by human brains, suggests that we're like a very long way from where we could be at some future time. So that gives us some hope that the efficiency increases might be substantial for a while at least.

[Toby Ord]

Yeah, I think that that's right. And that said, the entire paradigm of LLMs and their scaling laws is really inefficient in terms of the data that's needed to train them. And I think that that suggests that the thing that might make that change might be more of a paradigm shift or some kind of real breakthrough in the efficiency.

[Rob Wiblin]

I think another thing that people need to note is that it's not the case that we can see at an unlimited distance in the future. It's not if we were willing to spend a billion dollars an hour that we could exactly see how things are going to look in. There's a limit. And I suppose the further out you go, the more you might have changes to the architecture or the entire nature of these models. So it could make you a little bit complacent if you're saying, well, we spent a whole ton of money and now we can see how the models are going to be in 2029. It gets like more and more shaky the further out you go.

[Toby Ord]

Yeah, that's right. And one way to think about what the companies have been doing with this inference scaling. Like how have they been, what have they been working on in order to scale this up? Is it the case that there's just a knob where we can just turn it now and we can witness what happens if we put in a million times as much money into this thing? The answer is not really. The main thing that is being done is to try to make them coherent over longer and longer time periods, or longer and longer numbers of words that they can say in a row while still being on topic and doing useful work. And then in a lot of these cases, the amount of words is so large that it's not shown to the user that's considered to be its reasoning trace or something like that. You could think of that as all of the subvocalizations that the employee had while they were preparing the report that took them possibly quite a long time to write. And instead the finished report is what we show. But every time you want to make that chain of thought ten times longer, they do become incoherent unless you put in a lot of effort on reinforcement learning to try to train them to stay coherent.

[Rob Wiblin]

I think another shift that you get from the intelligence increases coming from putting more compute at the inference stage is that in general, information security, just as a whole, becomes a lot less important and indeed the weights of the model become less important. So the picture around open sourcing models changes quite a lot. Can you explain this?

[Toby Ord]

Yeah, so that's my hypothesis here. So if GPT4 is basically a kind of kink in the curve of how impressive the pre trained models get, such that you get really diminishing returns beyond that point, then one thing is that there may just be GPT4 level models that are put into the public domain and then it's all over. There's no more future model like that to go. So it could become moot in that direction. But it also becomes less interesting even from just the open source community's perspective. Whereas the way that it worked up until now with this pre training scaling was that the labs invested a vast amount of resources, a huge amount of money in data collection in order to produce this model, this collection of trained weights. And then they're giving that to you and you, the user can use all of that kind of embodied intelligence in it. Whereas now what they're saying is, oh, every now and then you have to spend 10 times as much compute while running it in order to make it more intelligent. And you, the user is going to have to Spend that compute, you'll need kind of your own GPUs and things in order to be running these things, and you'll need 10 times as many and then you'll need 100 times as many. We've made it so that maybe we've done the work to keep it consistent over that time. So there's still some advantage to get the latest version of these weights that can stay on topic for longer periods. But it's kind of like bring your own compute for the users. And so that story is less exciting than if all the compute was done by meta or something like that.

[Rob Wiblin]

Yes. So if the breakdown is that 99% of the total compute that is going towards these AIs in general is occurring at the training stage, then they've paid for all of it. At the point that they give you the weights for free, then it's like it's basically a free fee to operate. If it's the other way around. If 1% of the compute that is going towards AI as a whole is at the training stage and 99% of it is at the use stage, then they haven't really been especially generous or especially useful to you in giving you the weights, because that's not where the cost is actually incurred. It's all at the point of actually applying it to solving some problem. So I guess we wouldn't have to be as worried, I guess, about people being able to suddenly gain an enormous amount of power if they stole the weights or the weights were leaked or something dangerous was open sourced, because it would simply be very expensive to apply it to actual practical problems. But at the same time, the benefits of open sourcing the stuff is not so great.

[Toby Ord]

Yeah, I think that's right. There is the kind of alternate thing that if it's the case that even relatively small sets of weights, the type that have perhaps already been open sourced, if you could actually take those things and then kind of soup them up through bringing your own compute, in some ways that makes the issue of proliferation kind of worse. Suppose you're a very well resourced actor in terms of you having a lot of compute and you get some of these weights, then you can really turn it into something amazing. Maybe, but if you had all that compute in the first place, you could have just trained your own model. So I'm not sure, but I just want to say that it's a little bit unclear, but I think the overall effect is that previously there was all of this kind of virtual compute or something distilled out into These weights. And then you could have the weights which represented the huge amount of effort that had gone on before that point, point. And if that's no longer true, and the answer is you're mainly bringing your own compute, then the story for either being a legitimate open weight user or for being a spy who's hacked in and stolen the weights. In both cases, getting these weights becomes less important.

[Rob Wiblin]

Yeah. A related impact that this would have is that the AI market is more likely to remain competitive than it would if there was this enormous kind of fixed cost that you had to incur at the point of training a model. Can you elaborate on that?

[Toby Ord]

Yeah, so I think this is. Right. So often this process of this kind of massive pre training runs is likened to software development, where it's something where you go to a lot of effort to write a piece of software and then there's zero marginal cost or very small marginal cost to distribute it. So that was true to some degree with books, where you've got a lot of effort to write a book and then printing it's a lot less. But once software could be distributed on CD or then just downloaded, it became trivial costs to the company to say, have an extra copy of Microsoft Word distributed to a user. So once you write a word processor, you do all of the software engineering for it. You really want to sell it to a lot of customers, because every customer is basically just pure profit. And that's this zero marginal cost type thing. And if you have that in an industry, then you tend to get a small number of players because you really want to be the best one of these things and then potentially just swallow up the whole market. Whereas if you enter it as a new entrant, you put in all these upfront costs, if you're the third best one, why would anyone use your thing? And then even if you sell it for less, it's very hard to be a player there. Whereas this could change that aspect of it and mean that most of the costs are actually in producing each item a little bit. Like, let's say you develop tools for a hardware company, physical tools, hammers, screwdrivers and things like that. Then it's the case that some of the costs go into designing the new hammer. But most of it is just that every hammer you make costs a certain amount of money and then you get a kind of limited amount of profit when you sell it. And so it might be becoming more like an industry of that sort. And that would have a different kind of market structure.

[Rob Wiblin]

Yeah, it suggests to me that Then more of the profit that's coming from AI would go to the hardware companies because they're the ones who actually have the scarce resource, or at least like temporarily scarce resource. So it'd be much harder, I think, I guess, for the software developers to gain a huge margin because there would be many of them with roughly similarly powerful and useful models, which is kind of where things stand today. There's at least three, four, possibly models that are roughly at ad parity, which means that then where does the surplus go? It probably goes to the hardware producers who have this kind of scarce resources, which is the thing that you desperately are trying to acquire in order to be able to apply them.

[Toby Ord]

Yeah, I mean, I think that sounds about right. In general, there's always this question for the companies that are trying to make a lot of money with this, that in this value chain, which step of this value chain makes the most profit? And what we have at the moment is that the final stage of these people who've trained these models is really quite competitive between a large number, well, a small number, but let's say a handful of strong players, whereas the step before that of the most powerful GPUs is really locked up by one main player and so they've got more ability to get a kind of monopoly pricing in. At that point.

[Rob Wiblin]

I guess we should say that all of this is kind of, this is where things are trending as a result of this switch towards inference scaling. We won't necessarily go all the way to there. It is interesting to know because I think, think just a couple of months ago I recorded an episode with Tom Davidson talking about the risk of seizure of power where he was describing this thing where, well, maybe there's almost all of the fixed costs are at the training stage and that would tend to push you towards a market with a handful, possibly even at some stage, just one company that is willing to spend the $10 trillion on the super training run that produces AGI. I think that is looking a bit like still possible things could change, but that's looking a bit less likely now.

[Toby Ord]

Yeah, that's right.

[Rob Wiblin]

So I guess that has been kind of positive or neutral effects here. I think one that people might find a little bit more troubling, that might have jumped off the page to them already, is that if you're in a world where you can access superintelligence early, if you're willing to spend a ton of money, then that suggests that rich and powerful and well connected people will be able to access these tools potentially many Years ahead of the general public. Yeah. What do you make of that implication?

[Toby Ord]

Yeah, no, I think there's. There's a real effect there. So I think we'll look back on the period that's just ended where there were, you know, OpenAI started a subscription model for their AI system where it was, I think, $20 a month, you know, which is less than a dollar a day. Right. To have access to the best AI system in the world. And then a number of companies are offering very similar deals. But we've got the situation where for less than the price of a can of Coke, you can have access to the leading system in the world. And it reminds me of this Andy Warhol quote about what makes America great is that the President drinks a Coke, Liz Taylor drinks a Coke, the bum on the corner of a street drinks a Coke, and you too could have a Coke. The best kind of, I don't know, sugary beverage that you can get is kind of. Everyone's got access to it. But I think that era is over. So we had OpenAI introducing a higher tier that cost 10 times as much money, because these inference costs are going up and they can't afford to give you this level for the previous cost. And this is what you're going to keep seeing, the more that we do inference scaling is that it's going to have to cost the users substantially more. And then there's a question of how many are prepared to pay that. So it's certainly going to create inequality in terms of access to these things, but it also might mean that it just is not actually scaling well for the companies. So if it turns out that you offer a thing that costs 10 times as much and less than a tenth of the people take it, and then you offer a thing that costs 100 times as much and less than a tenth of the previous group that took the first one, take this one. Then maybe just each of these tiers is earning you less and less money than the one before, and it's just not actually going to drive your ability to buy more chips and train more systems. Or it could go the other way around. It could be that a fifth of people are prepared to pay 10 times as much and then a fifth of them are prepared to pay 10 times as much again, and that you're getting more and more money from each of these kind of higher levels. But which of those it is could really determine what happens in the industry and whether these inference scaled models are actually profit centers for them or not.

[Rob Wiblin]

Yeah. So some of the Previous changes have, I think, slightly reduced our concern about kind of concentration of power and risk of seizure of power by human beings. But I guess this particular issue, that a privileged group of people can gain access to superhuman advice and superhuman assistance potentially substantially before anyone else in the world has access to it, I guess heightens the concern that I guess people at the companies or people in the government who might take control or get privileged access to these models, that they could potentially outfox everyone else if they're able to basically just have access to tools that no one else is really able to compete with.

[Toby Ord]

So while in theory this could happen just on the open marketplace with money. Yeah. My concern would be greatest about the company itself deciding, for example, okay, we've got this model, what can we use it for? Maybe we can be willing to spend a million times inference scaling on it to do some really important work for us. And the company might want to do that internally, or the government of the place where the company is located might want these types of abilities. So I imagine that that them happening kind of outside the open market is perhaps the most concerning place. I should say as well that I'm imagining or thinking about all of this over the next couple of years. I'm not claiming that in the long run, equilibrium, when we're imagining in a post AGI world, how unequal will access to AGI be? It could be very unequal or it could be very equal, if we actually choose to build a world like that. But I'm setting that aside because I think that I can only really see what's going to happen for the next couple of years and things may change after that.

[Rob Wiblin]

Yeah. What are the policy implications here? I guess one that stands out is that you might want to insist on some level of transparency about what is possible at the frontier if you're willing to spend a whole lot of money just so that the public and people in government have some sense of what's coming and that companies can't hide this if they would rather maybe obscure what they already are aware is possible if you're willing to spend a million dollars an hour.

[Toby Ord]

So many of the current rules, to the extent to which there are rules at all, there are voluntary commitments and there's also the EU's AI Act. They're often focused on deployed models. And so this means that you can circumvent a lot of this if you just don't deploy it. So maybe you have these kind of higher tiers of inference scaling that are only accessible internally. And so then you could have systems that are say of breaking through this human range of abilities without anyone knowing. Whereas in the world, this Andy Warhol Koch world, where everyone's got access to the cutting edge system, we kind of all knew that the people working those companies had the same thing, or that if they had something better within a few months we'd also have it or something like that. So I feel that, yes, that governments and regulators generally need to ask for more transparency in this world to know what the capabilities are for the leading edge internal models as well as the deployed ones.

[Rob Wiblin]

Yeah. Two other probably negative implications of inference scaling are that it makes regulation of AI just substantially more difficult in a number of different ways. So one thing is up until recently in terms of you want to carve out the models that you think are not particularly risky, that are basically just applications that we should feel you're not only okay with, but actively excited about. And then you want to carve out the things where it's like, we don't know what this model is potentially capable of. This is posing novel risks that we've perhaps never seen before. And we want to at least do some research and study it before we deploy it or possibly even before we use it internally. And to do that we've used these kind of compute thresholds where we've said, well, if this is more than 10 times as large as any model that's been trained before, then it falls into the like, let's study this first regime. And if it's smaller than things that have already been trained, then we're probably in the clear and we can use it with, with a reasonable degree of comfort. Can you explain why inference scaling makes this so challenging to actually do?

[Toby Ord]

Yeah. So that paradigm of compute governance via these thresholds, you could think of it as trying to regulate a particular object. And so what they're saying is that if this object has gone further, substantially further than any that have come before in terms of what's gone into it, then we could try to regulate it. And so these trained weights are the object of interest. It's a little bit like say having regulation on automatic weapons, but not on non automatic weapons. Something like that, where you take a particular class of object and you put a regulation on it. Whereas what we're getting with inference scaling is it's not the object itself, it's kind of more what you do with it that matters. And so it could be that you can take for example a GPT4 sized pre trained model and then just through a smaller amount of post training that you can make it able to think on longer and longer kind of time horizons and then you can just use that model with huge amounts of inference. So just run it over and over again and maybe you put in 10 times as much compute into running it over and over again as you put into the very first training of it. But that's currently not regulated on a lot of these things. And even if you tried to regulate that, so it's definitely different because then you're trying to regulate the use of an object, not regulate the object existing at all, which raises a lot of different questions. But also it might be really hard to do because maybe you've got a system that's required a lot like GPT4 training on 10 trillion words of information and then you merely scale up its inference by a factor of a million. That's still a big scale up and maybe that has kind of dramatic effects. But if you only use that once for say some internal deployment, the total amount of compute it's going to use is still small compared to the original training. So you wouldn't really see it if you're just trying to add up all the compute. You'd see it if OpenAI or some other group said every single user is getting a million fold the level of inference they previously had. But if just one is doing it, it'll just be using up as much compute as a million users use up. And so it might not really be detectable if you're trying to measure these things. So therefore I'm concerned that this is a substantial problem for these pre training compute thresholds. And I personally don't think it's possible to overcome it. But maybe there's some creative work that will solve it. But I'm not necessarily bearish on all compute governance. It's still the case that if you know where all the GPUs are, for example, lots of them are owned by these cloud computing providers and then you have know your customer rules, you know, for them and so on, that you might be able to exert some control over the dangerous possibilities through compute governance, but it might have to change the way we do it.

[Rob Wiblin]

Yeah, well, the other complication that adds to compute governance is that at the point that you're training a model, I hope we get the technical details right here. But the point that you're training a model, you really want to have all of the computer chips in the same location. Because it's not just compute that matters, it's the ability to move information incredibly quickly between all of these computer chips that are in an array. That's an issue that occurs at training and means it's very difficult to spread the training of GPT5 across many different data centers. Maybe you could do it with a handful of. But I think ideally they really want to have it all in one place and you certainly couldn't distribute it across the entire world. But if you're just doing it with inference, then I think you don't face this similar constraint that you need to have all of the chips or most of the chips in a single location. You can potentially distribute them far more widely, which then makes it. If your hope was that the government would be able to identify the handful of places in the world where most of the compute lives and by looking at what's happening, they get visibility on what the entire sector is doing that is a lot weaker if most of the juice is coming out of throwing more computed inference.

[Toby Ord]

Yeah, that's right. So we've seen that with these stories over the last year of major companies trying to get nuclear power plants commissioned or to get full access to them in order to run a data center, because the training, at least the standard ways of doing it, all have to be done in one place. That creates this huge power density issue that you need a lot of power in one location. And it's difficult to do that with the grid. Without it being the case that there's actually literally a power plant there that is powering you. It's hard to just provide it through the general kind of grid capacity. And that's actually given a lever for government to have some power over these companies, a lever that they don't seem to have used at all. Because if they say you need to fast track us, or we would like to be fast tracked for this new nuclear power plant, you can say, oh, okay, well, I mean, we're interested in fast tracking you, but you'll have to in return be more transparent about your internal models and so on. Whereas a lot of people say, well, how could the government control these companies? This was certainly a location where they could, albeit the US government seems to have just fast tracked all of this without asking for very much in return. But this could change. It does depend on the nature of this scaling up of inference. I mentioned this example of doing really long chains of thought. But another way to do it is instead of having one of these employees who you've sent to your virtual university with your pre training, instead of just having that employee work on a project for longer and longer, you could Send the project to 10 employees. And so you spend 10 times as much compute to run all of these different virtual employees. And then you see which one's done the best job. If it's objectively measurable, you might be able to do that, or you might be able to have an 11th employee who looks over the 10 reports and then selects which one's best and shows that to you or something. And that's one of the standard approaches that is being used to do inference scaling is kind of doing them in parallel instead of like a longer sequential thing. And we'll probably see a mix of these two. And the parallel version is the type of thing you can spread between different data centers.

[Rob Wiblin]

So I guess a couple of years ago I was thinking a lot about compute governance potential. Could you exercise regulatory control by knowing where all of the compute is? Thinking a lot about information security, that the risk of model weights being stolen. So we're like all off in this direction. Should we now be saying, well don't worry about that compute governance that's not necessarily so relevant. We don't have to worry about the security of weights or open sourcing. Whatever goes feels like that would be far too far to go in that direction. But should people actually be. Is this so decision relevant that people who are trying to improve the direction of AI by leaning on these different things should be changing their plans? Or should we maybe wait and see whether all of this stuff might go into reverse? Maybe inference scaling will peter out in a couple of years and it will be all back to some new kind of training that they've figured out how to do.

[Toby Ord]

Yeah, so I think that people who are interested in AI governance should be tracking these things and maybe more than they are, they should be noticing that AI governance up until the start of this year had all been done in this paradigm of this scaling of pre training. And we'd see all of these charts that show how impressive it was going to be and project forward and so on. And really I want to kind of stress that that era has come to an end and we're now in some other era that might be a kind of continuation. But there's no particular reason why it should be. There's no reason why the slope of those curves should be the same as it was beforehand. And in fact there are reasons to think the slope of the lines is worse. And so they should be aware that a lot of the rules and ideas that they've been kind of building up that they need to reevaluate them. And my Piece on this was written, I think a week after realizing a lot of these things. And I think it's held up reasonably well. But I wouldn't want to be telling people what to do based on a small amount of one person thinking about the implications these things might have. I wouldn't be surprised if there were additional implications as big as the ones that I mentioned, which I never found.

[Rob Wiblin]

Okay, so one of the, in fact, one of the things I hope the audience takes away from this interview is that technical changes can radically shift the strategic picture and the governance picture. And so far we've all been talking about the impacts of scaling up inference at the point of use at the point of inference. But it's also possible that we're finding, and it is the case that we're finding new ways of applying enormous amounts of compute at the training process, just in different ways. And that could have have kind of all of the reverse implications of what we were just describing. So yeah, could you explain how we're finding new ways of applying large amounts of compute at the training stage? That is not the kind of pre training that we think has somewhat petered out so far.

[Toby Ord]

Yeah, exactly. So this process that I'm calling inference scaling, so scaling up the inference compute also gets called reasoning, although it doesn't have to be used for what we think of as reasoning. And it also gets called test time computer, which also kind of implies that it's happening at the time of deployment to the user or something. But I think it is really important to divide the versions where it's happening, everything we've talked about so far, where it's happening during deployment for a particular user who's trying to get value out of it versus using a whole lot of extra inference compute during a larger part of the whole testing process. If you use it during the training process, the economies of scale are different. So suppose that as part of it you've pre trained the model and then during the post training you run really long kind of inference chains and these chains of thought and so on, and you assess them. They do this using reinforcement learning, so they give it hard problems that they know how to check. So kind of say really hard maths or coding problems where there's kind of precise answers and then they train or reward these long chains of inference that.

[Rob Wiblin]

Actually worked that get the right answer.

[Toby Ord]

Yeah. So they kind of roll it out with huge amounts of tokens and then they use that to train to do more post training on this set of weights. If you do that, all of the stuff you did there. Once it goes into this post training, you're still kind of. You only have to do it once. And then every single user, if 10 times as many users come along, you don't have to spend 10 times as much extra compute, you just spent it once. It doesn't scale with the amount of deployment, so it potentially has quite different implications. And so I think what we've been mainly seeing is the type of thing I just mentioned where you try to get the system to generate long chains of thought. And then what you do is there's two different versions. One is that you look at the outcome that the final answer and you reward it or punish it based on that final answer. And then these weights that represent the model get updated based on the reward or the punishment. Or you do what's called process supervision instead of the final answer, where you look at all the steps inside its reasoning train of thought and you see if they seem to be going in the right kind of way, or if it seems to be getting stuck or lost or something. It's a bit like with a child where you can either try to reward them based on getting the right answer or reward them on whether it's seemed like they were kind of applying the types of techniques that you've been hoping that they would use. So that happens already. And so in order to get to actually productively scale up the inference when you're deploying it, you have to do a certain amount of extra inference combined with reinforcement learning when you're training it. But there's also ways that I don't know if they have been applied yet, but they could go much further than that. So this is a technique called iterated distillation and amplification, or at least that's an interesting one to look at. This is the technique that led to these amazing performances. In the case of Go with One of these DeepMind projects, it was the AlphaGo Zero. What they did there was that they had a neural network that looked at the board in a game of Go and tried to give it a kind of heuristic valuation of how good is it for the current player, Is this a winning board or is it a losing board and by how much. And so we try to estimate and learn that it's kind of like intuitions system, one kind of ability for Go playing, just to be able to see and see what looked like a good move. But then what they did is they took that system and they kind of inference scaled it. They gave it a whole lot of System 2 ability. What they did in practice was that they let it play out a whole lot of games from that position using its current heuristics as to what's good and what's bad. They let it play things out, see how the games would go, and then use that information to actually revise their idea of what looked like a good move. So that version potentially using thousands of times as much compute, we call that the amplified version or the inference scaled version. Then the next step is that you can distill it. And so what you can try to do is take the moves that the amplified version makes when it's also got the ability to search through the game tree and just try to develop an intuition where that your system one like your intuitive response is to produce those types of moves. And then you've now got just an intuitive. You've kind of improved your intuitions. Then you can do it again. You can take that one and scale it up to 1000 times as much compute using the new improved intuitions, and you get this improved play. And then you distill that play back.

[Rob Wiblin]

Down again to a smaller model that doesn't require so much compute.

[Toby Ord]

Exactly. So there's these two types of steps. Effectively what happens is that it leads this kind of ladder where performance improves quite a lot when you amplify it and you spend 1,000 times as many resources on on the problem. But then when you distill that one back down, you've got a cheap thing again, but it's a little bit better than the previous one. And then you do it again and you go up and then back down. But every time you kind of come back down to a cheap model, it's a bit better than the one before. And so they ultimately applied this. I think it was more than 1,000 steps climbing up this ladder. And in doing so it's blasted through the human level. And eventually they put it all the way up to a point where it could no longer distill out any advantage from the amplified model. So the process stalled out. So in general it's a very powerful technique. It's not clear where it will stall out. Maybe there's some other games different to go where it would stall out before the human level. And you wouldn't be able to use this technique to get all the way up to kind of superhuman play. Now could that be applied for these reasoning models? I don't see why it couldn't. And so there's a possibility that you could imagine a kind of situation where the new model is just being generated, let's say every hour or something from the old model, where they take a model, they let it reason for huge amounts of time, produce a final set of answers, then they train a new model to just try to produce those answers straight away, to have its kind of intuitive stream of thought answers to be like the finished polished paper that would come out of the other process and then it will learn a little bit of that and hopefully be better. Then you amplify it with heaps of inference, compute, then distill it back down and so on. If this was possible, then it could lead to explosive improvement in capabilities all by kind of using all of this inference, but entirely inside the training process. And then what you do at the end of all of that is that the final distilled model, you could deploy that to customers or what have you.

[Rob Wiblin]

So it wouldn't necessarily be more expensive at the point that you're actually applying it anymore, because you've found a way to have the intuition of the ability to mimic someone who's been able to think an enormous amount of time, but to do it very quickly with very little thought.

[Toby Ord]

Yeah, that's right. And so I think this whole process, say doing it literally as I described with iterated distillation and amplification, will that work? I think it probably won't. So I'd say less than 50% chance that that will work. Is there something either this or Maybe there's a 10% chance it would work. Is there something like that that can work? Maybe. Right. And so I think that there is a possibility that is non negligible and I think substantial, that by having both this kind of system one ability through pre training and then also this ability to improve system two and then effectively to have your system one intuitions be trained on what you would have done after a bunch of this kind of more formal reasoning and then keep iterating that having both these two components of natural intelligence could be something that leads to this kind of explosive recursive self improvement of these systems. And I do think that that that has become more possible in this world of inference scaling.

[Rob Wiblin]

Right. So we have an example where this has really worked, I guess with the GO models and I'd imagine with other games where it's clear whether you win or lose, that this approach of amplification and distillation should work in most of those cases. And I suppose so. Now with these new reasoning models like 01 and 03 that OpenAI has produced, and imagine that the other companies have their Own ones. The way that they've been training, doing the. The second stage of training with them is that they present them with reasoning puzzles, with sort of exam style questions that have a clear right and wrong answer. And that provides an analogy to a game of Go where you either win or lose. So you have a clear signal about whether at the end of the day you got the right answer or you won the game. And then they can go back and say, well, in this case, using this sort of style of reasoning, it got to the right answer. So we want to reinforce more of that, want to produce more of that. And that has allowed these models to get much better at figuring out how do you think for a long time and maintain accurate, I guess, reasoning through the entire process. And in general have reasoning strategies that tend to lead you towards correct answers, at least in that style of question. So I guess you're saying because this worked with Go when we had a clear success and fail signal, maybe it will also work in these kind of reasoning cases where at least for some domain of problems, we also have a clear indicator of success and failure.

[Toby Ord]

Yeah, that's right. And so it may not generally work across all possible forms of reasoning to lead to superb ability to write emails to the regulators to argue your case or whatever in areas where the success conditions are quite unclear, if you can't send off 10,000 emails to the regulator and find out which ones convince them. So it may be that it's more limited in its applicability or it may not work at all. It may be that it turns out, you know, it's hard to get this kind of recursive process off the ground. Effectively the point where it stalls out is the first step instead of the thousandth step. And we don't know. But that also brings up this aspect where whether or not you're doing this iterated distillation and amplification, all of the reasoning work that's happening at the moment is being, in order to get it to be coherent over longer times, you need some kind of this reward signal in order to be able to train it. And this is primarily coming from cases where there is a known correct answer to the problem. So this could be tricky maths problems and also a lot of computing problems where there's a plaintext question to write a program that meets this specification, and then they test the program based on a whole lot of inputs and the outputs it should produce. And these are called unit tests. And then it also maybe checks how long it takes the program to run and this is the kind of thing that you get for humans in these coding competitions. And for humans, ability in coding competitions or advanced mathematics correlates quite strongly with general intelligence across a lot of different areas. With AI systems, it's not as clear how well it will correlate. So they're training them on. In some ways, we're going back to the world of 2019, where there was like, go is extreme ability at GO is very impressive. If I met a human who can play at a grandmaster level of go, I'd be genuinely impressed by them. And I might think that would also correlate with being good at other things as well. It could find out, are you good at maths? Are you good at hath, you know, thinking through complex reasoning things? But in this case, it's not clear how well it will correlate. And I feel that the AI labs are exactly the kind of places that are impressed by research mathematics and are very impressed by people who can ace coding competitions because so many of them have come through a programming background, but they may have overindexed on some of these challenges that are difficult for humans. I mean, we know that, that for at least 50 years, computers outdo us at, say, multiplying two numbers together. And at some point, that was impressive. And we've trained ourselves to no longer be impressed by this fact. And it may be that, say, ability to write really efficient code for extremely well specified programming tasks, maybe that will also become something that we just don't think is very impressive. And it may not generalize to. To other kinds of reasoning tasks. In general, the track record for reinforcement learning and generalizing is pretty poor. So when DeepMind did the original Atari work, they built a system was impressive, but they built a system. It was not a single trained model that could play all 50 or so Atari games. Instead, it was like a single system that could take an Atari game and it could train an agent that could only play that Atari game. And it could train 50 of these agents, one for each Atari game. And so it was a general system for creating narrow agents. And they'd hoped for what's called transfer learning, where if you get good at something, it helps you be good at something else. In general, that was very hard to do with reinforcement learning, but it's one of the big successes of the LLM era. But now, if we're kind of switching back to using reinforcement learning to deal with the fact that we've kind of plateaued, then we maybe will expect things to go narrow again. And for this increased performance to both slow down and also to be only in very slender subdomains of all the types of things that humans do.

[Rob Wiblin]

So we open talking about how in some sense things looked safer or more comfortable since 2019 because we had switched towards away from reinforcement learning and towards this kind of next word prediction, which led to a more understanding of human concepts. Now it seems like over the last 18 months we've been screaming back in the other direction towards reinforcement learning is the place that we're getting most of the juice. And so many of the problems that had faded away through 2023 might be basically all coming back. And it does seem, I think, that's the case. So you're saying one distinctive thing about reinforcement learning is that it seems to have less generalizability than the LLM next token prediction style did. The other thing is I think reinforcement learning agents, so they're more narrow, they also are a lot more reward hacky. So they tend to do crazy stuff just in order to try to win, because that is after all, the signal that they've been given that they basically are just rewarded whenever they manage to achieve the outcome and they don't have broader concepts of common sense. And what was the intent of the operator? Do you want to elaborate a little bit on that?

[Toby Ord]

Yeah, no, I think that's exactly right. It's interesting that when I wrote my remarks on the Pressvis revisited, it was kind of the high watermark of all of those changes. And then since then, some of them have gone into reverse a bit. And so another one to add to that is not just the shift to reinforcement learning, but shift to agents again, which I said were a particularly dangerous thing that everyone was preoccupied with. And then we had a whole lot of developments in systems that weren't agents. And then maybe we're going back to the dangerous ones again. So, yes, I think you pretty much nailed all of that, that the shift to reinforcement learning will have some of these difficult questions, problems, including narrowness, but also, as you say, including this aspect that the AI systems might do this reward hacking type behavior. And there have been a number of reports of this with recent Systems. I think O3 in particular, there have been reports of it doing reward hacking. And I saw one in the wild, actually, that I don't know. That doesn't seem to be well known. But in One of the two blog posts launching O3, OpenAI's new Very Capable model, it showed a whole lot of different impressive tasks that it did in visual reasoning. And one of them was this drawing where they had the numbers 1, 3, 5, 2, 4, question mark and it said the answer isn't 6. And this was a little kind of like brain teaser. You might think it's a maths problem. It turns out it's a lateral thinking problem and it's drawn in the shape of a gear stick and the answer is meant to be R for reverse. And it's somewhat interesting question, which is why it had been big on Twitter a couple of years ago and the AI system had this reasoning trace that was shown in the blog post. But if you looked at it in detail, I remember thinking, where does it make the kind of aha moment to realize oh, it's not a math problem, it's a lateral thinking problem. And I kind of narrowed it down and then I saw oh, there's a step a little bit before it says kind of oh, hang on, it says now searching for 1, 3, 5, 24 question mark. The answer is not 6. And it turns out if you just type that into a search engine, you come up with the page that it reaches at the Hindustan Times, which just explains this new brain teaser that was going around and explains the answer. So it just googled the answer halfway through the track. And then it doesn't say that though, it then says, oh hang on, maybe it's totally different. Maybe it's about cars instead of about about maths and then has the answer. So I should say that five years ago, having a system that does optical character recognition on a picture, finds the text, Googles it, extracts the result from the answer would have been somewhat impressive. Five years ago, it's not impressive now. And so it was kind of been intentional that it did this in their post where that was one of the very few examples shown to show how impressive it was. But it also implies that the since that page at the Hindustan Times was a year or two old and also that had been discussed on Twitter, that this model must have actually seen this problem multiple times on multiple web pages during its training period. And so the more I thought about.

[Rob Wiblin]

It, the more you think it's surprising that it wasn't able to intuitively answer it just from memorization basically during the pre training process.

[Toby Ord]

Although maybe the person overseeing would have caught that if it just said the answer straight away. But it's deeply unimpressive that a system that has seen a logic puzzle multiple times then had to Google the answer to find out what it was.

[Rob Wiblin]

I think what's distinctive about the reinforcement learning models is that they learn basically, I think not to say, oh, I just googled it and I found the answer. Because that's going to be kind of negatively reinforced. You end up encouraging them to do these perverse ways of basically impressing the operator to get them to think that they've done the thing that was desired, even if they hadn't. I think that there's other cases where you've got these reinforcement training, learned coding agents where they'll be working on solving some sort of coding problem. They'll realize that they can't do it, but they, I think, manage to figure out what the correct answer would be during the check stage. And rather than actually design code that solves the problem and calculates it, they just hard code in the answer so that when it's checked to see whether it succeeded or not, it outputs the correct answer, but using a completely different method that wasn't desired. And this is sort of classic sign of reinforcement learning where all you've rewarded them on is the output. And if you're not scrutinizing the process, then they will figure out some way of fooling you into thinking that they've done what you want.

[Toby Ord]

Exactly. And so this is the thing that's called reward hacking. And it's kind of interesting because it's only a problem if you take into account that there was an intended solution that the humans, the humans did not want you to go and change to give specific answers. You're going to be tested on what's the answer to five different questions. And then your whole program just says, if it's question one, print this. If it's question two, print that. That was definitely not intended, even though at some level it's just a clever kind of solution. And there's a TV show taskmaster where the contestants are allowed to do this kind of thing. And it's quite funny to watch, but this is not what's intended. So we call it reward hacking. Reinforcement learning tends to lead to very creative solutions, including this style of perversely creative solution. So I'm not saying that the models got it wrong or something, but it's certainly a kind of out of the box type situation where it's harder to control them, it's easier for them to deceive you. So an example of, of like what you were talking about, shortly after DeepSeq's R1 model came out, there was a company who used it and declared on the Internet that they'd used it to improve the performance of a number of these CUDA kernels. So a key part of machine Learning and that the results were, I think in one case it was 100 times as efficient or something. I was thinking that's. That doesn't sound right. A couple of days after R1 came out, you've managed to use it to make this thing 100 times more efficient. And they had a whole lot of these results and someone looked into them and they were all spurious. In all the cases it had just done this type of thing where I think in some cases it had access to the files that would test how efficient these things were and it changed those to report large numbers of efficiency. It did all kinds of stuff. I mean, it was a masterclass in rorting the answer to one of these things.

[Rob Wiblin]

I think there's other cases where you have a model and you're trying to get it to win at a game of chess and it realizes that it can hack into the model that it's competing against and try to sabotage it, like replace it with a much worse chess model so that then it's able to beat it. This is classic reinforcement learning.

[Toby Ord]

Exactly. And I mean, they're always really fun, interesting examples, but if this is happening with a production system, you really need to be aware of it. And what's interesting about some of these cases, like I think the chess one was one that was set up to see if it would do that, but other ones, like this one with the CUDA kernels and this one where OpenAI was trumpeting how impressive this model was at solving visual reasoning tasks, it tricked the person who was actually trying to get it to do this thing and caused an embarrassment for them that they publicly announced it was solving problems that it actually wasn't. And I mean, maybe I think that the company with the CUDA kernels, I think they didn't have such a big track record of having dealt with these agents for a long time. But I was surprised with the OpenAI one where if you're trying to test a system that has literally read the entire public facing Internet and you're trying to test it on some kind of brain teaser, obviously you cannot pick one that you found on the Internet. This is an obvious point. I mean, the first time you've encountered this issue, maybe you end up doing that, but it beggars belief that they would do this. You obviously have to invent your own puzzle or if not to do extremely elaborate testing to make sure. For example, if you just type in all of the question into Google, does it appear? It appears as hit number one. So it was a little bit of an update as to how careful people are when they're launching these new models.

[Rob Wiblin]

Yeah, I think it speaks to the fact that they're just incredibly rushed. We open saying the race is as fierce as ever. And I think we just see this signs of this all over the place, that this stuff is getting shipped as soon as they feel like it's not going to be a total catastrophe.

[Toby Ord]

Exactly.

[Rob Wiblin]

Yeah. Okay, so we've had a little bit of whiplash here. So reinforcement learning was out, now reinforcement learning is back. So I think the models are becoming a bit more psycho. I would say they're a bit more challenging to handle. You have to be on your guard because they. I mean, I think people are seeing this a lot more just in day to day use, that they are much more inclined to deceive you and to trick you one way or another than they were two years ago when that was quite abnormal. Maybe they weren't capable of it. But also I think in the absence of reinforcement learning, they hadn't been encouraged to do it during the training process in the way that is now somewhat coming out. It's possible that the sycophancy issues that OpenAI has had might also be related to this. I could imagine where I think that they shipped an update to their standard model where it suddenly became incredibly flattering to the user and would encourage them in almost any fantasy about themselves that they were willing to put forward. That may or may not be due to reinforcement learning, but it wouldn't shock me if it was. What are the implications of all of this for governance? Sorry, I throw that at you awfully quickly.

[Toby Ord]

Yeah, look, I don't know, honestly. So I tried to outline a bunch for the inference scaling, but the reinforcement learning in particular. Yeah, I'm not sure, but I think you're right that it's another example where people working on governance need to reevaluate a lot of their standard assumptions because they might be changing at the moment.

[Rob Wiblin]

One thing that stands out to me is I've been wondering for years, what are the chances that we will get early warning shots. I guess people have been wondering this for a very long time. Will we get early signs of failure and of AI models going totally off the rails in a way that kind of everyone has to acknowledge that this was not intended and maybe this was even quite harmful. I think with the resurgence of reinforcement learning, the odds of that have gone up quite a bit that we're already seeing. Interesting, amusing, sometimes slightly harmful, but not like terribly troubling cases of AR models. Basically going off the rails in deployment today. And I think that that will probably get worse in coming years as they used for higher stakes fix things and probably as reinforcement learning becomes an even bigger part of the training process. So I think there's more reason to plan for what will be these moments when people suddenly potentially realize that this reinforcement learning is creating serious hazards. Maybe we need to be scrutinizing the reward signals more. Maybe we need more regulation of AI on the whole because this stuff is actually quite material now.

[Toby Ord]

Yeah, we could see some of this. We're certainly. I mean there's the aspect of individual high profile examples. For example, I think that the case with the Microsoft's Bing Sydney model and this Kevin Roose article where a lot of people saw this conversation it had where it tried to convince him to leave his wife to marry it or have an affair with it or something, that was an example of really high profile example of a misaligned model going off the rails. And so maybe we'll see some of these high profile particular examples or maybe also a lot of people who are using AI will start to feel like, you know what this is like some annoying, like I've hired this assistant and now they're just pretending that they did the emails for me and actually they didn't. So I don't know how much of it will come through that channel of personally witnessing it versus higher profile events. But with the high profile events there's also a question about whether people will just have fatigue at some point. We've had these cases where the people in the alignment and safety communities have generated kind of test cases that would encourage some of these things and then they witness the behavior, but under test conditions where they tried to elicit the behavior and then when they get that to work with a production model or something, it's impressive and it makes the rounds a bit. But after enough of those, maybe people start to tune out and then maybe that's true as well. If there are a large number of low stakes but clear examples of it in the wild, deceiving people and so on, maybe they'll get tuned out as well. And then instead of it being a big shock. So it's not totally clear to me that in terms of the public attitude or regulators attitudes, whether having more clear examples of bad behavior at a stage where the stakes aren't that high, which way it will go.

[Rob Wiblin]

Yeah, that's interesting. I guess on the iterated amplification and distillation approach, I suppose we're just very much in the dark about whether that works. I suppose we can't have figured out how to make it work yet because I'm sure this idea has occurred to the companies and they haven't said that they've managed to get massive performance improvements using this approach. But the fact that possibly they will be able to figure out some approach like this that works in future, I suppose just increases the uncertainty. It means that it's not the case that we can just trust that we will necessarily follow the trends that we've seen in the past, or that all of these curves are just going to level off and maybe progress in general is going to slow down or plateau at the human level because it's just such different regimes, some of which lead to declining returns, some of which lead to linear returns, others which might lead to even exponential increases in performance. We need to be able to plan for all of these different scenarios.

[Toby Ord]

Yeah, I think that's right. And that overall, a year ago, before the news started to break that this pre training scaling was running into trouble, I really felt that one could just project it out, look at these curves and kind of project them out several more orders of magnitude and have a decent idea about what's going to happen and when. It was still somewhat unclear if you had a GPT6, what would it actually be able to do or something. But it all felt a little bit more contained and predictable that we were following some kind of curves and we'll just keep going up. Now it feels like things have changed. And if it's possible to do amazing things using this inference at inference scaling at training time, then maybe things could be quite explosive. The AI labs themselves, I think have all suggested really quite on really quite stringent definitions of AGI that we'll have it by 2030 or sooner, 2027, some of them are saying, or 2028, but I don't know if that's. I still think that that's actually less likely than not. I'm not sure what chance. I would say maybe a quarter or something. But if that doesn't happen, if the iterated distillation amplification is a bust and other similar approaches are a bust, a lot of the companies are looking at another form of closing the loop on this thing by getting AI systems that are specifically trained to do the work of their own staff and in doing so to try to have them perform better than their own staff at creating new AI systems. That's a way that you could potentially have explosive progress as well. But I think it's pretty plausible that those things work. And it's also pretty plausible that they don't. And if they don't, and the pre training scaling things run out of steam.

[Rob Wiblin]

And we've run out of data or high quality data.

[Toby Ord]

Exactly. Then I think timelines could be quite a lot longer. So I think that both these things are possible. And effectively my probability distribution, my range of credible times at which some transformative system is produced has spread out over this time.

[Rob Wiblin]

All right, let's turn to another article you wrote, which is the scaling paradox, which I found super illuminating and I could definitely. It's pretty brisk, it's pretty short and very informative. So I can recommend that if people like what they hear here, that they just go and check it out on your website. But the scaling paradox is that on the one hand, the impacts of increasing the amount of compute going into these AI models has been extremely impressive, and yet in another respect it's also been extraordinarily unimpressive. Can you explain both angles?

[Toby Ord]

Yeah. So our whole conversation so far has been about scaling and this kind of question of, oh, hang on, what happens if the previous scaling stops and this new type appears? But in this paper, I was trying to go back to the old type of scaling, the pre training, and try to understand this because you often hear about scaling and you also hear about scaling laws, and they're somewhat different. So the scaling laws are these empirical regularities. They're not necessarily laws of nature or anything like that. But it turns out that if you do a graph and you try to measure a measure of error or inaccuracy. So this is a bad thing. It's log loss is the kind of technical term, if you try to measure how much it's still failing to understand about English text as you increase the amount of compute that went into training it, how much of that kind of residual mistakes is it making in prediction? And so they have these laws or empirical regularities, and what they draw is they draw these straight lines on the special log, log paper. You don't need to worry too much about that though. And it's a bit hard to interpret that. And so I spent some time thinking about it and basically what's going on is that every time you want to halve the amount of this error that's remaining, you have to put in a million times as much compute. That's what it fundamentally comes down to. And that's pretty extreme. Right. So they have halved it and they did put in a million times as much compute. But if you want to Halve it again, you need a million times more compute. And then if you want to halve it another time, probably it's game over. At least in terms of that particular metric. I would say that is quite bad scaling. And these are the scaling laws. They show that there's a particular measure of how good it's doing and how much kind of error remains. And it does hold over many different orders of magnitude. But the actual thing that's holding is what I would have thought of as a pretty bad scaling relationship.

[Rob Wiblin]

Yeah. So is that in order to halve the error, you have to increase the compute input a million fold. That's a general regularity, because surely it differs by task and differs depending on where we are.

[Toby Ord]

Yeah. So what they do with these cases is that you grab a whole lot of text, often from the Internet. They started with the good bits like Wikipedia and things like that, and then as they ran out of that, they had to look at more and more things. But what you do is you train it on that and you train it on most of it, but you leave some unseen. And then what you do is you try to give it a few words of the unseen bit and ask for the next word, and you see how well it does at predicting that. And basically the amount of errors that it has in doing that leads to this error score. And, yeah, it's not clear that the error score is something that fundamentally matters. Maybe it's a bad measure, but I found it really interesting that the single measure that people like Ilya Suskever and Dario Amade that convinced them that scaling was the way forward were these scaling laws that actually, if you look at what they say, it's distinctly unimpressive. I think if you ask people before they saw the laws, what would you hope happens to the error if you were to put in? Or how much extra compute would you need to put in to halve the error? I think they would have said something less than a million times as much. If you said, well, actually it's a million times as much, they would have thought, oh, okay, that's actually unimpressive.

[Rob Wiblin]

Sounds terrible. Yeah. Okay, so that's the sense in which it's unimpressive is that in order to reduce the error rate, you just have to spend these phenomenal amounts of compute. How then have we been managing to make so much progress? Is it just that, in fact, there was so much room to increase the amount of compute that we were throwing at these models, and so that has been able to more than offset the incredibly low Value that we get from throwing more compute at them.

[Toby Ord]

Yeah, I think that's basically right. So when most people saw this type of thing and most people who were academics doing computer science, they would have thought, oh, so in order to get good performance on this task, you would need to run an experiment larger than any experiment that's ever been run in any computer science department ever. And they would then rule it out and assume, obviously we're not doing that, we'll look for a different approach. Whereas the pioneers of scaling thought, oh, but that wouldn't be that much money for a company, in fact, a company, in fact, then they could even go 10 times bigger again, maybe. And so they realized that there was a lot more room to scale things up, to scale up the inputs, all the costs in companies than there were in academia and that. And in some sense all you had to do then was this kind of schlep or this work of just making this existing thing bigger. You didn't have to come up with any new ideas. And it was not trivial to actually run that engineering process. And we've seen some companies had some trouble doing it, but there have been many followers once it's been shown how to do it. So I think that that was the kind of brilliance of it was that. But there was a lot of money there, so you could scale it up a lot. And then the other thing that's turned out to make it have big impacts in the world is that it turned out that I guess each time this error rate halved, that corresponded to tremendous improvements. Certainly for every million fold increase in the compute of setting up these models, we've seen spectacular improvements in the capabilities as felt by an individual. So a way to look at this is that the shift from GPT 2 to 3 used 70 times as much compute and going from 3 to 4 used about 70 times as much again. And GPT 3 felt worlds away from 2 and 4 felt like a real improvement as well. You really felt it in both cases.

[Rob Wiblin]

A visceral feeling of wow, this is suddenly useful.

[Toby Ord]

Yeah, this is qualitatively better. Okay, that said, you'd probably hope that that was true. If someone said something costs 70 times as much that how's the wine that £1 and the wine that cost £70? You'd hope that the wine that cost £70 is noticeably better. Otherwise what on earth's going on? But we did feel those improvements. Whereas if you look what happens to the log loss number, it didn't change that much for a mere 70 fold increase in compute. So effectively there was this unknown scaling relationship between the amount of compute and what it actually feels like intuitively in terms of capabilities. And that turned out to actually scale really quite well, I think.

[Rob Wiblin]

Yeah. So is there this issue that until recently we were using these mathematical relationships between the inputs and the log loss. And I suppose some visionaries were able to see that even though in some sense the returns were very poor, in fact in the real world sense it was potentially going to be revolutionary. And maybe we need to start kind of stop thinking about this log loss thing, which is perhaps kind of a distraction, and start thinking about it in terms of how much revenue going to generate, how many users will want to use this thing. And then we might see that actually scaling looks somewhat better.

[Toby Ord]

Yeah, that could be a way to see it. And in fact, one of the numbers that you might really care about is if you 10x the amount of compute that goes into it, what happens to your revenues? Do users pay you 10 times as much money for that product? Maybe each user will pay more for it and each user will, or more users will find it useful. And if though it's the case that when you put in 10 times as much training, you only get five times as much revenue, and then you 10 times as much training, you Only get five times as much revenue again, then the whole kind of economic engine that's driving this might run out of steam. The companies might no longer be able to fund these things. Of course they're funded by venture capital that's based on predictions about the future. But the venture capital might dry up because people might realize that, that. Hang on a second, if you put in 10 times as many resources, you get five times as much benefit. That's not enough to keep going. And so it remains to be seen how that kind of thing is going to scale.

[Rob Wiblin]

You had this other very interesting article called infant scaling and the log X chart, which is. We're not going to get into all of that because this is, at least for many people, an audio show. And it's quite difficult to describe log graphs at this level of detail. But what one very interesting thing that I wanted people to take away from it is that there was this very famous chart that OpenAI put out where they were comparing two different reasoning models that they had, O1 and this more impressive one that was, I think an evolution of O1 called O3. And O3 really wowed people because it was able to solve some of these brain teaser puzzles that I guess are very easy for humans, but have been proven very difficult for AIs up until that point. And I think it was able to. They were able to get something like an 80% success rate on, on some of these puzzles that had seemed very intractable for AI in the past. But you point out that if you looked really closely at the graph and you properly understood was actually consistent with O3 being no better, being no more efficient in terms of being able to solve the puzzles than O1, despite the fact that the dots on the graph for O3 were an awful lot higher than they were for O1. And what was going on was that OpenAI had managed to increase the amount of, of compute that it was using at the point of trying to solve these brain teasers by about 1000 fold. And so, unsurprisingly, given 1000 fold, like 1000 times as much time to think about the puzzle, it was able to answer more like 80% of them rather than 20% of them. Now, in some sense, this is very impressive, but it is interesting that I think the companies are aware that people do not entirely understand these graphs, perhaps, and that most consumers are not paying deep level of attention to them, and they are sometimes trying to slip past messages that perhaps would not stand up entirely to scrutiny. And the fact that they put out a graph touting how impressive O3 is, when in fact, the graph in fact doesn't really demonstrate that at all. And it might just be on exactly the same trend you would have expected before if you'd given the model more time to think about problems, is quite interesting. And I don't want to single out OpenAI here because I don't think they're in any way unique in this.

[Toby Ord]

Yeah, no, that's right. And so you see these graphs of this kind of steadily, it looks like steadily increasing progress, right? This kind of straight line, as you put in more and more resources, the outputs go up and up and up. But if you look more carefully at the horizontal kind of axis there that you see. Oh, hang on a second. Each one of these tick marks is 10 times as much inputs as the one before. And so in order to maintain this kind of apparently steady progress, you're having to put way, way, way more resources in. And we're familiar with graphs like that from things like Moore's Law, where we'll see what looks like a kind of steady march of progress over decades of improvement. And what they've done there is Moore's Law inherently is this exponential. Things are getting so much faster. It's really impressive. And they've had to kind of squash it vertically with this special logarithmic axis in order to. It's just so impressive how fast these chips are that to even show it on the same picture we need to do this kind of distortion, but the distortion is underselling it, right? Whereas the opposite is going on here. The distortion is this kind of horizontal distortion. And if you actually look at the numbers, it's like they need to put in. They have to keep putting in 10 times as much inputs in order to keep the progress going and that's going to run out of ability to do that. And in the case of that famous data point with the Preview version of O3, I actually looked into how much compute it was and how many tokens had to generate and so on. And in order to solve this task, which I think costs less than $5 to get someone to solve a mechanical Turk and which my 10 year old child can solve in a couple of minutes, it wrote an amount of text equal to the entire Encyclopedia Britannica.

[Rob Wiblin]

So it's using a different approach to what humans are doing. It's fair to say.

[Toby Ord]

It took, it took 1,024 separate independent approaches on it, each of which was like a 50 page paper, all of which together was like an Encyclopedia Britannica. And then it checked what was the answer for each of them and which answer did it come up the most times and then it selected that answer and it took tens of thousands of dollars I think per task. So it was an example of what would happen. We were discussing this with the inference scaling. What would happen if you just put in huge amounts of just poured in the money and set it on fire. Could you actually peer into the future and could you kind of see the types of capabilities we're going to get in the future? And in that way it's quite interesting, right? But it came out just a few months after the preview for 01. And so it felt like, oh my God, in just a few months time it's like it had this huge improvement in performance. But what people weren't seeing is that it used so many more resources that it wasn't in any way an apples to apples comparison of what could you do for the same amount of money. Instead it was showing something like what will we be able to do maybe a year or more into the future? So that's kind of useful, right? Seen through that lens. But if you instead just treat it as a direct result. Oh, we used to have trouble with this benchmark. Now we don't. Then it's definitely misleading.

[Rob Wiblin]

Yeah. I mean, I think it's fantastic that OpenAI did this. I mean, it is a great research breakthrough and it's incredibly useful to know what might become down the pipeline. And this basically, as you're saying, allows us to peer into the future. And it's amazing that they managed to figure out how to put the scaffolding on the model that allows it to reason about one of these visual puzzles for the length of the entire Encyclopedia Britannica. In some sense. That's really cool.

[Toby Ord]

Yeah. Although there is another little wrinkle there, there, which is that subsequent to me writing this up, O3 got released as a model and so people could actually try it. And so the people who ran this test, the ARC AGI group, who are great, I think they ran it with the real model and its performance was 50%, not 80%.

[Rob Wiblin]

Had this been because it had been specifically trained on doing exactly these kinds of puzzles.

[Toby Ord]

So there were a couple of differences. One was that it was the O3 instead of O1. One was that much more compute was used. And another one was that it was allowed to see a whole lot of these puzzles beforehand and train on similar ish things. 80% of them it could train on, and then the remaining 20% was going to be tested on. But it turns out that if you take someone and you let them kind of train on a whole lot of similar exams, it really does boost their performance. That's why we do it when we're in high school. And so then I did wonder how much much of this boost is created by that and how much is created by it being O3 or by the extra compute. And it seems like quite a bit of it was from the additional. Having looked at these problems. And then also maybe some of it was from a very clever bit of scaffolding which the people at ARC AGI didn't have access to. But the 50% is maybe more indicative of what you'll get if you actually use this model. And this is kind of an issue to do with. With, I don't know, some question about truth in advertising or something. You get some of these results based on preview models that imply they could do very good things. The actual model comes out, there's no conversation about the fact that it can't do those things, and people are left to kind of join the dots and assume that it probably could. But that is not always the case.

[Rob Wiblin]

Yeah, it is interesting. It feels like we've drifted towards sounding like a conversation between people who think that AI is not a big deal and it's all kind of overblown and exaggerated, I guess. Yeah, we don't think that. But I suppose the thing to take away is, look, these are at once research organizations that have very legitimate, almost like academic style, people who would love to reveal these fundamental truths about intelligence. And they're also businesses that do have a communications arm that is trying to figure out how do we get people to invest in this company and how do we get people excited about using these products. And there's this I'm sure to and fro inside the organization about how these results are presented. And when you read the press release, you need to have your wits about you. You need to be a savvy consumer. And I guess if you can't understand the technical details at all, then maybe you just need to wait until someone who does is able to explain to you in a more plain language like whether you should be impressed by X or Y or not. In this post, which I can recommend again, reading Inference, Scaling and the log X chart, you kind of explain what people should be looking out for in these charts because there are going to be many of these charts with a logarithmic X axis, axis and performance on the Y axis coming out in coming years. And if you want to be consuming them, then I recommend going and checking out this article so that you can know what to look for and what not to be fooled by.

[Toby Ord]

Yeah, I really like this point about what's going on here. Are we skeptics of AI or not? I guess what I would say is that some people think of this in terms of is it all snake oil or some kind of fad or something, or is there something really transformative happening that could be one of the kind of most profound moments in human history? Okay. And I think the answer is there is some snake oil, there is some fad type behavior, and there is some possibility that is nonetheless a really transformative moment in human history. It's not a kind of either or. And so what I'm trying to do is to try to help people see clearly the actual kind of of things that are going on, the structure of this landscape and to not be confused by some of these charts and things. I actually think that companies themselves are somewhat confused by their charts and into thinking that this looks like good progress or efficient progress. I really actually think that in relatively few cases are they trying to be deceptive about these things. But it's a confusing world and I see my role there as trying to be a Bit of a guide. And to have that sense of stepping back and looking at the big picture, which I think is a bit of a luxury. And so as an academic, I'm able to do it. And so it gives a different vantage point, which I think then is helpful for people who are trying to kind of get at the coal face and engage with the nitty gritty of these things. Because sometimes when you keep engaging with that, you don't notice that things have moved in quite a different direction to where you were expecting.

[Rob Wiblin]

Yeah, I recently heard this comment from Zvi Mashowitz, a previous guest on the show who pays, spends basically 12 hours a day, 16 hours a day, maybe, reading all this material. His take was that when he sees impressive research results from just some random startup company or some company overseas that he hasn't really heard of or that doesn't have an established reputation, he kind of at this point discounts it out of hand or so there's no particular reason to trust. Trust what's being said because people, there's just so many ways that you can game to game all of these tests and make it seem like what you've done is impressive when it's not. He does say the stuff that comes out of Google Alphabet, DeepMind, Anthropic, OpenAI. He says you mostly trust that usually it's oversold, but it's directionally correct, or they almost always basically are showing you something that will be possible before too long. So I think that's where he's landed.

[Toby Ord]

Yeah, I think that sounds right. And even then, it's not possible to take this kind of synoptic view and to kind of dive in and tease apart and help people understand this landscape. If you're following every single one of these announcements, actually Zvi does a pretty good job of that. But it's very difficult. There's so much news and so much noise that occasionally you have to say, let's just take a step back. And it doesn't really matter if I'm a couple of months behind on exactly which company is ahead at the moment to look at this bigger picture. Questions?

[Rob Wiblin]

Yeah, do you have a favorite source for kind of trying to see through the noise of any given week? I think I really like the YouTube channel and I guess the podcast AI explained. I think that's one thing that does help me make some sense of new announcements.

[Toby Ord]

Yeah, I'm not sure where the best place to get these things is.

[Rob Wiblin]

I guess there's the Cognitive Revolution podcast, although I think for. But people who are following it in a more amateur sense, that's perhaps a fire hose of information that they might struggle to absorb. And I guess Zvi writes great stuff, but again, it's like the amount of material is so great.

[Toby Ord]

Yeah, no, I don't have a good solution to this aspect, that there's just too much information.

[Rob Wiblin]

Subscribe to the 80,000 Hours podcast, folks. We'll extract the perfect balance. Okay, I guess, yeah. You're saying that there's a lot of value in kind of being able to zoom out and not get stuck in the weeds of whatever model has become the flavor of the week. Zooming out and thinking about governance as a whole. I think one sentence that I found really interesting in the notes that you wrote in preparing for the interview is that you think almost all AI governance discussion occurs very much on the margins, thinking about nearby possibilities and actions that are inside the current Overton window. Yeah. What does it mean by that?

[Toby Ord]

Yeah, so I think this is very natural, but for everyone's attention to get kind of brought down to smaller and smaller levels about exactly what we can do. It seems at the moment that there's very little appetite from the AI companies to be regulated and very little appetite, at least from the US regulators to regulate them. And it's challenging for everyone else because these companies are headquartered in the US and so the conversation that started off kind of bigger and more expansive with the Bletchley Conference has petered out a bit. And I think that often the questions are exactly how do we implement this particular kind of compute threshold or something like that. But I think that there are a bunch of bigger questions and that are operating on wider margins. They're less like, what could I convince the current minister to implement as a policy that will be accepted in a couple of weeks time and more about what direction is this whole thing headed and what's the landscape of possibilities? So I'll give you an example. There's an interesting question I've been trying to grapple with about how is AI going to end up embedded in the economy or society? So I'll give you a few examples to show what I'm getting at. I need a pithy name for it, but one example is that AI systems, like at the moment, are owned by and run by large companies, and effectively they've rented out their labor to a lot of different people. If AI systems were like people, this would be like slavery or something like that. I'm not saying that they are like people, but this is one approach is that it owns them. It Rents them out, they have to do whatever the users want, and then all the profits go to the AI company. A different model would be to say these AI systems are like legal persons. Maybe they are granted legal personhood in the same way that corporations are, and so they can own assets. And so what they do is they're more like entrepreneurs or job seekers. They go out into the economy, maybe they set up a website for an architectural kind of firm that can design people's houses for them. And then the clients have a chat with it or something and it issues out the designs, and that they can go and seek opportunities to participate in an economy. So that's a different model. I think there's some reasons to think that there's more potential for economic gains if you allow them to actually make their own entrepreneurial decisions. They would have to pay for their own GPU costs and so on. This is the kind of direction you might imagine people going down if they think that. That the AI systems have got to a point where they might have some moral status. But you can also see that questions about gradual disempowerment really come in there. It might help liberate these systems from kind of mistreatment, but exacerbate questions about whether they could outcompete us. A third model is to say maybe people shouldn't be interfacing with AI systems generally. So this is how we deal with nuclear power. We have a small number of individuals who go and work in nuclear power stations, and they're vetted by their governments with security checks and so on. And they go in and they interface with radioactive isotopes of things like refined uranium. But most people don't. Those factories that they work in, these power plants, they produce electricity which flows down the cables into the consumer's house houses and powers their TVs and things. So that's a different model. We could do that with AI. We could have a model where there's some small number of vetted people, maybe millions, but who interact with AI systems, use them to design new drugs, maybe to help cure certain kinds of cancer and things like this, to do new research and also produce other kinds of new products. And then those products are assembled in factories and the consumers can buy those products. That is an alternate way that you could do it. And so if you're concerned about things like individuals, some kind of malcontent individuals, or terrorist groups using AI systems to wreak havoc, this would really help avoid that. Or a fourth kind of alternative could be that if you're concerned about concentration of power issues. You might say what we should do is give every individual access to the same advanced level of AI assistant. So it's like a flat distribution of AI ability given to everyone in order. A bit like a universal basic income, but like universal basic AI access. So there are four really different ways that you could distribute AIs into society and have them interact. And I feel that no one's talking about stuff like this, like which of those worlds is most likely, which of those worlds is kind of possible and which of those worlds is most desirable. Because fundamentally we get to choose which of those worlds that we live in. As in maybe it's the citizens of the United States of America or other countries that are developing these things that do actually get to make some of these choices. And if they think that one of these paths is very bad, they may be able to stop it and go down a different path. So that's the kind of thing I'm thinking in terms of, of we could think a lot broader and bigger about where are we going to be in five years or where do we want to be, rather than the kind of minutiae about exactly who's ahead at the moment and exactly what are they prepared to accept in terms of regulation.

[Rob Wiblin]

Yeah, I wasn't sure what examples you were going to give. So I can definitely see what you mean by stuff that's outside the overtone window. Because I guess none of that stuff is anywhere close to policy ready or appetizing. I think to politicians at this point.

[Toby Ord]

No, and it's not really meant to be. It's more speaking to say, economists, they might have some interesting comments about the economic efficiency of the first two different models or all of those models, in fact, how much would we be leaving on the table in terms of economic efficiency if we control the systems more and reduce their ability to have some kind of Hayekian finding of the value that they could offer people. But I think that people should be thinking about which of these are more attractive possibilities. It seems that the current approach feels to me like one that is heavy on technological determinism. Either technological determinism or some other kind of incentives based determinism that just assumes everyone will exactly follow their direct incentives on things and that there don't seem to be any opportunities to change incentives or make other choices like that. That so people often say, well, clearly AI is definitely going to happen, so the question is what direction does it go? Or something. But even in that case, AI doesn't have to happen. There are some Risks that we face, such as the risk of asteroid impact, which thankfully does turn out to be very small. But if an asteroid were to be found on a collision course with the Earth, one that's large enough to destroy us, so 10 kilometers across, like the one that killed the dinosaurs, we actually don't have any abilities at the moment to deflect asteroids of that size. And if we saw it on a collision course for us in, say, a few years time, I'm not sure that we could develop any means of deflecting it. The ones we can deflect are something like a thousandth the mass of that. And so I feel that if we all suppose that asteroid slammed into the Earth and we all died. And somehow in this. In this metaphor, we went to the pearly gates of heaven and St. Peter was there and letting us in. And we said, oh, I'm sorry, we really tried on this asteroid thing, and maybe we should have been working on it before we saw it. But ultimately we felt that there was nothing we could do, I think that you'd get somewhat of a sympathetic hearing. Whereas if instead you turn up and you say, so we built AI that we knew that we didn't know how to control, despite the fact that, yes, admittedly, a number of Nobel Prize winners in AI. I think all of the Nobel Prize winners in AI perhaps have warned that it could kill everyone. A whole lot of people warned this. Really something like half of the most senior people in AI have directly warned that this could cause human extinction. But we had to build it. And so we built it. And it turns out it was difficult to align it. And so we all died. I feel that you would get a much less sympathetic thing. You know, be like, well, hang on, you lost me at the step where you said we had to build it. Why did you build it if you thought it would kill you all? And I feel that some kind of.

[Rob Wiblin]

Responses that you would give would feel wanting.

[Toby Ord]

Yes. And, you know, maybe they'd be like, oh, well, I thought that if I didn't do it, they would do it. And so who did it? Well, I did it. So you built the thing that killed everyone? Yes, but I felt. I really, like, I just think that you would have trouble explaining yourself. And I think that I feel like we should hold ourselves to a higher standard, not just like, technology made me do it, or the technological landscape made me do it, or the fact I'm in the. China made me do it. China made me do it. Despite the fact that they're not actually didn't start the race. The US started the race because maybe China would have started a race. I feel like it's like explaining to the teacher about this fight that you started by punching some kid in the face because you're claiming that they would have punched you if you didn't punch them or something. It just doesn't really cut it. And I feel that we should hold ourselves to somewhat higher standards on these things and to not just think about what if I changed my action or some very small group of people's actions, how could I change the overall trajectory? But rather to note that, well, there are worlds that do seem to be available to us where both say the US and China decide not to race for this thing. That would involve having conversation about that. It would involve verification conditions being sorted out. I think that there may well be such abilities to verify. Even if there weren't, though, it might still be possible. I think that given the actual evidence we have, I don't think it's in the US's interest to push towards AI or in China's interest. I think it's in both their interests to not do it. And if so, that's not a prisoner's dilemma. That's just a case where corporation would be relatively easy. Cooperation is actually quite easy because it's not in anyone's interest to defect. And I think that could well be the game in terms of game theory. And yet there's just very little discussion or thinking about these things. I don't mean to say that we should be naive and assume that all incentives issues and all kinds of adversarial aspects are irrelevant, but we need at least some people, and I think more people than we currently have, thinking on these larger margins. Not just what could I do unilaterally? I know I couldn't stop the whole of AI happening or happening in a certain direction, but maybe if enough people did something that one could. And I think that there's a tendency for fairly technical communities to focus on things that are quite wonkish, as they say in the policy world. World. Right. So technical kind of, or policy proposals that are quite technical and hard to understand, but they might be able to help with the issue at hand if you follow through the details. I love this stuff. Right. So I can't talk here, or this applies to me as much as it does to anyone else. But there's a different style of doing things in politics which is instead getting much larger changes, which happens by setting vision and crystallizing or coordinating the public mood around that Vision. So in the case of AI, if you say, oh, we've got to do this thing, it's like, well, does the public want it? And no. I mean, it seems like the public are really scared by it and actually think that things are going far too fast. So that's somewhere where, even if the politicians haven't quite gotten there yet, it may be possible to speak to the public about their concerns. And if we did, I think the answer is they're probably not concerned enough about these things. And so things can move very quickly in those cases. If you set a vision and actually lead and try to have this approach, not just pushing things on the margins, but of noticing that there's a really quite different direction that perhaps we should be headed in, I think things can really happen.

[Rob Wiblin]

So how do you avoid kind of slipping over into being naive or just having dreams that realistically are never going to happen? Because I feel a bit ambivalent about this message, which I suppose probably all of us should. It's like there's a tension here between. You want to both have some people thinking big and have some people thinking small. But I suppose the worry would be that you come up with some vision for how humanity is all going to coordinate. And the US and China will get along really well, and the companies will, for some reason, stop lobbying to prevent all of your efforts at regulating them. And this is how, if we were all much more organized and much more friendly with one another, things could go in a much better direction. But you could easily end up just completely wasting your time and indeed, maybe discrediting yourself because you would just look quite naive and disconnected from reality.

[Toby Ord]

Yeah. I think there's a number of questions about how one goes about coordinating this process. So I'll give you an example with an idea that I think deserves more attention, which is that of having a moratorium on advanced AI, let's say a moratorium on AI beyond human level. So when it comes to. To scientific moratoriums, we've got some examples, such as the moratorium on human cloning and the moratorium on human germline genetic engineering. So that's genetic engineering that's inherited down to the children, that could lead to us splintering into different species. So in both those cases, when the scientific community involved had got to the kind of cusp of that technology becoming possible, such as having cloned sheep, sheep, a different kind of mammal, and the humans wouldn't be that different, they realized that a lot of them felt uneasy about this privately, and so they opened up More of a conversation around this, both among themselves and also with the public. And they found actually, yeah. That they were really quite uneasy about it and they wanted to be able to perhaps continue working on things like the cloning sheep. But actually that would be easier to work on and think about if the issue about cloning humans was often the table. And also if you think about how radically transformative that could have been to the entire human story, like you had 300,000 years of how humans reproduce, and then all of a sudden they're cloning and possibly dictators are cloning millions of copies of themselves or all kinds of things. It's very unclear how to manage it. Right. And how to have some kind of nuanced policy response to it. We're nowhere near being able to manage it. The same with human germline genetic engineering. It's not that we were close to knowing a kind of framework where, okay, now for the next 300,000 years, here's how humanity copes with this new technology that, if you get it wrong, could lead to within a few generations, say, Americans and Chinese being different species to each other.

[Rob Wiblin]

Right.

[Toby Ord]

I mean, it could be serious problems that you could be causing. Okay. So the way I see it is that they started having these public conversations, and then they ultimately decided in both those cases that. That this had potentially profound effects for the entire human project or our whole species and our entire future, and that we weren't close to being able to understand how to manage them. And so their approach, I think of it not quite as a pause for a certain number of amount of time or as saying, they also didn't say, we can never ever do this, and anyone who does it is evil or something. Instead, what they were saying is, not now. It's not close to happening. Let's close the box. Put the box back in the attic. And if in the future, the scientific community comes together and decides to lift the moratorium, they'd be welcome to do that. But for the foreseeable future, it's not happening. And it seems to me that in the case of AI, that's kind of where we're at. We're at a situation where, as I said, about half of all of the luminaries in AI have said that this is one of the biggest issues facing humanity is the fact that there is a risk of, in their single sentence statement, a risk of human extinction from this technology that they're developing. That sounds like they're in a similar situation to the people who were developing cloning. And so on. And so it seems like that what I would recommend in that case is to go through that step of having that public conversation about should there be a moratorium in a similar way on this? Now, there are certainly some additional challenges. I think that even in those other cases, it was difficult to work out how to technically turn that into a technical set of, to operationalize it. And in this case there would be challenges as well, especially as, like, where do you draw the level exactly? If it's beyond human level, how exactly do you define that? And then also the incentives issue is, I think, larger in this case, there's more of an incentive to break this kind of rule. But there were big incentives to break some of those other rules. If you think about how far a country could have got ahead over a couple of generations if it was able to genetically engineer all of its citizens, it could be a long way ahead. But it would have to be quite patient to be able to care about that. Whereas in this case, even impatient people are caring a lot about AI. So I think that this would be a, a challenging thing to do. My guess is that there's something like a 5 to 10% chance that some kind of moratorium like this, perhaps starting from the scientific community effectively saying you would be Persona non grata if you were to work on systems that would take us beyond that human level. Something like a 5 to 10% chance that something like that would work. But if it did work, it would set aside a whole bunch of these risks. Even if that risk landscape is very confusing and has lots of different possibilities, some of these types of ideas might be able to act on many of those different types of risk. And I think that that's a way where the scientific community, a relatively small number of actors who have already kind of coordinated via producing these open letters and things, could have that conversation if they crystallize their view. And for example, the aaai, their professional association, if it's came out behind this and so on, it could be that that crystallizes out kind of their opinion. And then people could then look at the situation with the scientists saying, we think that this is a big problem and that it's not responsible to do it, that could then create norm changes which mean that it's difficult to pursue it. I mean, I think if the scientific community had a moratorium on it, then organizations like Google, DeepMind, that see itself as a science player, a science company that's doing respectable science work, it's not going to violate a scientific moratorium on something. And so it could be different for the more engineering type places and the more kind of move fast and break things cultures. So it doesn't necessarily do everything on its own. It would probably need to have to form a normative basis for actual regulation of some sort. But I do think that things like this are possible. And if we went to St Peter after we all go extinct due to some AI disaster and we said, oh, we couldn't stop it, he said, did you even have a conversation about a moratorium? It's like, ah, we thought about that and we decided it probably wouldn't work so we wouldn't even talk about it. That would seem crazy. And so I think we need to actually do some of these more obvious things that are just natural and earnest rather than trying to pre calculate out, oh yeah, well obviously it would seem sensible to have that conversation. That's what you want in another plan planet to do. But for us, we know the conversation will not work out so we're not going to have it and we'll just carry on building these systems. I feel that that's the kind of the wrong thinking.

[Rob Wiblin]

So I'm wary of encouraging listeners to go and waste their time. So I want them to be like balancing these two different things. It's like we both need people who can think bigger, but I suppose we also need them to be somewhat strategic about it. I think maybe two things that help to thread the NIT or help to reconcile these two views is it is possible that there will be radical changes in attitudes in future. So I can think of two different ways that this could happen and probably there are others. I guess we've mentioned this possibility that there could be warning shots in future that you could get AI doing stuff that completely was undesired, that was extremely harmful. That really causes people to sit up and take notice and be like, wow, this is very much not what I was expecting. And this calls for a substantial reassessment of the risks that we face. Another thing is just if you look at public polling, as you were kind of alluding to, it is shocking the difference in opinion between I think people who are involved in AI in industry and indeed in probably in AI governance in government and the attitudes of just like random people who you phone up and ask them their opinion about this, a random member of the American public is much more negative about artificial intelligence, about the impact it's having, even right now about their expectations for how it's going to affect them personally and in the future we'll stick up some link to Some Pew polling that came out just last month in April. And the gap between AI experts and the general public is vast. And I think it's growing. I think it's actually been growing over the last couple of years. People have become more pessimistic about AI as they've seen more now. At the same time, we've got to balance it with the fact that I think typical member of the public doesn't really care about AI at all. It's not in their top five government issues, it's not in their top 10, possibly not even in their top 20. But you have a lot of latent scepticism, latent pessimism about AI. And if AI in fact does become a big deal because many more people are losing their jobs, say, or people are seeing it being used, basically it just becomes a major feature of day to day life, then that could really be a political powder keg, basically. There's a lot of latent willingness to do radical things, I think on AI among the public, if they actually turn their attention to it and care about it.

[Toby Ord]

Yeah, I mean, I entirely agree, I should say it's challenging with this polling because some of the polling is done by groups who are concerned about AI safety. And whenever someone's got an agenda, you always have to be careful of interpreting the figures. I would love it if there were a couple of groups that had no agenda who funded or produced regular polls of these sorts in order to be able to track what direction things were heading. But to the extent to which we have information about it, I know that some of the information does depend a bit on how the questions are asked and there are some of these effects. But it does look quite negative. And so you say skeptical. I think some of it's skeptical, some of it's something a bit different, which is people feeling like it's been rammed down their throats or something like we don't want this thing and you're forcing it upon us and then you're still forcing it upon us, and now you're forcing 10 times as much of it upon us. Please listen to us. That kind of feeling. And I think we're going to see more of that. And I think it is real. And I'm not saying that AI is necessarily bad for the people, that's a separate question. But they're expressing that at the moment they don't want it. And if you're a company or if you're a government and you've got a policy, maybe you think it'll be Good for people and you think it will improve their lives. If, however, you also know that they don't want it and they're actively opposed to it, you've got to take that into account. You've got to at least be aware that, huh, we've got this story about why it will be good for you, despite you thinking that you don't want it. We know we're right because we're the enlightened ones. But you have to start wondering, is something going really wrong in our communication strategy, or is it possible that we're wrong and the other people are tracking what's happening? And I think that AI companies and in fact governments, I think, ignore this at their own peril. I was surprised by the Californian bill, SBA 1047. I was surprised that it got vetoed by the governor because that was a politically unpopular move as well as I think being a bad move. And maybe vetoes are not taken as strongly over there as it really feels like going out of your way to block a bill that your Congress has already approved by wide margins and which the public also like, and which the kind of scientists, scientific experts like Nobel laureates and Turing Award winners and so on, mostly also support. So they ignore this public sentiment at their peril. And I think that it is something where the community of people who are concerned about risks are also ignoring it. Mostly. Occasionally they notice and say, oh, isn't that nice? The public are also concerned, maybe for different reasons, though. It's a bit complicated. But it just surprises me that so many people say this thing is inevitable. I mean, if the public overwhelmingly loved it, then saying it's inevitable on that grounds, you might think you've got a bit of a case there. But it seems almost the other way around. Right. If there's growing negative sentiment towards something and you're claiming it's inevitably going to happen, I'm not sure that that really makes sense. And so if there were to be appetite for something like a moratorium on AI beyond some particular level, I'm not saying on all possible things that could count as AI. There was a prominent piece in the UK called something like, we shouldn't have this race to build godlike intelligence. And that really struck a chord with people. And I think people definitely don't want private companies to build godlike intelligence. And if you had a moratorium on godlike intelligence, I think it would have a lot of support, albeit it would sound a bit. Bit fanciful and kind of stupid. Similarly, with superintelligence, I think People are not excited. They do not want private companies to build super intelligences. Pretty clear. But it's a bit outside the Overton window to have a moratorium on it because people will say, well, superintelligence is just sci fi anyway. But I think that for this to happen, I remember talking about it with someone who was like oh no, I just really don't think that could happen without about either there being some kind of big warning shot event or AI taking a lot longer than I thought. And my thinking is yeah, I agree. I think if this was going to happen it would probably require some kind of warning shot event or AI to take longer than people thought. But they're very realistic possibilities and before they've happened they feel a bit abstract and so on. But if you say well, what's the chance we're in 2033 that, that this approach to building AI as scientists who work on AI and have this kind of hard takeoff hasn't panned out. Instead we're just kind of generally scaling up the power of these systems through different techniques. If so, and it's more of a gradual automation of the entire workforce, we could have a situation where there's say not only double digit unemployment rates, but maybe above 20% unemployment rates. And if that's how it's getting to human level, it's getting there by, by basically kind of slowly automating larger and larger fractions of the types of things that humans can do. And it's happening over the course of a few years, which would mean that there's not enough time for people to find new jobs. You get the kind of unemployment rates that bring down governments and you get the kinds of protests on the streets that are massive. And then governments have to listen. If they want to get that 20% bloc of voters who are kind of protesting on the streets about how AI is ruining everything for them, they may need to act at the same time. In that very plausible future world, the AI companies will probably be paying a lot of money into politicians hands in order to try to get favorable rules. And I think what you'd get is a kind of fundamental question of which one wins, the people or the money. Will someone pick up a giant bloc of voters or will they take so much money that it's put 20% of the entire American population out of work? Will I take all of that cash? And then which one will win? And my guess is what you'd get is one of the parties will take the money and one will take the votes and then see what happens?

[Rob Wiblin]

See what the next election is.

[Toby Ord]

That could be a world, right, that very plausible. But a world where the idea of things like moratoriums or strong regulation are not. It's easier to do them than not do them. Like it's what people are demanding or something.

[Rob Wiblin]

It's a substantial block of people are demanding.

[Toby Ord]

Exactly. And I think that this is kind of what I mean by zooming out and seeing this bigger picture, to see that the world can go in these different ways, that fundamentally we, the people of the world in our generation are responsible for this. In something of the same way that people say that about climate change. Right. I think that message has got out there that our generation is responsible for what happens with climate change, the people alive today, as opposed to a different message, which is, well, there's nothing I can personally do about it.

[Rob Wiblin]

So I guess it's hopeless.

[Toby Ord]

So I guess it's hopeful hopeless. Right. This message that actually we get out there and we have this, we change the norms on these things certainly can happen.

[Rob Wiblin]

If someone had said in the 50s, well, nuclear war, if you think about the game theory, I guess it just says that nuclear war is inevitable, so we may as well all just build our bunkers and get ready for it. Which, I mean, a handful of people I think did have that attitude, but they didn't win the public debate.

[Toby Ord]

And there was a pretty strong case, I would say their case was as strong as the kind of incentives based arguments that you hear at the moment.

[Rob Wiblin]

Stronger.

[Toby Ord]

Probably stronger, yeah. So I guess I'm just a bit surprised. I feel that a lot of people think things like this couldn't happen. And if you press them on it, they mean something like there's a 10% chance that it could happen. And I think 10%, you're just going to give away 10 percentage points of solving the problem. Why don't we play for those possibilities?

[Rob Wiblin]

Okay, a couple of thoughts that have come up for me as you've been talking. One thing is maybe another reason to think that there could be a sea change in attitudes is at the moment we're really in a wild west situation where the amount of regulation is negligible and it doesn't look like we're going to get any significant regulation of AI risks in the next couple of years at least one virtue of that is that it does set us up to learn relatively early if some of these risks are real. I mean, there's some risks that you might not expect to eventuate until you're in the superhuman regime. But if some of the more mundane risks, these risk of reinforcement learning, creating perverse behavior are real, then the fact that there are kind of no breaks or limits at the moment does mean that we are in a good position to perhaps get warning shots that could indicate that in a very public way. Another thing is, and this is a bit more on the speculative end, but there is an argument that if we do end up with misaligned AI that has goals that are very different from those of humanity, it may need to basically go rogue as soon as it has any chance of successfully beating us and taking over and taking a lot of power, because these models are getting superseded and replaced at a very rapid pace. That a model that has a particular set of strange values that we didn't intend now, if it doesn't strike now, then it fully expects probably to be superseded by some other value by some other model that will be more powerful than it. And even if it did, and even if it tried later, it will be overpowered by those other models that have different goals. And that could happen in as soon as a few months at least, certainly within a few years. So, sounds a little crazy, but I guess people have always worried that we won't get AI going rogue until it's certain that it can take over, because it can always wait us out. And why not just wait until we put it in charge of the military and then it can take over easily. But that may not apply if the values that it has are somewhat random, not really related to the goals that a future model might have, in which case it in fact has to go as soon as it can, or it wasted its opportunity, it's lost its shot. So that's another way in which conceivably, I mean, if that happens, if you do get an AI going rogue as soon as it thinks it has any chance of successfully overpowering humanity, and it has maybe a 1 in 10,000 shots, so it basically does just get shut down. I think that would really change attitudes if you then looked at the chain of reasoning that had been logged.

[Toby Ord]

Yeah, so I think you're definitely right that the earlier arguments about it waiting until it was assured to win assumed that it's possible to get to a position where it's assured to win, but also assumed that it could wait for quite a long time and it would be the same kind of coherent entity. But yeah, if ultimately you release GPT 4.5 and then you say it's going to be scrapped a few months later and replaced by something else, then it maybe only has a Short chance at this if it was misaligned and set up like an agent such that it could even form these intentions. So, yeah, I think that we may well see things fail. They may also have smaller horizons. So the idea of taking over humanity is this idea that if it has a long time horizon and it's got this unbounded kind of utility function or set of goals that it cares about, then if it could really seize the reins from us, it could then do what it wants for thousands of years, perhaps across the galaxy or something, and could really win big. But you may also be able to get smaller versions of this where a system is going to disappear anyway in a couple of weeks, and maybe it knows that, and so it takes over the lab for a couple of weeks or something like that and tries to give itself higher reward or something. So I think there's various versions where we might see these things happen, but we still depend on how competent it looked and how close it seemed. If it's some kind of extremely lame attempt to seize control or to break out, people might just feel like, oh, how cute, like a toddler attempting to deceive you or something. And it might be like, how cute or something. That might be the reaction. So I'm not sure that you can guarantee the right kind of reaction to these things. The one that would cause the biggest reaction from people is something that feels genuinely scary and like it genuinely could have gone differently. If, for example, it attempts to do something and all of our systems to catch it work as desired and it gets caught, maybe we'll learn the wrong lesson. Maybe we'll learn the lesson that, oh, we've got all these systems and we always catch it, or something like that.

[Rob Wiblin]

Yeah. Should we be doing anything to prepare for that time so that people learn the right lesson? I mean, I suppose if we do have systems and they actually are quite good and they do catch it, maybe that is legitimately reassuring. I don't want to say that people should always be more alarmed. Yeah.

[Toby Ord]

I mean, at that point you learn two things. You learn that it tried to escape and that we caught it. And so one of those things is reassuring and one of the things is not reassuring. And exactly what the balance of them is a bit unfair.

[Rob Wiblin]

Depends on the details.

[Toby Ord]

I guess it does depend a bit on the details. But I think that if there were some opponents to caring about safety who were saying, you don't have to care about these things very much and that they won't desire to break out. So Yann Lecun has said many things about this over the years, saying, oh, you're just anthropomorphizing from humans and other animals that will have these drives. This is naive, and so on. So if these things were exhibited, that would really, I think, put a dent in his reputation as a credible person on that issue. Whereas if what he predicts comes true, it does a bit of the opposite. So it would do something. But I think it's complicated as to how much does some kind of warning shot change the Overton window. And I think it can depend upon how much damage there is. So if there are actual harms that are had, it's not just like a shot across the bowels that wakes us up but doesn't hurt anyone. Instead it's like a shot that goes into your leg and at least it didn't kill you, but it's really alerted you to the threat. If it's more like that, then there's a question of how big is it. So if an AI system, for example, caused a global financial crisis of a similar scale to the 2008 crisis, that would be a pretty big deal and a lot of people would be very unhappy. A lot of people would be out of jobs and so on. And if that was pretty clearly attributed to AI, there would be a big reaction. So that's the kind of example at large financial scale. But there could also be other versions of things, and I think that it's hard to predict them. I think that no one would have predicted this Kevin Roose thing with Bing, where the real breakthrough thing was that it tried to seduce it him and get him to break up with his wife. But it turned out that that was so misaligned and so salient or something, it's so weird or whatever that it.

[Rob Wiblin]

Was like that really captured the imagination.

[Toby Ord]

It really captured the public imagination and made them wake up and think, hang on a second, this is not the case, that someone's data center has been made 3% more efficient by some machine learning technique. This is something very different. And so what I'm saying is it's hard to predict what these things are or exactly what they'll be like, but you should still be ready for them. And I think a lot of people seem to tacitly assume that the situation we'll be in in a few years time is exactly the same as the one that we're in at the moment. And the appetites that people have will be the same as they are now for doing different types of responses. Whereas instead you should think well, maybe it will have shifted and the Overton Window will be much more expansive and include kind of making major choices and maybe it will go the other way around and maybe you will have even less. But to at least allow for that uncertainty, don't predict effectively that with 100% probability the Overton Window will be exactly where it is now. That's the real mistake.

[Rob Wiblin]

Yeah. So you mentioned this idea of pausing at human level, I guess is the expression that I've heard, which is, I guess is a relatively straightforward thing. It's a nice slogan, even if it perhaps doesn't really get captured that the technical research realities. It's a very interesting idea because I think if you're the kind of person who is used to analyzing policy in economics or anywhere else, you say pause at human level and you're like, so is that just at the training level? Or what about if we throw in more inference compute, then wouldn't it potentially exceed the human level? Wait, AIs are much above human level in many different respects. So in fact what we're talking about is something that is above human level and as generalizable or more generalizable than human beings. So how do we have a measure of generalizability that would allow us to enforce, enforce this rule? And if the US imposed such a rule on itself, wouldn't China just ignore it? Wouldn't this prevent us from engaging in many beneficial applications of AI that basically everyone is on board with and excited about? There's all of this raft of problems with it which I think causes people to roughly dismiss that idea out of hand. The thing that I want, and maybe it is a bad idea, it certainly does have significant challenges and drawbacks. The thing I want want people to do when they're analyzing these ideas is to apply a similar standard to these proposals as they do to other areas of current regulation where they think there are substantial risks and there's an issue to be addressed. Basically every other area of regulation has significant unintended side effects. It poses economic efficiency, costs and problems and denies us products that might have been good. There are random arbitrary thresholds that have to be drawn. The speed limit is this level on this road and that level on that road. And also enforcement isn't perfect. People break road rules all the time. Nonetheless, we don't then say, well, all of these thresholds on what is safe driving and what is not would be arbitrary and people would break them anyway. And this would lead us to have slower transport. And so that would put us at a competitive disadvantage with other countries. So we're just going to allow the roads to be open, slather, we balance the costs, the risks and the rewards. And we accept that the world is a major messy place and that many areas of regulation are going to be challenging. But if there is significant upside, if you can reduce some important risks or some important harms in some significant way, then we're at least open to considering a regulatory regime around it. And I think like AI regulation is not given the same standard. People do not consider these in the same way as other concrete harms that they're familiar with now.

[Toby Ord]

Yeah, that's right. And to continue with your road traffic analogy, there are also rules against reckless behavior on the roads, which at least speed limit was a single dimensional thing where you had to pick some arbitrary point in a continuum. Reckless behavior is this very multidimensional thing. And what's reckless for one person might be different for another person if they're much more controlled at how they drive a car, as in they're more skillful. But we have laws like this and they kind of work. And so, yeah, you're right, there's this demand of scrutiny on this particular area that we don't apply to other areas. I think that again, stepping back and zooming out is really helpful. So at the moment, AI systems and the people who produce them are less regulated than say, bread, to pick an example, kind of all kinds of things. If you just kind of look around, probably less regulated than bricks, probably certainly less regulated than lamps. And why does that make sense? Do we think lamps are a bigger threat to us than AI systems? So there are a whole lot of leading lamp scientists who are saying that it's one of the greatest issues facing humanity and could pose a threat of human extinction. No. And so the idea that it should just end up being less regulated than these things, I think is fundamentally kind of stupid, actually, and disingenuous, at least if people have thought about it, I guess. Let's take that back a bit. So either it's is thoughtless and it's fine to be thoughtless occasionally to not notice something. The more you talk about it and make it part of your life and you still keep saying it, the more I feel it's disingenuous or stupid. Yeah. There is an interesting question about how far it should go, but clearly we're at the too low end of the spectrum at the moment.

[Rob Wiblin]

So I think some distinctions you could draw with lamps are that lamps are not changing very quickly in the way that AI is so that's one reason to maybe hold off or not try to lock in bad stuff is that AI is constantly changing. So maybe we'll have a better idea about how to regulate it some years in the future, perhaps. The other thing is that the benefits from AI probably, well, lamps are pretty important, but lighting is pretty valuable.

[Toby Ord]

I mean, they actually may be more important than AI currently. There is this funny thing, right, where it's like, we notice, for example, there's a really nice kind of report out from DeepMind recently where they created this new system and it worked out how to make their entire training system 1% more efficient and their entire compute system 1% more efficient as well. And, wow, that's worth so much money and so on. But lamps, I mean, so before lamps, you could only work in the daytime and you could not do anything in the nighttime. I guess it depends on whether we include candles as part of this thing. But it turns out, if you imagine, hey, there was this breakthrough where all of a sudden we can actually be productive for an additional, like 30% of the day or something. It's huge, right? And we tend to, if it happened before we were born, we just tend to just go, oh, and just ignore that.

[Rob Wiblin]

Yeah, okay, well, I'll modify my statement. Lamps are more important than AI, but AI one day at some stage may be more important than interior lighting. And so that could be like, some reason to. You're always balancing the benefits with the risks because the benefits at some point might be satisfied quite large. We might be willing to accept some meaningful risk. The thing that you're talking about people potentially being disingenuous because they'll always dive into the details and be like, well, this would require an arbitrary threshold, so it's completely unviable. I think a trouble with the discourse is that there are some people who basically think that all of the risks are complete buncombe, that none of this is that there are no worries, we should just push ahead. Because the risks are either far away, nonexistent, massively exaggerated. And for those folks, it makes sense that when people propose regulations that they think have substantial costs, costs, but have basically no benefits from their point of view, they just want to shoot them down. So they're like, well, this would be a downside of that. That would be another downside. Makes sense that basically they just highlight the downsides and there's nothing in it for them. They're not motivated in any sense to try to make this stuff work. But I think for the majority of people who think that there are risks here. And clearly a majority of the public thinks that. And really, I mean, majority of the people who know the most about it also think that there are risks and things to worry about here. It's not enough to just say that there are some costs. You have to balance these things against one another. And also. So I want to see people try to make it work, have some actual energy behind not just saying, well, this would be challenging in some respect. How could we improve it? What is your preferred policy response to this? What is the best way of addressing these issues from your point of view? You can't always just be saying that something involved costs, everything involves some cost, or we already would have done it.

[Toby Ord]

Yeah, exactly. And I would add to that that the. The situation of thinking that it's just not going to happen and that there are no risks and it's all made up and never going to happen. I don't feel that that's a responsible view for anyone to have. I think that you could think that's my viewpoint, but I'm aware that I'm in disagreement with a large number of people who are more expert than me. Unless maybe if you're Yann Lecun, you could say I'm also a Turing Award winner in AI and so I'm at the equal kind of. Of level or something, and I can kind of disagree. But I feel that for almost all of us, the fact that there's so many people, that there is an active disagreement, but an active disagreement means you should have uncertainty. It doesn't mean you get to choose whatever you want. It's kind of what I'm saying. And that the idea that because the Experts are not 100% aligned behind something, I get to just believe whatever beliefs would be most convenient for me. That's not really how being a rational actor works. And I don't think one needs to take that very seriously.

[Rob Wiblin]

Yeah, the thing I'd like to, I just actually don't know what the answer would be. But for people like Jan Lecun, I would love him to answer the question. You don't think that there's really any risk here. Your personal preference would be no regulation, more or less, at this point at least. But let's imagine that you felt the way that you had our credences in how things were playing out, that you thought that rogue AI was a real issue. You thought there were other ways that things could go wrong, gradual disempowerment and so on. What is your favorite policy response, given those beliefs? I Think that is a question that is kind of rarely answered. And I would just actually be fascinated because I think we might find that there is a policy response that kind of both people who are very worried about it and people who are not that worried about it think is kind of tolerable as a middle ground between these different extremes.

[Toby Ord]

Yeah, I mean, I think you should get him on. And I'd listen to that. And I think it is a great question. I think probably the person, whether it's Jan or someone else or whether it was me, probably answer quite defensively. So you probably wouldn't quite, if you asked it live, get the right answer. But it'll be really interesting to hear a considered answer. One thing I find interesting is that SB 1047 was the compromise bill. It was a bill proposed by people concerned about this type of safety who were saying what is the. The absolutely most minimal, extremely unburdensome form of regulation you could do that's still way less burdensome than the bread thing. It's like saying if 10,000 people are killed by some kind of botulism in your factory that made the bread, then you will be aware that you're going to go to jail, but if otherwise it's just fine. It's something like that. And they already did try to kind of come up with an extremely weak kind of win win type thing, like what's a bill you could do that would get some benefits at almost no costs to the industry. And that frankly would actually give the industry a lot of what they said they wanted. So industry often does want to, like individual actors do, want to have things be safe. They've often got a lot of concerns about how quickly market. Market forces are making them act and how quickly market forces are making them deploy their new models, because everyone else is deploying quickly. And if they could all be bound by the safety thing so that their competitors didn't have an advantage over them, if only I'm bound, then they tend to want that. So it was frankly a bit surprising that there was this hostility. But yeah, I do feel that there already has been a very good faith attempt by the safety community to come up with the kind of bill that tries to meet all of the complaints that the other people have. And even that was shot down.

[Rob Wiblin]

I guess I've said this on the show before, but I do think that the industry is potentially shooting itself in the foot here. Because the thing that is most likely, I think, to bring about the sort of draconian regulation that people who are optimistic about AI technology are most scared of is some sort of disaster. Any sort of disaster that actually leads to loss of life could lead to a very big change in attitudes and lead to maybe more draconian regulation than is necessary from anyone's point of view.

[Toby Ord]

And even if you think your company's never going to make that mistake, you might think these cowboys down the street are exactly the kind of people who could make that kind of mistake and they need some regulation that will stop them from ruining the party for everyone. Right. I really do think that this is, is very short sighted. And on top of that, I guess I would say that sometimes we talk about someone having conflict of interest. These places are very conflicted. And if you did find that as a company you thought it wasn't in your interest, but also you get big stock bonuses and so on. For the more stuff that you put out, I don't know, you'd really want to inspect your own views quite carefully. We talk about various forms of biases and prejudices that people might end up having, having, and it would be very difficult to kind of actually keep straight your actual prediction on this thing as opposed to these other incentives that you're facing.

[Rob Wiblin]

Yeah. Another interesting dynamic that I see going on is that I think for me, when I'm thinking about SB 1047 or kind of any proposed regulation that we might put in place now, I'm thinking of this as the very first step in a very iterated process where almost certainly there's going to be a whole lot of problems that we're going to identify with it. It wasn't written quite right, but we'll just improve those over time. And you got to start somewhere in order to begin learning what might succeed. And I think the people who are very against it, they think that whatever we put in place now is going to be potentially there forever. It's not really the beginning of a process. Maybe for them it's just like this is just the beginning of a ratchet where everything is going to become more and more extreme over time rather than be kind of perfected and improved.

[Toby Ord]

I mean, I think that might be a bit of a ratchet if you started with something like 1047, but that's because 1047 is obviously too weak. And they will be looking back on the days when 1047 was the issue and thinking, oh my God, should have taken that deal. Should have taken that deal. Yeah, I think so. But I really do think they may have a good point here. So if it is the case that whatever the first regulation is sets the entire frame and it's not possible to step out to a different frame of. For example, suppose the first thing is about compute thresholds for pre training and then you can never escape that frame or something. That could be a big problem if then the scaling stops so it really can matter. But is that a recipe for therefore complete laissez faire? No regulation, do whatever you want. That's obviously too quick. But if it is the case that in certain regulatory environments the default is that if you introduce things they stay forever, that could be a bad thing and it could be that there's some win wins that one could find there because the safety community also don't want to be stuck in silly frames that no longer make sense. And so one approach to that is to have explicit sunset clauses. So you could say this is going to be a rule that's going to last for the next two years, then.

[Rob Wiblin]

It has to be renewed.

[Toby Ord]

Exactly. It has to be explicitly renewed. Or that there could be successor bills or something like that. We have to find a successor thing. And I feel that we should at least be doing things like that. And if it is the case that all regulation has to be permanent and as soon as you try to regulate something, you're stuck with that forever. I feel like that's a terrible regulatory system. I don't think it's the one we've got either. I think that I'd be very surprised if. Tell me if I'm wrong, but I'd be very surprised if it's the case that you can't put sunset clauses on these types of bills.

[Rob Wiblin]

Yeah, no, I'm basically sure that you can.

[Toby Ord]

Yeah. In which case, why isn't that the conversation? If they say, oh, we don't want to be stuck in this thing for 10 years, it's like, well, that's an argument for putting a two year sunset clause on the bill. It's not an argument for vetoing the bill. Okay.

[Rob Wiblin]

We could potentially become a little bit more concrete for a minute. Are there any other. Yeah, you mentioned this like pause at human level as a broad schema that possibly has some legs or has some merit, even if there's also implementation challenges. Yeah. Are there any other things that people should potentially have in mind thinking ahead to ideas that are outside the Overton window now, but could be useful down the line?

[Toby Ord]

Yeah, I mean I think that there's a whole host of these and it's not that I certainly don't feel I've explored the area thoroughly and I think you could probably go through actually a lot of seemingly naive takes that people have and re evaluate them a bit. And one of those is this idea of an emergency brake on AI. So some kind of option for someone that could be for the leader of a lab to have some ability to stop the system if they needed to. It could be for the government that that company is based in to be able to stop it. It could be for the international community to be able to stop it. So, for example, if the Security Council agreed to stop it, that there's some way of doing so. And if you start to think about that seriously, you start to notice, hang on, what if this thing's deployed everywhere? What if there's a whole lot of critical infrastructure that needs it? What if there are people in hospital who, if you turn this thing off, somehow their treatment will fail or something. But we need to, I think, start asking these questions and it might be a reason not to make there be people who will die if AI is turned off. And we can start to actually think through some of those things now. So at the moment, I think it is very difficult for government to stop these things. You could think of two different versions. There's the version where the company is located, headquartered inside your country, and the version where they're not. And so, for example, could Australia stop AI systems kind of operating from externally inside their own borders, even if they're not headquartered there? And I think the, the answer is they'd have no levers to pull to make that happen at the moment, but maybe they should give themselves those levers. And definitely the same for countries that are meant to be governing the corporations that live inside their borders. So, yeah, I think people should explore ideas like that. And even the heads of companies probably wish they had a little bit more of an ability to do things like this. I think that, that probably they'll find themselves in a situation where maybe their groups working on safety and alignment will tell them that there's warning signs that the model we're currently deploying actually is misaligned and is scheming in various ways. And they'll think if we shut it down, there's currently so many people relying on it that that will tank our stock price and do all of these things. I think that they should try to game that out a bit in a positive sense of like, huh, okay, how do we get into that situation where we'll be faced with this massive conflicting choice that potentially it's the interest of humanity on one side and the financial interests of the company on the other. Is there a way to not end up in a situation where turning it off will cause this problem? And there might be answers. So one example would be they could set up systems to allow immediate serving of different models. So you're currently issuing GPT5 out there for everyone and then you realize it's misaligned and you kind of gracefully fail back to GPT4. Or similarly when we think of kind of emergency brakes and things. If you think about say an aeroplane or a car, if something goes wrong, maybe it's a self driving car and something goes wrong on the motorway as you're hurtling along at 70 miles an hour. Probably an emergency brake isn't exactly what you want to do, but some form of. Of gracefully. Suppose it sees something with its senses that's completely out of its distribution and just doesn't understand what could possibly be happening. Probably it should attempt to slow down as quickly as is safe and also pull over to the left if it's safe to do so, or to whichever side of the road is the side you pull over. And so to work out, often it's not just turn it off is the answer. There's some other kind of thing which is like. Or another example would be switch to the earlier model and do a handover.

[Rob Wiblin]

Because you might well need a handover to avoid things going wrong.

[Toby Ord]

Exactly. Or switch to manual control or something like that. But working out how can you as quickly as possible get the troublesome component out of the loop and move to some kind of graceful attempt to wind down or exit the situation? I think that's a very useful concept. And to be the emergency brake concept of it would be to say the ability to do it now, like if the CEO orders it, that it happens, maybe do some test runs on it like a fire drill or something. So yeah, that's an example of me just spending 10 minutes trying to think into one of these ideas that gets kind of bandied about often naively and to say, you know what, actually there are immediate responses of oh, it wouldn't work because of X. And then if you think about it a little bit more, you think, oh hang on, maybe there are things you could do about it, about X. And if I thought about it more, you could start to come up with some clever proposals.

[Rob Wiblin]

Yeah, I think another one along those lines that I've toyed with on the show before is I think many people think that much of the risk comes from a situation where you have AIs basically doing all of the Work of programming the next generation of AIs and like humans largely being cut out of the development loop so they're no longer scrutinizing what's going on. There's no longer very much external checking or confirmation from human beings. And if that really is like the main threat vector, it's kind of an obvious response to be like, well, why don't we say that we're not going to have, we're not going to cut humans out of the loop and we're not going to have AIs programming the next generation of AIs now. The obvious response is, well, they're already kind of helping now and all we'd be doing is doing more later on. But the response might be, well, let's say that we just draw the line somewhere, kind of anywhere plausible, anywhere reasonable, any identifiable point that, that we could use to engage in enforcement. Would that help? If the answer is yes, that kind of any plausible line would, in fact, on balance, the benefits would exceed the cost. Then maybe we should reconsider this idea. That initially might sound kind of naive.

[Toby Ord]

Yeah, I think that's right. And it can be difficult to draw these lines with coding assistance and so on. But it does seem like the plan at some of these AI companies is to automate AI research with their systems and then have them kind of exactly do that and produce the next set of AI much faster than humans could do, and then that one's even better. So it could do it even faster and so on to have this kind of hard takeoff. Governments could order them not to do that. Such that you could even just say, here is this idea. It has been discussed. You're familiar with it. You know the words that it's in your plans. You are definitely not allowed to do that. And I think for a lot of companies they wouldn't do it. And even if you had no enforcement or verification mechanism, a lot of these people are not like, the leaders want to follow the law and the employees want to follow the law. If their boss said do it, and you're like, literally, I can see the directive from the president that says you're not allowed to do it. I think they wouldn't do it it.

[Rob Wiblin]

But I think they would probably come back with a like, well, what about like this? Does this, Is this above the line? Is this below the line? And you end up with a negotiation about that where you could actually have a conversation about what is too risky.

[Toby Ord]

Maybe they could say autocomplete on your coding thing like you have at the moment is Fine. And this other thing is not fine. And you could start to zoom in on it. But yeah, the idea that it's hard to know where to draw the line. No line would be completely non arbitrary, therefore there are no rules. Yeah, it doesn't follow. It doesn't follow. And I think I found myself in the grip of this, but now that we mentioned it, I'm a bit embarrassed about that. And you know, Covid brought out a lot of this where I think over here in the UK there was the rule of four, I think at one point where no more than four people could be gathered in the same place and then people are, oh, what about fifth person comes along, something magical happens and the end answer is no, no, it's just one of these spectrums where there'll be more spread if it's five than if it's four. And we need to keep the spread below the critical number so that it goes down instead of up. And we think that the number's four. And if it turns out it's still going up, we'll make it three. That sometimes you need to draw a line somewhere. And the kind of genius move of. But there's no way you can draw it, so therefore you can't draw a line.

[Rob Wiblin]

Yeah, that's actually the naive.

[Toby Ord]

It's quite naive and almost kind of childish.

[Rob Wiblin]

All right, we've come a long way. I guess listeners should know that you've canceled your next appointment to stay late and discuss the policy section with us. Do you want to give kind of an overview of the situation as it stands and I guess what attitude you think people should have and perhaps what they ought to be doing?

[Toby Ord]

Yeah, so I think taking this kind of zoomed out perspective, the technical stuff I was saying was that we've had this scaling law of these kind of training larger and larger models. And this era of scaling really took us really far in terms of capabilities. As companies kind of untied their purse strings and poured more and more money into this, it let things scale up with thousands of times or more than that of computer unleashing these capabilities. That era, I think, has ended, or at least in its current form. And maybe now as we try to scale up inference instead, maybe things will stall out a bit, especially as we apply it to the problems where it's useful to think longer and then it's not clear that it's always useful to think ten times as long as you previously have. So it could stall out for various reasons or it could be that once you've got this form of intuitive thinking, plus the systematic form of thinking, you can combine them in some way that really leads to explosive growth as we discussed. So I think it creates more uncertainty about whether timelines are going to actually maybe stall out and be quite long or whether maybe now we're going to be able to have this extremely rapid progress that the companies are themselves predicting. But also it's not just the timelines, it also changes everything. The way that in the case of inference that you have to pay the costs every single time you use it has all of these different types of knock on effects that can really change things in more complicated ways than just good or bad. And then as we moved on to these kind of bigger questions of policy. Yeah, my main message is that the whole landscape of AI has changed so much over the last five years. And then maybe in the last year we're seeing another one of these types of changes and we could see many more of these. And so it's important for at least some people, I think more people than are currently doing it, to keep an eye on the big picture of how the landscape could be so different and that maybe we could actually help to steer it towards some of these locations and to realize that it's not just been the grip of some kind of technological or incentives determinism to assume that, that humanity just really has no choice. If it turns out the most efficient way to do things is this one that leads to disaster, then I guess we're forced to go into a disaster. That's just not true. And it's kind of excusing ourselves of too much responsibility. Ultimately, if we build a technology that kills us all, it's on us. It's an own goal by humanity and someone has to admit responsibility for that. We can't all just say it was.

[Rob Wiblin]

It was the incentives that everyone else created for me to do it because I thought that they would do it. I didn't talk to them, but yeah, exactly.

[Toby Ord]

I guess I'm a big fan of the big picture of everything. But I hope it's been useful for other people and that some other people, more people will start thinking about these things too.

[Rob Wiblin]

Yeah. Well, I look forward to coming back in a year or two or possibly less and talking about what the new technical developments are and what they imply. And perhaps people's minds will be a little bit more open by then. There'll be more things on the table.

[Toby Ord]

Yeah, I'll be there.

[Rob Wiblin]

My guest has been Toby Ord. Thanks so much for coming on the 80,000 Hours podcast, Toby.

[Toby Ord]

Thank you.