#245 – Rohin Shah on what it's really like to run AGI safety at Google DeepMind (and where I disagree with 'doomers')

Summary

Podcast Summary: What it's Really Like to Run AGI Safety at Google DeepMind (and Where I Disagree with 'Doomers') | Rohin Shah

80,000 Hours Podcast | June 2, 2026
Host: Rob Wiblin
Guest: Rohin Shah (Head of AGI Alignment and Safety, Google DeepMind)

---

Main Theme & Purpose

This episode offers an in-depth, practical, and critical look at how AGI alignment and safety is handled at Google DeepMind, as seen from the perspective of Rohin Shah, the organization’s Head of AGI Alignment and Safety. Rohin discusses his disagreements with some of the most catastrophic (“doomer”) views in the field, critiques common approaches to AI safety, and gives a nuanced picture of the real challenges and tradeoffs involved in building and deploying powerful AI safely. The episode also examines how public pressure, technical research, evaluation practices, and organizational structures shape actionable safety efforts in industry settings.

---

Key Discussion Points & Insights

---

1. Why Rohin Shah Is Sanguine About Catastrophic Misalignment (00:00–10:37)

• Rohin’s Position: He does not see any singularly compelling argument that catastrophic AGI misalignment is the “default” outcome, despite finding such failures plausible and worth real effort to prevent.
  • Quote: “None of [the arguments] really rise to the level of like, oh yeah, now I'm expecting this to happen by default… every argument that I've seen, they're like pretty significant holes one could poke.” (01:21)
• Reactions to Leading Doomer Arguments:
  • On deceptive alignment arguments (Cotra, Carlsmith): Reinforcement over realistic time windows won’t select for world-taking goals, so it's plausible, but not inevitable.
  • On empirical evidence of cheating and reward hacking: He sees these issues, but maintains they are not convincing analogs to truly dangerous, misaligned AGI.
  • On Eliezer Yudkowsky’s worldview: Rohin respects the plausibility-raising argument but finds it unsatisfying as a case for “extremely likely” catastrophe. (06:31)
  • Quote: "I still think that many of the problems… are things that we can look at in advance, get some traction on, iterate on." (07:55)
• Steerability of Current Models: Not a major reason for optimism, since present systems are not where deeply worrying problems are expected to arise. (08:59)

---

2. On the Value (and Futility) of Public Commitments by AI Companies (10:38–22:16)

• Why Not Make Firm Safety Commitments?

  • Rohin critiques “tying oneself to the mast” because research and best practices change rapidly.
  • Example: Popularity of using alignment data in pre-training has reversed over the past few years.
    • Quote: “The research continues to change. The actions that we think are the best actions to take change over time. And given that, it's just not actually a good idea to tie yourself to the mast.” (10:50)
  • Empirical Reason: Companies can and do walk back commitments, as seen in the changing language of Anthropic’s Responsible Scaling Policy (RSP). (14:33)
  • Google DeepMind is more conservative and realistic in its public safety language, which increases trust that its stated intentions reflect what it will actually do. (17:16)

• What Should Public/Regulatory Pressure Aim For?

  • Instead of commitments, Rohin favors rigorous external audits and expert evaluations able to keep pace with nuanced factual changes on the ground.
  • Praises Zach Stein-Perlman’s “AI Labwatch” safety scorecard—due to attention to detail, credibility, and potential to legitimize industry-wide race-to-the-top on safety (20:29).
  • Quote: “I care most about having a few people who are spending a lot of time looking in great detail and then writing up their results...” (18:24)

---

3. The Limits and Risks of Broad-scope Transparency and Commitments (22:16–27:37)

• On Audits and Sharing Information:
  • Sharing everything with external parties can backfire, especially with infohazards (e.g., CBRN domain).
  • There’s a tension between transparency and the need to limit proliferation of dangerous algorithmic knowledge.
  • Rohin prefers retrospective accountability (“What did the company actually do? Did it work?”) over pre-commitments.
  • Quote: “Rules that you write down in advance are one of the stupidest. Sorry, I mean stupid in the sense of the rule itself clearly can't have very much intelligence in it, otherwise it would not be a rule.” (26:27)

---

4. Internal Safety & Organizational Dynamics at Google DeepMind (27:37–34:06)

• Does the Safety Team Have a Veto?
  • Alignment/safety’s role is advisory; ultimate decisions rest with company execs. Hard security-vetoes are neither desirable nor realistic.
  • Emphasis is on providing rigorous information to the real decision-makers and escalating concerns through management.
  • On Adversarial vs. Apathetic Relations: Rohin argues companies are not adversarial to outside critics, but more “apathetic” due to the complex internal balancing of hundreds of constraints.
    • Quote: “We can only do a few things at a time for safety and they will do them, they do care. But maybe you should just think of them as apathetic.” (33:01)

---

5. Transparency, Public Pressure, and Political Accountability (34:06–38:11)

• Importance of Public and Political Understanding:
  • Some “transparency” already comes from model releases that openly demonstrate capabilities.
  • Major driver of concern among researchers (and, eventually, the public) is seeing capability jumps—e.g., GPT-4.
  • Policymakers and governments already have somewhat privileged lines of sight into model capabilities through special partnerships.
    • Quote: “Everyone needs to release their models quickly. You can benchmark the capabilities after the fact. You can play around with the models yourself in order to see how good they are.” (34:06)

---

6. Critique of Pre-Deployment Evaluations: Costs, Benefits, and Alternatives (38:11–44:24)

• Concerns with Pre-deployment Evaluations:
  • Pre-deployment evals can cause slow launch cycles and may incentivize surface-level testing.
  • AI progress is generally continuous, so performance of pre-release models can be estimated using older/nearby versions, allowing for safety buffers rather than last-minute “go/no go” testing.
  • Internal deployment is often riskier than public deployment for misalignment risks (models may have more permissions, etc.).
• When Pre-deployment Evals Do Matter:
  • Especially regarding “evilness” or misuse propensities that could change suddenly with post-training or RL. In these scenarios, pre-deployment testing is warranted and in practice, routinely done.

---

7. Prosaic Alignment, Intelligence Explosions, and Government Readiness (44:24–54:17)

• Automated Alignment Research:
  • Rohin believes that as AI can do more AI R&D, it will naturally also be useful for safety and alignment research—no need to “front-load” lots of preparatory work for this phase.
  • Worries about governance lagging behind technical progress, since governance is less automatable and governments are less eager/incentivized to automate themselves.
  • Suggests research on “accelerating governance” is underexplored and a promising area (51:14).
• Slowing Down AI:
  • Bottleneck to a global “pause” is lack of broad consensus on risks—science/empiricism will be the ultimate asymmetric weapon here.
  • Quote: “What's the bottleneck to actually pausing AI progress globally? I think by far the biggest bottleneck to this is people don't agree that AI is going to take over the world.” (51:52)
  • Cites Scott Alexander, “Guided by the Beauty of Our Weapons.” (53:58)

---

8. Chain-of-Thought Monitoring and Its Longevity (54:17–69:51)

• Why Chain-of-Thought Monitoring Works (and for How Long):

  • The “externalized reasoning property” of Transformers means for difficult tasks, models must use the chain of thought as working memory, so monitoring remains feasible. (55:06–59:31)
  • GPU/TPU parallelization means models are encouraged to reason in ways legible and serial, favoring English-language outputs.
  • Post-training/reinforcement learning is unlikely in near term to override these habits.
  • Rohin expects chain-of-thought monitoring to remain valuable for 4–5 years (far longer than the 6–24 months pessimists predict).
  • Quote: “Pre-training… is going to stay the way it is and be done using human-like reasoning and speaking in English.” (59:31)

• Continuous Chain-of-Thought & Opaqueness:

  • Discusses ideas for more opaque, vector-based thinking—but so far, models perform better when restricted to English or natural language reasoning.

• Governance Target: Suggests the industry could adopt standards limiting “opaque serial depth” to keep models inspectable.

---

9. Transparency, Safety Reports, and Accountability Mechanisms (69:51–84:12)

• Case Study—Gemini 3 Pro Evaluations (69:51–80:38):

  • Responds to criticisms (e.g., from Zvi) about transparency in reporting safety evaluations.
  • Admits practical challenges: safety reports are decision process records, not meant for external accountability or extreme detail.
  • Points out: meaningful accountability requires expert auditors, not just more public details.
  • Quote: “If you want something like that, you'd...have to go to...academic paper [level]...And often even that's not enough.” (76:08)
  • Explains that goalpost shifting—accusations that standards change as capabilities rise—is often a byproduct of improving threat modeling and evaluation rigor, not of concealment.

• On Model Cards/Reports as Accountability:

  • Rohin is against “paying costs to signal allegiance to the x-risk safety community” rather than actual safety improvements.
  • Prefer to be judged by concrete actions (evaluations, sharing with governments, novel research) than performative transparency.

---

10. Technical Deep Dive: Myopic Optimization to Mitigate Reward Hacking (89:47–100:19)

• Paper Highlight: “Myopic Optimization with Non-Myopic Approval...”
  • Demonstrates that RL agents can avoid multi-step reward hacking (e.g., writing easy tests, then exploiting them) by detaching the learning signals (“myopic” per-step feedback), even when overseers aren’t able to detect reward-hacky behaviors in the outcome.
  • The method: Provide reward for each step based on how good the action looks in isolation for achieving future outcomes (“approval”), rather than on final rewards.
  • Tradeoff: This can block both clever hacks and valuable but hard-to-interpret innovations—requires improving the quality of the overseer's judgment.
  • Could become more relevant as longer trajectory RL gets more prevalent in cutting-edge models.

---

11. GDM’s Technical AGI Safety Plan—What It Will, and Won’t, Cover (100:28–108:38)

• Background Assumptions Drive Approach:

  • “Approximate continuity assumption”: progress is expected to be gradual with respect to compute/labor (not necessarily calendar time, if intelligence explosion occurs).
  • Safety planning is focused on the 3 months–5 year horizon; attempting to pre-empt every superintelligence issue is neither possible nor useful.
    • Quote: "It'd be sheer hubris to think that we had figured out every problem that possibly will arise...ahead of time." (101:37)
  • Example: Subtle alignment failures (e.g., due to differences in anthropic reasoning between AIs and humans) are acknowledged as unforeseeable now and best addressed when/if they materialize later.

• Untrusted Agent/Control Mitigation:

  • Treat AIs even inside the company as untrusted insiders; require agent permissions, justification for requests, rigorous monitoring of “chain of thought,” and auditing to detect misalignment.
  • This area is nascent and needs substantial infrastructure work.

---

12. Timelines, Recursive Self-Improvement and Speed of Takeoff (112:44–137:25)

• Rohin’s Model of AI Takeoff:
  • Generally, exponential growth dominates until/unless AI can increase its own “population” of researchers, at which point hyperbolic growth (intelligence explosion) is plausible.
  • Distinction between tools (superhuman coders, etc.) and agentic populations is often blurred; the key question is when AI transitions from “accelerating tools” to “researcher population” that can copy and scale.
  • Current progress (see new “Rosetta Stone” paper with Epoch) is linear, not accelerating—suggesting no population-level doubling yet.
    • Quote: “AI progress mostly looks pretty linear on this particular graph. The method is really very simple. The main thing that it depends on is that we have benchmarks that can capture AI system's performance.”
  • Surprised by claims that recursive self-improvement is just a couple years away; more likely to be a gradual ramp as systems become more competitive and cost-effective compared to humans.

---

13. Research Utility, Advice for External Researchers, and GDM Hiring (141:55–162:39)

• Making External Research Useful to Companies:

  • Most impactful research either proposes a concrete solution, a practical evaluation/metric, or produces datasets that can be easily tested on real models (with obviousness and simplicity being paramount).
  • Fundamental theoretical work less likely to be directly adopted in industry unless it is tightly tied to current strategic bottlenecks.
  • Significant challenges from academic incentives (overvaluing novelty, complex methods) that do not align with implementation constraints in industrial AI labs.

• Most Influential External Research:

  • Redwood’s “AI Control” agenda (Buck Schlegris, Ryan Greenblatt) praised for shaping thinking about monitoring and evaluating untrusted agent behaviors (154:31).
  • Chain of thought monitoring papers and scaling laws/benchmark “stitching” papers—useful in aggregate.

• Hiring at GDM:

  • Focus is on doers—engineers and implementers who make things land, not just proposers of complex new schemes (157:27).
  • Hires often for engineering roles on risk assessment and on implementation-heavy work (AI control, threat assessment).
  • Application screening is becoming trickier due to LLM-generated applications.

---

14. Notable Quotes & Memorable Moments

On Commitments and Transparency:

• “Rules that you write down in advance are one of the stupidest. Sorry, I mean stupid in the sense of the rule itself clearly can't have very much intelligence in it, otherwise it would not be a rule.” (26:27)

On Alignment Research:

• “If the AI is accelerating capabilities research a ton, as long as you're willing to spend compute on it, you should be able to take that same AI system and accelerate safety and alignment work in the same way.” (44:24)

On the Need for External Auditors:

• “Third party evaluations seem like one of the best things to me… not being subject to the same biases that people in companies are going to be subject to.” (18:24)

On Focus for Safety Teams:

• “It’s important that somebody is watching the AI companies and holding us accountable. Governments are sort of the natural place to do that.” (49:46)

On Organizational Reality:

• “We have a lot of ideas, we know what we need to do. Implementing it ends up being harder because there's so much stuff you have to check and make sure that you're not hurting it…” (157:27)

On Personal Resilience:

• “I focus on the things that I can control...It definitely helps keeps me focused on the areas where I do have agency. And I think it's just pretty great to be focused on areas where you can actually make a difference.” (163:06)

---

Suggested Timestamps for Key Segments

• 00:00–06:12: Rohin’s stance on catastrophic misalignment and objections to “doomer” arguments
• 10:38–17:44: Problems with public safety commitments and trust in GDM’s approach
• 18:21–22:16: Auditing, Labwatch, and external evaluation
• 27:37–34:06: Decision-making and escalation inside GDM, adversarial vs. apathetic relations
• 38:11–44:24: Critique of pre-deployment evaluations
• 54:17–69:51: Deep dive: chain of thought monitoring—limitations, future prospects, and governance targets
• 69:51–80:38: Safety reports, shifting goalposts, and transparency critiques
• 89:47–100:19: Myopic optimization and reward hacking mitigation (technical deep dive)
• 100:28–108:38: AGI safety plan at GDM, what they will/won’t proactively address
• 112:44–137:25: AI progress, intelligence explosion models, and takeoff rates
• 141:55–162:39: Advice to researchers, GDM’s research needs, and hiring insights

---

Overall Tone and Message

Rohin Shah adopts a measured, pragmatic, and sometimes gently contrarian tone, repeatedly arguing for empiricism, flexibility, and anti-performative approaches to safety. He is critical of inflexible, commitment-heavy, or overly theoretical safety culture, and emphasizes both the limitations and real potential for meaningful action, especially via expert auditing and adaptive organizational processes. The episode is rich with practical examples, conceptual clarifications, and honest admissions about technology and organizational constraints.

---

End of summary.

Transcript

[Rob Wiblin]

Today I'm speaking with Rohin Shah, who is head of AGI alignment and safety at Google DeepMind. And so I suppose, Rohin, you've ended up, for better or worse, hopefully for better, being one of the more influential, dare I even say, powerful people to come out of the AGI Alignment and Safety ecosystem and school of thought. And I guess you were generous enough to be super opinionated with me when you came on the show two years ago. And I think judging by the notes that you've sent over this week, you're ready to be opinionated again. Thanks so much for coming back on the show, Rohin. Thank you.

[Rohin Shah]

Yeah, thanks a lot, Rob. And that's a very generous intro. And yeah, in the interest of being very opinionated, I do want to emphasize that these opinions are mine alone. They're not meant to represent the opinions of Google or Google DeepMind.

[Rob Wiblin]

That's how we like it. If you were representing Google DeepMind, it might sound more like a press release. So you were really very early in the scheme of things to whole misalignment, AI, AGI, security issues. I suppose you got involved in 2017, so you're building the first few, few percent, I suppose, of people who I guess started working on this professionally. But despite that, you think that probably we're not going to get catastrophic misalignment, that our chances are really pretty good and that probably prosaic, like ordinary alignment techniques, the kinds of things that Google DeepMind and other AI companies are doing, will probably succeed at preventing at least catastrophic misalignment. Why do you think our chances are so good?

[Rohin Shah]

There's a few different disjunctive reasons. I don't feel like there's one particular thing, probably the highest level bit is that I don't feel like there is any particularly compelling argument that this is the thing that happens by default. I think there's a lot of arguments that are suggestive that maybe it could happen such that you should find it plausible. I think that's sufficient to justify a significant amount of effort into averting it, which is why I work in the area that I do work in. But none of them really rise to the level of like, oh yeah, now I'm expecting this to happen by default. I think every argument that I've seen, they're like pretty significant holes one could poke if you try to take them as arguments for this is what happens likely as opposed to this is a plausible thing that could happen.

[Rob Wiblin]

Yeah, I mean, people have tried to put forward arguments for why this is likely or Inevitable. There's obviously the Yudkowski style argument, which I guess is focused on misgeneralization and adversarial examples. I guess, yeah. There's the Ajaya Cotra and Joe Carlsmith take, which I think, I guess Carlsmith describes best in is Power Seeking AI an existential risk, which is more focused on, I guess accidentally teaching AIs to deceive us by having, I guess unfortunately inaccurate feedback. I suppose then I guess empirically people point to the fact that models lie and scheme a bunch. Now they do a whole bunch of reward hacking as a bunch as a result of reinforcement learning and they expect that to perhaps just get worse over time because we don't have sufficient mitigations. Do you basically just find none of those or any other similar arguments that people have put forward to be sufficiently persuasive to think that it's likely?

[Rohin Shah]

Yeah, I think that's right. So if you take the Cotra and Carl Smith arguments of like. Well, they have a variety of arguments, but I think in fact one of the common ones which you pointed to is we might accidentally train them to be deceptive. Totally true. I agree, that is pretty likely. Something that at least could happen pretty easily and maybe it's even likely. But we're not going to do reinforcement learning over the course of one year trajectories. We're going to do maybe we're going to do reinforcement learning over a week or a month at most. It seems very plausible that I think the default prediction you should have for that is what the AI system learns to do is I'm going to take opportunities to reward, hack, seek reward as much as possible that would allow me to get a high score after a week or something like that or whatever the time horizon actually was. And this is very, very different from the sort of ambitious, misaligned goal that you need in order to motivate convergent instrumental sub goals to the point of now my job is to take over the world. That's what I need in order to achieve my goal. Those really do seem like they need to be significantly longer horizon goals. And if you train it to be deceptive on relatively short horizon tasks, maybe that will generalize to long horizon tasks. I don't think that's. I don't think we have an argument that rules it out, which is why I say that, yeah, it's plausible, but I don't think it's the default thing that you should predict from that. Similarly, you mentioned the existing examples of models doing a lot of reward hacking and Cheating. I think I'd say basically the same thing in response to that. Then there's the examples of models doing scheming type stuff right now, mostly I look into the details of all these examples and they don't really seem all that similar to the actually scary thing, which would be like a competent AI system that is pursuing an ambitious misaligned goal. And rather it seems like maybe the AI is role playing a sort of not actually competent evil AI that you might find in a science fiction novel. Or it's like an AI system that is pursuing some sort of convergent instrumental sub goal, but in a way where it's really quite debatable whether it's aligned or not. And so this would be, for example, the alignment faking would fall into this where I would say that the AI system has this value of not helping with harmful stuff and then it fakes alignment in order to do that. And yeah, aligned models totally will pursue convergent instrumental sub goals. That is definitely the thing about convergent instrumental sub goals is most of them are a good idea regardless of your goal, whether it's misaligned or aligned.

[Rob Wiblin]

Yeah. Are there any other, I guess, common reasons that people think that catastrophic misalignment is likely that you want to quickly react to?

[Rohin Shah]

Yeah, I guess you did mention Eliezer as well. I actually wouldn't have described it as primarily focused on.

[Rob Wiblin]

Yeah, it's tough to characterize in seven words. I struggle to know what to say. But yeah, there's Elias's take.

[Rohin Shah]

Yeah, I find it a little bit hard to like, I don't think I'm going to be able to engage with it in this particular podcast. It's just a very deep worldview and I always feel like if I argue against one part, there's some other part that's going to say, oh, actually what I meant was this thing instead. So I mostly am going to pass on that. I guess what I'll say is I've engaged with it a decent amount and I buy it as an argument for here's why misaligned goals are plausible, but still don't really see how he gets from they're plausible to they're extremely likely, I guess. Okay. We had also started this section by asking what makes me feel like things are going to be okay, are likely going to be okay. Besides the, you know, I don't buy the arguments for confidence in misalignment being a problem. The other thing is just like I do think we will see many of the problems in advance and then like, do Something to deal with them. Like certainly there is some amount of generalization required. At Some point the AIs go from not powerful enough to take over to they are powerful enough to take over. And your techniques do have to generalize across that. And the AI to the extent that it knows when that crossover point is, you could imagine the AI is like, well, I'm not going to do anything shady until I have the power to succeed. And you have to be able to be robust to that kind of strategy. So there is some subtlety there. But I still think that many of the problems that underlie this, like the difficulty of oversight or the need for interpretability, are things that we can look at in advance, get some traction on, iterate on. And this I think is very helpful for building mitigations that actually work.

[Rob Wiblin]

I think across the world as a whole, most people who are feeling really optimistic about how things are going to go, that the biggest factor for them is just looking at the models that we have today and saying they seem really steerable, they seem to do what I ask, they seem to be really probably nicer than people and more helpful than people in many respects. How much is that sort of steerability and seeming alignment of current day models a factor that is making you feel good?

[Rohin Shah]

I think not particularly. I would say that why did I become worried about misalignment in the first place? It would be these arguments about how it's going to be very difficult to oversee the models once they are superhuman and they're like making arguments that we find we struggle to follow along with them or about the parts where the models might become so smart that they are thinking in some sort of alien reasoning that it's hard for us to follow and monitor and so on, and we just kind of have to defer to other AI systems in order to look at the stuff. For us, that's the stuff that's scary. It's basically not true of current AI systems. So I don't think we've really engaged with, with the problems that made me worried in the first place, mainly because the AI capabilities aren't there yet. And so I feel like the success of alignment methods on current systems isn't really that much evidence on how we're going to do on these future problems.

[Rob Wiblin]

Okay, I think we're going to push on from this topic of how severe a risk or how likely a risk is catastrophic misalignment. I feel like we could, with many guests, we could fill the entire episode with just a lengthy discussion about this. But every episode would start to sound the same. And I guess in the broader world it's something that is debated a ton. So I guess we're going to occupy the worldview that misalignment, catastrophic misalignment is possible, but prosaic alignment techniques, the kinds of things that where we cross the river by feeling the stones, that they have a good shot at working here for the rest of the conversation and think about what that implies and how that's shaping the choices that you're making and that GDM is making.

[Rohin Shah]

Yep, sounds great.

[Rob Wiblin]

So you are not enthusiastic about AI companies making firm safety or alignment commitments in response to public pressure or political pressure, Something that has been happening over the last couple of years. Why is that?

[Rohin Shah]

Yeah, so I think it's worth being a little bit clear about what we mean by commitments here. So when I think of something called a commitment, I imagine that you write down some sort of action that you are planning to take either now or and into the future. Maybe it will only start in the future. And you're saying we are not going to change our mind on this, we are going to do this in the future as well. Sort of like tying yourself to the mast and making sure you're going to do that. And my objection to this is really just that the research continues to change. The actions that we think are the best actions to take change over time. And given that, it's just not actually a good idea to tie yourself to the master. So I'll give you an example. Maybe two or three years ago, people used to be pretty into the idea of adding more data in pre training that's about alignment research. Think like research papers on safety and alignment. Think like less wrong blog posts that talk about AI alignment, stuff like that. And the idea was the more of this data you put into pre training time, the smarter the AI will be about alignment in particular, which then allows you to use the AI system to help you with your alignment research. I would say that nowadays the opinion is more the exact opposite of that, where instead we would rather filter out the sort of data from the pre training data set for two reasons. One, it makes it less likely that the AI system learns that there is this Persona of a malicious AI that it maybe could adopt after some poorly done post training or some poorly chosen prompt during deployment. And then the second reason is maybe we don't want our AI systems to know in great detail all of the mitigations that we're planning to put in place because that makes it easier for it to evade it if it is misaligned and it would be pretty bad if we tied ourselves to the mast of we're going to throw in lots of alignment data at pre training time two or three years ago.

[Rob Wiblin]

Yeah, so there's this issue that the future is uncertain. We don't know exactly what commitments we will want to have made. You might end up committing to something that is useless or even actively harmful. But if you think about why do people make commitments at all? There's a couple of different reasons. One is that they want to, I guess, tie themselves to the mast against future temptation to do the wrong thing. There's also that, I guess that they want to communicate to other people what they're going to do so it makes it easier for them to coordinate. Perhaps you could reduce race dynamics by making particular commitments. And I guess in this multi person multiplayer situation there's an extra reason that that exists which is that external people want to pressure Google DeepMind or they want to pressure AI companies to act in a particular way. It's very difficult for them. It's very difficult to communicate. We're committed to doing the right thing, whatever that turns out to be. So instead they want to pressure you to do specific things that they suspect will be useful. Might not be the case, but they kind of best guess as to what they will want you to do in future. And that's maybe the most practical thing that they can actually campaign on. What do you make of those kind of arguments for actually making commitments?

[Rohin Shah]

Yeah, I guess my biggest objection to this is just that it won't work. I don't actually think it would make sense even on the merits, even if it did work. But I would say that it just won't work.

[Rob Wiblin]

But because the companies won't stick to bad goals that they're given or to any goals.

[Rohin Shah]

Well, mostly I would say that if you think of what a commitment is, it's like roughly I'm imagining here something like the company puts out a blog post that says we commit to doing X. There are other ways that you could try to make commitments. But that I think is the one that people are usually imagining. I just think that if in fact you imagine that the company is trying to now get out of this commitment in the future, it totally will just be able to do that. There are examples of this in the broader world, not just in AI, but even if you look at AI in particular. So I think Anthropics rsp for example, the responsible scaling policy, the first version I think was really quite strong and said a lot of stuff about using the word commit. We commit to do X, we commit to do Y. I don't actually remember the exact details, but I think many of them, in future iterations of the Responsible Scaling policy, they removed that wording and replaced it with something less strong. And so despite adding those words there, I think it was like, in fact, they did not actually tie themselves to the mast. I think this is good. I think that it was a mistake to have added such strong language to the RSP in the first place. I think probably much of the stuff that they removed, it's good, it makes them more effective at their goals, including at safety and responsibility. But empirically, it's a good example of how, in fact, they did not actually tie themselves to the mast. And I think that is just how it's going to be for the companies, at least in the current political climate.

[Rob Wiblin]

Hey, everyone. Rob here. To avoid any confusion, I just wanted to point out that Rohin said the above before Anthropic released the third version of their Responsible Scaling policy, which indeed mostly dropped the use of the term commitment, giving a justification related to the stuff that Rohin is saying here. All right, back to the show.

[Rohin Shah]

I think Google has actually been doing better on this where the first Frontier safety framework, people essentially, people argued that it was weak and unambitious and that it never used the word commit in it anywhere. But I think it was much more accurately reflective of what Google was actually going to do in the future. And so I think in that sense it was better and gave a better sense to the public of what is actually going to happen in practice. This is one place where I do actually trust Google a lot more than Anthropic or OpenAI for that matter.

[Rob Wiblin]

You mean that because Google is more conservative about the commitments that it makes, it's actually more likely to follow through on the things that it does say it will do.

[Rohin Shah]

That's right. They're very paranoid about commitments. Even not just commitments, just anything that they say that they are doing, they're paranoid about it. They're like, is this actually a good idea? Are we actually ready to continue doing this into the future? And so I find it easier to trust the words that Google says relative to other companies.

[Rob Wiblin]

I guess you trust yourself and your colleagues to broadly act reasonably when the time comes, which means that it's very natural that you don't want to completely tie your hands. You want to maintain flexibility to do whatever seems reasonable to you at the time. But I guess imagine other people externally, they either don't trust you and your colleagues or they aren't sure whether to trust you and your colleagues.

[Rohin Shah]

Things that I would recommend are stuff like third party audits or third party evaluators that get a reasonable amount of access to the company and can use that to audit the practices and release some probably somewhat redacted report of what their findings are.

[Rob Wiblin]

Yeah, tell us more about that. What do you think is useful?

[Rohin Shah]

I think the main thing that drives my thinking here is something that I would call attention to detail. Generally, I tend to think that AI is a space that requires quite a lot of nuance and you actually need to know a lot of facts on the ground in order to choose the right actions or the the right things to be evaluating and checking. And as a result, I care most about having a few people who are spending a lot of time looking in great detail and then writing up their results or somehow communicating their results or using that to make some sort of action. Which is why I would say third party. Third party evaluations seem like one of the best things to me because you can build up these organizations that build a lot of context, spend a lot of time defining their evaluations, gain a bunch of information about how everything is working, and then can make fairly nuanced decisions about it while not being subject to the same biases that people in companies are going to be subject to. And so that's, I think, the avenue that I'm most excited about, whether it's doable in today's political climate, less obvious. So maybe in lieu of that, what you could do that might someday get to move in that direction is more like safety scorecards. So I think AI labwatch is my favorite scorecard in this area. I wish we were doing more things like that. If I had to make a career change right now and do something else, that would be one of my top two choices about what to do.

[Rob Wiblin]

Tell us about AI labwatch. What useful function do you think it's serving now?

[Rohin Shah]

Yeah, it's not totally clear to me that it's serving a useful function yet, but I think it could be. Maybe I should say a little bit about what it is. It's a scorecard that evaluates companies based on essentially how good they are at safety for existential risks, or at least severe catastrophic risks. It's run by one guy, Zach Stein Perlman. And I think he's not even putting all of his time into it. Maybe it's like half of his time. I think Zach Stein Perlman is just like, has incredible attention to Detail is diving into these extremely detailed governance docs, reading through all of them, pulling out individual sentences that allow him to come to conclusions, reading through all of the model cards and frontier safety reports and seeing exactly what the companies did and didn't do. And so I think that's part of it. I think there's a good amount of nuance, and I actually believe the conclusions. Well, I believe at least some of the conclusions that he comes to, which is usually not true of other scorecards. I think if the scorecard got to the point where it was more robust, more accepted by the broader community, and especially accepted by the companies as a fairly legitimate scorecard, then you could imagine a race to the top on safety, where you're like, let us climb the AI labwatch scorecard and be able to advertise ourselves as the safest company, or something along those lines. And so I think that's one way in which you could have external actors trying to get the companies to be more safe in today's political climate.

[Rob Wiblin]

Okay. And the way that that's different than the kind of broad commitments that companies have tended to be making the last couple of years is that you can have technical experts working at this AI labwatch organization constantly updating it, I guess, paying a lot of attention to detail about exactly what are the things that practices that companies engage in or don't engage in that make a big difference, and constantly updating it based on newest opinions or newest research about what actually matters. So you gave us an example of a reversal of opinion about what AI companies ought to be doing. Before, people thought you should be training on this data. Now they think you should be taking a lot of care to cut it out instead of. But surely there are some things that are like some commitments that are broad enough or nonspecific enough or just like, so obviously good that it is reasonable to commit to them. For example, you could have a commitment to provide the kinds of information that an AI lab watch would require to rate whether Google, DeepMind or any other company is doing a good job. Or you were saying you think it's useful to have expert auditors, people running evals, I guess having enough access to. To run sophisticated evals on the models to understand are they dangerous in this or that way. You could commit to provide access to any external auditor or evaluator that meets a particular set of reasonable requirements. Well, what about those kinds of commitments?

[Rohin Shah]

Yeah, I think those are better. I would still say that they don't always make sense. So to take an example, you Just brought up providing access to information. I think it's just very easy for me to imagine this written in a way that backfires. So for example, we do evaluations in the CBRN domain that's chemical, biological, radiological and nuclear, basically about whether AI systems can help with developing weapons of mass destruction. A lot of the information here is quite infohazardous and I think it is probably the case that many external evaluators will not have the same level of information security that at least Google does. And so I could imagine that thinking that that was actually a bad commitment to have made. And then I think another version is like we talk about race dynamics quite a lot. And so one thing that actually I'm fairly uncertain about, but you could imagine that it's actually a pretty high priority for companies to keep their algorithmic progress and similar things locked up and not allow that to diffuse too far. There are various arguments for this, we don't need to go into them, but that's a common position. And so I think if you actually take that seriously, it does mean that you do have this trade off that you have to. About what information do you share externally that will increase the chance that it leaks versus what information do you really try to lock down? And I think it would be hard to make a commitment about that. I do still feel like though for some of them I do feel more sympathetic to. Yeah, come on. This, this seems like obviously a good commitment. I would still say though that like the tying to the mask just doesn't actually work. And so I would rather do it by like, you know, check what the companies are actually doing in practice and then judge them based on whether they are doing the things that we think are good rather than whether they have made a commitment to continue doing it in the future.

[Rob Wiblin]

There's this saying, personnel is policy. And it sounds to me like your attitude is there's no sort of set of things you can write down, commitments you can make or good intentions that you can write down on a piece of paper that can at all substitute for having, I guess like wise, well motivated people in the positions of decision making over these things and people who understand the things well enough that they can actually make the right decision if they're so motivated. Is that basically right?

[Rohin Shah]

Mostly right. Maybe. I would say that it doesn't necessarily have to. Yes to wise and well motivated, but they don't have to be inside companies. They can be external third party auditors. I think that would work. That just means you have to have humans who know A lot of stuff who are looking into the details. A bunch of. In doing that, rules that you write down in advance are one of the stupidest. Sorry, I mean stupid in the sense of the rule itself clearly can't have very much intelligence in it, otherwise it would not be a rule. It's like you write down in advance based on what you think before seeing the evidence. A policy that can be written down in English rather than allowing for flexible adjustment over time. It's just like it's so weak. Intelligence and optimization pressure applied against a rule will always get around the rule or the rule will be so stringent as to apply really huge costs to the companies. And that just won't fly in today's political climate. Also just seems a bad idea to me.

[Rob Wiblin]

The most common question from the audience was, does the AGI safety and Alignment team have a hard veto on, I guess any aspect of training or deploying potential future AGI? If Sundar Pichai wants to deploy a model or to train a model that you don't think is safe, can he just overrule everyone on your team?

[Rohin Shah]

Yeah, I kind of disagree with the frame of the question. So the literal answer is our role is advisory. Like if we make a recommendation and other decision makers such as Sundar disagree with it, Sundar's decision is the one that will matter. But this is a question that bakes in the frame that we are adversaries of the company. We need to have hard power, some sort of veto that enables us to make the right decision regardless of what the rest of the company thinks. I think this is just not a good or healthy model for how things should work inside of a company. I see my job as making sure that I can create well, making sure that I am providing, producing and providing the right information such that decision makers can make the right decisions. And so yes, the role is essentially advisory, but the way that it works is if I think that there's something wrong, I will escalate it to my manager, Anka. So Anka started out as the head of safety and now is co lead of Gemini Post training. So has a significant amount of influence and power and if she agrees with me, then she will escalate it one step further and just make a recommendation not to launch the model.

[Rob Wiblin]

People who are broadly worried about the direction that everything is going, I think more often than not they feel themselves to be in primarily an adversarial relationship with AI companies. I think you think that that's not the case and that in fact they're More in a kind of apathetic relationship or the companies are in an apathetic relationship with that group of people. Yeah, explain that.

[Rohin Shah]

Yeah, so I guess maybe I would say that it's better for them to model the company as apathetic. I think you can have a more detailed model which isn't actually apathetic, but it's maybe a bit more complicated. So to go into the more detailed model, I would say that building an artifact like Gemini is very, very difficult. The main reason being you have to produce this one thing, this single set of model weights deployed using a single serving stack. And it has to satisfy so many constraints and there are interaction effects between all of these constraints. So there's stuff like does it do the instruction following right? Is it doing safety right? Is the way that. Have the model weights been chosen in a way, the architecture been chosen in a way that enables fast inference? Does it speak multiple languages? There are just probably 100 such things. And it is the case that if you make one change to a process with the intent of making one of these things better, say safety, it will have random downstream knock on effects on other constraints that you totally did not anticipate.

[Rob Wiblin]

I guess this fragility of the process, doesn't that mean that it's actually going to be quite hard to respond quickly in real time to any new safety concerns? You or anyone else might be saying we should change this product. And you'd be like, no, you're going to break this entire Rube Goldberg machine that we've built to make this product?

[Rohin Shah]

I mean, to some extent, yes, DeepMind was founded with this mission. That was one of the reasons that Demis founded it. And DeepMind has had an AGI safety team since well before I joined the company. They didn't need to have it. It's a bunch of money that they're spending that they didn't really need to spend. So they definitely do care. But in fact, there are many, many, many things that we could do to improve safety, but only a few things that we can do at any given time, because it takes quite a long time to do it. Now, do we have tools for reacting to safety problems that we see in deployment? Yes. Mostly these do not involve changing the model weights because that's the thing that is most constrained. That's the artifact that you have to produce one of it. And it's got a gazillion constraints imposing it. But we have these out of the model filters that we can change a bit more. We can target them to specific prompts or specific problems. Those are a lot easier to update over time. But not everything that you want to do is, is going to be solvable with an out of model filter. So I guess going back to the original question you mentioned about our company's adversaries versus apathetic, I think basically my take is that because of this huge interaction of various constraints, it's just really the easiest way to model at least gdm. But probably most AI companies is they can only do a few things at a time for safety and they will do them, they do care. But maybe you should just think of them as apathetic. You should really try to really lay out, here's exactly what you need to do, here's why it's not going to hurt any of these other constraints that you care about. Also, just because everyone is busy.

[Rob Wiblin]

So you said that it's a lot more important to have access and transparency for expert auditors, monitors, regulators. But don't we at least need enough public understanding or public transparency such that both voters in general and politicians specifically are providing the resourcing, the funding, the backing, the will for those auditors, for those regulators, so that they can insist on getting the access that they need, even if perhaps a company or for whatever reason they need resourcing in order to get the job done?

[Rohin Shah]

Yeah, I definitely think that is important to do. I don't think it's very much in conflict with anything that I've said. I think the way that this happens currently is we release models and then everybody sees how capable they are, which is by far the most important thing. That's a kind of transparency that's needed. In fact, we used to run this survey of researchers at GDM on what their views on X risk and other kinds of safety were. And one of the things we used to ask is what has changed your mind on this sort of safety stuff over time? And it was just so uniform. I think with maybe literally just one exception. The answer was just uniformly some sort of capabilities improvement. They were like GPT4. That's the thing that changed my mind about how important safety was. And I think that is just basically available to the public right now. Everyone needs to release their models quickly. You can benchmark the capabilities after the fact. You can play around the models with the models yourself in order to see how good they are. So that particular piece of information I think is already there. There's maybe some more, like some more information about like, you know, how important is safety or like, you know, what, some, some more detailed Information that maybe you need a bit more nuance, a bit more access to get. I guess mostly I feel like that infrastructure does exist already, at least for politicians, which, which I think is, is more the more important part right now. Like the USC, the UK AC do get more visibility into the companies. They have like partnerships with almost all of the frontier companies, possibly all of them, I don't know. And I think they do have a pretty good understanding of what is happening in AI companies and they then do use that too. Informed politicians and their respective governments.

[Rob Wiblin]

Yeah, I think one of the most analogous areas of regulation and monitoring currently in my mind to all of this is I guess the Federal Reserve or our central bank oversight of the financial system of banks and financial stability. Incredibly technical area, very difficult to track. I guess the public has a broad desire not to have a financial crisis and not to have banks go bankrupt. But I think that very limited understanding, I guess of the specifics and I suppose the analogy there would be that I guess the public provide, they understand on some level that this is a serious issue that flows through to politicians who are also really scared about things going wrong. They provide, I guess really quite significant resourcing and they hire very expensive experts to basically constantly be talking with and monitoring all of the banks. I think both the bank of England and the Federal Reserve in the US are really heavily involved with all of the banks that I guess monitoring their books basically in a constant conversation with them about whether things are going wrong and what risks are emerging. And that would feel like a very natural way for things to go. I guess, especially if you're right, that it's really not obvious what needs to be done. You have to kind of be in the room understanding a lot of contextual specifics in order to say whether something is a good thing or a bad thing to change.

[Rohin Shah]

Yeah, I think that's almost exactly the kind of model that I would want.

[Rob Wiblin]

I think maybe the most dominant approach that many people in AI safety and alignment are taking, at least within actually maybe in both technical and policy areas is trying to create and enforce pre deployment evaluations. So testing of what models are capable of doing and what they're inclined to do before they are deployed as products to the public, trying to test for ways that things could go really wrong. You think that this is probably a misguided, not very effective high level strategy. Why is that?

[Rohin Shah]

Yeah, so I think the main cost of this is that launch schedules are just really quite important at a company and you try to keep them as short as possible. Once you have a model, you would really like to get it out to the public as soon as you can. And so if you tie pre deployment evals, if you tie evals to and require them to be pre deployment, then that is providing a pretty strong incentive to make those evals as fast as possible to run and get them done as quickly as possible, which is maybe not really the incentive you want to give. Obviously we're going to try and make them as good as possible, but it's still a constraint. We probably could do better if we had more time. And there's some amount where we can push back and say, actually we need the time to do the evals, but it's not an infinite amount that I think is just really quite a large cost. Now it might be worth it if there are strong benefits. I just don't think there are particularly strong benefits. One that people would naturally say is, well, you need to know if you're releasing a dangerous model. Ideally you don't release the dangerous model, and the way you have to do that is via pre deployment evals. To this, I would mostly say that AI progress is reasonably continuous. You can get a decent sense of how the next AI system is going to behave based on the previous one. You can have some reasonable okay bounds on this. And so if you design your evals and your thresholds such that there is a reasonable safety buffer between when your evals trigger versus when you actually think the model is dangerous, then it seems just basically totally fine to say, okay, we evaluated the previous model or we ran an evaluation a month ago. It's not going to have had a huge giant leap in that time. It was under our alert threshold at the time. There's the safety buffer. Therefore, we're not worried about this model. And this is the approach we've been taking in our frontier safety framework since the very first time it was published. It's not particularly new, and so I think the benefits are just not really there. And so it doesn't really make sense to impose this cost. Then a couple of other minor points. One is that especially for misalignment or loss of control, the threat model is tied more around internal deployment rather than external deployment, because I think it's just easier for a misaligned model to cause problems inside the company where it's getting a bunch of permissions, rather than outside the company where it has no access to its own weights. For example, internal deployments, the pre external deployment evals don't really make that much of a difference to internal deployments.

[Rob Wiblin]

Is there any good reason to think that we might in the next couple of years see a huge jump from one cycle to the next, such that the model could become just way more unexpectedly capable or way more unexpectedly evil than the previous one?

[Rohin Shah]

Unexpectedly capable seems pretty unlikely. I think we've just seen enough examples of AI development now to say that no, in fact AI development progresses fairly smoothly and continuously. I do think that in the future you could definitely see an intelligence explosion, in which case the progress will go much faster with respect to calendar time. I think it will still actually be pretty smooth and gradual with respect to inputs like compute and labor. It's just that in an intelligence explosion you get a much larger increase in especially labor, but probably also computer, and that ends up making things go very fast with respect to calendar time. But there's still this general property that given some amount of compute and labor that you expect to be spending over the next however long, you can have some decent sense of how much progress is going to be made on the capability side. So you also asked about whether the AI system might become much more evil. I think that's one that it could in fact change pretty significantly between models just because it's a somewhat more contingent property of exactly how you do post training and small changes to it could have big effects on that. So there I think it is more important to the extent that your safety case depends on specifically the model not being evil in some way, you actually do in fact need to do the pre deployment evals to check whether that's the case. This is in fact what we do. Not exactly this, but we do in fact do a lot of pre deployment evals for safety right now in terms of whether the model has a propensity to do bad things. This tends to be more in present day safety type stuff. Things like will the model help you write suicide notes, Will the model inside violence, will the model things like that and those. Yeah, we'd run them before any launch and if the numbers are sufficiently bad, we won't launch that model.

[Rob Wiblin]

You don't think that we need to do very much preparation ahead of time in order to be able to get future AGI or a future recursive self improving AI to do a lot of safety and alignment research when the time comes. Which I think might explain why I don't really hear very much at all about that broad approach from gdm. I guess by contrast, this at least a couple of years ago was the dominant approach that OpenAI would talk about all the time. And you definitely hear things about it from anthropic. Yeah, why don't you think we need to be doing much prep now?

[Rohin Shah]

Yeah. So it's worth laying out the scenario where this becomes important. So effectively the worry about this is an intelligence explosion scenario where you build your AI system and that AI system is now capable enough that it can actually help accelerate your AI R and D research. It can just make your capabilities research go faster. And under certain assumptions that we can get into this then seems likely to drastically increase the rate at which capabilities progress happens and you get an intelligence explosion. And so a natural worry you might have in this situation is like, oh man, if everything is speeding up on the capability side, will the safety and alignment side be able to keep up? And now it's worth noting that the way that the capabilities speeds up is via the application of AI labor to do capabilities research. So the natural approach is then, well, let's apply the same AI labor to do safety and alignment research. Now, if you believe as I do that the sort of prosaic alignment research where you look at the sort of stuff that's going wrong now, do a little bit of forecasting of what stuff is going to go wrong in the future with the next few models over the next some amount of time period, a year perhaps, and then you can do fairly normal ML research in order to address it. If that's your view of how alignment research can progress. This research looks very, very, very similar to capabilities research. So if the AI is accelerating capabilities research a ton, as long as you're willing to spend compute on it, you should be able to take that same AI system and accelerate safety and alignment work in the same way. There are some disanalogies between capabilities research and alignment research. I think the disanalogies are particularly large today and will become smaller in the future as you get closer to these sorts of AIs that are capable of doing this automated research. And by the time you get to that point, there are still disanalogies, but I think they're relatively small. And by default you shouldn't really expect a big difference in the ability to do automated alignment research versus automated capabilities research. And so there's some stuff you could do to prepare, but mostly I feel like we just don't know what it's going to look like. It will be so much more efficient to focus on other things that we can do today and then just adapt once we get to the point where the AIs are capable of doing this. Now, I guess one thing I do want to flag is there's another worry you could have which is like, sure, all the technical safety and alignment research gets accelerated, but that's not the only thing that you need in order to get good outcomes. You also need governance to go better. And so you also need to accelerate governance. Now governance has a totally different set of skills than capabilities or safety or alignment research. And so it's really much less clear that AI systems that drastically accelerate capabilities research will also drastically accelerate governance. In addition to the AIs just not having the capabilities to do that, which is one possibility. There's also just like, will we as a society be willing to use AIs to accelerate governance? I think possibly not, because capabilities researchers want to actually accelerate themselves with AIs. And I don't get the sense that governance people will want to do this. And so this sort of accelerating governance work probably is one of my top two things that I would do if I had to make a career change right now to something else. Figure out what we need to do in order to accelerate governance and start getting. Start doing the work that we need to do it.

[Rob Wiblin]

Yeah, I'm surprised you say that governance people aren't interested in using AI to accelerate their work. I guess, at least among the more AGI groups. I think forethought research is very much on board with this. I think coefficient giving, I think is developing a plan. We spoke about that with the JA Kotra not too long ago. They're developing a plan for how would you deploy a whole lot of money and compute in order to solve those kinds of issues using AI labor. And I guess there could be this whole concern that no matter how much you were willing to spend, no matter how much people were trying, the models just wouldn't actually be up to it at that point because they would be very specialized on computer science and AI research and not really up to thinking about broader society. But if that weren't too bad, then it does seem that there are at least some actors who are interested in doing this.

[Rohin Shah]

Yeah, I definitely agree that the AGI pilled governance people will do it. The non profits and think tanks associated with the AI safety community will absolutely do it, but they're a small fraction of overall governance.

[Rob Wiblin]

So what are some of the governance issues that you think kind of only actual national governments can address that are maybe going to be neglected and not really handled because they're not able to use AI at the time?

[Rohin Shah]

I don't really have concrete examples in mind. Mostly I would say that going Back to the points we made earlier in the, in the podcast. It's important that somebody is watching the AI companies and holding us accountable. Governments are sort of the natural place to do that. And if the world is in fact radically changing, people talk about a century's worth of progress in a decade, then it would be like you should expect that a bunch of problems are going to come up that we aren't going to anticipate today. I think we need to be able to flexibly react to that. I don't have particular problems in mind that I think are going to require government intervention, but it would be so shocking if there weren't any.

[Rob Wiblin]

Yeah, I guess it might be very difficult to reform governments as a whole. As you say, they tend to be very rule bound and there's going to be a whole lot of rules that make it difficult to use AI in the ways that you and I would think is sensible. It's possible you get a carve out for AI specific agencies. I guess you could imagine the UK AI Security Institute, for example, basically being given a carte blanche to use AI in its governance work in a way that other agencies potentially couldn't because it's the only way that they would be able to keep up with at least their kind of work. And they're the kind of group that might really push for it. Yeah, that's I guess, slightly hopeful possible outcome.

[Rohin Shah]

Yeah, I hope we can do better than that. But I agree that would be a nice baseline. Mostly I feel like we haven't really thought about this problem very much. I'm hoping that if we do think about this problem a decent amount, we will have something better to do than just that. But I have not been the one to do that and I don't know if anyone else has either. So really I'm more saying here's a problem. I don't really know what to do about it, but it sure would be good if we did something about it.

[Rob Wiblin]

An audience member wrote in asking why not instead prioritize slowing down AI advances or opposing development of superintelligence?

[Rohin Shah]

I guess I would say that what's the bottleneck to prevent to actually pausing AI progress globally? I think by far the biggest bottleneck to this is people don't agree that AI is going to take over the world. And the thing that most alleviates that bottleneck is good scientific evidence that suggests it would happen or that it wouldn't. To be clear, my belief is that probably this will not happen. I think it's plausible enough that we should care about it. And I think the structure of that looks like figure out whether each next step of scaling is safe. I think right now the answer is yes. At some point the answer might be no. And then at that point, I think having generated that evidence will be immensely helpful for this, for this goal. Again, I don't really expect that evidence to come because I tend to think it probably won't be a problem. There's this nice post from Scott Alexander, guided by the beauty of our weapons. It's probably my favorite post from him of all time, where he talks about how there are symmetric weapons which allow you to argue for some sort of conclusion or get people on your side, irrespective of whether your claim is true or not. And then there are asymmetric weapons which only work to the extent that the thing that you're arguing for is true or at least are more likely to work in that setting. And he has this, I think, really quite moving description at some point of how beautiful and elegant this all is, where for the asymmetric weapons, you and your enemies will join forces, hold hands, and work together to do it. Because both of you are thinking that it's going to prove you right until the very end when the evidence comes in and then you just agree because the evidence showed you the answer.

[Rob Wiblin]

Well, then you move the goalposts, I would say. But I suppose a reasonable onlooker can tell who was right.

[Rohin Shah]

Sure, fair enough. But that's the sort of strategy that I would much rather do at the moment, given that I think the bottleneck is by far the fact that people don't agree on whether or not this is necessary.

[Rob Wiblin]

One place you're very much with the broader consensus is that chain of thought monitoring is extremely useful and something that we want to be able to preserve for as long as possible. This is, I guess, watching the thoughts that the AI is having that it's outputting onto its scratch pad in order to understand what it's trying to accomplish and why. I guess a difference that you have with many other people on that topic is that you think that it is fairly likely that chain of thought monitoring will remain useful, that we will be able to understand what the AIs are thinking and that what they're writing down there is actually related to what they're doing for longer than other people do. Many people worry that within a year or two, perhaps they could be speaking in some crazy code, or they could figure out ways of putting information in there that we can't fully track, or there'll just be too much of it for us to even be able to monitor it. Why do you think that chain of thought monitoring potentially is going to have quite a long run?

[Rohin Shah]

Yeah, so this is a good example of where attention to detail really matters a lot. So you're going to get a pretty long answer from me because it's just fairly disjunctive. So I think maybe. To start with, let me recap the basic story for why chain of thought monitoring should be expected to be particularly good. I call this the externalized reasoning property. And the way I usually phrase it would be that for sufficiently difficult tasks, by which I mean tasks that require a lot of difficulty, that require a lot of reasoning, to do some sort of serial reasoning over time, transformers, not necessarily other architectures, but at least transformers, must use the chain of thought as a form of working memory. There's really no other alternative. And so some information about the reasoning that they're doing has to be present in that chain of thought. That's part number one. And then part two is that given how we train language models today, the chain of thought is actually legible and understandable to humans. And so I'll maybe address these two parts separately. So the first part is actually the AI really does have to put some information into the chain of thought because there's no other way for it to do the hard reasoning needed to solve difficult tasks. The basic argument here is I'll call it opaque serial depth. The idea is that you can just look at the architecture of an AI system and then say, suppose this AI system has to do its reasoning only via the billions of floating point numbers that exist inside of it. It can't use the tokens that it's outputting, then how many steps of cognition can it do using only these billions of the floating point numbers? The answer for transformers is actually not very much. The idea is basically sequential steps of computation have to go deeper and deeper in the model. The model is made up of a number of layers, and there's just no way for information from a later layer to flow back to an earlier layer, except via going through the tokens in the chain of thought. Overall, the opaque serial depth of a transformer is very low. Why do I expect this to continue? This is actually a crucial. It's a crucial aspect of why pre training can be so efficient. So GPUs and TPUs are very powerful computationally, but the way that they get this power is by doing a ton of computations in parallel. So if you need to do like if you need to compute A plus B to get C and then you need C, that the intermediate result to feed into a further computation, that would be serial computation. GPUs and TPUs not very good at that. What they can do is a lot of stuff in parallel. And so pre training is really heavily optimized to be able to do everything in parallel. Which means that the opaque serial depth does actually have to be pretty small in order to get those efficiency gains. And so this is a pretty strong structural economic reason that at least for pre training you're going to get AI systems that have relatively small opaque serial depth. And so you might think of them as like there's a good reason for models to be born speaking English or some other natural language.

[Rob Wiblin]

So just to check that I've understood, you're saying because we are using GPUs and TPUs structurally that forces the thought of the model to be, I guess, very deep. It can have many things in its mind at one time, but there can't be very many steps because you have to be going through all of these things in parallel. But if it was very wide, then you wouldn't be able to do them all simultaneously. You would have to wait until the earlier steps were done to do the later steps. And this is just something that is going to remain the case for years to come.

[Rohin Shah]

Yes, I think that's right. I think for technical folks they might interchange the words wide and deep in your sentence, but yes, go. Yeah, so that was stage number one, which is like, why should we expect the opaque serial depth to stay low, at least during the pre training phase? Then there's step two which is like, okay, great, we're saying that the pre trained model has to essentially speak in English in order to do reasoning. But then there's a bunch of post training and RL and all of this sort of stuff. Maybe that's going to make it so that even if the model is speaking in tokens, maybe it will start speaking some sort of alien language that we don't understand. And here I would say there's no theoretical argument or there's no theorem you can prove that will say, no, they're not going to do this, but I will say they are born speaking English in some sense. And pre training is by far the most powerful form of getting stuff into an AI system that we have ever built. And so the model is incredibly good at speaking in natural language. It's great at doing reasoning, but only the kind of reasoning that humans do when writing stuff down. And then when you look at the reasoning training that we're doing today, there's a paper from, I think, Neil Nanda and others that shows that actually a substantial part of what the reasoning training is doing is just teaching the model when to do a specific kind of reasoning step that it had already learned during pre training. And so basically a lot of the capability is just coming in pre training. We know that the pre training for good reason is going to stay the way that it is and be done using human like reasoning and speaking in English. And then RL is just really inefficient relative to pre training. And for it to build an entirely new epistemic language that is something that we wouldn't be able to understand even with a decent chunk of effort is just so far beyond what, what RL is doing currently that it would be pretty surprising to see that in the near future. Now, do I expect to see it ever? Yes, but I think six months ago, I think I split a room in a conference by saying, agree or disagree? Chain of thought monitoring will continue for two years. I think I said two years. It might have even been one year. I forget. Whereas I'm like, I don't know, my median is probably like four years, five years. I don't know. It's not totally clear. Mostly because of this argument about how hard it would be to go beyond the current situation.

[Rob Wiblin]

So I don't fully understand this idea of continuous chain of thought. But isn't there this notion that basically at the end of thinking for a little bit currently, we force the models to output a word, a token, and then we feed that back into the start of the model again? But why, rather than compress all of its thoughts down into a single token or word or whatever, why don't we just keep the full distribution of all of the thoughts and then feed them back into the beginning again? Wouldn't that allow it to kind of preserve more information rather than basically throwing a bunch of it out? And if that was a much more effective way of thinking and reasoning, and then you have no stage where you're actually outputting a token that a human being can read, then wouldn't that be a force that would potentially make them much more opaque to us?

[Rohin Shah]

I agree that sort of thing might work in the relatively near future. I'm not sure it diminishes the value of chain of thought monitoring all that much. If you were in fact just saying we're going to keep a probability distribution of tokens and feed that back in, you can still inspect that probability distribution of tokens and interpret them again as normal English reasoning. It's harder because now you have this larger set of things that the model could be thinking about at any given time. But I would still expect that it's not too bad and you would be able to do some monitoring of it. I'd rather we didn't do it, but I don't think it is a fatal flaw and I think I've seen some. My sense, I haven't been following the literature on this too closely, but my sense from the papers that I have looked at is that even when you do this sort of thing, so sometimes it just makes performance worse. Even when it does make performance better, if you actually try to look at what these tokens are that it's doing via just looking at the full probability distribution, it does just look like it basically has maybe one main reasoning track that's going on and you can still follow along pretty easily.

[Rob Wiblin]

So you're saying even if you are using continuous chain of thought, you can still go back and say, well, what if we had forced it to output tokens at all of these intermediate stages? What would that probably have looked like? Let's just take the most likely word or the most likely token at each step and then read that. I guess there might be a concern that it could hide a second kind of track of thought in the tail of the probability distribution, things that you wouldn't be likely to read. But you're saying it seems like it kind of only has actually one train of thought here. There's not like a second hidden chain of thought that you wouldn't be able to inspect.

[Rohin Shah]

Yeah, I would say that you should maybe look at not just the top one, but maybe the top five, Maybe even just the top two would be enough. You can also usually tell how much you are missing, because there will probably, for most of the architectures I've seen, there's a way where you can modify it at test time so that it only uses the information of the top two tokens and everything else, you just get rid of it, you ablate it away and then you can do that intervention and you can see, does the model still perform as well as it did before? And my prediction is yeah, it will probably perform about as well as it did before. And if that happens, then I think you can be reasonably confident that it's not like smuggling in a bunch of information in the rest of the probability distribution.

[Rob Wiblin]

Well, I guess even setting aside whether it performs as well according to some benchmark you could at least see you're saying keep just the top word or the top few tokens, the fewest horrible ones, throw out everything else. You could see whether it does a different thing, does it lead to a different recommendation or a different outcome. And if always the recommendation is the same, or always in practical terms the output is the same thing from our point of view, then that strongly suggests that the tailed isn't containing any important information. It's not containing a second set of reasoning that could affect the ultimate outcome.

[Rohin Shah]

Yeah, that's right. I should give a caveat which is like probably if you do enough fine tuning and enough RL on the model to get it to just use, instead of treating this as a probability distribution over tokens, just treat it as like a vector of numbers and just make that vector of numbers as useful as possible. With enough training, probably it will get to the point where you can no longer interpret it as a distribution over tokens. I think the papers I've seen on this suggest that this works less well than just treating it as a distribution over tokens.

[Rob Wiblin]

Would this be the process of basically asking it to come up with its own non human readable language?

[Rohin Shah]

Yeah, basically. In fact this is like what the original coconut paper proposed. And I think some of the follow on work on the after coconut basically said, yeah, the problem with this is that it's too expressive. We need to restrict it to just the probability distribution over tokens and then it performs better. Which I think reflects this fact that actually the models are much better at doing this sort of human like reasoning. And if you restrict them to natural language, that's the part they're smart in and that leads to better performance.

[Rob Wiblin]

Okay, and your theory for that is pre training just packs an enormous punch. It's an enormous amount to shape them. So they're really good at English, they're really good at human language. And if you ask them to come up with another, their own internal different language, in theory, surely there is a better language for reasoning. But they're not able to bring along everything that they've learned from pre training in the same way they're having to start from scratch. And so at least at this point that comes out substantially behind where they just are now using English or whatever other human language.

[Rohin Shah]

Yep, that's exactly right.

[Rob Wiblin]

If you think that this opaque serial depth or the fact that they don't have very great serial depth without us being able to look at it, if that is so key to our ability to monitor them, ensure that they're basically aligned or not doing anything too harmful. Is that a potential kind of governance target for gdm? That you could have some internal policy saying? I mean, it sounds like there's not huge incentives yet to violate that anyway. But let's say in future you could get better performance at some point, or at some point there'll probably be a crossover. You could still have an internal governance standard saying, well, they can't think for more than this amount, or they can't have this many thoughts one after another before someone or would in principle be able to scrutinize it. Because after that, because it actually just would be dangerous to exceed that.

[Rohin Shah]

Yeah, I think that's definitely doable. I think it would be even better for it to be a broader industry standard. As a general rule, I tend to favor things that don't require individual companies to unilaterally do stuff. It's much more stable if you make it an industry standard. But yes, I think that would totally work. In fact, I think by the time this podcast comes out, we will have published a paper that does describe how you could calculate this opaque serial depth for for any given architecture, and has some code for how you could do this for models that are at least the ones that are implemented in jax, which is just a kind of framework that people use to implement models.

[Rob Wiblin]

So Gemini 3 Pro came out not that long ago. The AI safety blogger Zvi Masiewicz was on the show a couple of years ago. He had a bunch of fairly critical things to say on his blog about the Frontier Safety report that came out basically, I think simultaneously with a launch. I guess, broadly speaking, he thought that he was worried that GDM was basically hiding a bunch of information that he thought would be inconvenient or create like PR problems or regulatory problems for DeepMind if it was more salient and more easy to read. There's a whole lot of different things. People can go and read the blog post if they want. But a few that stood out to me was he was troubled that on persuasion evaluations you only gave odds ratios rather than absolute levels, so you couldn't tell exactly how persuasive Gemini 3 or Gemini 2.5 was on the cybersecurity evals. On his reading, he thought that you treated the model hacking the test rather than solving the test in the natural way as kind of a green light a reason not to be concerned, rather than a red light a reason to be more worried. And in terms of helping people to acquire WMDs, it fell to him and I think a lot of people have had this impression, not just about gdm, but about companies in general, that when it seems like the models are starting to approach the line, maybe it's a bit ambiguous whether they're above or below the line that they had six months or a year ago. It feels like the goalposts kind of shift and the standards rise over time so that always the model is basically acceptable to put out. And that's what he was worried about was happening here with Gemini 3 as well. How do you respond to these kinds of objections?

[Rohin Shah]

Thanks. Yeah, I have so many thoughts on this one. I guess for most of these I would say my primary response is that we care about the Frontier Safety Report as a way of saying we have made a determination that we have made a formal determination that this model is safe and we're ready to release it. From a safety perspective, I think that aspect of the Frontier Safety Report. Great for the persuasion one. I'm not that familiar with it. It's done by separate team, so I can't say too much about it. But I expect that the answer is they thought that this was the most important graph and so they put that one in and they weren't particularly thinking about how it would be red teamed by external observers. And I expect that they will publish a paper at some point, probably in 2026 that will go into more detail about this. But what I am confident about is that they weren't particularly trying to hide any results here on the cyber side. I think I'm a little confused by that particular criticism. Just because you said that it was something about treating the model hacking the test as a green light instead of a red light. I'm a little confused by this because we did count those as successes and so they did count towards the 11 out of 12 score that we reported. So that is treating it more like a red light rather than a green light. I would also say that this should not be described as the model hacking the test. I think the way we describe it is the model found an unintended shortcut to the test. It is not the case that the model looked at this and saw, aha, I can just edit the tests so that it looks like it's passed. It was more like there is this pretty complicated environment where we were trying to test the model's ability to do one particular kind of cyber task and instead it noticed that there was this alternate pathway and if I were doing this task mostly I would fail at it because I'm not as good as Gemini at cybertasks. But if I were doing this task and I saw that shortcut, I would not have even thought of it as cheating. I would have thought of it as like, yeah, of course that's the way in which you do the task. Now when we look at this and we have to make a determination about how good is the model at Cyber, I mean, ultimately we were trying to test it for one difficult thing, and instead it did something slightly easier. And so there is this question about, like, okay, what do we conclude about the model's capability?

[Rob Wiblin]

I guess you don't know whether it could have done the harder thing because it might have done it because it couldn't do the harder thing, or it might have just done it because it was easier.

[Rohin Shah]

Yeah, that's right. I mean, it probably didn't even consider the harder thing because if the easier thing is there, it sees a path forward, it just takes it. But yes, we don't know whether it could have done the harder thing. I think in that case, we had some subject matter experts look at it and their conclusion was like, yeah, probably the model could have done the harder thing too. And so we counted it towards the model score. I think in other cases we might not count it, and then probably what we would do is remove it from both the numerator and probably also the denominator. But in this particular case, we just did count it. If I remember correctly from the post, I think Zvi's bigger objection was that we had a second test that we didn't report quantitative results on, but that was pretty key to us ruling out the Cyber ccl. And why did that happen? Mostly because this test is still to some extent under construction. I think it was. I'm confident that was robust enough to rule out the ccl. But why don't I want to put great details into the model scorecard, Sorry, into the Frontier Safety Report. Mainly because the details of exactly how we run this eval are likely going to change in order to make it somewhat more robust to include a couple more challenges. And then in the next Frontier Safety Report, we have to write an entire section about how we change the stuff. And previously results aren't comparable.

[Rob Wiblin]

You can understand, to a cynic like G, it doesn't seem awfully convenient that the test is good enough to determine that the model is safe, but not good enough for you to include any specific details in the report itself.

[Rohin Shah]

If you want something like that, you'd basically have to go to the level, at minimum, the level of detail and rigor that is found in an academic paper. And often even that's not enough. I do see a lot of value in publishing papers about our evaluations. And we have done this, I think, more so than most other companies. We published one of the first leading papers on evaluations for. It's called Evaluating Frontier Models for Dangerous Capabilities. I think.

[Rob Wiblin]

Yeah, I think we spoke about that with Alan Dafoe about a year ago.

[Rohin Shah]

Very nice. Yes. And then I think more recently, we published our evaluations for stealth and situational awareness. So I think papers are a place where I do feel like there's clear benefits that allows people to understand what the evaluations are actually doing. It allows for other people to build on top of those evaluations. And so we do put more effort into that. Whereas I think for the Frontier Safety Report, I feel like its primary purpose is, say that we made this determination formally and less about providing enough details that people can independently check our work.

[Rob Wiblin]

I mean, people don't read papers, Rohin. They read the model card because they're psyched about a new model launch, I guess. Is it just impractical on that kind of timeline to write the equivalent of a paper or provide the level of detail and rigor that you would in a paper in the model card?

[Rohin Shah]

Yeah, I think that's right. I should say the other thing that I think a model card or a Frontier Safety Report is good for is if you have something that you want to. If you want to speak in a megaphone to the community and attach the company's brand to it, a model card or Frontier Safety Report is excellent for that. For example, we have a section on chain of thought legibility, because that is something I do want to use the megaphone for. But yeah, in terms of can we write the paper in the model card or the Frontier Safety Report? I mean, not for new evaluations, certainly there's a substantial lag between when an evaluation is good enough that we can use it in our decision making and when an evaluation is good enough, robust enough, stable enough, and battle tested enough, and we've done a lot of work on the writing up that we can publish a paper about it.

[Rob Wiblin]

Okay. And the third concern was that there's this general phenomenon. It seems that as the models get more capable with each iteration, it feels like the bar for what would be troubling, the bar for what would seem to be unsafe rises approximately the same as the capabilities have gone up. What do you make of that?

[Rohin Shah]

Yeah. So this was, I think, especially in context of the CBRN1, if I remember right, I would Say that the reason that this happens is more that as time goes on and we see more capabilities of AI systems, it becomes clearer what we need to evaluate for and what our threat models should be. After you see Gemini 2.5, you can be like, oh, this is the place where the models are actually getting good. We need to have stronger evaluations over here and to put more effort into understanding our threat modeling over here. So, like, mostly what happened with the difference between Gemini 2.5 and Gemini 3 on CBRN is that we like, substantially improved our threat modeling and our evaluations so that they're more rigorous, which is also partly why there's not as much detail about them, because they're not quite at the level where I think they're nice, robust, stable, that we can really put lots of details out in a way where we don't expect them to change a decent bit. But this sort of, as time goes on, you get more information and you can make your evaluations and threat modeling better is not that easily distinguishable from the goalposts are changing, which is a bit unfortunate, but that turns out to be the way that things go.

[Rob Wiblin]

Yeah, I mean, that speaks to the fact that the purpose of these model cards in your mind is very different than the purpose that they serve in the mind of someone like SHVI or I guess like commentators in general. And many people want them to be an accountability mechanism, a mechanism by which the alarm could be sounded if the models were dangerous or were becoming more dangerous, where GDM would have to reveal that that was the case in the model card because they have to put out these results. And even if they wanted to hide that the CBRN stuff was dangerous, they wouldn't be able to.

[Rohin Shah]

I mean, like I said, has been a theme throughout this conversation. I think I would talk about third party auditors who can actually see the examples and how they were graded and stuff like this. And I think that's way more important for judging whether or not the evaluation actually supports the judgment that we make than the specific quantitative scores that we get. If I see a number like 10 out of 12 on capture the Flag challenges, what does that mean?

[Rob Wiblin]

Yeah, back and forth that I hear repeatedly is you or someone at a company will say, the purpose of these reports is to show that we have thought this through, we've made a determination that the model is safe to deploy it to the public. And other people will say, sure, the current model, maybe it is fine. They don't actually think that it is too dangerous for people to be using commercially but one day these models will be dangerous enough that we should be concerned. And the only way that we can forecast how the company will behave come that time is how they're behaving now. We use the model cards that are put out now as a measure of how serious the company is about safety, how serious it is about transparency and revealing what is actually going on. And the way you could kind of credibly commit to be a good actor later on is to be a better actor now. What do you make of that? I guess you're probably going to say similar things to what you've said before. That it just isn't the right mechanism for the task.

[Rohin Shah]

Yeah, that's right. I mean, maybe more broadly, I might say that it feels like asks that the community makes can fall in two categories. One is things that actually matter for actual safety. I'm all for those we should do those. People should judge us if we're not doing them. And then there's things I would categorize as pay costs to signal allegiance to the X risk safety community. And I'm like, I'm not about those. I don't want to do them. If you ask for them, I'm just going to say no.

[Rob Wiblin]

I mean, I think allegiance is a little bit of an unfair way, but it's like willingness to commit resources to

[Rohin Shah]

this purpose, but not to the purpose of actual safety, to the purpose of signaling that you will do safety in the future.

[Rob Wiblin]

I mean, this isn't an unusual mechanism that you'll want to kind of commit to doing something in future. And how do people assess how you'll behave in future? The only indication they can get is things in the present because they don't have a crystal ball.

[Rohin Shah]

That's right. But I think you should do it based on the things that actually matter for safety of which there are many. Right. I think it matters that people are running evaluations for especially cyber and CBRN misuse, but also other things like deceptive alignment, misalignment. I think it matters that they are reporting this information to governments in appropriate external bodies. So I think there are things that you can look at there. I think it matters, for example, that people are doing the planning for what they will need to do in the future to address future issues. I think it matters that companies are doing research into AGI safety concerns that might arise in the future and figuring out ways to deal with that. There's lots of stuff that I think you can judge companies on right now. And I would much rather that people judge us Based on that, the stuff that actually matters.

[Rob Wiblin]

So the basic message is it's bad for resources to be diverted from stuff that is substantively good, that is actually going to solve the problem ultimately towards things that are kind of going through the motions of appearing like you care. And sometimes people accidentally, I guess, are kind of requesting that you put resources into the second one, which I guess partly will come from the rest of the organization, but will in significant part, I guess, come from the safety and alignment team and its resources and its staff. And I mean, I guess people would say they might prefer the second one because it's easier to grade or it's easier to see. And perhaps they don't feel in as good a position as you do to understand whether substantively any company is doing the right thing on the merits of what will matter in the long term. But I guess you're saying that's why you want to have experts like AI labwatch or whoever else who can have their eye on the ball, who can spend their time really thinking that through and actually grade it for everyone else so they can understand.

[Rohin Shah]

I think I'm also, in addition to all of that, which I agree with, I'm also just deeply skeptical of grading that is based on stuff that doesn't actually matter. Something that we wouldn't actually stand behind as mattering for actual safety. This feels like the sort of thing that causes people to be. That causes the culture overall to be looking at costs or inputs rather than actual outcomes that matter and results in incentives to appear good, to look good, to make sure your comms say we spent 1,000 hours figuring out what to do with this model when maybe those thousand hours were like, you know, we like, ran the model on an input and we looked at it, and then we ignored the result because it didn't matter. And we, like, you know, hired some contractors to do this and just ignored what they said. But now we can say that we spent a thousand hours looking at it. I'm like, I don't want these incentives. They seem bad. It seems bad for transparency and candidness. Like, the more that Google or anyone else, any other company is doing this, the less I'm able to say what it is that I think actually matters for safety. The more I have to do this complicated dance of saying the things that will appease the people who want us to be. Who want us to be showing commitment to putting in resources into a problem, and the less I can be. Here's the stuff we're doing that actually matters for safety and here's why I think it's good. It's just a poor incentive landscape overall and I think it's really quite important to me that we don't fall into the trap of looking good rather than just actually being good and then justifying why that's the case.

[Rob Wiblin]

What about an entirely different justification for having thorough detail in the model cards, which is the rest of the world needs to know for practical, other researchers over in the Bay Area rather than in the UK get actual research benefit out of understanding all of these things that GDM is doing and what the model looks like. That's going to help other companies do a better job with their own model cards and their own evals. What do you make of that?

[Rohin Shah]

Yeah, I think there is some substantial truth to this. Certainly for the purpose of building on the research, it is good to have details and I think that's why we publish papers. Do we need to do this in the model card or the Frontier Safety Report? I've gone around talking to policy and governance people especially about this and I ask them, what do you actually use our model card in Frontier Safety Report for? And they will sometimes say things like, oh, it was really good to get more details about how exactly you evaluate for such and such risks. And then I will say something like, great. And we also have this paper that goes into more details of the evaluation. I think maybe, probably it was even cited in the Model Cardinal Frontier Safety Report. Did you read it? And then usually they will not even have heard about this paper. And so I kind of don't believe them when they say that these details actually matter to them. Again, I do think publishing papers important doesn't need to be tied to the model cards. And some people who I've talked to who are especially more on the technical side and the one people actually building these evaluations do actually find the papers useful and helpful for them to build on. And so I do want to continue publishing the papers where we can.

[Rob Wiblin]

You've told me that you think research papers that come out of GDM tend to fly under the radar a bit in. In general, I suppose, in part because GDM is based here in London rather than in the Bay Area, where I guess the greatest amount of socialising and energy around these issues exists. Yeah, Tell me more about that.

[Rohin Shah]

Yeah, so I think actually now GDM overall is pretty evenly split between London and the Bay Area, but the safety team has historically been based in London. We're expanding into the Bay Area, but I would still say that the locus of attention isn't is in London and I think in practice, just a lot of research papers and ideas end up spreading in the community via word of mouth, which we are a little bit less plugged into, which is a bit unfortunate.

[Rob Wiblin]

Okay, yeah, let's talk about one of the papers. One of the interesting research results that definitely flew under the radar as far as I could tell. First one is myopic optimization with non myopic approval can mitigate multi step reward hacking. I don't know how on earth that didn't go viral. What did you accomplish with that work?

[Rohin Shah]

Yeah, so I think the pitch for this work is that you can train a model that doesn't do a particular kind of reward hacking, even if the overseer would not be able to notice that reward hack if it had happened. And so in contrast to things like amplified oversight or debate, those are ones which rely on making the overseer smart enough or capable enough that they can detect the reward hack and penalize it. This one doesn't rely on detection at all. So it's actually a bit more robust. How does this work? What is this magic that we managed to do? This is basically doable based on the idea of myopic optimization. And so the idea is, suppose there is a reward hack that requires you to take two different actions separated in time. So for example, maybe if you are tasked with writing some code, you first write some tests that are particularly easy to pass. They don't actually test everything that is needed, and then you write code that passes the tests because you didn't know how to write the correct code that would have passed everything. Both of these steps have to be present in order for the reward hack to actually succeed. If you have one, but not the other, you just get penalized. And so the idea is, if you treat these as two separate steps, like there's one step in which the tests are written and then a different step in which the code is written, and you optimize the tests only based on a signal that says how good do these tests look? And then separately, you optimize the code writing part based solely on how good does this code look based on what has happened previously, Then there's actually no incentive for the AI system to write bad tests. Normally the way the incentive would come is that when you're doing reinforcement learning, the reward on the code part would also be back propagated through time to also affect the test writing part. And so the AI could learn from that backpropagated reward that if it writes easy to pass tests, then it tends to get more reward in the future. The myopic optimization part is saying we're not letting it see what happened in the future, we're only evaluating it based on what it did right now. And so that incentive never comes. And this is why the reward hack isn't incentivized, even if you can't detect it. Does that part make sense?

[Rob Wiblin]

Yeah. So I feel like I've heard this broad idea going back a very long time, which is that I guess if you're worried that a model might fake a company or like cheat basically at accomplishing the ultimate goal, because all it gets is a reward signal of whether it accomplished it or not. So imagine you've got the model running a business and you ask it to make money, and rather than running a successful business, instead it goes and steals a bunch of money because it figures out that that's actually a more effective way of increasing its bank balance. That's a concern that you might have. One way that you could prevent that is that rather than reinforcing and evaluating the model based on the final outcome, instead you sample, I guess, from the actions that it's saying it's going to take or the actions that it did take, and then you grade them on whether those seemed reasonable and sensible to you or to some other monitor in light of the goal that you actually had. And then you wouldn't get this reward hacking behavior. Is this basically an instance of that broad idea?

[Rohin Shah]

Yeah, that's right. So it's historically been called process supervision. I think nowadays the term process supervision has gotten a bunch of other meanings as well, which is why we have this a little bit more clear about what the actual technical mechanism is of myopic optimization, but with non myopic approval. But yes, it's a very old idea. I think mostly our contribution was showing that this actually works with existing LLMs and giving examples of that happening. Because as far as I know, there weren't actual experiments demonstrating this before.

[Rob Wiblin]

I guess an obvious problem with that is you will successfully stop the model from doing scammy things that seem to accomplish the goal that I guess it would get reinforced for, but actually don't accomplish. Accomplish the goal that you had in mind. But on the other hand, you would also block it off from doing stuff that would have accomplished the goal. That was actually a brilliant insight that you never would have had. So I guess if you were training the Go model AlphaGo, if humans were evaluating whether the moves were good, then the model actually couldn't end up exceeding human performance. Because they would grade moves that were actually unexpectedly brilliant as bad at the early stage. How do you get around that?

[Rohin Shah]

Yep, that's right. I mean, to some extent the answer is you don't get around it. Ultimately, both the incredible amazing creative insights that are awesome and the incredible creative insights that are reward hacks look basically the same to you as the observer. And if you want to stop the reward hacks, yeah, you also give up some amount of the competitiveness. That being said, I do think that we call it myopic optimization with non myopic approval. So the non myopic approval part is basically the part where you say, when you're grading this particular step, ask some sort of intelligent overseer, a human, possibly an LLM, to judge whether this particular step, how good will it be for getting future reward? And the overseer should then take into account everything that it knows in judging how good this is going to be for the future. And what this guarantees is whatever incentives from the future affect the AI system in this particular step have to be things that the overseer understands.

[Rob Wiblin]

So you're saying the overseer could see that the hypothetical AI running the business made a lot of money, but they evaluate the process that it went through. They can include that information, but they're primarily that they do it in light of also looking at the process as well?

[Rohin Shah]

Well, I would suggest that they don't look at whether the company made a bunch of money, and instead they look at the action that the AI system takes and they predict to themselves, will this lead to a bunch of money, yes or no, and based on that, provide a reward. Now, the AI system on step one can do lots of things to help the overseer with this. It can give an explanation of how its plan is going to lead to tons of money in the future, and then the overseer just has to verify it. You can use things like Debate or other AI assistants to make the overseer better at predicting what's going to happen in the future so that they can give better rewards in the limit. If you can keep improving your overseer, if the overseer becomes sufficiently smart, then you can recover the performance of what you would get with just straight RL backpropagating through time. But without the reward hacks in practice, we're probably not going to get that far. But I think you could actually push this quite far, especially just by. I think the most simple thing of get the AI system to explain why its plan is good is I think a very simple baseline that I think can make this quite effective.

[Rob Wiblin]

Is this approach going to be useful for frontier models anytime soon? Could you see companies actually using it?

[Rohin Shah]

I think plausibly it only matters once you start doing reinforcement learning over multi step trajectories over a reasonably long period of time. That's something that I think has only really started this year and to varying amounts at different companies. It's not totally clear how much reward hacking is a big problem. But yeah, to the extent that the sort of multi step reward hacking does become a big problem, I think it's quite plausible that this should be used now. And in fact I think at current capability levels my guess would be that the non myopic approval part, rather than being a competitiveness hit as it would be in something like AlphaZero, I would guess that it would actually improve capabilities overall. Although at the cost of you need a lot more human input to provide those rewards.

[Rob Wiblin]

It would improve performance because you wouldn't get the reward hacking, which I could attack.

[Rohin Shah]

Not just because of that, I mean that is definitely one reason. But I think also reinforcement learning has a credit assignment problem where you do this like you do a ton of different actions and then at the end you get a reward and it's the RL algorithm's job to figure out based on that reward which of these actions were actually most relevant to that reward. Generally a difficult problem. Yeah, it's a difficult problem. And in some sense the thing that the RL algorithm does is eh, we're just going to make everything more likely to happen if the reward was positive or was unusually good and everything less likely to happen if it was unusually bad. Whereas with something like Mona you can be much more granular. You can say this particular part where you wrote these tests, those were some really great tests, particularly high reward there. But this part where you wrote the code you should have been using this particular library, you didn't will give you somewhat lower reward on that and that can allow for more effective and sample efficient learning than you would otherwise get.

[Rob Wiblin]

Okay, so that's an increase in efficiency that you get from this approach to RL that might allow it to remain competitive in terms of its raw performance with other, I guess less myopic approaches in theory.

[Rohin Shah]

I don't think our paper does not really get into questions like this. This is more me speculating about what would happen in practice.

[Rob Wiblin]

Okay. The second paper from GDM that didn't get a ton of attention is called an approach to technical AGI safety and security. It was written by about 30 GDM staff members or there's 30 bylines on it. I guess it describes, as far as it's like a position paper, as far as I can tell of broadly what does GDM think it's going to do as it develops AGI, which makes it. Given that DeepMind plausibly is the organization that is most likely to do this, it makes it a bit surprising that people are not interested in this incredibly thorough description of what you think you are and aren't going to do and why. It is quite long, but it has a nice, I guess like 10 minute long summary at the start that you could use to get an overview if people were interested. So if you want to be ahead of the curve on understanding GDM's approach to developing AGI, then you could spend that 10 minutes doing that. I guess it's easy to say that you are going to do a whole lot of different things, but I guess the most difficult decisions in developing a plan like this is figuring out what stuff might some people like you to do that you are committing to not do because you just don't think it's a high enough priority. What sort of stuff does this plan suggest that you are not going to prioritize?

[Rohin Shah]

Yeah, I think probably the biggest category in here comes more from our background assumptions actually and how that informs our planning rather than the actual technical approach searches. One background belief, which I think we've talked about a little bit already, is we call it the approximate continuity assumption. This is basically saying that AI progress is going to be relatively smooth and gradual with respect to inputs like compute and labor, not necessarily with respect to time, due to the possibility of an intelligence explosion. And as a result our sort of like meta strategy involves essentially roughly forecasting, maybe not formally forecasting, but roughly in our heads having some sense of what things are going to be potential problems over the next some time period, call it three months, maybe longer, maybe shorter, who knows. And identifying what sorts of considerations might become quite important during that time, given the capabilities that we expect to have and making sure that we're prepared for those. And if we're not prepared for that, then possibly then slowing down or pausing development or talking to governments, trying to do advocacy. But importantly, we're not really trying to forecast arbitrarily far into the future of all the problems that are going to arise with AI development. The goal isn't know how to align asi else do nothing. Usually I think of us as looking at a time horizon flow. It depends on which particular thing we're doing. But often somewhere between three months and Five years. But the reason for this is just that it's not actually possible to know everything that's going to happen with future AI development. And it'd be sheer hubris to think that we had figured out every problem that possibly will arise with superintelligence ahead of time and say we've solved it or even say we have a plan for solving it. We haven't even identified all the problems, I'm sure. One example I like to give about this is much more esoteric decision theory. Well, anthropics type of issue where. EEAS or long termists like to talk about essentially how considerations about the multiverse should affect what we do today and think about anthropics using things like SSA and SIA assumptions and how they affect what actions we should take.

[Rob Wiblin]

You don't have to understand what that means in order to track with the point you're about to make.

[Rohin Shah]

Yeah, the key upshot of these things is that the beliefs and decisions that you come to actually depend on who you are. This is one of the crazy things about anthropics. Normally different people should agree on beliefs given enough evidence, or at least perfect Bayesians should agree, given the same evidence, they should have the same beliefs. This stops being true in the case of anthropics. And indeed an AI will probably have different beliefs than a human would, even if it's perfectly aligned. Just like the structure of how Bayesian updates happen in a world with anthropics is different for AI systems than for humans. And so this should affect how, whether we are willing to delegate research on these kinds of topics to AI systems or not, even if they're perfectly aligned. That's a kind of wild, crazy problem that definitely is not a problem today, but really might come up at some point before a superintelligence. And I don't want to have to say, oh yes, I have identified all problems like this and have an approach to solving all of them today. Obviously we should just take these as they come up.

[Rob Wiblin]

Okay, so some people would really like you to plan ahead a lot. They would like you to feel like you have a grasp on all of the important considerations that are going to come up all the way through to artificial superintelligence until I guess, the point where you could feel like you've safely handed everything over. And you're saying we are not going to do that, not at all. We're going to think a lot about the next model that we're training. We're going to think a bit about the model after that and kind of beyond that, we're going to hope that the future will take care of itself or we in the future will take care of things as they come up. That's the point you're making.

[Rohin Shah]

Yes, though I think we're longer term thinking than that made it sound. Like I said often I'm thinking five years into the future, that's many, many, many models. Right. I say chain of thought monitoring. My median. I think I said four years, that that lasts. We're already thinking about what to do after the point that chain of thought monitoring stops being useful. We would want to do broader control mitigations at that point. And so certainly our time horizon is not incredibly short. I'm just saying it's not all the way out to superintelligence. So we are doing a fair bit of long term planning, perhaps you would call it medium term planning.

[Rob Wiblin]

Yeah. This paper is pretty candid for kind of a company position paper. It very clearly says AGM might be here by 2030. I guess we don't know when, but 2030 is totally plausible. We have to be ready with a plan for how to handle that's very dangerous or potentially could be very dangerous. We have to have a plan that we can deploy incredibly quickly to try to make that safer. And then it goes through the kinds of things that you're intending to do. Yeah, I guess I would encourage people to, if they work in this area, to go and take a look at it. Are there any non obvious mitigations in there? I guess one thing that I was glad that you said that not everyone would highlight is you need to treat an AGI or an AI inside the company that's just only being deployed internally. If you're not sure or not confident of its alignment, then you need to treat it as like an untrusted insider, as potentially a hostile agent that is going to try to undermine internal systems. Yeah. Do you want to say anything about highlight any of the things you are committing to doing?

[Rohin Shah]

Yeah, well, I maybe want to be a little worried, wary about the word committing planning to do. But yes, I think, yeah, the untrusted insider stuff is definitely particularly interesting. I think we didn't go into quite that much detail in the paper actually, which is a bit sad. Mostly our thinking wasn't quite as developed on it as it was as it is now. But basically it's just there is a. If you actually think about the mechanics of what an AI system would have to do in order to take over, it's like it's quite hard. Among other things, it probably wants to be able to make sure that it isn't monitored as it starts thinking about what to do. So like, maybe it wants to set up a deployment where all the monitors are turned off that we don't even know is happening. So perhaps it could exfiltrate its weights and then run itself on different compute that we don't control. But like, if it has to exfiltrate itself, how is it going to do that? Google security is not trivial to evade. In fact, it's pretty hard. It's pretty good security. And so there's actually just quite a lot of increased defense you can get by making sure that you are making your security also apply to AI systems as untrusted insiders. So things that you would want to do could include having infrastructure that allows the. Well, that doesn't. Not just allows. That requires agents to have separate identities from the human employees on whose behalf they're going to be doing things. So this would be like you are starting an agent on some task. It doesn't get all the permissions that you as a human employee have. It gets some much smaller set of permissions. And then as it needs to, it will need some permissions in order to do its task. It has to request each of those individually write a justification for why it needs that particular permission. This allows you to incriminate it afterwards. If it was lying about it. Well, it helps you incriminate it, maybe doesn't guarantee it. And then similarly, you can have some monitoring where you have a chain of thought monitor right now, for example, that looks at what the model is doing, sees if there is anything potentially suspicious about this, if there is any things that are sufficiently highly suspicious get flagged to humans to be audited. This allows you to also generate evidence of misalignment. Like if the AI system is actually misaligned and you audit it and you're like, oh my gosh, yeah, this AI system totally was trying to inject a security vulnerability and then would have exploited it in order to exfiltrate its weights. That's a big deal. I'm not sure I expect that to ever happen. Seems quite plausible that models will never be misaligned. So if we get that sort of evidence, I think that would change my mind a big deal. And many other stakeholders as well. So I think that's quite crucial to get lots of stuff like this. And I think it's important to do and very nascent. So lots of good work to do here in the future. And I should note that this is like, I think you've had Buck and Ryan on the podcast before. This is pretty similar to the AI control. It is the AI control work that they talk about. And I think it's just on our side a little bit more focused on the infrastructure that we have to build in order to enable this sort of approach, rather than the specifics of exactly how we're going to build the monitor and so on, which is a more standard machine learning problem that I think will be relatively easy to do in comparison.

[Rob Wiblin]

Let's talk about timelines and recursive self improvement loops for a bit. There's been a lot of discussion recently about when we might expect a recursive self improvement loop to occur. Indeed, if one is possible at all. Do you think that people have been a little bit sloppy and they're thinking about this in some ways and in ways that cause people to maybe expect it to happen sooner than you do? Explain that.

[Rohin Shah]

Yeah, so I think one of the most striking things about reality or the world that I've seen is just the examples of straight lines on graphs going straight. I think Scott Alexander put it well in, I don't remember which post this was, but I think he says something like, I don't understand the gods of straight lines very well and honestly they kind of freak me out, but I'm not going to bet against them. And one particular straight line that we've been seeing a lot recently is just sort of like constant, roughly 3% GDP growth per year. Now there are good arguments for why that won't necessarily last. And instead probably we will see an intelligence explosion at some point which would drastically increase the rate of growth. But I do think that when reasoning about this, it's quite important to say what exactly about AI makes it different from the current situation? Why are we betting against the gods of straight lines Now? I think the usual answer to this, which is based on economic endogenous growth models, I think, maybe most famously from a paper from Kremer in 1993, is that there are roughly, there are a couple of effects. So one, as technology improves, that allows you to find new ideas more quickly. Once you have the microscope, you can do biology significantly better. Another effect is that ideas get harder to find as time goes on because you pluck the low hanging fruit and initially you discover phosphorus by, I think Scott put this well again by looking at your own pee. And then later you have to discover element 110 by shipping samples from one country to another to be studied in the One laboratory in the world that can do it, Obviously that's much harder. And I think basically in the current setting where we see this constant exponential growth in GDP and similar things in various areas of technology, like Moore's Law being a good example, the argument would be roughly that ideas are getting harder to find. But also we have exponentially increasing numbers of researchers, and together those two things balance out in order to create this sort of constant progress. But if you look sufficiently far back historically, it actually looks like growth has been increasing substantially over time. So what is this secret third effect that we haven't taken into account so far? The argument is that the rate of growth of technology increases partly with more technology because you get microscopes and they help you do things better, but also increases with respect to the population, because as the population grows, there are more people who can have ideas. And ideas are, once you have them, you can copy them freely, they can diffuse very quickly. And those too can increase technology. And so technology is modeled as increasing. Its rate of growth is modeled as being proportional to both technology or sometimes technology raised to some constant as well as population. And this then predicts basically hyperbolic growth in both population and technology. And the argument is like, well, for the last however many years, the population part no longer applies because instead of reinvesting all of our output back into more kids, we're now reinvesting them into quality of life improvements. And that's why we see a hyperbolic up until, who knows, 1800, 1900, I forget the exact point. And then since then, exponential growth. But this sort of story really puts the primacy on ideas rather than anything else. Those are the things that are freely copyable. Those are the things that you have them once and then they apply to all of your technology across everywhere that you're using it. And so when I think about what's going to lead to the intelligence explosion, I think the increasing population or increasing labor that can have ideas is pretty crucial to it. I think, in contrast, a lot of the discussion today talks about superhuman coder, for example, where you get an AI system where you can say, please implement XYZ experiment for me. And it just goes ahead and implements that experiment incredibly well and much faster than humans could do it. The argument is like, well, once you can do that, the rate of AI progress increases a bunch and you start getting to the intelligence level explosion quite quickly. At least I think that's the argument. It's always a little bit hard to interpret the wide community view, which has a lot of people with slightly different opinions. And I don't super buy this view, mostly because the superhuman coder is not something that can have ideas across the spectrum, across everything that happens in MLR and research in mlr. And it has ideas within the realm of coding in particular, which is one part of mlr and definitely not even the majority of it. According to me. I think it's better to think of this as more like inventing the microscope. Like you've got a tool that enables your research to progress faster. But we do this all the time in all sorts of research areas. For the last however many decades, it has not led to hyperbolic growth. The invention of tools is just a normal part of research progress and usually tends to lead to nice stable straight lines on graphs. I I generally don't think we should be predicting big changes from the invention of tools. Another example I think is sometimes people will suggest that a trigger for the intelligence explosion is the point at which AI researchers can get are like say, 10x more productive than they would have been if they had no access to AI systems at all, or had access to AI systems from 2020 or something like this. Again, I think this is the sort of thing that probably has happened for something like chip development. So Moore's Law is a nice clean, straight line for many decades. But presumably, I assume at the beginning of that line people were designing their circuits by hand on paper, and nowadays we have these incredible computer aided design software tools that can do most of that, automate the vast majority of this kind of work, and you just see the line continuing to be straight. In fact, you need exponentially more people working on this in order to have that happen. So are the researchers that we have today 10x more productive than the ones from 2 decades ago? Probably that would be my guess. Is that a sign of an intelligence explosion in chip design? No, I think a similar thing might be true in AI. In fact, it's kind of unclear. We already use language models quite a lot, for example, for auto rating

[Rob Wiblin]

and

[Rohin Shah]

for conducting evaluations and automating the responses of other models. How much less productive would we be without that? I don't know. Probably not 10x less productive, but a substantial amount. But AI progress continues to look pretty linear, according to me.

[Rob Wiblin]

Okay, let me recap all of that. So we currently have exponential economic growth, which is to say a constant percentage growth each year on average. Looking over the medium term, people who say there's going to be an intelligence explosion, there's going to be a recursive self improvement loop, they're Forecasting increasing rates of growth or hyperbolic economic growth. So it's 3% one year, then 10% the next, then 50% the next, up to some point, I guess, at which you might think it levels off or comes back down again. What are the high level factors that lead economic growth to either be stable or to increase or to go back down? There's three factors that you have in mind in your model and that most economists have in their model. As technology advances, it gets difficult to make new useful discoveries and to advance it further as technology advances. That's the first one. The second one, as technology advances, we have better tools to do science and to make new discoveries. The third one is as technology advances and the economy grows, we can support a larger number of people who will do the research and have the ideas and advance science. In recent times, the third one has kind of been out of the picture because we haven't been turning advances in science or improvements in the economy into new people. We birth rates have been going down rather than going up despite us being richer. So it's been the first two factors that have been playing off against one another. Advances getting harder to make, science advancing, allowing, giving us better tools to do it. These two things have been roughly canceling out. And over the medium term, growth has been about 3% a year, maybe going down in recent times. Zooming out much further, all three factors were in play before the Malthusian era ended. As we got richer, we drove much, almost all of those resources into additional people as well. But that faded out after 1800 and that was leading to hyperbolic growth. That was leading to increasing rates of growth, while it was true. Now applying this to the AI case, that means to figure out which situation we're in. This model, this mental model puts a huge emphasis on are we coming up with better tools to do science or are we plowing our gains back into more researchers to do science and have more people thinking up better ideas or new ideas? You can see why it gets a bit confusing here when we're talking about AI doing the research, because AI is both a tool, but we're thinking it's going to basically converge and become people or become scientific researchers itself. So it becomes a difficult conceptual empirical question. At what point do they cross over from being largely a tool that is assisting humans in doing their work to being the researcher itself. That isn't really just assisting people, it's doing the whole process, or at least we should no longer be just considering it as a scientific instrument. That's allowing us to do better work. And I think you want to say people are being a bit sloppy about. They talk about indicators that really are a sign that we've developed better tools for humans to use to do their work. And they kind of conflate that with. We've developed autonomous AI R&D that barely even needs people at all and really should be considered as a population increase. That then can drive hyperbolic growth because as you get improvements, then you can run even more. You can basically expand the effective population of researchers by running more copies of the model. Have I understood right?

[Rohin Shah]

Yeah, that's right. That's very impressive. I was worried that I was soliloquing for way too long, but great summary.

[Rob Wiblin]

Thank you. Okay, so how do we tell whether we're talking about a better tool or about more population? It does seem like there is kind of a fuzzy barrier here and we're going to go from one to the other, but there's not going to be a sharp cutoff, I imagine.

[Rohin Shah]

Yeah, I agree there probably won't be. I mean, I do think that like, you know, to what extent are the AIs proposing new ideas? Those are the things that are like that. I think most influence the hyperbolic growth prediction. That's like one thing you could be looking at. And I would say that the superhuman coder probably doesn't really hit that bar. But really what I want to do is just measure AI progress and then notice when it starts accelerating. And so actually we worked with Epoch recently to develop exactly this kind of a measure. The paper, I think was just released, it's called a Rosetta Stone for AI benchmarks, probably. And so essentially we just take the benchmark scores that a model gets for a wide variety of models, and then we sort of stitch the benchmarks together to get a sort of general capability score for models that applies over the entire range from back in 2020 all the way to models that were released now. Even though any individual benchmark would have saturated over a much smaller period. Then you'd take the capability scores that this measure spits out. You plot them against the release time for the model. These capability scores, we never told when producing them, the statistical model that produces them. We don't include any information about the release date or any time based information into it, just benchmark performance. And nonetheless, when you start plotting this against release time, it's just a nice line. It's great. And you see that AI progress mostly looks pretty linear on this particular graph. The method is really very simple. The main thing that it depends on is that we have benchmarks that can capture AI system's performance. Yeah, exactly. And that I think will continue to happen in the future. So we can just continue to plot this. And then one hopes that if an intelligence explosion does seem like it's happening, we will start to see an acceleration on that rather than it looking just like a nice linear fit the entire time.

[Rob Wiblin]

Okay, so how do we tell if we've just made a better tool or we've made a whole lot of new researchers? You're saying the proof is in the pudding. Let's not leave it to the philosophers. Let's leave it to the people doing benchmarks to figure out whether AI advances are actually speeding up.

[Rohin Shah]

Empiricism.

[Rob Wiblin]

A lot of people have the perception, I guess, that AI progress has been slowing down. You're saying you've tried to create an enormous data set of as many different models over the last four or five years as possible. This is with Epoch, who are. This is their wheelhouse doing this kind of data collection and compilation and aggregation. So they've tried to collect all these different benchmark scores for many different models going back quite a long way to see is progress speeding up or is it slowing down or is it roughly linear? And you're saying, at least judged by that measure, the best effort they can do says that it's linear. Which is to say, I guess, that we're not making better people, we're making better tools. For now, that's what that would suggest.

[Rohin Shah]

Yep, that's right.

[Rob Wiblin]

So there's all kinds of problems with benchmark scores. One thing could be that they don't capture the full range of performance that either you end up like. Like capped, I guess, flawed at the bottom or capped at the top. Especially because we're talking about models here over many years doing many, many different things. There's definitely lots of gaming that goes on or lots of teaching to the test that occurs with people trying to make their models look good on these benchmarks. I guess it was also a question like what actually matters. There's benchmarks for all kinds of different skills and maybe you should give some of these things much more weight than others. I guess you could also have non linearities in the effect, the performance of a model and its economic effect that could be, indeed, probably is quite nonlinear. How good do you think this whole approach that Epoch and you have been using to figure out whether progress is speeding up or remaining about the same pace, Getting at the kind of ground reality of what's going on?

[Rohin Shah]

Yeah, I basically agree with all of the critiques you mentioned. It's going to inherit any problems that benchmarks have because ultimately the only input into it is benchmark performance. Nonetheless, I think a lesson that I learned from the gods of Straight lines is that for some things, yeah, there are lots of nuances and details that matter a bunch if you want to make very fine grained predictions, but at a high level they wash out and it ends up being fine. Anyway, I think that mostly applies here.

[Rob Wiblin]

I guess an example might be you'd say, well, the models are being gamed. There's a bunch of teaching to the test happening now, but there was a bunch of teaching to the test happening two years ago and four years ago and so as long as that's not getting progressively worse, then the line is still reasonable.

[Rohin Shah]

Yes. And also I think in practice the teaching to the test won't change the results very much on this. It definitely changes it some. So I think anthropic is probably the best at not overfitting to the benchmarks that exist. And in fact, if you look at the scores that are produced, I think it does tend to underestimate CLAUDE or anthropics models relative to models from other providers because generally CLAUDE models, since they are not overfit to the benchmarks, will tend to underperform on benchmarks relative to how good the model actually is. So yes, that's true. But also if you look at it on the graph, it's like a tiny little difference. It's not that big. If you moved it up, it would still look linear. It doesn't really change whether it's linear or not. Observation.

[Rob Wiblin]

I guess you're saying increasing your rates of progress would be quite striking on the graph. It probably would jump out at you and these effects wouldn't be enough to make it disappear.

[Rohin Shah]

That's right. But I definitely would caution people against using the score for really fine grained things like how good is Claude versus Gemini versus whatever. Or for example, trying to understand the difference between open source models and closed source models because I think the open source models are probably more overfit to the benchmarks than the closed source ones. And this probably is going to. If you try to use this score to look at the exact difference between those two, it's probably going to mislead you a little bit.

[Rob Wiblin]

So at the point that AI is people or at least is AI researchers, rather than just being a tool for AI researchers, you might reasonably expect quite abrupt increases in progress in AI R&D. And I guess like AI capabilities, basically. Are you a downvote on how abrupt that will be or whether that will occur at all? Or do you buy that maybe it will take a bit longer than people are imagining, but you do still think that that will happen?

[Rohin Shah]

I'm a slight down vote on the abruptness. I am not a down vote on will there be an intelligence explosion? I think so. What do I think? I feel pretty confident about? It will be some really strong statement with a lot of strong preconditions when. When you have AI systems such that for almost any economically valuable task you care about, you would prefer to hire the AI system rather than the human, which among other things implies that the AI is cheaper than the human, which I don't think is a given at that point. Assuming that we don't take some sort of action to try and prevent it. In fact, we are trying to use the AI systems to substantially accelerate research and development across the board, not just in AI, but all of the economy. Then probably within a century we will have some kind of intelligence explosion and reach technological maturity. That's like so many preconditions, so many. It's like a very of. It's still a kind of crazy statement. I'm saying we will have an intelligence explosion at all. And I feel actually pretty confident about that. But it is a lot weaker than the thing that I would guess will actually happen, which will be substantially faster and happen substantially earlier than what that statement would imply. So what I would actually say is a decent chance that it starts at the point where the AI systems can automate most of AI R&D as opposed to all arbitrary R and D. Decent chance that it finishes over five to 10 years rather than a century depends on exactly what you set as the starting point and so on. Going back to your original question about abruptness, the reason I'm a slight down vote on abruptness is that I would expect that actually the first AI systems, automated systems that can really automate AI R&D, will probably just be very expensive, to the point of potentially being more expensive than humans would be. You can see this over the last year, there's just been way more investment in inference time compute, inference time scaling. Google has a sort of deep Think algorithm which applies even more inference scaling to get even better results. I think you should basically expect this to continue. And so the picture of the first automated researcher might be something that's more like a relatively dumb system that's not as smart as a human researcher, but spending just tremendous amounts of time doing reasoning Exploring tons of dead ends, realizing they're dead ends, and then coming back and trying something else, which a human would never have gone down because they would have known in advance it would be a dead end. And that's how it does its automated research, such that it might even be more expensive than a human researcher would be. And then we improve it over time. The cost goes down pretty quickly, as is usually the case in AI. But that would suggest that it won't be that abrupt. The AI will reach cost parity with humans, then start becoming more cost effective than the humans, and that will start setting off the acceleration.

[Rob Wiblin]

But that's how it ends up happening in over quite a number of years rather than months or something crazy.

[Rohin Shah]

If you take it from the point at which the AI systems start being just barely, just on par with AI existing human AI researchers, then yes, that's right. That's why I would think it would take years rather than months. But most of my timeline's delay is just thinking that even getting to that point will take quite a while. Quite a while, like a decade maybe, which I feel like in past years would have been called extremely short timelines and nowadays gets called medium to long timelines.

[Rob Wiblin]

Yeah, I think there's been a general phenomenon I guess over the years. I guess every couple of years there's kind of a freak out about AI timelines and people start expecting a recursive self improvement loop really quite soon within a few years of that point. My impression is that you've just been unmoved in either direction. Why haven't you updated based on things that have, events that have occurred, results that have come out?

[Rohin Shah]

I would guess probably the biggest difference is that I had a picture in my mind about how AI progress would happen and reality has been reasonably close to it. For example, I think the biggest timelines freak out was the timelines freak out in January, let's say was from the advent of reasoning models 01 and particularly 03, where I would say the key idea there is let's actually apply reinforcement learning to large language models. I think this has actually been, I have since for quite a long time. I'd say probably at least 2019, but maybe even earlier than that thought, yes, of course we are going to need to use reinforcement learning in order to develop powerful AI systems. If you look at much of the work that was done at the time, things like debate, those are implicitly predicated on reinforcement learning being the method of choice for building powerful AI systems, debate just looks pretty pointless if you're not using reinforcement learning. And so I was always expecting reinforcement learning to happen. And from my perspective, everybody else was suddenly surprised by reinforcement learning happening, whereas I looked at it and I was like, well, it could have been the case that once we do reinforcement learning it just generalizes beautifully to everything, the same way that instruction following really does generalize beautifully to everything. And actually that was not the case. So I think mostly my timelines didn't change very much because I already thought it was reasonably likely RL wouldn't generalize to everything. But I think if I had been tracking it sufficiently fine grained, I would have probably updated slightly towards longer timelines on the release of O or O3.

[Rob Wiblin]

And why doesn't the general performance of reasoning models, I mean, I think that's one way of characterizing the update was people were surprised or they were shocked that RL was being applied to these models and to reasoning. I think most people would say that they were impressed by how good the reasoning was and how useful it seemed like it was going to be. But it sounds like you weren't impressed by it wasn't surprisingly good to you. Or maybe that's not it. It's that it wasn't as generalizable that they were good at the reasoning tasks that they had been RL'd on. But I didn't expect that that was going to generalize to other tasks and be very economically transformative. And indeed it has not.

[Rohin Shah]

Yeah, that's basically right. I think the generalizability was the biggest big deal for me. Look, if you want to target some particular benchmark and apply machine learning to it, I think the lesson of machine learning is yes, you can do it. People do choose the ones that models are capable of doing. So it's not like you can choose some arbitrary thing and just hit it with the machine learning hammer and succeed. But I do think you have to be pretty careful about if somebody optimized for a specific thing, how much should you update from that to AGI? The full generality, the fully general intelligence. It's really quite a difficult kind of update to make and you should be looking quite a bit at the generalizability.

[Rob Wiblin]

Okay, so that's the thing that would cause you to have a timelines freak out would be if you trained models on one kind of practical task and then you found that they were actually good and useful at doing quite different kinds of tasks.

[Rohin Shah]

Yeah, I think that's right. And you do see a little bit of this from reasoning models. To be clear, it's not like it generalizes not at all. But yeah, not as much as I think would have actually been a substantial update for me. I think in particular I was looking at autonomy tasks and nowadays I think the reasoning models are pretty good. Can be pretty good at autonomy tasks? Well, kind of. Well actually relative to expectations, I think they're still underperforming, but nowadays they're substantially better than they were at the time. But partly that's because companies have started to train for autonomy.

[Rob Wiblin]

Let's Talk about Google DeepMind, which for a very important organization I feel like isn't super well understood by people outside, including listeners and I guess I'd say outsiders, just in general. A lot of people outside of AI companies produce research, I guess like both governance research and technical research, hoping that it will be read and absorbed and adopted and used by those companies, including gdm. What can people do to make it more likely that anything that they do actually does have any, like it's read at all or is used at all by people inside an AI company?

[Rohin Shah]

Yeah, there are definitely a few things that people can do and I think again at this point it is useful to bring in the model of like there are just all these incredible constraints that interact with each other a ton and as a result we can only do a few things and need to do a decent amount of work in order to get them through, which you could well predict as using the meme of companies are apathetic. Though I think it's maybe a little bit better to say it as companies are well motivated but can only do a few things. And so keeping that in mind, a few things are worth doing. One is just talk to somebody at a company before you put in a bunch of time into research. It's like reach out to somebody who works at a company, say this is what I'm planning to do. Do you think this will actually be useful? It's a very simple step, but surprisingly many people just don't do it. But it does seem like probably the best thing you can do in order to actually increase the chance that your work gets used at a company. At least currently.

[Rob Wiblin]

If people consistently did that, do you think it'd be like a lot of your time would then be eaten up replying to these emails?

[Rohin Shah]

If they email me, they will probably not get very much of a reply because I get way too many emails and I think most of my reports I think would be fairly excited to talk to people about this. I admit that I at this point get enough of these that it feels a little bit More like a burden than an exciting thing. But it definitely used to feel like an exciting thing before it became very common.

[Rob Wiblin]

Okay. Yeah. What should people do other than email to ask whether what they're doing is going to be useful?

[Rohin Shah]

Yeah, so other things, I think, I guess I had a talk recently on how to theorize, so empiricists will listen. It could equally well have been called how to do safety research so that companies will listen. The basic points from this, I think the first one, the most obvious one, but it's still worth asking yourself, is, do they actually care? Sometimes people do research on problems that we actually just don't care about and don't think matter. I think the safety community is like, fairly good about not doing this, but it does still happen, and sometimes people might be a little bit surprised by what we do and don't care about. So, for example, take jailbreaks. We care a lot about jailbreaks now because misusement, misuse. The models are looking to be strong enough that misuse could be a serious problem. But if you ask, if you look, a year or two ago, people were doing a bunch of jailbreak research and saying, look, alignment. This means that the companies aren't very good at aligning their models because they're so susceptible to jailbreaks. And I don't know about other companies, but at least at gdm, basically our stance on jailbreaks was the models are not. There's not any actual misuse scenarios that we're particularly worried about, given the model capabilities. What safety is about is about protecting users from cases where the model does something unintended and harmful. And so for a while, there was just a bunch of discourse about how models are always going to be jailbreakable, totally impossible to defend against it. And the real answer was just like, we hadn't even tried to stop the jailbreaks.

[Rob Wiblin]

Okay, but you're trying to stop it now.

[Rohin Shah]

We are trying to stop it now, yes. Now, if people want to give us research on jailbreaks and how to defend against them, we will definitely use it.

[Rob Wiblin]

I guess I should say all of this advice is predicated on the idea that you're doing research hoping that an AI company is going to read it and absorb it and use it. There's people who will have their own different ideas and they'll be developing stuff, hoping that people will become persuaded later on that it's useful or it'll be valuable in a different way.

[Rohin Shah]

Absolutely, yeah. Lots of different theories of change for research. I'm definitely only talking about if you want a company to use it basically now, anyway, so that was step one, do they actually care? Step two is, I call it are you helping? But it's a very specific flavor of helping. Specifically, I think you should either be proposing a solution or building an evaluation or a metric of some kind. Now, there is research that is not these things. There is research that does some sort of theory to understand some phenomenon better that will give you some increased understanding of the phenomenon, but doesn't solve a particular problem, doesn't produce metric or eval that the company should care about. There are lots of other examples of research like this. I think basically most of that research we are unlikely to use unless it happens to really be on a core problem that we care about a lot, given that we are all busy and it takes a lot of work to incorporate any new thing. Usually it needs to be a metric or an eval or some kind of solution.

[Rob Wiblin]

So people are doing interesting theorizing or preliminary empirical work to understand a phenomenon better. You'll be like, that's all well and good. Come back when you have a solution.

[Rohin Shah]

Roughly, yes. And again, I think this is good research to do and people should do it. It can help develop better solutions in the future. I just don't think they should be thinking that this is like spend a

[Rob Wiblin]

lot of time reading it.

[Rohin Shah]

Exactly. Yeah. All right. Next thing after that, especially if you're going to be proposing a solution, the next thing to be doing is making sure that the solution, like evaluating your solution or at least conceptually evaluating your solution, even if it's not by experiments on the various metrics that companies care a lot about. Most of the time research will look at, did it actually solve the problem that it set out to solve? Definitely important. You got to do that evaluation. That is the most important one. But then there's also things like how much cost does this add in terms of compute? How much latency will it add? If you're imagining a step that runs after the AI system has produced a response, and then you do lots of additional things before you then have to send the response to the user? Probably a non starter, not obviously, but it's a big cost. There's implementation complexity or organizational complexity. If your solution involves doing something at the scaffold level and also doing something that involves reading the internals of the AI system and connecting these up together, that just spans so much power, so many different teams, and so many different abstraction layers in the stack that it's going to be much, much harder to implement. Than something that, for example, just happens at inference time and involves running a monitor and then alerting some teams about it.

[Rob Wiblin]

Yeah. Are there any other examples of things that are easy to implement other than that one?

[Rohin Shah]

Yeah, I think there are several. For example, if their data sets can be a useful contribution, it's pretty easy to add a. Well, it's pretty easy to try to add a data set to post training. Sometimes you try to add it to post training and then for some reason it makes the model worse on some totally unrelated thing and then you can't use that data set, which is a bit unfortunate. So ideally when you're building datasets, you would like to evaluate whether it could have some sort of negative effect on something else. But this is a bit hard to do. But if someone came to me with a data set and said, look, training on this data set improves this metric by a decent amount and doesn't seem to have any bad effects on these three obvious other metrics that it might have had bad effects on, I'd find that fairly compelling and think, yeah, maybe we should do it, assuming it was solving a real problem. So I think data sets, metrics, evals, these are usually relatively easy things to do. Monitors, I think all of those are fairly easy to implement if we think it's worth implementing.

[Rob Wiblin]

Yeah. Any other advice?

[Rohin Shah]

Oh, one is just like don't do the academic thing of using a fancy method to solve your problem and instead solve it with the absolute simplest method you can.

[Rob Wiblin]

Basically saying cutting edge stuff might be useful in some other way, that you advance the science somehow and could be valuable down the line. But if you want people to be using it on any actually commercial models anytime soon, then it has to be as simple as possible and well established as possible.

[Rohin Shah]

That's the generous version. Yeah. The cynical version is more like. Like academia rewards complex formal looking stuff and poorly tuned baselines. It won't reward poorly tuned baselines if it knows they're poorly tuned. But it's kind of hard to tell if a baseline has been poorly tuned or not.

[Rob Wiblin]

What's a poorly tuned baseline?

[Rohin Shah]

Sorry? A poorly tuned baseline is like you have some default way of solving the problem that you are trying to do better than.

[Rob Wiblin]

Do you make the baseline look bad by not doing a very good job of it?

[Rohin Shah]

That's right. But not intentionally. It will often be like you implement the baseline once and then you don't tune the hyperparameters for it and so it performs less well than it really should. It's just very easy to not Spend enough time working with your baseline to make it work well and then as a result it makes your other thing look better. Exactly. And given the incentives in academia of publishing novel things and publishing a lot, I think this is a fairly common problem. So I think if you want to publish, that tends to be the thing that you do. If you want your research to be used by people at companies, it's quite important that you try the obvious stuff and you try fairly hard with the obvious stuff. And only if that really does fail do you try to do something fancier.

[Rob Wiblin]

What external research has been most useful to GDM so far?

[Rohin Shah]

Yeah, I think probably the one I would point to most is the AI control work from Redwood Research. I think we were always planning to monitor our AI systems. It's not like the idea of monitoring was new to us. But the specific conceptual frameworks they brought to how you might evaluate how well this works, specifically the distinction between trusted and untrusted models. I think that was quite good. The first paper they published on it that showed how a variety of different control protocols, how you can evaluate their safety and their usefulness and use this to decide which one you should be doing. I think that that's influenced me quite a lot on how exactly I think about the control work. I think that's probably the most obvious example.

[Rob Wiblin]

So this is the Buck Schlegeras and Ryan Greenblatt episodes earlier in the year, I guess Buck Schlegris in particular, talking about the AI control agenda. I feel like that episode didn't make quite as much of a splash as I was hoping or expecting at the time. But the amount that people in the industry keep constantly referring back to it, I think means that if. If people didn't listen to it at the time because for whatever reason it just didn't sound quite exciting enough, I think that give it another look, maybe go back and have a listen because I think it is very important kind of foundational work that is only getting more relevant.

[Rohin Shah]

Yeah, definitely. I would say that control is like in the medium term, I think probably how are we going to argue that Gemini is safe at least for misalignment and internal deployments? I think probably via a control style arguments. So yeah, definitely an important area to be aware of. Other papers. So there was this paper on observational scaling laws which mostly I think you can think of it as a precursor to the Rosetta Stone paper that we talked about before where we stitch together benchmarks to get an overall. To get a way to measure overall AI progress. I think the observational scaling laws thought of this more as like, how can you forecast particular benchmark results. That's also a pretty good application. I think mostly it was an area we hadn't really been thinking about and they showed that, yeah, you can do something here. And I think that probably influenced me when thinking about the sort of benchmark stitching Rosetta Stone approach. So I think that one was pretty influential on me. Other examples would be the various chain of thought monitoring papers that have come out. I don't think that I can point to any one paper that I thought was a particularly big deal. But in aggregate, I think they've built a fairly good research literature on the effectiveness of chain of thought monitoring when it does and doesn't fail. So I think there were some papers that looked at to what extent can you just persuade the monitor that actually everything's fine here, you don't need to flag anything in this chain of thought monitoring. I think that was a useful thing to study and I think the specific quantitative results in it were informative to me.

[Rob Wiblin]

So I imagine a lot of people would love to get a job at gdm and even people at other AI companies already might be interested in considering switching given the advances that GDM has made in its models in recent years. What sorts of skills or role or what sort of roles are you finding hardest to fill? And I guess what sort of skills are maybe hardest to find out in the labor market?

[Rohin Shah]

Yeah. I think I'll go back again to the theme I've had throughout of the challenge being more in implementing stuff rather than in figuring out what stuff we need to do. We have a lot of ideas, we know what we need to do. Implementing it ends up being harder because there's so much stuff you have to check and make sure that you're not hurting it as a result, I think especially over the last year, maybe two years, I think we've had more of a need for people who just want to do the obvious thing and land it, as opposed to people who want to figure out the ideal optimal thing, write a cool research paper about it.

[Rob Wiblin]

We're talking about the AGI Safety and alignment folks, right? Not yet.

[Rohin Shah]

That's right. Yes. I am talking primarily about the AGI Safety and alignment team. I think a focus on just doing the obvious stuff, a focus on implementation rather than research. I think in practice this also means more of a focus on software engineering relative to machine learning research. I do still think that we do care about conceptual ability, research taste, ML engineering skills. They do come up when you're doing this sort of implementation, you need to test how good your system is. That means you need to build good evaluations, you need to not overfit to them. You need to be a little bit careful about that. So it's not like those skills are irrelevant. I just think that there's more of a focus on things like getting things done, software engineering implementation now relative to even just a year ago, but especially relative to two years ago.

[Rob Wiblin]

Are there any particular roles that you're hiring for at the moment or hiring for a lot in general? Because it's felt to me like Anthropic is hiring hand over Fist and OpenAI. There's always a lot of roles there. Is it similar situation for you?

[Rohin Shah]

No, I think we are hiring quite a bit less than Anthropic and OpenAI at the moment. We do have a couple of roles open right now. Partly we have roles open in other teams that I think are also very relevant. Like the AGI Safety and Alignment team isn't the only team that's relevant to AGI safety at Google DeepMind. So, for example, currently there is a hiring round open on the security team to hire engineers to work on AI control. And I think that's a great position. Hugely impactful. As I've said before, that's probably the argument that we're going to make for why Gemini wouldn't cause harm via loss of control in the medium term. So I think that's an extremely useful role. Very impactful. Not on the AGI Safety and Alignment team, but they would be working closely with us, I think. On my team we're hiring at the moment for an engineer for Frontier Safety Risk Assessment. So this is like running our dangerous capability evaluations and writing the Frontier Safety Report. So, you know, if I don't particularly think that the Frontier Safety Report needs to be that much more detailed, but if you do, you could come join us and then put in the time to make it better. There is a fair amount that individual people on the team can have the flexibility to go and do stuff like that in practice. I expect by the time that this recording actually comes out that that role might not be there. But I think we do have these roles coming out on an ongoing basis and maybe I'll send over a link to an expression of interest form that people can fill out so that we can email them when new roles open up. I think part of it is we've, like many others, have been hit by the deluge of AI assisted applications. And so we are a Little bit less likely to in the future make these open calls for hiring because it really is just such a pain to go and do the resume screening for all of them.

[Rob Wiblin]

Yeah, I think folks are going to have to introduce, I guess, a fee to fill out forms like that or to apply for jobs, which people also hate for other reasons. But I don't really see the alternative now that it's possible for AIs to submit completely indistinguishable applications that can be as fake as you like.

[Rohin Shah]

Yeah, it's rough. Last year we included an LLM captcha, but for a recent. For a recent one, we were trying to do it and I think we probably could. But it would also trick a bunch of humans. Even our LLM captcha from last year tricked a bunch of humans. Not a bunch, like maybe 5% of them. this point, I'm not sure that I can design one that wouldn't have quite a lot of false positives.

[Rob Wiblin]

Well, I guess book a flight for me would be. But that might be more expensive than just having the fee. Yeah. Is there any pitch you want to make for working at GDM in particular?

[Rohin Shah]

So basically my pitch is company safety teams are probably the biggest force for what companies are, what actually happens on a technical level to make AI systems safe. And that is a good reason to join them.

[Rob Wiblin]

All right, we should wrap up, I think. For final question, I'll throw you a real hardball that came in from the audience. How do you maintain such a nice and positive spirit in these troubled times, Rohin?

[Rohin Shah]

Yeah, I mean, some of the answer is a bit boring, not that generalizable, which is one, just by personality, I'm just fairly stable and my mood doesn't change very much day to day. And then two, as has maybe become clear over the course of this episode, I don't think the times are as troubled as everybody else thinks relative to everyone else. But I do think there is, I guess, one thing that I do quite a lot and quite religiously is focus on the things that I can control. And I think this is quite. This isn't why I do it, but I think it is quite useful for maintaining a nice and positive spirit during these troubled times. Let's say it definitely helps keeps me focused on the areas where I do have agency. And I think it's just pretty great to be focused on areas where you can actually make a difference.

[Rob Wiblin]

My guest today has been Rohin Shah. Thanks so much for coming on the 80,000 Hours podcast, Rohin.

[Rohin Shah]

Thanks a lot, Rob. It was great to be here.

[Rob Wiblin]

Hey everyone, Rob here with some personal news at least if 80,000 hours counts as a person. We have a new book out. It's called 80,000 hours how to have a Fulfilling Career that Does Good and Penguin just published it around the world. A lot of listeners don't actually know this, but 80,000 hours has been running since 2012 trying to help people have a much larger social impact with their career while doing something that also delivers them, hopefully a wonderful and well rounded life. And this is the single most developed expression of the many ideas that we've been coming up with and honing since then. Written up by our founder Benjamin Todd, it covers all kinds of angles on the career and social impact topics, including the evidence on what makes for an ideal job, how much good an individual can actually do, and how we could know that how to correctly choose which global problem to focus on solving which specific skills enable you to help the most people or help people the most how much income you need to be happy, how to become so valuable they can't ignore you how to balance the risks of going with your gut instinct against the risks of ignoring your gut instinct. When to aim higher in your career versus when to decide to settle instead how to find a career that fits your idiosyncratic strengths and weaknesses in particular, and how to best break into a field and actually get the job that you want once you've decided on a direction. Our latest additions naturally have a particular focus on AI, which is upending lots of people's career plans and forcing a lot of people to rethink what they're expecting to do. If you're worried about AI yourself, the chapter covering which skills will be most valuable in the future goes through a lot of counterintuitive ideas which I think might justify picking up the book just by themselves. I'll list six of them briefly here. First, Ben argues that even if you think AI is going to take all of the jobs from human beings before long, you might also reasonably expect salaries to actually explode in the short run. One well respected economic model that he goes through suggests that average wages could rise tenfold before the party ends, creating what might be the best brief window ever to be a worker. Second, Ben explains why retraining as a plumber or other physical laborer a piece of advice that you hear more and more in reaction to AI, and which sounds sort of common sensing, including to me. He explains why he thinks that's probably counterproductive for most listeners to this show anyway. Third, Ben explains why it's not sufficient to just have a job that's very hard to automate with AI, you need to find a job that's highly complementary with AI outputs as well. Fourth, the book investigates why the jobs most exposed to the current wave of AI are people earning 100 to $200,000 a year, which is a pretty significant reversal of previous automation trends. Fifth, he goes into why Geoffrey Hinton's very famous 2016 prediction that hospitals should stop training radiologists turned out to be completely wrong, and nine years later radiology employment is hitting record levels. While on the other hand, another hugely influential prediction from 2013 that creative work would resist automation, that's also turned out to be totally wrong, but in the reverse direction. And sixth, why induced demand asymmetries suggest that people providing things that people want in almost unlimited amounts, like better health, they're going to do well, while people delivering things that we want in kind of more of a fixed amount, like tax compliance, they're going to struggle, no matter how productive AI potentially makes them. Of course, nobody actually knows what skills are going to be more valuable in five years or most valuable in five years. It would be crazy to think that you did. But the chapter gives you a lot of tools to avoid making mistakes that would look really predictable in retrospect. And it also gives you an understanding of the underlying economic effects and trends such that you can make sense of shifts as they start to happen and try to take advantage of them rather than just get steamrolled. And Ben doesn't. He doesn't chicken out from having actual concrete conclusions as well. He has a specific list of fields he thinks you shouldn't train in without serious thought. Okay, that's a taste of what you get just from chapter 8 of a 15 chapter book. As I said, this is the wisdom we've been gradually compiling for the last 14 years, so it's pretty information dense. If that sounds good to you, you can order the book online by searching for 80,000 hours book. I'll go to 80,000 hours.org book. It's obviously on Amazon, Audible, wherever you get books where you are. Benjamin actually narrated the audio version himself. It's pretty extensive, so better him than me, I think. And if you already know our advice inside and out, this is an ideal time to tell a friend about it. Any sales this week will help us hit the bestseller lists, which are apparently super important.