Dwarkesh Podcast: Ilya Sutskever – From the Age of Scaling to the Age of Research

Date: November 25, 2025
Host: Dwarkesh Patel
Guest: Ilya Sutskever (Co-founder, SSI and AI research pioneer)

Episode Overview

This episode of the Dwarkesh Podcast features a wide-ranging, deeply technical, and philosophical conversation with Ilya Sutskever, one of the most influential figures in AI. Together, they explore the transition of artificial intelligence from the "age of scaling"—where progress was largely about making models bigger and training them with immense amounts of data—to the new "age of research," where breakthroughs will increasingly depend on fresh ideas, understanding generalization, and novel training recipes. Ilya shares candid reflections on how models work, where the field is at an impasse, and what may be required to achieve safe, robust superintelligent systems.

Key Topics & Discussion Points

1. The Surreal Normalcy of AI Progress

[00:00–01:30]

• Normalization of AI breakthroughs: Both note how the rapid advancement of AI technology can feel strangely ordinary, despite being straight out of science fiction.

  • "Another thing that's crazy is, like, how normal the slow takeoff feels. The idea that we'd be investing 1% of GDP in AI…" (B, 00:13)

• Perceived impact: While headlines announce huge investments, most people do not tangibly feel the impact of AI in daily life—yet.

2. Model Capabilities vs. Real-World Impact

[01:30–05:00]

• Disconnect between model benchmark success and economic value:

  • "How to reconcile the fact that they are doing so well on evals ... but the economic impact seems to be dramatically behind?" (A, 01:38)

• Overfocusing on benchmarks: Researchers may unintentionally "reward hack" by tuning models too closely to evaluation metrics instead of real-world robustness.

  • "The real reward hacking is the human researchers who are too focused on the evals." (B, 05:00)

• Possible explanations: The current training regimes, especially RL (reinforcement learning), may narrow model focus, diminishing generalization.

3. Analogy: Model Training vs. Human Learning

[06:08–09:39]

• Two types of learners:

  • The "grinder" who over-trains on narrow tasks vs. the naturally talented learner who generalizes widely.
  • "The models are much more like the first student … if it's so well trained... it will not necessarily generalize to other things." (A, 06:56)

• Pretraining as a supercharged form of grinding: The vastness of pretraining can result in huge knowledge acquisition, but not necessarily deeper understanding.

• Lack of a clean human analogy:

  • "I don't think there is a human analog to pre training." (A, 08:31)

4. Human Learning: Pretraining, Evolution, and Emotion

[09:39–17:15]

• Pretraining analogies: Is it like childhood or evolution?
• Depth vs. breadth: Humans assimilate less data more deeply; already as a child, one avoids model-like blunders.
• Emotion as value function: Emotion may act as a built-in value function guiding decision making.
  • "Maybe what it suggests is that the value function of humans is modulated by emotions in some important way that's hard coded by evolution…" (A, 16:12)

5. Defining Value Functions in ML and the Brain

[13:36–17:15]

• Reinforcement learning and value functions defined:
  • "The value function lets you short circuit the wait until the very end..." (A, 13:39)
• Speculative future: Value functions will be critical for more efficient RL.

6. The Era of Scaling and Its Limits

[19:00–24:10]

• "Scaling" as paradigm:

  • "Scaling is just one word, but it's such a powerful word because it informs people what to do." (A, 19:00)

• Recipe for progress: Scaling pretraining (more data, compute, parameters) has worked, but is finite.

• Transitioning to research: As scale hits limits, the frontier returns to searching for new ideas—"the age of research" is back.

  • "So it's back to the age of research again, just with big computers." (A, 21:50)

7. The Crux: Generalization & Sample Efficiency

[24:35–32:27]

• Biggest challenge: Models generalize much worse than humans, despite scale.

• Human sample efficiency:

  • Evolution gives a rich evolutionary prior for sensory-motor tasks, but not for abstract domains like language/maths—in these, humans still far outperform models in robust, flexible learning.

• The unsolved principle:

  • "There may be another blocker though, which is there is a possibility that the human neurons actually do more compute than we think..." (A, 31:27)
  • "There is a principle … unfortunately circumstances make it hard to discuss." (A, 31:27)

8. The Return to Research and Community Culture

[36:38–42:02]

• Scaling ‘sucked all the air out of the room’: Everyone focused on the same paradigm.
• Bottlenecks shifting: As compute is no longer the rate-limiting factor for research, ideas become critical again.
  • "If ideas are so cheap, how come no one's having any ideas?" (A, quoting Twitter, 36:38)
• Mid-to-large scale is sufficient to explore new concepts—vast compute is only needed to be competitive at the absolute frontier.

9. SSI’s Approach, Funding, and “Straight Shot” Plan

[42:44–50:43]

• SSI is research-focused: Sutskever explains that most frontier labs’ massive funding is earmarked for inference at scale, not pure R&D. SSI’s comparatively modest budget suffices for innovative research.

• Straight shot to superintelligence:

  • Pros: Avoid market distractions to focus squarely on the core challenge.
  • Cons: Public exposure and gradual introduction of advanced AIs is valuable for global preparation and safety.
  • "The counterpoint is, hey, it is useful for the world to see powerful AI because that's the only way you can communicate it." (A, 44:55)

• Conceptual legacy: Terms like “AGI” and “pretraining” have shaped expectations (for better or worse), sometimes overshooting the true target compared to continual learning in humans.

  • "A human being is not an AGI … Instead, we rely on continual learning." (A, 47:08)

10. Continual Learning, Deployment, and Intelligence Explosion

[50:43–56:07]

• Superintelligent learning agents: Envisions AIs that can learn any job, then amalgamate all learned skills across instances, potentially catalyzing an “intelligence explosion.”

  • "You basically have a model which functionally becomes superintelligent even without any sort of recursive self-improvement in software..." (B, 51:51)

• Economic and governance implications: Rapid economic growth and regulatory gaps are anticipated—details of the rollout remain unpredictable.

11. Alignment and Safety: Sentient Life and Robust Generalization

[56:07–67:56]

• Sutskever’s evolving alignment views:

  • Now favors incremental and visible deployment, so society can better grasp the impact and implications.

• Long-term equilibrium:

  • "Care for sentient life" (not just humans) could be a unifying alignment goal for superintelligent systems.
  • "It will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient." (A, 58:10)
  • Recognizes the risks: if most sentient beings are machines, human priorities may be overwhelmed.

• Power capping: Proposes limiting the absolute power of the most advanced AIs to avoid runaway risks.

12. The Mystery of Social Desires and Evolutionary Alignment

[70:44–76:53]

• Brain desires vs. genome: It's mysterious how evolution hardcodes high-level, abstract desires (like social standing) in the brain.

  • "It's harder to imagine the genome saying you should care about some complicated computation that your entire brain does." (A, 74:08)

• *Speculations about brain region mapping are likely wrong—this remains a major open question in neuroscience and AI.

13. Diversity and Parallelism in AI

[87:16–92:40]

• Will one company take it all?
  • Sutskever’s intuition is that while it’s theoretically possible, practical specialization and market forces will drive diversity among both companies and AI agents.
• Model diversity:
  • Lack of agent diversity is a byproduct of similar pretraining; RL and post-training offer opportunities for differentiated agent behaviors.

14. Research Taste and Inspiration

[92:40–95:29]

• Top-down aesthetic and brain-inspired guidance:
  • Sutskever describes his “research taste” as being guided by beauty, simplicity, and elegance, often derived from understanding how the brain works.
  • "If you just trust the data all the time... sometimes you can be doing a correct thing. But there's a bug. But you don't know that there is a bug. How can you tell...? It's the top down." (A, 94:39)

Notable Quotes

• On normalization of AI breakthroughs:
  "We get used to things pretty fast. Turns out." (A, 00:27)

• On overfitting to benchmark evaluations:
  "The real reward hacking is the human researchers who are too focused on the evals." (B, 05:00)

• On why humans generalize efficiently:
  "I think it's the it factor." (A, 07:48)

• On the value function:
  "The value function lets you short circuit the wait until the very end... As soon as you conclude this, you could already get a reward signal a thousand time steps previously..." (A, 13:39)

• On scaling and research: "From 2012 to 2020, it was the age of research. Now from 2020 to 2025 it was the age of scaling... But now the scale is so big ... it's back to the age of research again, just with big computers." (A, 21:50)

• On the bottleneck in AI: "If ideas are so cheap, how come no one's having any ideas?" (A quoting Twitter, 36:38)

• On the hardest open problem: "These models somehow just generalize dramatically worse than people. And it's super obvious that seems like a very fundamental thing." (A, 24:35)

• On continual learning as a paradigm:
  "A human being, yes, there is definitely a foundation of skills. A human being lacks a huge amount of knowledge. Instead, we rely on continual learning." (A, 47:08)

• On aligning future AIs:
  "...there is merit to an AI that's robustly aligned to care about sentient life specifically." (A, 58:10)

Key Timestamps

• 00:00–01:30: The surreal and normalized pace of AI progress
• 01:30–05:00: Disconnect between model benchmarks and economic impact
• 06:08–09:39: Training analogy: human learners vs. machines
• 13:36–17:15: The role of values, emotion, and value functions in learning
• 19:00–21:50: "Scaling" as recipe; approaching natural limits
• 24:35–31:27: The crux: poor model generalization and sample inefficiency vs. human learning
• 36:38–42:02: Research culture: bottlenecks, innovation, and compute
• 42:44–47:08: SSI strategies, fundraising context, and the “straight shot” plan
• 50:43–56:07: Continual learning agents and intelligence/societal transformation
• 56:07–67:56: Alignment, incremental deployment, and societal effects
• 70:44–76:53: Evolutionary alignment of desires—open mysteries
• 87:16–92:40: Diversity in AIs: market, specialization, and agent variation
• 92:40–95:29: Research taste; beauty, simplicity, and brain inspiration

Conclusion

This episode offers a rare, candid, and wide-angle look at the current and future state of AI research. Ilya Sutskever grapples at length with the core remaining challenge—generalization—while envisioning near-future learning agents that could transform the world in unpredictable ways. He advocates for continual learning as a more human-inspired paradigm, calls for incremental deployment for safety, and sees alignment as a problem with profound ethical stakes. SSI's distinct approach is rooted in a search for new principles behind robust generalization and AI values, while remaining pragmatic about the limits and risks that lie ahead.

Podcast link and further reading at www.dwarkesh.com