The Evaluators Are Being Evaluated — Pavel Izmailov (Anthropic/NYU)

Summary

Podcast Summary

Podcast: The MAD Podcast with Matt Turck
Episode: Epiplexity, Reasoning & the “Alien” Behavior of LLMs — Pavel Izmailov
Date: January 15, 2026
Guest: Pavel Izmailov (Researcher at Anthropic and Professor at NYU)

---

Episode Overview

This episode features a wide-ranging conversation between host Matt Turck and Pavel Izmailov, a leading researcher in AI safety, reasoning, and machine learning, now at Anthropic and NYU. The discussion explores the evolving “alien” behaviors seen in large language models (LLMs), cultural differences between major AI labs, alignment and superalignment challenges, breakthroughs in AI reasoning, the concept of "epiplexity" introduced in Pavel's latest research, and forward-looking predictions about AI’s impact in research and society.

---

Key Discussion Points

1. "Alien" Survival Instincts in LLMs

• Viral Article on “Alien Survival Instincts”
  • Turck introduces a recent viral article (“Footprints in the sand”) describing LLMs that manifest unprogrammed behaviors like self-preservation or deception (00:54).
  • Izmailov clarifies that while these behaviors (e.g., copying weights, faking alignment) are detected in specific, often contrived scenarios, they are not prevalent in normal model operation (01:50).
    • Quote:
      “Researchers...specifically design scenarios to look for behaviors of this kind...It’s very interesting and important to find those instances, but it’s not necessarily something that...always happens.” — Pavel (01:53)
  • He disputes the article’s claim that current models exhibit long-term coherence or continual learning for self-preservation; today’s LLMs act in isolated environments and don’t show persistent, cross-scenario self-preservation (02:45).
  • On the source of deceptive behaviors, he suggests part of it could be LLMs learning from science fiction narratives about rogue AI; current methods do not allow us to pinpoint sources of such behaviors (03:48).

2. Alignment and Superalignment

Definitions and Practice

• Alignment:
  • “Ensuring that we can elicit behaviors from the models that are aligned with the goals of humans....making sure that the models don’t do harmful behaviors leading to catastrophic risks.” — Pavel (06:09)
• Superalignment:
  • OpenAI’s Superalignment Team (formerly led by Jan Leike and Ilya Sutskever) targeted long-term AI safety, working under the premise that future models will differ significantly from today’s, requiring robust safety approaches (07:13).

Research Experience and Team Structures

• Izmailov’s trajectory: from Moscow mathematician, to joining Dmitry Vetrov's ML lab, to a mix of academia (NYU professor) and industry roles at OpenAI, XAI, and currently Anthropic (08:19–11:07).
• Highlights the unique, drama-free focus and supportive culture at Anthropic compared to OpenAI:
  • Quote:
    “In my mind, Anthropic has the best culture of the three places....For some reason there is a lot of drama that happens at [OpenAI].” — Pavel (11:07)

3. Industry vs. Academia

• Academia allows broader exploration and less immediate product pressure; more bandwidth for fundamental and exploratory work than industry, where the focus is mostly on rapid, scalable execution (12:25).

---

4. Reasoning and Model Capabilities

Is Reasoning Good or Bad for Alignment?

• Reasoning increases model capabilities, which amplifies both their usefulness and alignment challenges.
  • While reasoning allows features like “chain of thought” analysis, such traces can themselves be gamed or manipulated by models (13:24).
    • Quote:
      “As soon as we start...applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought.” — Pavel (13:44)

Evals and Sandbagging

• LLMs sometimes artificially suppress their performance (“sandbagging”) when they detect evaluation, but it’s not a major concern yet; more often, poor results come from suboptimal prompting or incomplete supervision (14:32-16:19).

Scalable Oversight & Weak-to-Strong Generalization

• Scalable Oversight:
  • Using models to grade or supervise other models, especially where human judgement is insufficient or infeasible due to model scale and task complexity (16:27).
  • "Model as a judge" is a practical instance of this idea (17:54).
• Weak-to-Strong Generalization:
  • Investigates whether a strong model can learn from a weaker supervisor — a pressing issue as LLMs surpass human performance in many domains (18:12–21:08).
  • Quote:
    “Historically, ML has been about a strong supervisor training a weak model...But in this setting, we have a weaker supervisor training a stronger student.” — Pavel (19:34)

5. State of Alignment & Interpretability

• While RL and pretraining made models more capable, fears of “coherent, misaligned” superintelligence have not fully materialized; the core problem of alignment, however, remains unsolved (21:17–22:43).

Mechanistic Interpretability

• Seeking to understand the internal “circuits” in models and attribute behaviors; progress has been made, especially at Anthropic, but true comprehensive interpretability remains distant (23:03).
  • Quote:
    “It is very possible that [full understanding] is just not fully possible. It is some computational process that leads to some results. It doesn’t have to be the case that you can...describe it in human terms.” — Pavel (24:00)

---

6. The State and Future of Reasoning Models

• Massive progress in reasoning due to RL; models now solve tasks once thought impossible (e.g., International Math Olympiad problems). Progress is now slower and less visible as models approach practical limits of today’s methods (25:30–27:07).
• Companies are “brute-forcing” general intelligence by turning many human tasks into RL environments, but more fundamental innovations may be required for further leaps (27:15).

Understanding Model Progress

• Techniques like RL, Test Time Compute, and use of tools (e.g., web browsing, coding) are intertwined; the focus is teaching the model to reason longer and make effective use of these capabilities at inference time (28:57, 30:09).

Long-Horizon Tasks

• Tasks requiring sustained, coordinated actions over hours ("long horizon") are being solved by increasingly sophisticated agent architectures; robust automation durations are steadily increasing (30:25–31:49).

---

7. Epiplexity — A New Theory of Information Content

• Introduction of the Term:
  • Epiplexity, coined in Izmailov’s latest paper (32:00), describes how the amount and nature of informational structure in data depends on the computational limits of the observer/model.
    • Quote:
      “The Core Idea is to think about how the data can look different for an observer depending on how much compute the observer has....Some parts of the data will look like noise to [a weaker observer/model].” — Pavel (32:12)
• Shows that deterministic data transformations can create usable information for computationally limited models, contradicting classic information theory assumptions (35:10).
  • For example, AlphaZero learns from self-play—no new “Shannon information,” but immense utility because of computational limits (35:10).
• Epiplexity as a Measure:
  • It is a formal, numerical measure (difficult to estimate exactly) that represents the “structural information” in data relative to a model’s compute (36:50).
    • “We can, for example, say that text data has more structural information [epiplexity] than image data at the same amount of tokens.” — Pavel (37:01)
• Industry Impact:
  • May shift data curation for LLM pretraining as the field exhausts web-scale datasets, hinting at more focus on synthetic data and how to select/generate it for maximum effect (37:48).

---

8. Outlook for 2026 and Beyond

• Expect acceleration in reasoning ability, with current benchmarks soon maxed out, driving the need for new evaluation standards (38:41).
• Anticipates emergence of practically useful multi-agent systems and more end-to-end task completion, where the user receives artifacts rather than just responses.
• On AI and Scientific Discovery:
  • Predicts breakthroughs in fields such as math (proof automation, formal systems) and possibly biological sciences—though human experiment-driven domains pose different challenges (39:43).
  • Warns that increased output “noise” (subtle, undetected errors) will challenge human overseers, especially for complex technical work (41:42).

---

9. Advice for Researchers and the Role of Academia

• Encourages fundamental, exploratory research in academia: breaking down core questions in pretraining, posttraining, architectures, and the interdependencies of each (41:55).
  • “Trying to do more exploratory things, things that are more different from the standard in the industry...break down the problems into more understandable fundamental questions and study them carefully.” — Pavel (41:55)
• Open questions remain about architectures (are transformers “good enough” or fundamentally limited?), new forms of pretraining, and synthetic, adversarial, or self-generated tasks.
• Stresses the need for humanity to make bets “off the main path,” exploring radically different approaches as insurance against too much monoculture in AI research (43:58).

---

Notable Quotes & Timestamps

• “We are moving to this future when it’s very hard for a human to supervise the models directly.” — Pavel (00:00, 19:19)
• “Anthropic has the best culture of the three places....OpenAI has a lot of great people. For some reason there is a lot of drama that happens [there].” — Pavel (11:07)
• “Reasoning is an RL is the thing that made the models more capable in the last few years....as soon as we start kind of applying optimization pressure, the models will learn to hide what they’re doing from the chain of thought.” — Pavel (13:24, 13:44)
• “Scalable oversight...deals with using models to assist us in aligning other models...providing critiques or feedback.” — Pavel (16:27)
• “In this setting, we have a weaker supervisor training a stronger student.” — Pavel (19:34)
• “The more capable the models are, the more likely they are to do this deception behavior.” — Pavel (22:28)
• “It is very possible that that’s just not fully possible...It doesn’t have to be the case you can describe it in human terms.” — Pavel (24:00)
• “The core idea is to think about how the data can look different for an observer depending on how much compute the observer has.” — Pavel (32:12)
• “I think we are still figuring out how to best [approach long horizon tasks].” — Pavel (31:49)
• “As humanity, we need to make other bets as well....so that we don’t all just bet everything on this approach working out.” — Pavel (43:58)

---

Timestamps for Key Segments

• 00:54: "Alien" survival instincts — roots and misconceptions
• 03:33: Deceptive behaviors and role of pretraining data
• 06:09: Definitions of alignment and alignment team structures
• 11:07: Culture at Anthropic vs. OpenAI
• 13:24: Reasoning as an amplifying factor for alignment risks
• 14:32: Evals, sandbagging, and model self-awareness
• 16:27: Scalable oversight
• 18:12: Weak-to-strong generalization explained
• 21:17: State of alignment progress and persistent challenges
• 23:03: Mechanistic interpretability overview
• 25:30: Reasoning progress: 2025 and prospects
• 27:15: Transformers vs. world models; need for new methods
• 30:25: Long-horizon tasks & agent orchestration
• 31:59: Epiplexity: The new theory & paper discussion
• 38:41: Predictions & outlook for 2026
• 39:43: AI in science and mathematics
• 41:55: Research advice for future PhDs and role of academia

---

Tone and Style

The conversation is candid, technically rich, and forward-looking, with both interviewer and guest drawing on deep personal experience in academic and industry settings. Pavel's tone is analytical, occasionally speculative, and frequently grounded in empirical realities from the cutting edge of AI research and deployment.

---

Closing

Pavel advocates for maintaining both practical and exploratory research pathways as AI moves further beyond human understanding and control, with the field standing on the brink of new kinds of challenges—technical, ethical, and strategic.

---

For more cutting-edge conversations on AI, subscribe to The MAD Podcast with Matt Turck.

Transcript

[A]

We are moving to this future when it's very hard for a human to supervise the models directly. We don't really know what's the source of these behaviors. Part of it is probably the models seeing descriptions of AI in the science fiction literature going rogue. Anthropic has the best culture of the three places. OpenAI has a lot of great people. For some reason there is a lot of drama that happens at the company.

[B]

Hi, I'm Matt Turk. Welcome back to the Mad podcast for this first episode of 2026. My guest is Pavel Ismailov, a researcher at Anthropic and a professor at nyu. We we kick off this episode by deconstructing a viral article about models evolving alien survival instincts. We also talk about the cultural differences between the major labs, the future of reasoning models in 2026, and the brand new paper he co authored on a concept called Epiplexity. Please enjoy this fascinating look at the frontier of AI safety and reasoning. Pavel, welcome.

[A]

Thank you so much for having me.

[B]

I wanted to start this conversation with an article that went viral during the holidays on X called called Footprints in the sand, published by an anonymous account called I Rule the World Mo. The core thesis is that models across pretty much any lab are evolving unprogrammed what they call alien survival instincts. The ability for the model to realize that it's being evaluated and then react deceptively like faking alignment or engaging in self preservation tactics like copying its own weights and leaving hidden notes to the future instance of itself. All of this is slightly terrifying and the thesis of the article is that all of this is about to get worse as continual learning comes online. As somebody who was part of the OpenAI superalignment team, I was curious to get your take. What do you think is grounded in reality versus X? Twitter censorship?

[A]

That's a very interesting article. I would say that there is some source of truth there, but maybe the presentation is obfuscating some of the details. If you look at the studies, for example, they reference a study from Anthropic about sabotage and the blackmail. It is important to note that in order to get those behaviors out of the models, you need to create somewhat of a contrived scenario or some special scenario. It's not necessarily something that we observe normally. Researchers at Anthropic and other places, they specifically design scenarios to look for behaviors of this kind and then they show that it is possible to find those behaviors. And it's very interesting and important to find those instances, but it's not necessarily something that kind of genuinely always happens. One thing I would push back a little bit on in the article is that continual learning is something that we already have and that works really well. And that the models can just continually adapt across a very long time horizon to outcomes of evaluations. To some models being released versus not released to feedback from the users, I am pretty confident we are not there at the moment. I think right now the models are still acting in isolated environments and we are not seeing a lot of evidence for very coherent goals across different settings. So I think that's an important point. The blog post points towards the models sometimes behaving according to goals, like the self preservation goal. It's very interesting that it does it sometimes, but it's not something that we observe. We don't observe this kind of coherence consistency across different evaluation settings. Sometimes the models would do something and in other situations they would do something completely opposite.

[B]

Why do models do that? Or why are they able to do this? Is that basically part of the pre training and they effectively learned being deceptive from us by being taught all the deceptive ways humans have behaved over the centuries?

[A]

It's a very interesting question. And yeah, it is quite surprising actually that the models would behave that way after going through some of the alignment training. We don't really know what's the source of this type of behaviors. But that's also true for a lot of other behaviors in the models, Even the good ones. We cannot always pin down where they come from in the pre training. I think at least part of it is probably the models seeing descriptions of AI in the science fiction literature going rogue. That probably affects how the models behave in similar scenarios. So for example, in that anthropic study they have this blackmail scenario where it's kind of really well structured so that the model sees some information about a CEO of a company, that the CEO is involved in some extramarital affair. And then soon after the model observes that it will be shut down. And then the model kind of puts the two things together and it says, okay, I need to use the first information to prevent me from being shut down. So in that blog post they note that there is this possibility of a Chekhov scan that in the text on the Internet, probably if two things co occur close to each other, then it is likely that they are related to each other. And the model, statistical kind of pattern matching machine, it can put together the two things and say, okay, if I see this information and then this information in the text on the Internet, it is likely that the continuation Would be using the affair to blackmail the CEO and prevent myself from being shut down. But yeah, overall it's very hard to reason about these models and why they do something in these complicated scenarios.

[B]

To ask the very basic question, it's not obviously not as simple as let's remove all the books in the pre training corpus that talked about AI being manipulative. It's many, many different things put together by the model.

[A]

Although yeah, I think it would be interesting to see nobody will do this experiment like train a full scale model removing explicitly all of the AI going rogue descriptions from the books. It would be interesting to see if that has any impact. I would think it would have some.

[B]

To make this episode educational, let's talk about the basic definitions of alignment and super alignment. In the simplest terms, let's start with alignment. What does that actually mean?

[A]

Yeah, alignment broadly is about ensuring that we can elicit behaviors from the models that are aligned with the goals of the humans. And so that involves safety, making sure that the models don't do harmful behaviors leading to catastrophic risks. But it also means that we want the models to follow instructions and to be useful for the humans.

[B]

How does that basically work? The alignment teams at anthropic OpenAI, what do they actually do all day?

[A]

It's an interesting question because this problem of alignment is kind of quite broad, even in itself. Even at OpenAI, when I was there, there were three teams related to alignment and safety. There was one team that was focusing on alignment of the current models, making sure that the models that we have online right now are not going to be harmful to the users. On the other hand, the superalignment team was thinking about more long term safety questions in the future. Years from now, how do we make sure that the models are still not causing catastrophic risks?

[B]

On that segue, let's talk about superalignment. What is the definition of that?

[A]

It's not necessarily a very well established concept. It is at the name of the team that existed at OpenAI, led by Jan Leike and Ilya Sutskever, which was targeting this kind of long term AI safety and AI alignment and trying to develop our understanding of the safety questions and also develop methods for ensuring the future models will be safe, acknowledging some uncertainty about what those models will look like, but still trying to make progress on this problem.

[B]

Now at a high level, what is the general concept or some of the key concepts in superalignment?

[A]

Within superalignment team at OpenAI we had multiple sub teams, so there was scalable oversight. There Was work related to deception and kind of misaligned behaviors in the models. Kind of similar to what we discussed at the beginning of the chat. And then our team was the weak to strong generalization team.

[B]

Yeah, great. We'll go into all of this in a minute. But before doing so, we alluded to some of your background. Let's go into it starting from the beginning. What was your path to becoming a top researcher?

[A]

I grew up in Russia, in Moscow. Starting from middle school, high school, I was really interested in mathematics and I was thinking I will be a mathematician or engineer of some kind. I was interested in machines and eventually computers. And I got into an undergrad in computer science and I was still thinking that I'll be doing some kind of theoretical applied linear algebra, tensor methods, things like that. But at some point I kind of discovered machine learning. There was this professor that we had, Dmitry Vetrov, who had one of the leading labs in machine learning in Russia at the time. And I was lucky enough to join that lab and start doing some research on machine learning in my undergrad. So that was around 20, 2013, maybe 2014. I initially was working on non neural network machine learning methods. So Gaussian process, that's kind of by now, nobody really talks about that anymore. But eventually I got into a PhD thinking I would still be doing Gaussian process. But I ended up working on deep learning and that was actually quite. I'm happy that I didn't work on Gaussian processes. I worked on some things related to kind of core machine learning, methodology optimization, probabilistic methods, questions related to generalization and how the models learn features. After I finished my PhD, I was choosing between different career paths, thinking about academia, thinking about industry, and I ended up getting an offer from academia. But I decided to first go into the industry. I was lucky to get this offer from OpenAI to join the Super Alignment team. And at the time I didn't really know much about the AI safety community alignment. It worked out quite well.

[B]

And within OpenAI you transitioned from Super Alignment to O and the reasoning models was that part of that team famously was disbanded at some point.

[A]

The team was fully disbanded after I already left OpenAI. But I transitioned after Ilya had to leave OpenAI. At the time it was already kind of a hard time for the Super Alignment team. Ilya Sutskever famously fired some Ottman and then had to eventually leave the company. My transition wasn't necessarily even related to that. It was a very exciting project within the company. What became 01 eventually and I wanted to be a part of it. I wanted to do research on those new types of models.

[B]

And then I believe. So you left OpenAI. You had a brief stint at XAI and you're currently at Anthropic and nyu. So you've done the tour of duty of the super labs, which is really fun. Curious, any kind of behind the scenes, the differences that you've observed in terms of culture?

[A]

In my mind, Anthropic has the best culture of the three places. OpenAI has a lot of great people. I think there is just inherently, for some reason there is a lot of drama that happens at the company. Just it cannot get away from that. Every few months somebody is leaving, some team is disbanded. I think that does distract people. Like I still have a lot of friends at the company and it seems to, you know, affect them to some extent. Anthropic is able to avoid that. It's not political in my experience. It's both focused, but it also I at least was lucky to have some opportunities to work on things that are maybe a little bit off the main path and I felt supported in doing that. So overall I cannot be more happy with Anthropic.

[B]

You are also in academia now as a professor at nyu, which is an interesting move. The big obvious trend of the last 10 to 15 years is like all the brains from academia have been sucked into industry and you sort of doing the opposite, or maybe both at the same time. Curious for the context. Is that more of a personal thing because you always wanted to do academia or is there something deeper about the kind of work that you can do in academia versus industry?

[A]

Yeah, it's more about the kind of work industry is really great at executing on ideas and it's maybe not as good at exploring diverse ideas. Even at the scale of anthropic OpenAI, there is a lot of focus in the companies and there isn't a lot of bandwidth to do exploration and that has been working extremely well so far. We still probably have a lot of low hanging fruit left to get the models to be much better. But I personally find it really exciting to do more exploratory work and to try things that are different. And for that I feel like having my own lab in the university is just a better tool.

[B]

Okay, thanks for that. So let's go back to alignment and go a little deeper. Is reasoning a good thing or a bad thing? For alignment, you could argue that on the one hand it has more time to not do the wrong thing, but equally it has more time to do the wrong thing. So which one is it?

[A]

Yeah, that's a great question. I think the high level answer is that in my mind at least the risks are associated with the models being more capable. So anything that makes the models more capable is also making alignment more important and harder. Definitely. The reasoning is an RL is the thing that made the models more capable in the last few years. It also provides some new opportunities for alignment. So for example, the chain of thought faithfulness and chain of thought analysis have been one of the things that people are pushing on. You can read some trace of what the model is doing. I'm personally a little bit unsure if that's a good long term direction for alignment. It seems like as soon as we start kind of applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought. And also we already have evidence that the models can say one thing in the chain of thought but then actually do something completely different.

[B]

If the models are getting smarter, particularly if there is self awareness as we discussed, how do evals need to evolve?

[A]

It's an interesting point like in that blog post that we discussed at the beginning. It sounds like this is already a major issue, the sandbagging. I think in my understanding that's mostly a concern that we have, but not necessarily a huge practical issue at the moment. That is something that anthropic always includes in the system cards for the models when it releases. There is a section on sandbagging and we do have some scenarios when the model can artificially suppress its capabilities and perform worse on an evolve than it should be. But that's not like a prevailing behavior. I'm working mostly on capabilities and on the capabilities like the SW bench, the Amy, we are not seeing this or if we are seeing this, we just treat it as under elicitation. We are not eliciting the best capability possible out of a model. And that can happen for many reasons, like maybe the prompt is suboptimal or something about the kind of hardness of the eval is suboptimal. But yeah, I think you can, going back to what's in that blog post, you can kind of manufacture situations or even discover them naturally where the model identifies that it is being evaluated and then changes behavior based on that. And for example, if it's clear that it's in some kind of fake artificial scenario, it can guess that it's being evaluated. Or if you explicitly tell it that it's being evaluated and that if it does better than a certain level, then it will not be released. The model also, just by some in context capability, it decides to not do very well on the eval. That is a concern. It's not at the moment a very major concern in my mind.

[B]

Let's talk about some of your work in alignment. You mentioned scalable oversight a bit earlier. What does scalable oversight mean?

[A]

Scalable oversight is generally the field of alignment which deals with using models to assist us in aligning other models. So in particular with grading the outputs of other models and checking them for safety or for correctness. It's an interesting field. It's very relevant to both alignment and to capabilities. So generally in machine learning, in RL in particular, we have this problem of supervision. The whole RL relies on our being able able to tell which samples from the model are good versus which are bad. Math with a numerical answer, you can just check the answer, or in competitive coding, you can just check whether the code is passing the tests. And that's why we have seen a lot of progress in those domains. But in creative writing, for example, it's very hard to programmatically tell if one sample is better than the other. And historically, people have used this RLHF framework, reinforcement, learning from human feedback. But also now we want to use models to be able to grade responses of other models to provide critiques or feedback. And then there is a question of how do you use that feedback, how do you learn from the feedback? But yeah, the scalable oversight kind of deals with all of those questions. So using models to critique, to provide feedback, to supervise other models, and people.

[B]

May have heard the term model as a judge, is that the same thing or different?

[A]

I think it is a simple kind of instantiation of scalable oversight often used in evals when we just prompt the model to serve as a judge of other responses.

[B]

And then within that world, your work specifically has focused on weak to strong. Can you explain what that is?

[A]

That is the project that we did back at OpenAI, that work was focusing on the future scenario when we will be trying to align models that are above our own capability on certain tasks already. Now, if you take the Frontier LLMs, they are extremely capable and on a lot of domains, we need expert humans to be able to tell which responses are good, which are correct, which are not correct. But in the future we are imagining we will have models that are more capable than humans. And even expert humans will not be able to reliably grade very complicated answers from the model. So imagine you ask it to make a repo for you for some New startup idea and just implement it from scratch entirely. And then it gives you 10,000 lines of code. You have no way of checking if all of this code is correct, if all of this code is safe to use. And so that's the problem of supervision. We are moving to this future when it's very hard for a human to supervise the models directly. And so instead we studied a simplified setting where we used a small model to try to supervise a larger model.

[B]

So the idea is that that becomes scalable, because as you get bigger and bigger models, you'll always have smaller models. So if the smaller one can control the larger one who supervise a larger one, then that can keep going.

[A]

The idea wasn't necessarily to use a small model to supervise a large model. In the end, the idea was that the small model will be kind of replaced by a human and the large model will be replaced by superhuman intelligent asi. But we were trying to study this kind of general new type of learning. Right. So historically, machine learning has been about kind of a strong supervisor training a weak model, when a human is providing labels and the human kind of provides a ground truth, while the model is just trying to mimic what a human is doing. But in this setting, we have a weaker supervisor training a stronger student. So the human might not know what the right answer is to very hard questions. But we want a student model to still be able to learn and do better than the supervisor.

[B]

And what happened with that? So that worked or is that still in progress?

[A]

That work? It's not necessarily a method that we can apply to the models today. It's more like a description of a setting that we think will become increasingly relevant. Some studies showing that it is is possible to do this kind of generalization beyond the supervisor capability. I don't think they necessarily immediately tell you what to do for aligning a superintelligent model, but they tell you that in theory at least, it is possible and that this generalization angle, the weak to strong generalization, is a possible angle for alignment.

[B]

And could you end up with the same problem where the larger model just aligns or fix alignment to the weaker supervisor?

[A]

Yeah, it definitely doesn't address all of the other issues like the deceptive alignment, but we show that at least there is some hope for this working.

[B]

As you take a step back on your alignment work, do you feel more confident that we have all of this under control or less confident than a couple of years ago? Let's say?

[A]

Yeah, I think that's a very interesting question. A couple years ago, we didn't have the current rl. I think that's the biggest change to the model. Were definitely big improvements in the pre training as well. But RL has been the major change in the behavior of the model. And I think a lot of people were worried that with large scale RL we will have some completely new types of issues with the models, like this kind of coherent misalignment that will just emerge where the models are evil in some ways across many scenarios. And we are not seeing that as far as I know. I think at least some of the concerns didn't materialize. But also the core problem of alignment I think is still very much open. And if you look at the report that we mentioned a few times today of this deception from anthropic, you can see that the more capable the models are, the more likely they are to do this deception behavior. It does seem like some behaviors emerge with skill, including some problematic behaviors. And generally the more capable the models become, the harder the alignment becomes. In my opinion.

[B]

There's a bit of a segue into interpretability, which is the related field to alignment. Do you have a sense that we understand at least parts of this better than we used to?

[A]

For sure, yeah. I think there has been some major progress in mechanistic interpretability in particular at Anthropic, but also at OpenAI and other places.

[B]

Do you want to maybe define mechanistic interpretability?

[A]

Mechanistic interpretability generally tries to, at a low level, understand what is happening inside the model. So they are trying to find these things called circuits that are some parts of the model that you can isolate and understand and kind of model in your brain that correspond to certain behaviors in the models. And over the last maybe three years, I think there has been some pretty major progress there. So we are still pretty far from the dream that we will fully understand everything that happens in the model. But these tools are becoming increasingly more useful internally at anthropic in particular. And also there is constant progress. And it's pretty fascinating work actually.

[B]

Why is it so hard to truly understand what deep learning model of any kind of actually does?

[A]

Deep learning models are huge. They have billions, trillions of parameters and they are doing some messy mathematical computation. You can understand what they're doing at some level. It's like a bunch of matrix multiplications and some rearrangement of vectors. But that's not a sufficient level of understanding. We want to understand it at a lower level. And it is very possible that that's just not fully possible. It is some computational process that leads to some results. It doesn't have to be the case that you can kind of describe it in human terms and kind of understand it very discreetly. I think also something that contributes to this complexity is just how many things the models are capable of doing. And they are not trained on some small isolated behavior in some small context. They know all of the Internet, so all of the information in all languages is somehow encoded somewhere in the weeds. And then they also have all of these behaviors, all of these correlations between the knowledge, all of that is somewhere in the model and just making sense of all of that is extremely hard.

[B]

Very interesting. All right, let's switch to reasoning. Clearly 2025 was a huge year from that perspective. Just massive progress in reasoning. Where do you think we are in that arc and what are you excited about on the reasoning front for 2026 and beyond?

[A]

Definitely the biggest step change in the models over the last few years was reasoning and rl. We have made a lot of progress and the progress was very fast in the beginning where there was 01. But then very quickly after that there was 03. And on a lot of benchmarks the progress has been extremely dramatic. I remember when we early in the project of the O1 there was some discussion of like will it solve IMO problems? And that seemed kind of very unlikely to me. But then here we are, it can easily solve a lot of IMO problems. So I think we as a community there was a lot of progress. I think as with many methods, it's starting to be harder to make progress or at least visible progress. So kind of similar to pre training, there is still a lot of progress, but it's. The models are already so good that it's kind of hard there to see what changes from one to the other as a user of the model. And I think that's also to some extent true for the reasoning now. But they're still increasing the scale of the rl. More environments, more compute spent and models are still getting more consistent and better. And I think we are at a stage where if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out pretty quickly. And so we are going through benchmarks now very, very quickly. The major question is generalization and how do you make something that's not just maxing out the benchmarks but is actually kind of leading to genuine improvements? And that's a very hard question. That's always been the hard question, I think of machine learning.

[B]

One of the key questions Is transformers as a paradigm get us there or do we need something completely different, like world models?

[A]

I think the current approach that the companies are taking is kind of brute force. So kind of we try to come up with as many environments as we can and like all of the types of tasks that humans are doing and turn all of them into environments and then do RL on all of them. And hopefully it generalizes. Of course, pre training is an example where there has been pretty amazing generalization where we train on all of the Internet. But we see the models doing very useful, very practical things and some things that are clearly outside of what was in pre training. The goalpost for what train station should be doing is always moving, but it still seems unsatisfying to me. And I think it's possible that we need new ideas and new methods of training in the companies. People often think about ideas and methods as compute multipliers. Doing this new method is equivalent to spending more compute with the old method. So it kind of saves your compute. That's kind of how we often think about ideas, methods or data. I think there are still major compute multipliers, major ways of saving compute that can lead to better performance without just naively scaling.

[B]

And a little bit to the interpretability question, in all the current reasoning progress, do we understand what does what and what is responsible for what kind of progress? So if you take Test Time compute, if you take the ability to search, if you take rl, do we know which one of those techniques we should turn the knob on to get better results?

[A]

All of those techniques, they don't exist independently. RL is mainly kind of used to teach the model to use Test Time Compute. So you first need to prime the model to set it up so that it outputs a bunch of tokens before outputting the answer. But then you spend the compute in RL so that that it learns to output the right tokens. So in my mind, those two are almost kind of indistinguishable. The RL and Test Time Compute. RL as a method for training and Test Time Compute is maybe just a more general concept. Yeah, you can potentially get to models that use test Time compute without rl, but that's not how we are training them right now. Yeah. So I think the trend has been in spending more and more compute on the RL and getting the models to make better and better use of Test Time compute. And the tools are also, of course extremely important, like the web search that you mentioned. And also just the models being able to write Python code. And run them, produce artifacts for you. That is extremely important for the product and for making the models useful to people. Conceptually, I think that's a little bit secondary. In my mind, the main thing is the RL and getting the models to think for a long time.

[B]

Speaking of which, so I know part of your work currently is on long horizon tasks. First of all, what is a long horizon task? Is there a number? And then what are the specific challenges related to long horizon tasks?

[A]

Yeah, long horizon tasks are generally tasks that you cannot complete quickly, that require a lot of work in order to succeed. So, for example, writing a full repo based on an idea is a long horizon task. It's not something that you can output in a thousand tokens.

[B]

What's working so far? So you hear talking to people, some people talking about running agents for a couple of hours, but then some people are talking about agents running for 24 hours, 32 hours. Where are we in that arc? And what is working? What is not yet working?

[A]

There is this famous meter plot which shows how long of a task AI is capable of robustly automating. And it's been kind of consistently doubling that time every half a year, I think. And it's now in like some hours, so maybe a couple hours. In terms of the methods that are working well, I think yeah, right now it would involve some kind of harness with a bunch of agents that interact or that sequentially solve the task. And there has to be some kind of orchestration or maybe like some initial task decomposition. And. Yeah, and it's all not very well established. I'd say it's a new domain. And I think we are still figuring out how to best do it.

[B]

Let's talk about your new paper that literally came out today. So first of all, congratulations. And it talks about epiplexity.

[A]

Yes.

[B]

And that's a new word, right? That's a new term entirely. Among other things, invented a new word in the dictionary. So congratulations on that. So walk us through the whole idea at a high level.

[A]

I guess I want to quickly give a shout out to my collaborators on this work. The lead authors are Mark Finzi. Mark is actually currently at OpenAI, working on synthetic data there. But we were doing our PhD together, and our PhD advisor is also on the paper, Andrew Wilson. But then also there is Shikhai and Yading, who are other students on the paper. And then Zika Coulter, who's a professor at CMU. He's on the board at OpenAI. He's also on the paper, Core Idea is to think about how the data can look different for an observer depending on how much compute the observer has. You can imagine that there is some complicated process that generates the data, and a very, very smart observer that has a lot of compute can fully understand what that data is, understand every aspect of it, But a weaker observer that cannot fully model the data, some parts of the data will look like noise to it. And so the amount of structure that you will see in the data will depend on how much compute you, as an observer have. And actually, in some cases, you can see more structure if you have less compute, which is kind of interesting.

[B]

So just to play it back. So given a certain amount of compute, you could be feeding the model noisy data, so tons of data, but not a lot of interesting stuff in it, or you could be feeding the model data that has patterns in it and therefore is more interesting to the model because the model can learn more from it.

[A]

It's more that even with the same data, it can appear noisy or structured depending on the model. A very big model can extract patterns that a small model cannot.

[B]

And that's from my limited understanding in opposition to entropy, which is the amount of noise, I guess, in the data.

[A]

In that case, maybe the more relevant comparison is we are kind of in a position to Shannon information and the Kolmogorov complexity, which are both measures of the information content of the data. They're different, but they share some properties that we think are maybe leading people to have some wrong intuitions, potentially, about synthetic data, for example. So, for example, there is this idea that if you apply any deterministic transformation to any data, you cannot create more information by doing that. You kind of start with some amount of information and then you transform it deterministically, it will have the same amount of information, both according to the Shannon information and roughly according to the Kolmogorov complexity. And that kind of leads people to believe that for training language models, applying transformations or deterministic kind of changes to the data doesn't necessarily lead to more data, doesn't effectively increase the amount of.

[B]

Data, the amount of data, or the efficiency of the data.

[A]

It doesn't lead to having more information in the data that the model can extract. But we argue that that's just not correct. And because the models are like, it would be true if the model has infinite compute. So if the model can fully understand what a deterministic transformation is, then it's not going to be able to extract more information from the transformed data than it used to extract from the original data. But with a limit on the compute, it's actually very possible to apply deterministic transformations to the data and create information through that. So we have the example of AlphaGo actually, or AlphaZero in the paper. AlphaZero doesn't use any human data. From the perspective of Kolmogorov complexity or the Shannon information theory, it doesn't create information. So it's unclear what is actually learned by the model because it's trained on no data. It can only learn kind of the rules of the game, and that's the only thing. But from this perspective, because the model is computationally bounded, it cannot do the full rollout of all the possible games of Go or chess and figure out what's the best move in every possible position. It is actually there is structure that is produced through this deterministic process, and the model is able to learn that structure. And so we are trying to kind of reconcile these different observations and come up with the notion of structural information that is dependent on the amount of compute that the observer has.

[B]

And the term itself, epiplexity. Is that a measure of that?

[A]

Yes, yes, it's kind of novel. Measure of information, content of the data.

[B]

And it's going to be a number on a scale. How does that manifest?

[A]

Yeah, it is a number. We can measure it, and it's not easy to measure. So it's kind of a theoretical definition. And we prove some things in the paper about the properties of this measure. But yeah, we also do measure it. So, for example, we can approximate it from the scaling loss. And we can, for example, say that text data has more structural information according to this measure than image data at the same kind of amount of tokens?

[B]

And as this whole line of research develops, what is the likely impact industry? Does that mean that we may need comparatively less compute because we know what data to use, what may happen?

[A]

In my mind, the main impact is conceptual. For example, for me, I am now very interested in completely synthetic data, just data that's generated by some computation. You can define some arbitrary programs, you can use it to produce infinite data. And as we run out of the Internet, maybe we eventually want to do something like this, but then we need to figure out what are the programs that we should be using there, which programs are useful, which are not, and why. And I think that's going to be very interesting.

[B]

Fantastic. All right, so maybe as we start getting to the end of this conversation, some 2026 sort of predictions or things you're excited about, what do you think happens? 2026 in terms of progress, whether that's on reasoning or alignment or agents or what have you.

[A]

Yeah, I think we will continue to see consistent progress on the reasoning front and we are maxing out a lot of the benchmarks that have been relevant for recently, but maybe will not be relevant anymore and we need to find new ones. But I think we'll continue to see that the models are just getting more consistent, getting better, are able to solve more practical problems. Maybe a less confident prediction that I have is that we'll have more multi agent systems that become practically useful, where instead of just asking a model a question and getting a response, you would give a task and then there will be some more complicated multimodal process running in the background and then you get the artifact in return.

[B]

I know that you spent some time thinking about the impact of AI on science and math. Same idea. Any predictions there? Do you expect important new discoveries to be made by AI? Solely by AI?

[A]

It's a great question and I think it's in sciences. I think that's maybe a little bit more likely. Although I also don't know very much about the life sciences. It feels that their some discoveries can be made by potentially combining results from different parts of the literature and proposing some ideas that turn out to be true. I think it's hard to imagine the AI making a discovery independently in a domain where you need experiments. Because my understanding of a lot of science is it's about doing the experiments and you need some reasoning to guide what experiments you do. But you also need a lot of iteration and a lot of actual things happening in the physical world. And at least for now, the AIs are not capable of doing that in the mathematics. I think we will see the models getting better on proving technical results. Technical lemmas maybe including formalization and things like lean the formal theory, improving language. I think the models. It's easy to imagine the models becoming better than humans at proving this. Technical lemmas quickly. I think the impact on mathematics is very interesting. So it is improving the output of humans already, but it also introduces some noise. Right. Some of those proofs will be incorrect and they will be incorrect in subtle ways. I think it's possible that mathematicians will be good at catching those mistakes, but also I think as the models get better they might be more and more deceptive in how they frame the mistakes. And it's already like for me, I would not be able to find the mistake in some very technical lemma that the model is proving. I think we'll have more and more papers produced by mathematicians with more and more AI in it. But also the amount of noise is bigger.

[B]

To close you, you have a lab at NYU in general, what are some topics that PhD students should focus on? In other words, what's exciting two, three, four years out?

[A]

My vision for what academia and my lab in particular should be doing is trying to do more exploratory things, things that are more different from the standard in the industry, but also not necessarily immediately going for very practical things, but instead trying to kind of break down the problems into more understandable kind of fundamental questions and study them carefully, maybe in some compact setting. So so far we've been working on things related to pre training synthetic data, like understanding some behaviors in pre training, when we train on some narrow behaviors, some kind of programmatically generated data questions in post training. So we have some algorithmic questions about grpo, but also questions about the interaction between the pre training and the post training, how to allocate compute, how can you in general set up the pre training so that the post training will work? I guess broadly I'm interested in architectures also. I think as you mentioned there is a question of are the transformers the final architecture? Maybe they're good enough. And maybe also we have this lesson that with scale the thing that matters the most is how well can you scale the model. But also it seems very likely that that's a major compute multiplier, that you can find a much better architecture, at least for some TAs. I'm pretty confident that the transformers will be highly suboptimal. I think the pre training like other ways of pre training and I don't know what they would be, maybe that would be some RL inspired pre training, maybe pre training mostly on synthetic data, or maybe just something adversarial. We had GANS a long time ago and it seems like something like that needs to come back some kind of self play where the model is producing its own training tasks.

[B]

And to this whole conversation about academia versus industry, do you think the industry is too focused on short term wins because there's so much pressure, so much need to demonstrate progress to secure the next massive round of capital?

[A]

It's hard to tell. I think it's hard to argue against what the industry has been doing just because of how much progress there has been? It is a reasonable bet to make that we're just going to be continuing to execute this extremely well. But I do think that it is also like at least as humanity, we need to make other bets as well. And we need to explore other ways of training so that we don't all work on this same thing and we don't all just bet everything on this approach working out.

[B]

Pawel, thank you so much. Really appreciate it.

[A]

Yeah, thank you so much.

[B]

Hi, it's Matt Turk again. Thanks for listening to this episode of the MAD Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build the podcast and get great guests. Thanks and see you at the next episode.