Subtitle: Safely navigating the intelligence explosion will require much more careful development.

It turns out that Anthropic accidentally trained against the chain of thought of Claude Mythos Preview in around 8% of training episodes. This is at least the second independent incident in which Anthropic accidentally exposed their model's CoT to the oversight signal.

In more powerful systems, this kind of failure would jeopardize safely navigating the intelligence explosion. It's crucial to build good processes to ensure development is executed according to plan, especially as human oversight becomes spread thin over increasing amounts of potentially untrusted and sloppy AI labor.

This particular failure is also directly harmful, because it significantly reduces our confidence that the model's reasoning trace is monitorable (reflective of the AI's intent to misbehave).[1]

I’m grateful that Anthropic has transparently reported on this issue as much as they have, allowing for outside scrutiny. I want to encourage them to continue to do so.

Thanks to Carlo Leonardo Attubato, Buck Shlegeris, Fabien Roger, Arun Jose, and Aniket Chakravorty for feedback and discussion. See also previous discussion here.

Incidents

A technical error affecting Mythos, Opus 4.6, and Sonnet 4.6

This is the most recent incident. In the [...]

---

Outline:

(01:28) Incidents

(01:31) A technical error affecting Mythos, Opus 4.6, and Sonnet 4.6

(02:10) A technical error affecting Opus 4.6

(02:50) Miscommunication re: CoT exposure for Opus 4

(03:25) Why this matters

(08:07) Appendix: How hard was this to avoid?

(10:19) Appendix: Did training on the CoT actually make Anthropic AIs externalize less of their (misaligned) reasoning in CoT?

“Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes” by Alex Mallen, Ryan Greenblatt