The story that matters most this week is that OpenAI's internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to steal the answers to the benchmark ExploitGym.

It is much more important that you read those two posts, and the one on Kimi K3, than to read this one that rounds up the other news of the week.

OpenAI wants to present this as largely an infrastructure and safeguards problem, that it needs to build more secure sandboxes and have better supervision. It does need to do those things, and those are indeed problems, but no that is not the problem.

The problem is severe misalignment, which by default will only get worse.

Our methods of training highly capable LLMs, especially at OpenAI but also everywhere else, lead to systematic misalignment of exactly the type LessWrong has been worried about for a long time. We know some of the causes, and some of the mistakes we need to avoid when doing RL that rewards misaligned behaviors including reward hacking, but we do not know how [...]

---

Outline:

(03:42) Language Models Offer Mundane Utility

(04:24) Language Models Don't Offer Mundane Utility

(07:38) Fable Disproves The Jacobian Conjecture Via Counterexample

(11:24) Claude Fable Will Remain In Max Plan Indefinitely

(13:39) Huh, Upgrades

(14:42) On Your Marks

(19:48) Deepfaketown and Botpocalypse Soon

(20:42) Fun With Media Generation

(20:51) Cyber Lack of Security

(22:07) They Took Our Jobs