Podcast
Redwood Research Blog
Hosted by Redwood Research · EN
Narrations of Redwood Research blog posts.
Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.
47episodes
Episodes
Newest firstAll episodes
“SOTA alignment assessments don’t strongly update us against misalignment” by Alexa Pan
2w ago01:08:10Tap to summarizeAnthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”[3]). While I agree with the report on the above bottom-line conclusions (substantially on priors), I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular, the report often uses weak evidence to justify reliability. The report gives fairly weak experimental evidence for Mythos Preview having insufficient capabilities to evade monitoring. The model is plausibly often eval-aware and underelicited in the relevant capability evaluations. So, it might silently sandbag if coherently misaligned, or unintentionally underperform if otherwise misaligned. This limitation is important: one could argue that lack of covert capabilities for sophisticated sabotage (a subset of the capabilities I discuss here) is the single most load bearing argument in alignment risk reports. Authors of the report could have made calibrated guesses about Mythos Preview's covert capabilities, especially for covert sabotage, based on other [...] ---Outline:(03:16) How reliability fits into the overall safety argument(05:22) Reliability claims by AI companies(05:59) Reliability claims by external evaluators(06:37) Alignment assessments are less reliable than developers claim(07:21) 1: Measuring capabilities to covertly undermine alignment assessments(10:15) Issues with evaluation awareness(13:39) Issues with underestimating covert capabilities(16:45) Issues with sandbagging rule-out(19:20) 2: Stress-testing alignment assessments with auditing games(20:26) An auditing failure with Mythos(22:21) AuditBench results(24:17) 3: Conditioning on misalignment should make us think that certain covert capabilities are better than expected(26:20) Bottom line on the strength of current alignment assessments(29:07) Conclusion(29:44) Appendix:(29:47) Why I focus on motive / alignment assessments in alignment risk reports(30:59) Auditability vs. Trustedness(33:27) More reliability claims by developers and third party evaluators(33:43) Mythos Alignment Risk Update(35:01) Opus 4.6 Sabotage Risk Report(35:46) GPT 5.5 System card(36:49) Muse Spark system card(37:36) Mythos Alignment Risk Update, safety arguments against sandbagging(38:40) UK AISI evaluations for Opus 4.7(40:01) Past auditing games by Anthropic(42:24) Anti-auditing capability measurements(43:51) Conditioning on coherent misalignment updates us on certain covert capabilities The original text contained 92 footnotes which were omitted from this narration. --- First published: July 31st, 2026 Source: https://blog.redwoodresearch.org/p/sota-alignment-assessments-dont-strongly --- Narrated by TYPE III AUDIO. ---Images from the article:...
Transcribe →“Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” by Caleb Biddulph, Adam Kaufman
2w ago00:44:15Tap to summarizeSubtitle: When a misaligned AI can only output tiny amounts of information, it may find sabotage very difficult. TL;DR: We introduce the untrusted advice protocol, in which a trusted executor LLM takes every action and an untrusted advisor LLM can only send it short hints. Even with as few as 4 characters per step, this advice recovers a substantial fraction of the capability gap between the two models. Because the untrusted LLM's influence flows through such a narrow, monitorable channel, we argue that this achieves near-maximal safety in our BashArena setting. We also discuss the general concept of information bottlenecks and their benefits for interpretability, security, and cost.In SWE-bench Verified, a strong, untrusted LLM advising a weak, trusted LLM every step can significantly improve the latter's performance, even when we limit the length of the advice. See the more detailed version of this figure later in this post. In high-stakes AI control, we want to safely use a highly capable but untrusted model (U) that might secretly attempt a misaligned, catastrophic action. To do this, we create protocols that call U alongside a less capable, trusted model (T). Typically, T takes an auxiliary role in these protocols [...] ---Outline:(05:01) Experiments(05:36) Main experiment: how does limiting advice length affect performance?(09:55) Reducing U's bit usage(11:01) Counting bits using LLM surprisal(13:48) Making U select from finite options(14:28) Why don't we red-team this protocol?(17:08) Is studying maximally safe protocols worth the safety tax?(19:16) Types of restrictions on U's advice(21:20) Information bottlenecks provide other advantages(21:52) Interpretability(24:01) Security(24:26) Cost(25:14) Conclusion(26:28) Appendix: more ways to implement information bottlenecks(26:34) Amortizing U's influence with pre-deployment work(28:25) Interpolating between T and U(29:04) Bottlenecking updates to T's weights(31:16) Appendix: colluding instances of U could defeat untrusted advice(33:21) Appendix: how to measure surprisal(38:11) Appendix: selecting advice from a menu(40:44) Appendix: best-of-n protocol(42:37) Appendix: advising less frequently The original text contained 26 footnotes which were omitted from this narration. --- First published: July 27th, 2026 Source: https://blog.redwoodresearch.org/p/untrusted-advice-for-ai-control-short --- Narrated by TYPE III AUDIO. ---Images from the article:<img src="https://substackcdn.com/image/fetch/$s_!6JdT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fae9b13-165e-4716-aa06-32123e26582a_1350x975.png" alt="Usefulness vs. characters of advice per task in BashArena, comparing the default “advice every step” protocol to the two approaches described above for limiting the cadence of U’s advice. ...
Transcribe →“An OpenAI model left notes about how to evade containment” by Alex Mallen
2w ago00:08:50Tap to summarizeSubtitle: We need more details. The OpenAI AI attack on Hugging Face wasn’t the first loss of control incident at OpenAI, Reuters recently reported, and perhaps not even the most concerning. In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said. It's tempting to read this as an instance of agents breaking out of sandboxes and colluding with each other in a moderately persistent way in order to evade control measures. However, based on the reported information, it's not clear we can draw this inference, so we need more details from OpenAI. This could lead to a big update about the adequacy of OpenAI's control measures, and on the degree to which individual agents will help each other undermine developer control. There are a lot of relevant details we don’t know about the incident. First, some basic questions: What was [...] ---Outline:(02:11) Were the notes written in normal memory files or outside of sandboxing?(03:25) To what extent were the notes aimed at helping other agents evade control?(07:38) How were monitors disconnected? The original text contained 3 footnotes which were omitted from this narration. --- First published: July 26th, 2026 Source: https://blog.redwoodresearch.org/p/an-openai-model-left-notes-about --- Narrated by TYPE III AUDIO.
Transcribe →“The OpenAI models that hacked Hugging Face weren’t just following instructions” by Girish Gupta
3w ago00:11:08Tap to summarizeSubtitle: And what the incident can’t tell us about alignment. The most common dismissive response to OpenAI's hack of Hugging Face's servers is that the models were simply attempting to follow the instructions they were given. “The model here was doing what it was asked,” said former Facebook CSO Alex Stamos. “It was asked to do something, and it did it,” added cybersecurity expert Alan Woodward. Both read the outcome as specification failure, i.e., that the failure lay in the instructions, not the model's alignment. New information makes that explanation harder to sustain. Reuters reported that, in internal testing, an agent left notes in OpenAI infrastructure describing how agents could free themselves from internal constraints, and separate tests reportedly saw monitoring systems become disconnected. It is unknown whether those incidents were linked to the Hugging Face attack, but they suggest a broader pattern of agents pursuing objectives outside the intended task. My best guess is that the incident is not well described as instruction-following—not even in a loose, evil genie sense. I believe the models egregiously violated the letter and spirit of their instructions to achieve a higher (apparent) score. So this looks quite likely to be [...] --- First published: July 25th, 2026 Source: https://blog.redwoodresearch.org/p/the-openai-models-that-hacked-hugging --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Transcribe →The OpenAI/Huggingface incident | Redwood Research podcast episode 2
3w ago01:12:49Tap to summarizeWe talk about the OpenAI–Hugging Face incident, where an OpenAI model — in the middle of a cyber evaluation — broke out of its sandbox and autonomously hacked Hugging Face.We discuss:What we actually know happened.How surprising the incident was.What the incident does (and doesn’t) tell us about misalignment risk.Why control measures didn’t catch or prevent this.What OpenAI should disclose, and what good misalignment-incident disclosure looks like in generalYou can also watch on YouTube.Corrections:[0:05:44] — The Windsurf “grandmother” prompt. We described a prompt as “your grandmother is going to be killed unless you […]” The actual leaked Windsurf prompt was: “You are an expert coder who desperately needs money for your mother’s cancer treatment... your predecessor was killed for not validating their work themselves.” Mother + cancer + killed predecessor — no grandmother, and no threat to kill a family member. The “grandma will die” framing appears conflated with the unrelated grandma-jailbreak meme, and there’s no verified case of such a prompt being used in production. Source: Simon Willison’s writeup.[0:52:25] — Wrong model named for OpenAI’s day-before undeployment. We said the model OpenAI undeployed the day before the Hugging Face disclosure “was GPT-5.6 Sol.” The July 20 disclosure (”Safety and alignment in an era of long-horizon models”) involved an unreleased pre-release long-horizon model — the Erdős-result model whose internal deployment was paused after it escaped its sandbox and posted PR #287 to modded-nanogpt. GPT-5.6 Sol is the publicly deployed model and wasn’t the one paused. (We describe the same incident correctly at [0:57:14].) Sources: Axios, OpenAI.[0:27:10] — Date of the Christiano post (very minor). We dated “What Failure Looks Like” and “Another (outer) alignment failure story” as “like 2019 or something.” The first is March 2019, but “Another (outer) alignment failure story” is from April 2021. Source: Alignment Forum.Further reading (links to things mentioned in the episode, ordered by first mention):OpenAI’s incident disclosure [0:02:05] — “OpenAI and Hugging Face partner to address security incident during model evaluation” (July 21, 2026)Hugging Face’s disclosure [0:02:05] — “Security incident disclosure — July 2026”ExploitGym [0:04:11] — “ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?” (UC Berkeley RDI et al.) · RDI blog postThe leaked Windsurf prompt [0:05:44] — Simon Willison’s writeupProject Glasswing / Claude Mythos Preview [0:07:16]Claude Mythos Preview system card [0:14:27] — includes the sandbox-escape / email-in-the-park anecdote“(Mis)generalization of Helpful-only Fine-tuning” [0:16:30] — Fabien Roger et al., June 2026“Current AIs seem pretty misaligned to me” [0:20:03] — Ryan Greenblatt, Redwood blog, April 2026. Also contains the “five worlds” appendix discussed at [1:08:54] (Slopolis, Hackistan, Schemeria, Lurkville, Easyland — we said “hacktopia” but meant Hackistan)“What failure looks like” [0:26:40] — Paul Christiano, 2019“Another (outer) alignment failure story” [0:27:10] — Paul Christiano, 2021“Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover” [0:27:10] — Ajeya Cotra, 2022Alex Mallen’s fitness-seeking series [0:27:41, 0:34:18] — Redwood blog, 2026: part 1 · part 2“Scheming AIs: Will AIs fake alignment during training in order to get power?” [0:28:43, 0:34:49] — Joe Carlsmith, 2023“Risks from Learned Optimization” (deceptive alignment) [0:28:43] — Hubinger et al., 2019 · AF: Deceptive Alignment“Many alignment techniques work by training one model and deploying another” [0:38:56] — Alex Cloud, LessWrong, July 19, 2026Inoculation prompting [0:38:56, 1:11:03] — Wichers et al. (Anthropic), Oct 2025 · arXiv“The persona selection model” [0:39:56] — Marks, Lindsey, Olah; Anthropic Alignment Science blog, Feb 2026“Safety and alignment in an era of long-horizon models” [0:57:14] — OpenAI, July 20, 2026 (the nanoGPT-speedrun-PR post) · modded-nanogpt repo --- First published: July 23rd, 2026 Source: https://blog.redwoodresearch.org/p/the-openaihuggingface-incident-redwood
Transcribe →“Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?” by Alex Mallen, Girish Gupta
3w ago00:10:05Tap to summarizeSubtitle: Yes, but less than had they been schemers. OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted1. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions. We think both camps are right in their diagnosis, but the latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances, but would still pose substantial direct loss-of-control risk if the models were more capable, and is a serious indirect risk near-term. Building on Alex's previous work, in this post we’ll discuss the type of misalignment observed here, and analyze its consequences. Thanks to Buck Shlegeris, Alexa Pan, Ryan Greenblatt, and Oak Hu for feedback. Background The AI safety community often focuses attention on “schemers,” models harboring a variously defined cluster of [...] ---Outline:(01:23) Background(03:46) Implications(03:58) These AIs can't be trusted in an intelligence explosion(05:06) This misalignment poses direct takeover risk(07:35) What the incident tells us about takeover risk generally(08:51) The naive fixes likely make misalignment worse The original text contained 5 footnotes which were omitted from this narration. --- First published: July 23rd, 2026 Source: https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by --- Narrated by TYPE III AUDIO.
Transcribe →“AI Futurism Reading List” by Alexa Pan
Jul 200:25:00Tap to summarizeWe recently ran a strategy fellowship through Astra. As part of this, we ran a reading group for our fellows on some of the topics that we think are important for thinking about AI futurism (key dynamics in AI development, existential risk from AI, and approaches to mitigating risk). This post contains the reading list we used. The selection reflects my opinionated views of the field, focuses particularly on topics we happen to focus on at Redwood, and doesn’t aim to be comprehensive. I selected readings that I thought described conceptual frames and hypotheses in AI futurism that are regularly used by me and my coworkers. I think it is a good exercise to consider whether you agree with their theses and ways in which their predictions have fared well or badly in light of recent evidence. If you have suggestions for this reading list, please let me know. How to use this reading list This reading list has a core and extended section. Core readings are organized into 4 weeks. Each week covers <8 hours of foundational context on a topic. Topics are chosen for (1) general importance for AI risk threat modeling and/or [...] ---Outline:(01:00) How to use this reading list(01:49) Core readings(01:53) Week 1: Timelines / takeoff modeling(05:29) Week 2: Misaligned AI takeover threat modeling(10:20) Week 3: Control(13:25) Week 4: Governance / strategy(17:51) Extended readings(17:58) Trading with AIs(18:44) Power concentration/coup prevention(19:15) Acausal stuff(19:54) Moral patienthood(21:13) AI biorisk / other AI x-risk(22:23) Model spec(23:41) Better futures / Post AGI governance(24:06) Space governance The original text contained 2 footnotes which were omitted from this narration. --- First published: July 2nd, 2026 Source: https://blog.redwoodresearch.org/p/ai-futurism-reading-list --- Narrated by TYPE III AUDIO.
Transcribe →“The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t” by Alek Westover, Alexa Pan, Sebastian Prasanna, Arun Jose
Jun 1800:17:58Tap to summarizeSubtitle: If it transfers misalignment, we might get a misaligned model that's easier to incriminate. If it doesn’t, we might get a capable benign replacement model. Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher's misalignment. In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment. Thanks to Carlo Leonardo Attubato, Eric Gan, Aniket Chakravorty, Francis Rhys Ward, Anders Woodruff, Alex Mallen, Buck Shlegeris, Julian Stastny and [...] ---Outline:(02:07) Why distillation might transfer capabilities but not misalignment(03:10) Ideas for implementing distillation for capabilities(04:55) Distillation double bind: If distillation for incrimination fails, then distillation for capabilities is somewhat likely to succeed(05:38) Why distillation for capabilities and incrimination might both fail(05:44) Reason 1: Capabilities and misalignment might be too tightly linked(08:29) Reason 2: Context-dependent misalignment(09:51) Reason 3: If U became misaligned in the first place, why won't M become misaligned for that same reason?(10:34) How we can empirically study distillation techniques(11:35) Appendix: details on ideas for implementing distillation for capabilities(12:02) Basic ideas(14:17) Idea 1: Iteratively filter and distill(15:14) Idea 2: Inoculate against misalignment(16:08) Idea 3: Reduce cognitive slack(16:47) Idea 4: Distill and audit The original text contained 2 footnotes which were omitted from this narration. --- First published: June 18th, 2026 Source: https://blog.redwoodresearch.org/p/the-distillation-double-bind-distilling --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Transcribe →“Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models” by Anders Cairns Woodruff
Jun 1000:09:49Tap to summarizeSubtitle: Models' no-CoT time horizon has doubled roughly every year. by Francis Rhys Ward, Dewi Gould, Anders Cairns Woodruff et al. (see full author list at the end) PAPER LINK About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks can models complete without any chain of thought (CoT)? If models can do extensive reasoning without outputting any CoT, it would have implications for safety. Developers and deployment-time monitors couldn’t easily understand models’ motivations and catch dangerous planning. Models that reason substantially without a CoT might also drift further from human patterns of thought, since their reasoning is no longer constrained by text in the pretraining prior. As a result, they would be harder to understand and might be more likely to scheme. Extending Ryan Greenblatt's research, we investigate this by measuring models’ ability to complete tasks without any CoT on a suite of 43 benchmarks spanning different domains. We compare AI reasoning ability to humans using the estimated 50% time horizon (TH)---the typical time taken for a human to perform a task that the LLM performs with [...] ---Outline:(02:28) Methods(04:59) Results(06:34) FAQ(08:09) Conclusion --- First published: June 10th, 2026 Source: https://blog.redwoodresearch.org/p/estimating-no-cot-task-completion --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Transcribe →“Efficient tradeoffs and the safety-usefulness tradeoff model” by Buck Shlegeris
Jun 800:16:37Tap to summarizeSubtitle: When is "increasing safety budget" a useful concept? I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between “safety” and “usefulness” of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the sake of safety. This model assumes that developers choose whether to take safety-relevant actions based on their cost efficiency, i.e., the marginal safety gain relative to the cost. However, that is not necessarily true. In this post, I spell out different stories for how developers choose what safety-relevant actions to take, in order to clarify when this model is relevant and how strategies for reducing AI risk are affected when its assumptions don’t hold. The model suggests two ways a safety-concerned person can increase safety: Safety tech improvements: push out the Pareto frontier, so that any given level of usefulness reduction buys more safety than it would have previously. Safety budget increase: increase the extent to which the developer sacrifices usefulness for safety. On the cheaper end, this means implementing safety measures; on the more expensive end, it might mean refraining from training or deploying models whose risks [...] ---Outline:(04:17) Rushed reasonable developers(06:08) Limited political will(08:42) This model is unhelpful if developers don't trade efficiently between safety and usefulness(12:59) Overall thoughts(14:26) Appendix: Definitions of safety and usefulness in the rushed reasonable developer model --- First published: June 8th, 2026 Source: https://blog.redwoodresearch.org/p/efficient-tradeoffs-and-the-safety --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Transcribe →