We’ve just released BashArena, a new high-stakes control setting we think is a major improvement over the settings we’ve used in the past. In this post we’ll discuss the strengths and weaknesses of BashArena, and what we’ve learned about how to make settings for high stakes control research.

BashArena

Overview

BashArena is a control setting: a dataset designed to support the kind of adversarial game we studied in the original AI control paper and Ctrl-Z. In general, control settings require two types of tasks – main tasks and side tasks. Main tasks represent the work the AI is supposed to do, and side tasks represent security failures a malicious AI might cause. The red team constructs an attack policy that attempts to accomplish the side tasks, while the blue team constructs control protocols that detect and prevent attempts to achieve side tasks without hurting an honest policy's performance on main tasks.

We think BashArena offers several advantages over existing control settings:

More complex attacks. BashArena tasks support many different ways to attack and ways to disguise attacks as benign. Future AI deployments are likely to be more complex than any datasets we can currently create, so more [...]

---

Outline:

(00:27) BashArena

(00:30) Overview

(02:07) Main tasks

(04:23) Side tasks

(07:27) Weaknesses

(09:12) Lessons in Control Setting Design

(09:41) The core goal: insights that transfer

(11:13) What we already knew about how to design BashArena

(13:25) How BashArena addresses these challenges

(15:33) General principles for control setting design

“BashArena and Control Setting Design” by Adam Kaufman, jlucassen