Subtitle: Reward-seekers are supposed to be safer because they respond to incentives under developer control. But what if they also respond to incentives that aren't?.

Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?

If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively[1]tightly control local incentives—the reward signal during training and deployment—but they can’t prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not because of a flaw in its local training, but because it's responding to incentives developers don’t control.

I focus on three sources of distant influence on a reward-seeker: (1) human adversaries (e.g., a competing nation) who claim that they will retroactively reward or simulate the [...]

---

Outline:

(02:48) Remotely-influenceable reward-seekers

(09:03) The selection pressures that shaped the AIs cognition were local

(10:20) But distant incentives will likely avoid conflicting with immediate incentives

(11:54) Reward-seekers might follow distant incentives once immediate incentives are relatively weak

(12:52) What can we do to avoid remotely-influenceable reward-seekers?

(16:47) What can we do to mitigate risk given remotely-influenceable reward-seekers?

(19:59) Appendix: Anticipated takeover complicity doesnt require strong remote influenceability

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
February 16th, 2026

Source:
https://blog.redwoodresearch.org/p/will-reward-seekers-respond-to-distant

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Diagram showing observation box linked to three action boxes connecting to reward R.

Diagram showing observation leading to actions, then diverging paths to present developers and future AIs or humans.

Diagram showing observation leading to actions for present developers and distant AIs or humans.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Will reward-seekers respond to distant incentives?” by Alex Mallen

Powered by Wave AI