1.1 Tl;dr

Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people's agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human's goals are themselves under-determined and manipulable, and it's awfully hard to pin down a principled distinction between changing people's goals in a good way (“providing counsel”, “providing information”, “sharing ideas”) versus a bad way (“manipulating”, “brainwashing”).

The manipulability of human desires is hardly a new observation in the alignment literature, but it remains unsolved (see lit review in §3 below).

In this post I will propose an explanation of how we humans intuitively conceptualize the distinction between guidance (good) vs manipulation (bad), in case it helps us brainstorm how we might put that distinction into AI.

…But (spoiler alert) it turns out not to really help, because I’ll argue that we humans think about it in a deeply incoherent way, intimately tied to our scientifically-inaccurate intuitions around free will.

I jump from there into a broader review of every approach that I can think of for writing a “True Name” for manipulation or [...]

---

Outline:

(00:13) 1.1. Tl;dr

(02:04) 1.2. Bigger-picture context: why is this issue so important to me?

(04:48) 2. How do humans intuitively define empowerment, agency, manipulation, etc.?

(04:56) 2.1. Background: human free will intuitions

(09:20) 2.2. Our free-will-infused intuitive notions of empowerment, agency, manipulation, corrigibility, responsibility, etc.

(12:00) 2.3. Another dimension: counsel vs manipulation as an emotive conjugation

(13:07) 3. If the intuitive definitions of manipulation etc. reside in a messed-up ontology, has the alignment literature found any alternative, better way to define these concepts?

(13:49) 3.1. Compare what the human wants to what the human would want under the null policy?

(15:32) 3.2. The AI learns self-empowerment and generalizes to other-empowerment?

(17:14) 3.3. Vingean agency?

(19:03) 3.4. The AI doesnt care about (is not optimizing for) what the human winds up wanting?

(21:01) 3.5. Impact minimization?

(21:44) 3.6. Attainable utility preservation?

(22:03) 4. Even more ideas (that dont really solve my problem)

(22:15) 4.1. Game theory and incentive design?

(22:47) 4.2. The persons judgments of what kinds of interactions are good vs bad?

(24:14) 4.3. Its a messed-up ontology, but who cares?

(25:35) 5. ...But doesnt this analysis equally disprove the possibility of human helpfulness?

(30:14) 6. Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/vzHtHHBJoKATi5SeK/empowerment-corrigibility-etc-are-simple-abstractions-of-a

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Political cartoon showing opposing sides labeling each other's views negatively.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.