Q1: What are you saying?

A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that's choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that's just an utterly terrifying thing that you’re doing. You’re playing around with algorithms that, if they work at all, would tend to create ruthless, callous AGIs, AGIs which would happily exterminate humanity and run the world by themselves, given an opportunity.

Mercifully, large language models (LLMs) today are not in the category of “algorithms that choose actions via RL & search”. At least, not primarily—see LLMs are (still) mostly powered by imitative learning, not RL. So LLMs are outside the scope of this post. However, lots of other researchers and companies around the world are enthusiastically trying to build AGI in the maximally terrifying way, as we speak.

Q2: So you’re saying, don’t build AGI based on RL and/or search & planning?

A: In principle, it's entirely possible that something is terrifying, but we should do it anyway.

…Like space travel! Space travel is: “Let's fill a tank with 1000 tons of the most flammable substance imaginable, and then light it [...]

---

Outline:

(00:21) Q1: What are you saying?

[... 13 more sections]

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/KHyBocZncAmtu4Jbc/rl-and-search-is-a-terrifying-way-to-build-agi-an-faq

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Screenshot excerpt from a much longer list of specification-gaming examples from the literature.

Comic: goose chasing person, asking about pseudocode reward function.

Images that maximize an output of a learned image classifier. (These are subject to a constraint that most nearby pixels are similar; if we drop that constraint, the results just look like random static.) Source: “Inceptionism” blog post by A. Mordvintsev, C. Olah, M. Tyka (2015)

Excited man beside reinforcement learning algorithm labels.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.