Introduction

Some people talk about "hard-to-verify tasks" and "easy-to-verify tasks" like these are both natural kinds. But I think splitting tasks into "easy-to-verify" and "hard-to-verify" is like splitting birds into ravens and non-ravens.

Easy-to-verify tasks are easy for the same reason — there's a known short program that takes a task specification and a candidate solution, and outputs a score, without using substantial resources or causing undesirable side effects.
By contrast, "hard-to-verify tasks" is a negative category — it just means no such program exists. But there are many kinds, corresponding to different reasons no such program exists.

Listing kinds of hard-to-verify tasks

I might update the list if I think of more, or if I see additional suggestions in the comments.

Verification requires expensive AI inference. A verifier exists and works fine, but each run costs enough compute that you can't afford the number of labels you'd want.
- Given two proposed SAE experiments, say which will be more informative. Running both to find out costs $100–$1000 per comparison.
- Given two research agendas (e.g. pragmatic vs ambitious mech interp), say which produces more alignment progress. Same structure, but each comparison costs millions.
Verification requires expensive human time. The verifier [...]

---

Outline:

(00:10) Introduction

(00:55) Listing kinds of hard-to-verify tasks

(04:38) Implications

---

First published:
April 20th, 2026

Source:
https://www.lesswrong.com/posts/NEscrkxr9SxHpGayB/9-kinds-of-hard-to-verify-tasks

---

Narrated by TYPE III AUDIO.