The US and China are racing to build AI systems that could lead to superintelligence before any enforceable global rules exist to inspect them, limit them or verify what they can do.

Mark and Jeremy Think On Paper about The Hard Question of AI, an essay by Planet CEO Will Marshall. Marshall argues that AI governance now needs the same kind of verification, trust and global protection as nuclear weapons.

That means more than voluntary safety tests and China and America laying down their AGI swords. And since that is unlikely, it could mean monitoring chips, tracking data-centre construction and energy use, inspecting frontier AI labs, protecting whistleblowers and agreeing international red lines around recursive self-improvement.

But the nuclear comparison has many challenges. Weapons-grade material is scarce and difficult to hide. Just ask Iran. AI software can spread quickly and cross borders. Any Tom, Dick or Harry could end up being the first to create superintelligence.

At the centre of the episode is one question: what would AI’s Cuban Missile Crisis look like? What event would force the US and China to cooperate? Really cooperate. Lip service won’t calm existential risk.

We also discuss Sam Altman, Dario Amodei and Demis Hassabis, the incentives behind AI safety warnings, the limits of voluntary regulation, the RAND verification framework, recursive self-improvement, the Fermi paradox and whether governments can build the institutions required to manage AI superintelligence.

Hint: they can’t.

Please enjoy the show. And don’t have nightmares.

Thinking on Paper is a technology podcast about AI, Space, quantum computing, science, and the systems shaping your life.

🏠 ⁠Buy us a beer on Substack⁠

🫵 C⁠hoose your own technology adventure ⁠

📺 ⁠Watch our beautiful faces on YouTube ⁠

🎧 R⁠emember Steve Jobs on APPLE⁠

📺 ⁠Get clips and exclusive videos on Instagram ⁠

Chapters

00:00 Trailer

01:38 AI And Existential Risk

02:40 OpenAI, Anthropic & Google