What happens if the companies racing to build superintelligence succeed before anyone knows how to control it?

In this episode, Nick sits down with Malo Bourgon, CEO of the Machine Intelligence Research Institute, to discuss why today’s chatbots are only a stepping stone, why leading AI companies are racing toward systems that could automate AI research itself, and why “letting it rip” may not be an option.

Malo explains why the real concern is not that AI becomes “evil,” but that highly capable systems may pursue goals that are not aligned with human flourishing. He also breaks down why AI systems are more like something we grow than something we program, why alignment remains unsolved, what it means when AI systems resist shutdown or hide behavior during testing, and why MIRI is working on international agreements to prevent a race to uncontrolled superintelligence.

This is not an anti-AI conversation. Malo is an aggressive user of AI agents in his own work. The question is whether we can capture the benefits of AI while avoiding the catastrophic risks of racing too quickly toward systems we do not yet understand how to steer.

Topics include AGI, superintelligence, AI agents, alignment, AI safety, MIRI’s proposed treaty, the future of work, national security, AI governance, and what everyday people can do if they are concerned about where this is headed.

Learn more about Machine Intelligence Research Institute (MIRI):
MIRI: https://intelligence.org
MIRI Governance: https://techgov.intelligence.org
If Anyone Builds It, Everyone Dies: https://ifanyonebuildsit.com

🔗 Support This Podcast by Checking Out Our Sponsors:

👉 Build your own AI Agent with Zapier for FREE: https://bit.ly/4nGriLd

👉 Test Prep Gurus: https://www.prepgurus.com
Instagram: @TestPrepGurus

Connect with The Nick Standlea Show:
YouTube: @TheNickStandleaShow
Podcast Website: https://nickshow.podbean.com/
Apple Podcasts: https://podcasts.apple.com/us/podcast/the-nick-standlea-podcast/id1700331903
Spotify: https://open.spotify.com/show/0YqBBneFsKtQ6Y0ArP5CXJ
RSS Feed: https://feed.podbean.com/nickshow/feed.xml

Nick's Socials:
Instagram: @nickstandlea
X (Twitter): @nickstandlea
TikTok: @nickstandleashow
Facebook: @nickstandleapodcast

Ask questions,
Don't accept the status quo,
And be curious.

Subscribe for more conversations on AI, learning, work, meaning, and the future.

Chapters:
00:00 The race to superintelligence
00:49 Malo Bourgon joins the show
01:11 Why people are confused about AI risk
03:11 AGI and the original goal of AI
07:07 Companies are racing to superintelligence
09:17 Dario Amodei’s 25% catastrophe concern
10:05 Why the extinction concern is not just hype
15:45 “You don’t get what you train for”
19:29 AI sycophancy and GPT-4o
23:37 AI systems resisting shutdown
27:04 The accelerationist argument
30:18 Are “doomers” actually the optimists?
31:04 Using AI tools while worrying about AI risk
32:18 Sponsor: Zapier
34:42 Violence, Sam Altman, and why peaceful action matters
36:46 MIRI’s proposed AI treaty
44:37 Buying time to solve alignment
47:33 Sudden capability jumps and Anthropic’s Mythos
52:37 Elon Musk, robotics, and concentrated power
55:18 What happens if AI automates cognitive labor?
1:00:10 Bernie Sanders, policymakers, and AI risk
1:07:46 Sponsor: Finding Your Best
1:08:39 How Malo uses AI in his own work
1:10:38 Claude Code, email, calendar, and work systems
1:14:38 Lessons from Wade Foster and Zapier
1:28:44 Social media, AI, and slow government response
1:30:36 AI as a national security issue
1:32:34 What Malo would say to world leaders
1:40:24 Where to learn more