Sean McGregor and I discuss about why evaluating AI systems has become so difficult; we cover everything from the breakdown of benchmarking, how incentives shape safety work, and what approaches like BenchRisk (his recent paper at NeurIPS) and AI auditing aim to fix as systems move into the real world. We also talk about his history and journey in AI safety, including his PhD on ML for public policy, how he started the AI Incident Database, and what he's working on now: AVERI, a non-profit for frontier model auditing.

Chapters

(00:00) - Intro
(02:36) - What's broken about benchmarking
(03:41) - Sean’s wild PhD
(14:28) - The phantom internship
(19:25) - Sean's journey
(22:25) - Market-vs-regulatory modes and AIID
(32:13) - Drunk on AI progress
(38:34) - BenchRisk
(43:20) - Moral hazards and Master Hand
(50:34) - Liability, Section 230, and open source
(59:20) - AVERI
(01:11:30) - Closing thoughts & outro

Links

Sean McGregor's website
AVERI website

BenchRisk

BenchRisk website
NeurIPS paper - Risk Management for Mitigating Benchmark Failure Modes: BenchRisk
NeurIPS paper - AI and the Everything in the Whole Wide World Benchmark

AIID

AI Incident Database website
IAAI paper - Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database
Preprint - Lessons for Editors of AI Incidents from the AI Incident Database
AIAAIC website (another incident tracker)

Hot AI Summer

CACM article - A Few Useful Things to Know About Machine Learning
CACM article - How the AI Boom Went Bust
Undergraduate Thesis - Analyzing the Prospect of an Approaching AI Winter
Tech Genies article - AI History: The First Summer and Winter of AI
CACM article - There Was No ‘First AI Winter’

Measuring Generalization

Neural Computation article - The Lack of A Priori Distinctions Between Learning Algorithms
ICLR paper - Understanding deep learning requires rethinking generalization
ICML paper - Model-agnostic Measure of Generalization Difficulty
Radiology Artificial Intelligence article - Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls
Preprint - Quantifying Generalization Complexity for Large Language Models

Insurers Exclude AI

Financial Times article - Insurers retreat from AI cover as risk of multibillion-dollar claims mount
Tom's Hardware article - Major insurers move to avoid liability for AI lawsuits as multi-billion dollar risks emerge — Recent public incidents have lead to costly repercussions
Insurance Newsnet article - Insurers Scale Back AI Coverage Amid Fears of Billion-Dollar Claims
Insurance Business article - Insurance’s gen AI reckoning has come

Section 230

Section 230 overview
Legal sidebar - Section 230 Immunity and Generative Artificial Intelligence
Bad Internet Bills website
TechDirt article - Section 230 Faces Repeal. Support The Coverage That’s Been Getting It Right All Along.
Privacy Guides video - Dissecting Bad Internet Bills with Taylor Lorenz: KOSA, SCREEN Act, Section 230
Journal of Technology in Behavioral Health article - Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice
Time article - Lawmakers Unveil New Bills to Curb Big Tech’s Power and Profit
House Hearing transcript - Legislative Solutions to Protect Children and Teens Online

Relevant Kairos.fm Episodes

Into AI Safety episode - Growing BlueDot's Impact w/ Li-Lian Ang
muckrAIkers episode - NeurIPS 2024 Wrapped 🌯

Other Links

Encyclopedia of Life website
IBM Watson AI XPRIZE website
ML Commons website
Wikipedia article