Podcast
Chain of Thought | AI Agents, Infrastructure & Engineering
Hosted by Conor Bronsdon · EN
AI is reshaping infrastructure, strategy, and entire industries. Host Conor Bronsdon talks to the engineers, founders, and researchers building breakthrough AI systems about what it actually takes to ship AI in production, where the opportunities lie, and how leaders should think about the strategic bets ahead.
Chain of Thought translates technical depth into actionable insights for builders and decision-makers. New episodes weekly.
Conor Bronsdon is an angel investor in AI and dev tools, Technical Ecosystem Lead at Modular, and previously led growth at AI startups Galileo and LinearB.
Disclaimer: All views, opinions and statements expressed on this account are solely my own and are made in my personal capacity. They do not reflect, and should not be construed as reflecting, the views, positions, or policies of my employer. This account is not affiliated with, authorized by, or endorsed by my employer in any way.
24episodes
Episodes
Newest firstAll episodes
Data Federation, Not Centralization, Is What Enterprise AI Needs
6d ago00:52:42Tap to summarizeJitender Aswani was customer zero for Presto at Meta, where a billion daily active users generated queries that took hours to return. He watched that drop to minutes, scaled the same technology at Netflix across 300 million subscribers, and now runs engineering and security at Starburst, the $3.35 billion platform built on Trino.His argument: every enterprise AI project that stalls is fighting the same hidden battle. The agents can query the model fine. They just can't reach the data. The average enterprise runs 52 to 200 data sources, and a decade of moving all of it into one lake produced ETL debt, governance problems, and pipelines that break whenever a SaaS vendor adds a column.Federation is the only model that scales with entropy.We cover:Why Presto changed what Meta could experiment on, and how that compounded product velocityWhat broke when Jitender took the same technology to enterprises running 52 to 200 data sourcesWhy centralization stopped working once data grew faster than the ability to move itWhat happened to Starburst's query volume the day they shipped an MCP serverThe FinOps agent that fired queries for 30 minutes against data it never hadHow AIDA turns ad hoc analysis into workflows using skills and MCP serversWhy a context graph is different from a knowledge graph, and why ontology decides agent accuracy(0:00) Enterprises run on 52 to 200 data sources(0:25) Intro(2:18) Customer zero for Presto at Meta(9:50) Scaling to trillions of events at Netflix(15:11) Taking Trino from Silicon Valley to 10,000 enterprises(20:24) The 2011 research that predicted conversational analytics(28:54) Why centralization can't scale with entropy(32:26) The agent query explosion and what MCP did to volume(41:44) Inside AIDA, Starburst's conversational analytics product(46:56) Context graphs versus knowledge graphs(51:17) Where to follow Jitender's workConnect with Jitender Aswani:LinkedIn: https://www.linkedin.com/in/jitenderaswani/Starburst: https://www.starburst.io/Connect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
You Can't Secure an AI Agent with Software
3w ago00:52:46Tap to summarizeCharles Guillemet is CTO of Ledger and the founder of the Donjon, Ledger's internal offensive security lab whose job is to break the company's own products before attackers do. He spent a decade in cryptography and hardware security before Ledger, including designing secure integrated circuits.His argument is blunt: you cannot secure an AI agent with software alone. As agents start moving real money, API keys and trust scopes leave no physical verification layer, and Charles makes the case that hardware has to sit in the loop.This one turned into a wide-ranging thought piece (and some debate) on what the agentic economy actually looks like, and how to stay safe inside it.We cover:Why Charles thinks "securing an AI agent" with software permissions and API keys is a false promiseThe economic asymmetry between attackers and defenders, and how AI is collapsing itHow a policy engine plus a hardware-enforced signature can delegate rights to an agent safelyWhy Charles thinks the agentic economy settles on blockchain rails over Visa and MastercardSecure elements, HSMs, and zero-knowledge proofs as execution-integrity guaranteesHow Ledger uses hardware authorization internally for passkeys, signed releases, and multisigA practical way to classify assets by threat model and match security to value(0:00) Why securing an AI agent in software alone is impossible(0:30) Delegating execution power inside your security perimeter(2:28) The attack-defense asymmetry AI is erasing(6:00) The alignment problem and delegating rights to agents(9:24) Policy engines, intents, and hardware-enforced signatures(13:19) From developer experience to agent experience(15:12) Secure elements, HSMs, and execution integrity(20:00) Zero-knowledge proofs, proving without revealing(27:24) Convincing the skeptics on agent-driven payments(34:49) Why Ledger bet on dedicated hardware(36:15) Hardware as a determinism layer for agents(38:52) How Ledger uses hardware authorization internally(43:42) Classifying assets by threat model(46:55) When attack and defense become symmetric(48:44) Deepfakes, voice cloning, and the scam wave(50:04) Closing thoughts on staying safe in the agentic economyConnect with Charles Guillemet:LinkedIn: https://www.linkedin.com/in/charles-guillemet/Twitter/X: https://x.com/P3b7_Ledger: https://www.ledger.comConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
Stop Token Maxxing: Find Where AI Actually Pays Off | Jiaona Zhang
4w ago00:57:35Tap to summarizeJiaona Zhang(JZ) is the Chief Product Officer at Laurel, where the team runs its own product on itself to see exactly where AI helps and where it doesn't. Before Laurel, JZ built products at Airbnb, Dropbox, Webflow, and Linktree, and she has taught product management at Stanford for nearly a decade.Companies are spending billions on AI tooling, but most still can't say where it returns time or revenue. Jiaona breaks down how to get that visibility, why blanket AI mandates backfire, and what it takes to re-architect a team so anyone can ship.Her argument is simple: stop token maxing and start measuring time back.We cover:Why most organizations can't see where AI is actually working, and how Laurel uses time data to fix itThe token max trap that "use AI everywhere" mandates create, and how to drive efficient use insteadWhy former managers make the best operators of agent fleetsHow Laurel lets PMs, designers, and customer success ship features end to endThe bottom-up plus top-down playbook for re-architecting a team around AIWhy technology moats are falling away while brand and data moats endureLaurel's bet on returning time to people instead of replacing them(0:00) The token max trap(1:47) Why companies can't see where AI is working(5:03) What Laurel does: turning time into data(8:53) Agents as an extension of the workforce(13:43) Why former managers make the best AI users(18:23) Lean teams and shipping end to end(22:29) Enabling non-engineers to ship features(28:30) Re-architecting teams: bottom-up and top-down(32:09) Keeping your professional identity as AI shifts work(38:53) The context layer is the new race(42:06) Fundamentals plus tinkering: how to learn(48:45) Brand and data moats when tech moats fall away(54:31) Laurel's movement: returning time to peopleConnect with Jiaona Zhang(JZ):LinkedIn: https://www.linkedin.com/in/jiaona/Laurel: https://www.laurel.ai/JZ's Linktree: https://linktr.ee/jzConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
Most of the Web Will Never Get APIs for AI Agents | Dhruv Batra
Jun 1800:54:44Tap to summarizeMost of the web will never get APIs for AI agents. School district sites, small business pages, government offices, and the long tail of e-commerce were built for humans, and they will keep working that way for years. So how do agents actually get things done across the web?Dhruv Batra is co-founder and chief scientist of Yutori, the company building specialized browser and computer-use agents. He previously led embodied AI at Meta's FAIR lab, training robots in simulation and shipping the image question-answering model on Ray-Ban Meta glasses. His bet: the web is a shared roadway, much like roads split between human drivers and self-driving cars, and agents will be built to use it the way people already do.Pixels in, clicks out. That is the API.In this conversation:Why the long tail of the web won't re-architect itself for agentsHow Yutori's Navigator perceives pixels and writes JavaScript on the fly to shorten task trajectoriesWhy Navigator runs 2-3x faster and 4-5x cheaper than Opus 4.7 and GPT-5.5 on browser tasksLearning from live websites, and using URL query parameters as privileged verifiers instead of cloning sitesWhat the shift from American to Chinese open-weight models means for startupsHow smart glasses and robots share the same perception-action loopWhy demand for inference compute is pushing models smaller and onto devicesChapters:(00:00) Pixels in, clicks out (01:37) Why most of the web will never get APIs (08:47) Aggregation, specialization, and human friction (11:39) Digital niches and specialized models (16:41) The web's heavy tail and where browser agents win (20:40) Inside Yutori's Navigator and Scouts (24:08) N1.5: writing JavaScript to cut trajectory length (27:45) Training on live websites (33:29) Open source: FAIR's legacy and the Chinese frontier (37:22) Agent frameworks: OpenClaw, Hermes, heartbeats (40:57) How non-technical users adopt agents (44:25) Smart glasses, robotics, and embodied AI (50:57) Compute demand and smaller on-device models (53:12) Why the company is called YutoriConnect with Dhruv Batra:LinkedIn: https://www.linkedin.com/in/dhruv-batra-dbatra/X/Twitter: https://x.com/DhruvBatra_Yutori: https://yutori.comConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
The First Fully Autonomous AI Attack Is 18 Months Away | Kristin Lovejoy
Jun 1100:45:36Tap to summarizeKristin "Kris" Lovejoy has spent her career inside the systems the global economy runs on: banks, hospitals, energy grids, governments. Today she is Global Head of Strategy at Kyndryl, the world's largest IT infrastructure services provider, working with mission-critical enterprises across more than 60 countries. Before that she ran security businesses at EY and IBM, founded the AI security company BluVector (acquired by Comcast), and now sits on the board of Dominion Energy.Her prediction: the first fully autonomous AI attack, where an AI takes down an enterprise network with no human driving it, lands within 18 months.Conor and Kris dig into why 62% of enterprise AI initiatives are still stuck in pilots even as spend climbs 33% year over year, why attackers chaining low-risk vulnerabilities changes the patching math, and why she has a fraught relationship with policy as code.We cover:The electricity analogy: we can build the models, but the transmission lines for industrial AI don't exist yetProductivity AI vs mission-critical AI, and why banks and healthcare systems aren't running agentic AI at production scaleWhy deterministic policy as code clashes with autonomous systems, and "human on top" vs human in the loopThe 18-month prediction: chaining low-risk vulnerabilities, outcome-oriented agents that take systems down by accident, and insiders armed with AI attack toolsThe data center build-out from a Dominion Energy board member: PJM load forecasts that miss by double digits every year, water use, density, and rack optimizationPrivacy as a double-edged sword: data combinations that suddenly become PII and the shift to continuous complianceWhat's next: open source everywhere, sovereignty as control, autonomous robotics, and quantumChapters:(00:00) Meet Kris Lovejoy: Kyndryl, EY, IBM, and Dominion Energy (02:09) Why 62% of AI initiatives are stuck in pilots (03:07) The electricity analogy: models without transmission lines (04:23) Productivity AI vs mission-critical AI (06:53) Vintage systems, hybrid data, and the risk gap (11:03) Policy as code and "human on top" (16:25) Data centers, energy, and the grid build-out (24:44) Data center design: density, cooling, rack optimization (26:54) Privacy, continuous compliance, and sovereignty as control (32:06) The first fully autonomous AI attack: 18 months away (38:06) Predictions: open source, robotics, and quantum (42:32) Control planes for agentic AI: closing thoughtsConnect with Kris Lovejoy:LinkedIn: https://www.linkedin.com/in/klovejoy/Kyndryl: https://www.kyndryl.comConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
The AI Framework Era Is Over: Why Context Is the Moat | Jerry Liu
Jun 300:52:42Tap to summarizeJerry Liu built one of the most installed pieces of AI plumbing of the last three years. LlamaIndex became the indexing and retrieval layer a whole generation of RAG apps were stitched together with. Then he started arguing that the framework era he helped create is over.Jerry is co-founder and CEO of LlamaIndex. In this conversation he walks through the company's pivot from open-source framework to managed document infrastructure with LlamaCloud and LlamaParse, and why he is betting that context quality is the one moat that compounds as agent loops get good enough to absorb the scaffolding.If you are a founder worried a frontier lab or a coding agent is about to eat your product, this is the playbook for reinventing your ICP without losing the thread.In this conversation:Why Jerry says the AI framework era is over, and what actually survivesHow agent harnesses like Claude Code collapsed the old framework patterns into the modelWhy context quality is the durable moat, not the agent loopHow LlamaParse beats legacy OCR and frontier models on document accuracy and costWhy 95%+ accuracy is the real bar for legal, insurance, and financial document workHow LlamaIndex disrupted its own product and reinvented its ICP to stay aliveJerry's take on agent memory, model personalities, and why LLMs are still bad writers(0:00) Is the AI framework era over? (1:56) What died and what survived (6:31) Why context quality is the moat (8:12) Defining the context layer (13:18) Coding and vision as the abstraction layer (18:13) The bet that context compounds (23:59) Which verticals are adopting (25:14) Why 95%+ accuracy is the real bar (29:49) The file system as an agent primitive (34:33) Surviving your own pivot (37:15) Reinventing strategy and hiring (42:00) Agent memory as persistent context (44:41) Model personalities and cultural memory (47:51) Writing with AI (50:19) Closing thoughtsConnect with Jerry Liu:LinkedIn: https://www.linkedin.com/in/jerry-liu-64390071/Twitter/X: https://x.com/jerryjliu0LlamaIndex: https://www.llamaindex.aiLlamaIndex careers: https://www.llamaindex.ai/careersConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
We Built Agents, Nobody Built HR | Tyler Akidau, Redpanda
May 2700:50:50Tap to summarizeTyler Akidau spent 12 years on streaming systems at Google and five years at Snowflake before joining Redpanda as CTO. He wrote the O'Reilly Streaming Systems book most of the field has on its shelf. His new piece on O'Reilly Radar (Post-Human: We All Built Agents, Nobody Built HR) argues that enterprises are stuck in the prototype-to-production gap because they're applying human-era identity, auth, and observability tools to a workforce that's unpredictable in structurally novel ways, runs at machine speed, and follows bad instructions to a fault. Inline guardrails like CLAUDE.md work until they don't. Governance has to be enforced through channels the agent can't see, modify, or override.We cover:Why AI agents are a new kind of co-worker (unpredictable, machine-speed, directable to a fault) and what that means for enterprise infrastructureThe four pillars of agent governance: identity, authorization, observability and explainability, accountability and controlWhy task-scoped, short-lived identity is the foundation everything else builds onAuthorization that's deny-capable and intersection-aware (Tyler's "guest badge" model)Why OpenTelemetry is the right starting point for recording every prompt, tool call, and responseHow Redpanda's Agentic Data Plane combines streaming topics, Oxla SQL, and Postgres under the hoodTyler's academic paper with a psychologist on the neurobiological systems humans have that AI agents are missingChapters:(00:00) Why nobody built HR for AI agents(02:12) Three ways agents differ from human employees(07:53) The four pillars of out-of-band governance(10:29) Identity: task-scoped, short-lived, chained to humans(14:40) Authorization: deny-capable and intersection-aware(18:57) Observability: record everything via OpenTelemetry(24:24) Redpanda's agents and the $1,000 trade limit example(30:10) Accountability and the kill switch(34:02) The Agentic Data Plane: streaming, Oxla SQL, Postgres(41:20) Should we stop chasing model alignment?(44:04) Building human-like value systems into agents(47:25) Tyler's 12-24 month outlook for agent governanceConnect with Tyler:LinkedIn: https://www.linkedin.com/in/takidau/Redpanda: https://www.redpanda.com/Post-Human article: https://www.oreilly.com/radar/posthuman-we-all-built-agents-nobody-built-hr/ Connect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show
Transcribe →
How Superhuman Built AI Into a 100ms Product | Loïc Houssier
May 2200:50:11Tap to summarizeLoïc Houssier leads engineering at Superhuman, the email client Grammarly acquired for ~$825 million in July 2025. Before Superhuman he was CTO of OpenTrust (acquired by DocuSign), ran engineering at ProductBoard, and started his career in applied cryptography for France's defense industry, including work on nuclear submarine systems. Loïc joined Superhuman in early 2024 and within 30 days was leading a six-week sprint to ship AI Inbox.Superhuman's brand is built on speed: every interaction under 100 milliseconds. LLMs do not run in 100 milliseconds. So Loïc walks Conor through how his team retrofitted AI into a product that was already winning without it: pre-caching context for the mobile voice feature, starting every feature on the smartest available model and only then fine-tuning down to cheap dedicated infrastructure, treating "look foolish" as a P0 bug class, and refusing to auto-send any email even when their agents could.This is a practitioner's tour of what it actually takes to put AI on top of a product that has to stay fast, stay quiet, and never embarrass the user.We cover:The model-routing strategy: Opus and frontier models to prove a feature, then fine-tuned BERT classifiers on dedicated inferencePre-caching voice and tone context separately from dictation to keep the mobile voice feature feeling fastWhy eval engineering at Superhuman is owned by PMs, and how a single "how much time did I spend in Waymo last month" query exposes the eigenvectors a feature has to coverWhy "look foolish" is a P0 bug class, and where the boundary between agent agency and agent laziness actually sitsHow Superhuman's pod structure (PM, tech lead, designer) and a central AI platform team support aligned autonomyHiring for AI fluency: how interview questions are changing and what self-augmenting engineers look likePattern detection as the leadership skill that transfers from nuclear submarines to AI emailChapters:(00:00) Cold open: pattern detection beats new tools (00:18) Loïc's path: cryptography, OpenTrust, ProductBoard, Superhuman (02:13) Retrofitting AI into a 100ms product (04:08) Voice on mobile: pre-caching LLM context to keep the feel fast (07:46) Frontier first, then fine-tune: model strategy across features (11:04) The "double-dipping" trick that worked on GPT-4 and stopped working (12:25) Cognitive load and staying current as a leader (16:59) Balancing YC founder urgency with peer CTO grounding (19:28) Pods, AI Guild, and aligned autonomy (23:15) Managing models vs. managing people: delegation in reverse (28:27) The Waymo example: eigenvectors of evaluation (32:15) Day 30 onboarding: leading the AI Inbox sprint (35:04) Why email is the killer agent use case (38:51) Auto-draft, never auto-send (39:57) Agent agency vs. agent laziness (43:07) Hiring for AI fluency (45:55) Pattern detection is the leadership skill (47:21) Nuclear submarines as engineering reference points (48:37) Closing thoughts (49:38) Superhuman is hiringConnect with Loïc:LinkedIn: https://www.linkedin.com/in/houssier/Superhuman careers: https://superhuman.com/careersSuperhuman: https://superhuman.comConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.showThanks to Galileo — download their free 165-page guide to mastering multi-agent systems at galileo.ai/mastering-multi-agent-systems
Transcribe →
The AI Hiring Doom Loop: Applications Up 239%, Hires Down 75%
May 600:57:07Tap to summarizeJob applications are up 239% since ChatGPT launched, tech layoffs show no signs of slowing down, and the market for technical talent is a topsy turvy mess. Greenhouse has a unique vantage point to understand all of this: they process 22 million job applications a month across 7,500+ companies including HubSpot, Anthropic, Coinbase, and the NFL. CEO Daniel Chait has had a front-row seat to the strangest hiring market in decades, and he's here to advise us all on how to navigate it.Daniel coined the term "AI doom loop" for what's happening: applications up 239% since ChatGPT launched, resume hacks like white-fonting and prompt injection up 500%, and 75% fewer applications reaching the hire stage. 91% of recruiters have spotted candidate deception. 38% of job seekers walk away from processes that include an AI interview.It's the worst job market for candidates and the hardest hiring market for recruiters.Daniel explains how technical talent can break the loop.We cover:Why software engineers, according to Greenhouse data, are the worst auto-appliers and what to do insteadThe North Korean infiltration problem: deepfakes, laptop farms, and why companies are flying candidates in for in-person interviews againHow AI screener interviews open up the funnel when companies are transparent about using them, and break it when they aren'tGreenhouse Dream Jobs: how a single high-signal application a month converts at 5x the rateWhy take-home assignments don't survive contact with AI and what Greenhouse uses insteadWhat a coding interview looks like when leetcode is dead and engineers run 10+ Claude Code sessions in parallelThe case for killing the resume entirely and rebuilding hiring around AI conversationsChapters: (00:00) Cold open: 239% more applications, 75% fewer hires (02:14) Galileo (03:05) The AI doom loop, defined (04:01) How we got here: remote work, ZIRP, and ChatGPT (07:51) Are software engineering jobs really in trouble? (12:46) The trust crisis: 91% of recruiters spot deception (15:52) North Korean spies, deepfakes, and laptop farms (19:34) Can AI fix the problem it created? (20:52) AI screener interviews and the uncanny valley (26:33) Greenhouse Dream Jobs: one signal, 5x conversion (28:31) Why auto-apply doesn't work (and what does) (30:18) Communities, building in public, and the early-mover advantage (37:08) Gen Z lost trust, and the bias problem (39:04) Kill the resume: rethinking hiring from scratch (43:34) How Greenhouse changed its own interview process (48:47) Coding interviews in the agent era: leetcode is dead (51:33) Predictions: more proof, more conversations, less noise (54:34) Where job seekers and hiring teams should startConnect with Daniel:Greenhouse: https://www.greenhouse.comMy Greenhouse (for job seekers): https://www.mygreenhouse.comLinkedIn: https://www.linkedin.com/in/dhchait/Connect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.showThanks to Galileo — download their free 165-page guide to mastering multi-agent systems at galileo.ai/mastering-multi-agent-systems
Transcribe →
Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI
Apr 2900:42:40Tap to summarizeAlex Ratner co-founded Snorkel AI out of Chris Ré's Stanford lab and helped establish data-centric AI as a field. Today, Snorkel is a $1.3B company shipping thousands of data sets and environments a week to frontier labs and vertical AI teams like Harvey.In this conversation, he argues our ability to build AI agents has outpaced our ability to measure them. That gap is what's keeping most enterprise agents stuck in demo purgatory.If you can't measure it, you can't improve it. And you can't deploy it.In this conversation:The three axes of the evaluation gap: input complexity, autonomy horizon, and output complexityBig Law Bench: how Snorkel and Harvey benchmarked legal agents on deep-research tasks that take lawyers 10-15 hoursWhat Snorkel's $3M Open Benchmarks Grant is funding, and why "benchmaxxing" critiques don't kill the case for public benchmarksWhy 40-50% of Snorkel's data work is still review and labeling, even with the best models in the loopThe "expert-agentic" era, where domain expertise (law, finance, coding, even woodworking) is the new bottleneckWhy self-supervision is a dead end outside narrow cases like distillationThe false dichotomy between data and environments, and why pure-environment vendors miss how AI actually worksChapters(00:00) Intro: Alex Ratner and Snorkel AI (02:50) What the evaluation gap actually is (06:05) Moravec's paradox and the jagged frontier (08:46) Where AI agents fall down in enterprise work (10:40) Big Law Bench: benchmarking Harvey's legal agents (12:00) The three axes: input, autonomy horizon, output (18:31) Snorkel's $3M Open Benchmarks Grant (22:33) From "janitorial" to epicenter: 15 years of data-centric AI (29:26) The expert-agentic data era (34:54) The false dichotomy between data and environments (40:05) DoorDash Tasks and expert data at scaleConnect with Alex Ratner:X/Twitter: https://x.com/ajratnerSnorkel AI: https://snorkel.aiConnect with Chain of Thought host Conor Bronsdon:Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.showThanks to Galileo — download their free 165-page guide to mastering multi-agent systems at galileo.ai/mastering-multi-agent-systems
Transcribe →