Brew Markets: "Clinical Trials For AI?" with Prof. Karim Lakhani

Podcast: Brew Markets (Morning Brew)
Host: Ann Berry
Guest: Professor Karim Lakhani, Harvard Business School
Date: October 24, 2025

Episode Overview

This episode explores the unprecedented scale of investment in artificial intelligence (AI)—projected at $1.5 trillion in 2025—and what that means for productivity, ROI, and the future of work. Host Ann Berry speaks with Prof. Karim Lakhani about his research pioneering "clinical trials" for AI in real business settings. They discuss how AI deployments are measured, their true impact on organizations, challenges in change management, and what the future might hold for workers, companies, and ethical standards in an AI-powered economy.

Key Discussion Points & Insights

1. Setting the Stage: The AI Investment Boom

• Massive global spend: $1.5 trillion on AI in 2025, expected to hit $2.2 trillion next year.
• Spend is concentrated in companies like Microsoft, Alphabet, Amazon, Meta, and OpenAI.
• AI is being integrated across all industries and functions—not just headline tech giants.
• Pressing questions: Is all this AI spend productive? Are organizations and workers ready? ([00:50])

2. The Concept of "Clinical Trials" for AI

• Prof. Lakhani likens generative AI to a new drug: "We actually don't know the right dose to take, the efficacy of the product, the side effects, the toxicity, or even the right diet to follow when we're using this drug." ([04:11])
• To get real answers, his team runs randomized, controlled studies: Some people get access to new AI tools; others do not; then they measure impact.

  "We basically think of ourselves as clinical trial specialists. We come in, we randomize who gets access to the tools and who don't. And then we study what happens." — Prof. Karim Lakhani [04:42]

3. Procter & Gamble (P&G) Clinical Trial: AI as a Team Member

• Study focus: Early-stage R&D and commercial teams, with and without AI assistance.
• Findings:
  1. AI improves productivity.
  2. AI fills missing expertise between collaborators (e.g., marketing getting R&D insights).
  3. People feel better working with AI: Contrary to popular concerns, participants reported more positive emotions and fewer negative emotions.

  "AI is transforming from a tool to a teammate." — Prof. Lakhani [06:00] "People have more positive emotions and less negative emotions when they're using AI." [06:50]

4. Boston Consulting Group (BCG) Clinical Trial: The Jagged Frontier

• 800 consultants studied, testing both creative and analytical (spreadsheet-based) tasks.
• Key results:
  • Creative tasks: AI boosted performance by ~40% (median performers advanced to 95th percentile).
  • Analytical/spreadsheet tasks: AI led to a 20% drop in performance two years ago (due to hallucinations/errors).

  "AI has this jaggedness... in some cases it'll be incredible, in other cases they'll have sharp edges and you'll bleed." — Prof. Lakhani [09:36]

  • Tools have improved since, but new "jagged" risks constantly emerge.

5. ROI and Organizational Challenges

• The "J Curve" effect: When new tech is introduced, performance often dips before rising as organizations learn to integrate it effectively.
• Massive productivity wins possible (e.g., reducing creative message development from 100 hours to 1.5 hours), but only if workflows are transformed—not merely automated.

  "If you come in blindly saying I'm gonna expect ROI without changing workflows, you're gonna be in a load of trouble." — Prof. Lakhani [11:50]

6. Change Management & Organizational Culture

• True gains: ~30% from technology, ~70% from leadership and change management.

  "Tech for me is actually getting easier. And 70% [of success] is change management, leadership, providing the right mechanisms, rethinking jobs, rethinking workflows..." — Prof. Lakhani [12:34]

• Leaders must "get their hands dirty": Executives need to personally use AI tools to understand their capabilities and limits.

  "You're the only person that will know how to use these tools to the best of your capabilities and improve your capabilities." [13:03]

7. The AI Boss & AI-Native Organizations

• Today’s algorithmic management: Uber, Instacart, etc.—millions already answer to algorithms for task allocation and evaluation.

• Ant Financial (China): 1.2 billion users, 10,000 employees; everything designed to be instant and touchless (the “3-1-0 mantra”: three minutes, one minute approval, zero human intervention).

  "The basis for competition is not people doing tasks, it's organizing machines to do all the tasks." — Prof. Lakhani [18:45]

• Question: How soon will we see the "one-person unicorn"?

  "I would say at least a decade before that shows up at that level of scale." — Prof. Lakhani [20:17]

8. Shifting Skills and Brain Drain

• Software used to “eat the world,” but AI threatens even established software giants like Salesforce and Adobe.
• AI brain drain: Top computer science talent is moving from academia to industry due to the data/computation required.

  "The bleeding edge of computer science is in companies and not inside of academia. And that creates a big risk for us." — Prof. Lakhani [22:43]

• Long-term expectation: Academia will differentiate, focusing more on foundational ethics and next-generation problems.

9. Ethics, Content, and Intellectual Property

• Academia should become the “soul and consciousness” of AI, focusing on ethical deployment and workforce impact.

  "We can't just be handmaidens of AI. We actually have to be the soul and the consciousness of AI." — Prof. Lakhani [25:26]

• Concerns about content misuse: e.g., OpenAI’s opt-out (not opt-in) for IP usage, lack of transparency.

  "Opt out seems, seems not the most ethical way to deal with this kind of, it's less transparent." — Ann Berry [28:01]

• Paths forward may look like the evolution in music (Napster ➔ iTunes ➔ Spotify).

10. Are We in an AI Bubble?

• Prof. Lakhani hesitates to declare a bubble, but notes the risk/reward is organizational:

  • Technology is improving at an exponential rate, while many organizations adopt only linearly, creating a widening gap between tech potential and realized benefits.

  "If you don't stay on the same exponential curve, you're going to be linearly behind." — Prof. Lakhani [29:53]

• Companies leading the way: J.P. Morgan (integrating AI at the highest executive level) and Procter & Gamble (deploying AI throughout supply chain, marketing, R&D).

  "The average firm executive team is mostly lost today." — Prof. Lakhani [32:03]

Notable Quotes & Memorable Moments

• "Gen AI to me is like a drug that has been introduced to our economy... we actually don't know the right dose to take, the efficacy... or even the right diet to follow when we're using this drug."
  — Prof. Karim Lakhani [04:11]
• "AI is transforming from a tool to a teammate." [06:00]
• "AI has this jaggedness... in some cases it'll be incredible, in other cases they'll have sharp edges and you'll bleed." [09:36]
• "It's 30% tech... 70% change management, leadership." [12:34]
• "If you come in blindly saying I'm gonna expect ROI without changing workflows, you're gonna be in a load of trouble." [11:50]
• "We can't just be handmaidens of AI. We actually have to be the soul and the consciousness of AI as well." [25:26]
• "The average firm executive team is mostly lost today." [32:03]

Timestamps by Section

| Segment | Description | Timestamp | |---------|-------------|-----------| | Introduction and AI spend context | Market landscape, why test AI before investing | 00:50–04:11 | | Clinical trials for AI explained | Prof. Lakhani’s research approach | 04:11–05:16 | | P&G clinical trial | Study design & findings | 05:56–07:20 | | BCG clinical trial & jagged frontier | AI sharply helps/harms certain tasks | 07:30–10:06 | | ROI, workflow change, and J curve | Performance dips, workflow redesign | 10:17–12:21 | | Change management | 30/70 tech to management; leaders must use AI | 12:34–14:01 | | AI as a boss and Ant Financial | Algorithmic management, AI-native firms | 15:20–19:55 | | One-person unicorns | Timing, shift in startup dynamics | 20:14–21:53 | | Brain drain and academia | Shifting role, risk, and differentiation | 22:29–24:49 | | Ethics and IP | Academia’s ethical role, market-based solutions | 25:02–28:37 | | Are we in an AI bubble? | Exponential tech, linear absorption, company examples | 29:13–32:03 | | Closeout | | 32:03–end |

Overall Tone & Style

The conversation is both candid and analytical, combining business pragmatism ("put your ruthless money hat on") and academic rigor. Prof. Lakhani uses clear metaphors —AI as a drug, the "jagged frontier," the "J curve"—to explain complex phenomena accessibly. The mood is optimistic but cautionary, emphasizing that realized value from AI will depend on much more than the technology itself.

Summary Takeaways

• AI’s biggest benefits are unlocked not merely through adoption but through business process and culture transformation.
• Measured, experimental "clinical trials" are necessary before AI's potential can be responsibly and profitably tapped in organizations.
• AI delivers spectacular gains in certain tasks (creativity) but can hinder others unless carefully managed—a "jagged" performance landscape.
• The ethical, organizational, and societal implications loom large. Academia and industry must collaborate, but also maintain critical distance and advocacy for responsible deployment.
• Companies leading today (e.g., J.P. Morgan, P&G) pair technical investment with top-down integration and change management.
• The greatest risk to most firms: falling exponentially behind as AI tools improve far faster than most can adapt.

Recommended for: Executives, investors, technology strategists, and anyone interested in AI's real impact on the way we work and compete.