Podcast Summary: HBR On Leadership

Episode: Cultivating an Experimental Mindset in Your Organization

Release Date: October 1, 2025
Host: Amanda Kirsey, Eric Jorgenson
Guest: Stefan Tomke, Harvard Business School Professor
Topic: Building a culture of experimentation to drive innovation and better decision-making in organizations

Overview

This episode focuses on how leaders can foster an experimental mindset within their organizations. The conversation, featuring Stefan Tomke—author of "Experimentation: The Surprising Power of Business Experiments"—dives deeply into why disciplined experimentation leads to better business decisions, what blocks widespread adoption of experimentation, and how companies of all sizes can practically embed experimentation into their culture.

Key Discussion Points & Insights

1. The Value and Power of Business Experiments

• Experiments as Engines of Value: Experiments can deliver surprising and often substantial value by testing assumptions rather than relying solely on intuition.
• Example from Microsoft Bing:
  • A minor change by an employee resulted in over $100 million in additional annual revenue, demonstrating the unexpected benefits of empowering employees to experiment.
  • "It was the most successful experiment that was run at Bing. So what made the difference? The employee had the power or the authority to run the experiment." — Stefan Tomke (03:36)

2. Barriers to Widespread Experimentation

• Cultural Resistance: Many managers are risk-averse or reluctant to admit uncertainty.
• Lack of Infrastructure & Skills: Some organizations lack tools, resources, or encouragement to test ideas rigorously rather than running informal "trials."
• Overestimated Risks, Underestimated Upsides:
  • Managers often worry more about potential downsides (losing traffic, negative customer reactions) than the possible gains.
  • It requires humility to say, "I don't know," and let experiments speak for themselves.
  • "The higher up you go, the more you get paid...you want to be a decision maker. But we get it wrong most of the time." — Stefan Tomke (05:49)

3. Rigorous Experimentation: Not Just for Tech Giants

• Applicability Beyond Digital: Controlled experiments are critical in offline settings too (e.g., retail opening hours at Kohl’s).
  • "Calculating the cost savings is easy, but the big question is, what's actually going to happen to our revenue?...We won't know until we actually do it." — Stefan Tomke (06:38)

4. Tools and Infrastructure: Doing Experimentation at Scale

• From Custom Tools to Third-party Solutions:
  • Large companies (Amazon, Microsoft, Netflix, Booking.com) built their own experimentation platforms.
  • Now, scalable third-party tools make robust experimentation more accessible, even for brick-and-mortar organizations.
  • "It's gotten a lot easier than if you wanted to start five or ten years ago." — Stefan Tomke (08:17)

5. Building a True Culture of Experimentation

• Symptoms of a Poor Culture:
  • Results of experiments are ignored or rationalized away if they don't fit management intuition.
• Cultural Transformation:
  • At Booking.com, experiments are a daily routine, running over 30,000 per year, and the entire organization is empowered to generate hypotheses.
  • Small companies with less data can still experiment—just with larger, riskier changes to achieve statistically meaningful results.
  • "At Booking.com...running experiments is like breathing. You do it every single day." — Stefan Tomke (11:10)

6. Experimentation in Unlikely Industries

• Creative Industries & Entertainment:
  • Even highly creative fields can benefit, e.g., Netflix uses rigorous testing to guide content and user experience decisions.
  • Experiments add transparency and challenge assumptions—even when "creative genius" dominates.
  • "Creative talent is really important, but that doesn't create certainty...experimentation allows us to create hypotheses and then rigorously test them." — Stefan Tomke (14:01)

7. Counteracting ‘HiPPO’ Culture

• HiPPO: "Highest Paid Person’s Opinion"
  • Too many organizations default to following whoever argues most convincingly—rather than real evidence.
  • "We call them hippos. And everyone knows hippos are very dangerous animals." — Stefan Tomke (15:44)

8. Scaling Experimentation Across the Organization

• How Much Experimentation is Enough?
  • Most companies under-experiment: The concern should be increasing testing, not avoiding excess.
  • "Right now, most organizations test too little. So I don't think you should be too worried about testing too much." — Stefan Tomke (17:33)
• Organizational Models:
  • Centralized: A single team runs experiments for others; good for starting out and maintaining expertise.
  • Decentralized: Empowering everyone; fastest for scaling but can dilute best practices.
  • Center of Excellence: Hybrid—central capability-building plus distributed experimenters; found to work best at scale.
  • "Companies found that's actually a very good compromise..." — Stefan Tomke (19:58)

9. Recognizing Cultural Success

• What Does Success Look Like?
  • Experiments become routine—like financial analysis in meetings.
  • "You know it's working when someone asks, 'Where's the experiment?' Or when someone actually walks in and says, 'Here is the experiment.'" — Stefan Tomke (20:30)

10. Tips for Individuals at Any Level

• Start Small, Create Awareness:
  • Run small experiments within your area and share the results; culture change starts with grassroots proof.
  • "I've talked to organizations that actually started this way and then got bigger and bigger...but you got to get started, don't wait." — Stefan Tomke (22:02)

11. The Modern Manager’s Role in an Experimental Organization

• Grand Challenge: Set ambitious strategic direction to focus hypotheses and experiments.
• Systems & Resources: Build infrastructure that makes running experiments easy and routine.
• Role Model: Lead by example—embrace humility, act on findings, and democratize decision-making.
• "Managers need to live by the same rules...That's a different style of leading." — Stefan Tomke (25:06)

Notable Quotes & Memorable Moments

• On experimentation over intuition:
  "Rather than trying to follow our intuition or our opinions, why not just run the test and let the test tell us what works and doesn't work?"
  — Stefan Tomke (03:51)

• On humility and leadership:
  "Walking into a meeting and...just go into the meeting and tell everybody, listen, quite honestly, I don't know what's going to happen, so let's just find out."
  — Stefan Tomke (05:33)

• On organizational change:
  "You empower people to do things on their own. At the same time, you actually have someone who centrally owns this capability as well."
  — Stefan Tomke (19:57)

Timestamps for Key Segments

• [02:42] — Why businesses must experiment more
• [03:50] — Microsoft Bing example and power dynamics
• [04:52] — The risks managers fear and humility in uncertainty
• [07:16] — Physical world experimentation (Kohl’s case)
• [08:37] — The evolution from internal to third-party tools
• [10:01] — Booking.com’s experimentation culture
• [13:02] — Creative industries and Netflix’s experimentation approach
• [15:44] — The danger of ‘HiPPOs’ in decision-making
• [17:14] — How much (and where) to experiment in organizations
• [18:09] — Centralized, decentralized, and Center of Excellence models
• [20:29] — Cultural test for experimentation
• [22:43] — Grassroots experimentation and change
• [23:06] — The evolving role of managers in an experimental company

Conclusion

The episode makes a powerful, clear case: experimentation should be foundational in modern organizations not just for tech companies, but everywhere decisions and innovations occur. Building a true culture of experimentation demands humility, structural change, and leadership by example. Individuals at any level can help by starting small and spreading a results-focused mindset. Managers must evolve, setting direction, building infrastructure, and modeling data-driven curiosity for their teams.

“You wouldn't make a big decision without running the numbers—you shouldn't do it without an experiment either.”

For further resources, check out Stefan Tomke's HBR article "Building a Culture of Experimentation," or his book, "Experimentation: The Surprising Power of Business Experiments."