Podcast Summary: “Testing as Measurement—Why Bug-Hunting Misses the Point”

Podcast: Scrum Master Toolbox Podcast
Episode: Miniseries on the Evo Method, Episode 3
Guests: Tom Gilb, Simon Holzapfel
Host: Vasco Duarte
Date: December 10, 2025

Main Theme

This episode dives into the common misconceptions about testing within Agile and software development, arguing that focusing on bug-hunting and defect identification is missing the point. Instead, Tom Gilb and Simon Holzapfel advocate for shifting focus to measuring value and incremental improvement, aligning directly with business goals. The discussion centers on how the Evo method addresses these issues and provides practical advice on true value delivery and continuous measurement.

Key Discussion Points & Insights

The “Near Death” Realization

• [01:58] Simon shares a pivotal moment when Tom’s explanation of testing and value opened his eyes to wasted efforts in error-finding.
  • Simon: “I realized I had wasted and just, you know, in a well meaning way wasted hundreds of hours in boardrooms arguing about errors of which we were aware of perhaps 10%.”
• The anecdote underscores how organizations often obsess over known bugs while missing the bigger picture of business result delivery.

Tom Gilb: Why Testing-as-Bug-Hunting is a Distraction

• [03:11] Tom explains that the fixation on finding and fixing bugs is “completely wrong and irrelevant” from a management perspective.
  • Tom Gilb: “Managers aren't really interested in bugs… They're interested in results, in improvements in their business, what we call business value.”
  • He points out that testing should be about “measurement of enhanced or increased value at every cycle” rather than just defect discovery.
• If you don’t quantify what value matters to the business, you can’t measure improvement or progress.

The Importance of Value Measurement

• [04:01] Tom Gilb: “We need to talk about measurement of enhanced or increased value. At every cycle. Measurement, measurement, measurement.”
  • The conversation shifts toward defining and quantifying critical stakeholder values before you can measure or improve them.
• Managers care about progress against business goals, not just defect metrics.
  • “It's not about reducing the bugs to zero. That's a programmer's view of the world. The manager's view of the world is, am I going to look good to the board of directors or to my CEO? And, and it's got nothing to do with bugs, okay?” — Tom Gilb [04:30]

Short-Cycle Measurement and Real Agility

• [04:50] Agile should be about measuring value increments week by week:
  • Tom: “If you did, move ahead, more of the same. Keep on going with confidence. This whole idea of testing as we have it is irrelevant for the management and value delivery level.”
  • If value doesn’t increment as expected, it’s a signal to adapt or pivot strategy.

Analogy: “Burnt Toast” and Current Testing Culture

• [05:58] Host Vasco shares Deming’s “American toast” anecdote:
  • Vasco: “We burn the toast and then we pay someone to scrape off the black bits off the bread. And that's what testing really is in software today.”
  • The analogy illustrates how finding and fixing defects after the fact is an inefficient and reactive mindset.

“Table Stakes” and Raising the Bar

• [06:35] Simon: “The stable stakes. That's what you're getting paid for.”
  • The group agrees that software simply working should be a default expectation, not the measure of success.
  • True progress is about moving the needle on real value for stakeholders.

Fighter Pilot Example: Micro-cycle Measurement & Decision-making

• [06:56] Vasco introduces a metaphor from a fighter pilot’s mission planning—a process that hinges on constantly asking:
  1. Where are we going?
  2. Where are we now?
  3. Where should we have been at this point?
  4. Why is there a gap?
  • Tom Gilb: “And this is so excellent, Excellent summary.” [07:27]
• Importance of regular checkpoints, abort criteria, and real-time adaptation in both aviation and Agile.

Comparison to Soviet-Style Economic Planning

• [07:50] Host argues that rigid sticking to plans and punishing deviations produces fake compliance and late error detection—mirroring 1970s Soviet planning mistakes.

The Real Problem: Obscuring Error Detection

• [08:38] Simon: “The system runs on hiding and obscuring error detection, guaranteeing crappy output by definition.”
  • This points to a cultural issue where lack of transparency and adaptability leads to poor outcomes.

How to Enable Better Decision-Making

• [08:48] Simon details his consultancy approach:
  • Don’t try to convince—listen for the client’s natural language and decision-making systems (OODA, PDCA, Shewart, Evo, etc.)
  • Meet them where they are to co-evolve value delivery and run learning cycles together.
  • Simon: “I listen for signals from their point of view… That's the sense and adapt. It's an evolutionary perspective to the interaction.”
• Importance of adaptation and mutual learning in value delivery.

Notable Quotes & Memorable Moments

• [02:38] Simon Holzapfel: “I realized I had wasted… hundreds of hours in boardrooms arguing about errors of which we were aware of perhaps 10%.”
• [03:34] Tom Gilb: “Managers aren't really interested in bugs... They're interested in results, in improvements in their business, what we call business value.”
• [04:01] Tom Gilb: “We need to talk about measurement of enhanced or increased value. At every cycle. Measurement, measurement, measurement.”
• [04:30] Tom Gilb: “It's not about reducing the bugs to zero. That's a programmer's view of the world.”
• [05:58] Host (Vasco): "We burn the toast and then pay someone to scrape off the black bits off the bread. And that's what testing really is in software today.”
• [06:35] Simon: "The stable stakes. That's what you're getting paid for."
• [07:27] Tom Gilb: “And this is so excellent, Excellent summary.”
• [08:38] Simon Holzapfel: “The system runs on hiding and obscuring error detection, guaranteeing crappy output by definition.”
• [09:17] Simon Holzapfel: “First I don't convince them of anything because that's doomed to fail. What I listen for is what is their approximate language?... That's the sense and adapt. It's an evolutionary perspective to the interaction.”

Important Timestamps

• [01:58] Simon’s transformative realization and introduction of the problem
• [03:11] Tom Gilb details the flaw in today’s testing focus
• [04:01] Measuring for value, not bug-minimization
• [06:56] Fighter pilot’s process as an Agile analogy
• [07:50] Comparison to Soviet-style rigid planning
• [08:38] Simon: Organizational resistance to visible error detection
• [09:17] Simon’s consultancy approach: meeting clients in their own language

Summary

This episode challenges traditional views on testing within Agile, presenting a compelling case for replacing “bug-hunting” with business value measurement. Tom Gilb and Simon Holzapfel urge listeners to define, quantify, and measure stakeholder value in frequent, short cycles, adapting quickly when results do not meet expectations. The lively discussion connects software development challenges to historical and cross-industry insights, ultimately promoting a mindset of adaptability, open error detection, and a relentless focus on meaningful business outcomes rather than technical perfection.