Testing as Measurement—Why Bug-Hunting Misses the Point With Tom Gilb and Simon Holzapfel

Summary

Podcast Summary: “Testing as Measurement—Why Bug-Hunting Misses the Point”

Podcast: Scrum Master Toolbox Podcast
Episode: Miniseries on the Evo Method, Episode 3
Guests: Tom Gilb, Simon Holzapfel
Host: Vasco Duarte
Date: December 10, 2025

---

Main Theme

This episode dives into the common misconceptions about testing within Agile and software development, arguing that focusing on bug-hunting and defect identification is missing the point. Instead, Tom Gilb and Simon Holzapfel advocate for shifting focus to measuring value and incremental improvement, aligning directly with business goals. The discussion centers on how the Evo method addresses these issues and provides practical advice on true value delivery and continuous measurement.

---

Key Discussion Points & Insights

The “Near Death” Realization

• [01:58] Simon shares a pivotal moment when Tom’s explanation of testing and value opened his eyes to wasted efforts in error-finding.
  • Simon: “I realized I had wasted and just, you know, in a well meaning way wasted hundreds of hours in boardrooms arguing about errors of which we were aware of perhaps 10%.”
• The anecdote underscores how organizations often obsess over known bugs while missing the bigger picture of business result delivery.

Tom Gilb: Why Testing-as-Bug-Hunting is a Distraction

• [03:11] Tom explains that the fixation on finding and fixing bugs is “completely wrong and irrelevant” from a management perspective.
  • Tom Gilb: “Managers aren't really interested in bugs… They're interested in results, in improvements in their business, what we call business value.”
  • He points out that testing should be about “measurement of enhanced or increased value at every cycle” rather than just defect discovery.
• If you don’t quantify what value matters to the business, you can’t measure improvement or progress.

The Importance of Value Measurement

• [04:01] Tom Gilb: “We need to talk about measurement of enhanced or increased value. At every cycle. Measurement, measurement, measurement.”
  • The conversation shifts toward defining and quantifying critical stakeholder values before you can measure or improve them.
• Managers care about progress against business goals, not just defect metrics.
  • “It's not about reducing the bugs to zero. That's a programmer's view of the world. The manager's view of the world is, am I going to look good to the board of directors or to my CEO? And, and it's got nothing to do with bugs, okay?” — Tom Gilb [04:30]

Short-Cycle Measurement and Real Agility

• [04:50] Agile should be about measuring value increments week by week:
  • Tom: “If you did, move ahead, more of the same. Keep on going with confidence. This whole idea of testing as we have it is irrelevant for the management and value delivery level.”
  • If value doesn’t increment as expected, it’s a signal to adapt or pivot strategy.

Analogy: “Burnt Toast” and Current Testing Culture

• [05:58] Host Vasco shares Deming’s “American toast” anecdote:
  • Vasco: “We burn the toast and then we pay someone to scrape off the black bits off the bread. And that's what testing really is in software today.”
  • The analogy illustrates how finding and fixing defects after the fact is an inefficient and reactive mindset.

“Table Stakes” and Raising the Bar

• [06:35] Simon: “The stable stakes. That's what you're getting paid for.”
  • The group agrees that software simply working should be a default expectation, not the measure of success.
  • True progress is about moving the needle on real value for stakeholders.

Fighter Pilot Example: Micro-cycle Measurement & Decision-making

• [06:56] Vasco introduces a metaphor from a fighter pilot’s mission planning—a process that hinges on constantly asking:
  1. Where are we going?
  2. Where are we now?
  3. Where should we have been at this point?
  4. Why is there a gap?
  • Tom Gilb: “And this is so excellent, Excellent summary.” [07:27]
• Importance of regular checkpoints, abort criteria, and real-time adaptation in both aviation and Agile.

Comparison to Soviet-Style Economic Planning

• [07:50] Host argues that rigid sticking to plans and punishing deviations produces fake compliance and late error detection—mirroring 1970s Soviet planning mistakes.

The Real Problem: Obscuring Error Detection

• [08:38] Simon: “The system runs on hiding and obscuring error detection, guaranteeing crappy output by definition.”
  • This points to a cultural issue where lack of transparency and adaptability leads to poor outcomes.

How to Enable Better Decision-Making

• [08:48] Simon details his consultancy approach:
  • Don’t try to convince—listen for the client’s natural language and decision-making systems (OODA, PDCA, Shewart, Evo, etc.)
  • Meet them where they are to co-evolve value delivery and run learning cycles together.
  • Simon: “I listen for signals from their point of view… That's the sense and adapt. It's an evolutionary perspective to the interaction.”
• Importance of adaptation and mutual learning in value delivery.

---

Notable Quotes & Memorable Moments

• [02:38] Simon Holzapfel: “I realized I had wasted… hundreds of hours in boardrooms arguing about errors of which we were aware of perhaps 10%.”
• [03:34] Tom Gilb: “Managers aren't really interested in bugs... They're interested in results, in improvements in their business, what we call business value.”
• [04:01] Tom Gilb: “We need to talk about measurement of enhanced or increased value. At every cycle. Measurement, measurement, measurement.”
• [04:30] Tom Gilb: “It's not about reducing the bugs to zero. That's a programmer's view of the world.”
• [05:58] Host (Vasco): "We burn the toast and then pay someone to scrape off the black bits off the bread. And that's what testing really is in software today.”
• [06:35] Simon: "The stable stakes. That's what you're getting paid for."
• [07:27] Tom Gilb: “And this is so excellent, Excellent summary.”
• [08:38] Simon Holzapfel: “The system runs on hiding and obscuring error detection, guaranteeing crappy output by definition.”
• [09:17] Simon Holzapfel: “First I don't convince them of anything because that's doomed to fail. What I listen for is what is their approximate language?... That's the sense and adapt. It's an evolutionary perspective to the interaction.”

---

Important Timestamps

• [01:58] Simon’s transformative realization and introduction of the problem
• [03:11] Tom Gilb details the flaw in today’s testing focus
• [04:01] Measuring for value, not bug-minimization
• [06:56] Fighter pilot’s process as an Agile analogy
• [07:50] Comparison to Soviet-style rigid planning
• [08:38] Simon: Organizational resistance to visible error detection
• [09:17] Simon’s consultancy approach: meeting clients in their own language

---

Summary

This episode challenges traditional views on testing within Agile, presenting a compelling case for replacing “bug-hunting” with business value measurement. Tom Gilb and Simon Holzapfel urge listeners to define, quantify, and measure stakeholder value in frequent, short cycles, adapting quickly when results do not meet expectations. The lively discussion connects software development challenges to historical and cross-industry insights, ultimately promoting a mindset of adaptability, open error detection, and a relentless focus on meaningful business outcomes rather than technical perfection.

---

Transcript

[Vasco]

Hey there, agile adventurer, just a quick question. What if, for the price of a fancy coffee or half a pizza, you could unlock over 700 hours of the best agile content on the planet? That's audio, video, E courses, books, presentations, all that you can think of. But you can also join live calls with world class practitioners and hang out in a flame war free and AI slop clean slack with the sharpest minds in the game. Oh, and yes, you get direct access to me, Vasko, your Scrum Master Toolbox podcast. No, this is not a drill. It's this Scrum Master Toolbox membership. And it's your unfair advantage in the agile world. So if you want to know more, go check out scrummastertoolbox.org membership. That's scrummastertoolbox.org Membership. And check out all the goodies we have for you. Do it now. But if you're not doing it now, let's listen to the podcast.

[Host]

Hello everybody.

[Vasco]

Welcome to our third in our five.

[Host]

Episode miniseries on the Evo method. And we have with us the whole week. It's so great to have Simon and Tom. Welcome back.

[Tom Gilb]

Thank you.

[Host]

So Simon has a great story to share with us about a realization he had during a conversation with Tom. We're going to talk about execution, the big focus for everybody. Everybody's focused on execution. We're going to talk about it from the perspective of the Evo method that Tom has written a lot about, links in the show, notes as usual, and how testing has always been understood wrongly. So, Simon, take us away.

[Simon]

Yeah, this is really kind of almost a near death experience in which I was driving down New York City to meet with a business school dean. I knew Tom maybe a few weeks by then, but I was complaining about something and he then kind of stopped me and says, wait, just listen to this. And he said like 10 sentences in a row, kind of like a geometric proof. That just so blew my mind. I almost drove off the road. I started laughing to the point that I realized I had wasted and just, you know, in a well meaning way wasted hundreds of hours in boardrooms arguing about errors of which we were aware of perhaps 10%. I didn't drive off the road, you know, everything that that day changed my life and I'll leave it there.

[Host]

So thankfully you didn't drive off the road, otherwise we wouldn't have the pleasure to have you here. But Tom. So tell us, what did you tell Simon at that time?

[Tom Gilb]

Well, first, as he alluded to, I went through what I call a design.

Process of about 15 steps and there's a paper on that which I can supply if somebody reminds me now the.

Let'S go straight to the.

Idea there of testing, okay? I, I believe the whole focus of it, people on testing and finding bugs and correcting them and defects is completely wrong and irrelevant. And it is the doom of agile people. Because managers aren't really interested in bugs.

They can be annoying. They're interested in results, in improvements in their business, what we call business value.

We need to stop talking about testing. Somebody has to do it at some level. Okay, but that's not the essential thing. We need to talk about measurement of enhanced or increased value.

At every cycle. Measurement, measurement, measurement. Now, if you have not quantified your critical values, the stuff that management really cares about, you can't measure what you haven't defined and people haven't defined their value. So it starts with the, the simple idea that it's not about reducing the bugs to zero. That's a programmer's view of the world. The manager's view of the world is, am I going to look good to the board of directors or to my CEO? And, and it's got nothing to do with bugs, okay? So we got to get away from the bug finding, testing, you know, does the logic work? That's appropriate at a lower level. Okay, but where we're failing is to Quantify the top 10 critical stakeholder values, whatever they be, and then design, as we mentioned in the previous episode, strategies that will help you get the values. And then we need to measure in short cycles, like every week that the values are incrementing. If they only increment 2% at a time, well within one year, by definition, you've got 100% of what you want. Okay, but you've got to measure. We expected the values to move forward by 3% next week. You need to measure that you did move forward approximately 3%. Or not. Or not means you've got the wrong strategy. Time to pivot, time to change, time to learn. Okay? If you did, move ahead, more of the same. Keep on going with confidence. This whole idea of testing as we have it is irrelevant for the management and value delivery level.

[Host]

I think that's very important. So I remember this anecdote by Deming. He called it the American toast. How do we make toast in America? I mean, he was American, so he was talking about America. He said, we burn the toast and then we pay someone to scrape off the black bits off the bread.

[Tom Gilb]

Well put.

[Host]

And that's what testing really is in software today. But as Tom was mentioning, the goal is not to make sure that the software works.

We expect a very low bar for delivering software. Right. Like that. It should work. Of course it should work.

[Simon]

The stable stakes. That's what you're getting paid for.

[Host]

Stable stakes, exactly. But we should be focusing is, are we progressing towards the value?

[Vasco]

Right.

[Host]

And I think Tom explained very clearly that the cycle should be short and we should have a clear direction. And I remember interviewing a fighter pilot here on the podcast who have this.

Of course, they have doctrine in the armed forces. This is a fighter pilot. They have a specific process they go through before they get onto the plane to complete the mission. And at every point in the mission. So we're talking about a couple of hours or even a couple of minutes, this depending on the mission. Right. At every step of the mission, at every critical step of the mission, they were trained to ask four questions. Where are we going? Where are we now? Where should we have been at this point? And why is there a gap?

[Tom Gilb]

And this is so excellent, Excellent summary.

[Host]

Exactly. And they do this at the micro cycle. So not just the very short cycles that Tom is talking about, but they do this every minute. Every checkpoint. The mission has many checkpoints. Right. At which they might decide to abort. This is a very important aspect. Like today's strategy implementation does not have abort criteria. This is disastrous for most organizations. And I would say, and I would be interested in Simon's take on this, I would say the reason why we're there is because we're trying to imitate the economic planning model of the 1970s Soviet Union. We're not even being capitalists. Right. We're not even paying any homage to the capitalist theorists of the world. We're repeating the same mistakes that the Soviet Union had in the 1970s in which we're creating a plan and then we're punishing everybody that doesn't follow the plan, which guarantees fake compliance and too late discovery of the errors that both of you have mentioned. Simon, what do you think about that?

[Simon]

I mean, the system runs on hiding and obscuring error detection, guaranteeing crappy output by definition.

[Host]

I think that's how it is. Right, but how do we. Simon, you've consulted many people who have to make critical decisions on a regular basis. How do you help these people not be afraid of making decisions faster with more, let's say, detailed and specific information about where they are versus where they want to be? Like, how do you convince them to step out of the. I've planned it so it will work kind of mindset.

[Simon]

So first I don't convince them of anything because that's doomed to fail. What I listen for is what is their approximate language? Do they speak OODA loop? Do they speak pdca? Do they speak Shewart cycles? Do they speak evo? Do they speak blah blah blah blah blah, right? I listen for what language is closest to them. If they're in the military, we go right to OODA loops. If it's about a social technology, we go to Captain Marquette. So I just, you know, I'm a history teacher and a, you know, dork. So I listen for signals from their point of view. So instead of trying to jump on their shoulder like, you know, some sort of revolutionary Che Guevara wah wah wah wah. It's like, no, no, where are you now? You just said the four questions, right? I'm just going to listen and respond. That's the sense and adapt. It's an evolutionary perspective to the interaction which says we're going to co evolve some sort of value thing together first and over time run the OODA loops, run the did it learning cycles, then you step up that value without it costing or hurting more.

[Vasco]

Talking about stepping up of value.

[Host]

Please stay tuned. Tomorrow we're going to talk about how do we actually get to measure all of this. We're going to talk about something that Tom Gilb Quality 5.0. So stay tuned and talk to you tomorrow. Simon and Tom, thank you for being here.

[Simon]

Pleasure as always.

[Vasco]

All right, I hope you liked this episode, but before you hit next episode, here's the deal. This podcast is powered by people like you. The members who wanted more than just inspiration. They wanted real tools and real connection to people who are practicing agile. Every day we're talking access to over 700 hours of agile gold, CTO level strategy talks, summit keynotes, live workshops, E courses, Deep dive interviews, books, and if you're into no estimates, we got the pioneers of no estimates in those deep Dive interviews as well. Agile business intelligence, creating product visions, coaching your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners, plus a private slack coach community which is free of all of that AI slop you see everywhere. And of course without the flame wars, It's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together. So if this podcast has helped you before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the [e-mail] membership because listening is great. It's important. But doing it together, that's next level. I'll see you in the community.

[Host]

Slack.

[Vasco]

We really hope you liked our show.

[Host]

And if you did, why not rate.

[Vasco]

This podcast on Stitcher or itunes?

[Host]

Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.