Scrum Master Toolbox Podcast: Agile Storytelling from the Trenches

Episode: Measuring What Matters Beyond Velocity and Story Points
Guest: Natalia Curusi
Host: Vasco Duarte
Date: December 18, 2025

Episode Overview

This episode explores what truly defines success for Scrum Masters, moving beyond traditional metrics like velocity and story points. Vasco Duarte speaks with Agile coach and Scrum Master Natalia Curusi about evaluating team performance, the value and pitfalls of metrics, and the critical role of high-quality team conversations. Together, they reflect on soft versus hard metrics and the evolving responsibilities of Scrum Masters in fostering self-improving, resilient teams.

Key Discussion Points & Insights

Retrospective Formats: Going Beyond the Standard

• Natalia’s Favorite Format:
  • Natalia recommends the Spotify Health Check, a team review event held every 2–3 months, which is more extensive than a regular retrospective ([01:40]).
  • The Health Check covers 11 dimensions, including psychological safety, fun, boredom, learning opportunities, quality, and empowerment.
  • Insight: This format consistently brings “aha moments” for both Natalia and her teams:

    “Every single time when I run this with a new team, I have some kind of aha moments, some kind of discoveries.” (Natalia, [02:43])

  • She appreciates how this exercise uncovers both expected and surprising team dynamics and uses it with every team she supports.

Defining Scrum Master Success: Leaving Things Better

• Natalia’s Core Principle:
  • Success is about making organizations and teams better—and fostering an environment that continues improving after the Scrum Master leaves.

    “We need to aim to let the place where we work a little bit better than it was and to make sure that this place could improve itself.” (Natalia, [04:35])

• Replaceability vs. Irreplaceability:
  • Good Scrum Masters make themselves replaceable, building teams that could function well even in their absence.

    “We don’t make ourselves irreplaceable, we make ourselves as replaceable as possible.” (Natalia, [05:09])

  • She contrasts this with a management mindset that equates dependence with leadership.

The Role and Risks of Metrics

• Initial Recommendations:
  • Natalia notes the necessity for agreed-upon metrics with leadership. Suggested metrics include predictability, team morale, customer satisfaction, and technical excellence.
  • She firmly rejects velocity as a meaningful measure:

    “Velocity is not a metric from the start. I even don't want to mention that.” (Natalia, [06:17])

• Gameability Warning:
  • Metrics are easily "gamed" by smart teams; velocity is especially susceptible.

    “They can game what we measure in kind of couple of weeks. They already know how to do that.” (Natalia, [07:11])

Gemba Walk: The Power of Observation

• Gemba Walk (Go See):
  • Natalia advocates for the Gemba Walk, from Toyota’s Lean approach, as a way for Scrum Masters—and leaders—to ground their understanding in real work and context.

    “They need to go out where the work is done and to see how that work is done, to see if that metrics are realistic or not.” (Natalia, [07:44])

Vasco’s Reflection: Soft Metrics and Conversations

• Vasco introduces the idea of measuring the quality of conversations in and around the team, referencing business philosopher Esko Kilpi.
• He sees collaborative, decision-making conversations as fundamental to knowledge work.

  “The quality of the conversations inside the team and between the team and its stakeholders…” (Vasco, [09:54])

What Makes a “Quality Conversation”?

• Contextual Quality:
  • Natalia points out that quality depends on context (pressure, personalities, timing).

    “The quality conversation could be very different depending on the context, depending what happened, …what is your personality type, and so on.” (Natalia, [10:51])

• Indicators of Quality (Vasco):
  • For tough deadlines: Good conversations surface impediments, generate agreements, and foster collective action.
  • Early in projects: Openness, exploration, and “better questions” matter more than decisions.

    “Are we discussing several options—or are we converging too early? Converging too early means there’s probably a power imbalance in the team.” (Vasco, [12:17])

Realism Check: Not Every Day is a Good Day

• Natalia emphasizes that not every team, especially new ones, is capable of consistently high-quality conversations.

  “You really need to assume that people are humans. ...Sometimes you need to step out and say, okay, you are not prepared today for that conversation...” (Natalia, [13:19])

• High-performing teams are rare and even they have off days. Sustained poor interactions, though, are a red flag.

Soft Metrics Matter

• Vasco suggests Scrum Masters often overlook “soft metrics,” like the team’s ability to communicate and adapt.

  “There’s a lot of soft metrics that help us understand the ability of the team to make decisions, to adapt, to deliver, that we very often don’t consider.” (Vasco, [14:38])

• Encourages listeners to look beyond the “hard” numbers.

Notable Quotes & Memorable Moments

• [02:43] Natalia: “Every single time when I run this with a new team, I have some kind of aha moments, some kind of discoveries.”
• [04:35] Natalia: “We need to aim to let the place where we work a little bit better than it was and to make sure that this place could improve itself.”
• [05:09] Natalia: “We don’t make ourselves irreplaceable, we make ourselves as replaceable as possible.”
• [06:17] Natalia: “Velocity is not a metric from the start. I even don’t want to mention that.”
• [07:44] Natalia: “They need to go out where the work is done and to see how that metrics are realistic or not.”
• [10:51] Natalia: “The quality conversation could be very different depending on the context, depending what happened, …what is your personality type, and so on.”
• [12:17] Vasco: “Are we discussing several options—or are we converging too early? Converging too early means there’s probably a power imbalance in the team.”
• [13:19] Natalia: “You really need to assume that people are humans. ...Sometimes you need to step out and say, okay, you are not prepared today for that conversation...”
• [14:38] Vasco: “There’s a lot of soft metrics that help us understand the ability of the team to make decisions, to adapt, to deliver, that we very often don’t consider.”

Timestamps for Key Segments

• [01:40] – Natalia describes the Spotify Health Check as a retrospective format
• [04:35] – Natalia’s definition of Scrum Master success
• [06:00] – Role and selection of metrics; velocity rejected as useful
• [07:44] – Gemba Walk: Importance of observation over pure measurement
• [09:54] – Vasco introduces measuring the “quality of conversations”
• [13:01] – Natalia challenges the assumption of “ideal” teams; the human reality
• [14:38] – Reflection on soft vs. hard metrics

Conclusion

This episode urges Scrum Masters to redefine success through the lens of team resilience and self-improvement, to rely less on gameable hard metrics (like velocity), and to pay more attention to qualitative indicators—such as conversational quality, team morale, and organizational learning. Both hard and soft metrics have their place, but fostering an environment of honest reflection, adaptability, and trust is the true mark of success.