Measuring What Matters Beyond Velocity and Story Points | Natalia Curusi

Summary

---

Scrum Master Toolbox Podcast: Agile Storytelling from the Trenches

Episode: Measuring What Matters Beyond Velocity and Story Points
Guest: Natalia Curusi
Host: Vasco Duarte
Date: December 18, 2025

---

Episode Overview

This episode explores what truly defines success for Scrum Masters, moving beyond traditional metrics like velocity and story points. Vasco Duarte speaks with Agile coach and Scrum Master Natalia Curusi about evaluating team performance, the value and pitfalls of metrics, and the critical role of high-quality team conversations. Together, they reflect on soft versus hard metrics and the evolving responsibilities of Scrum Masters in fostering self-improving, resilient teams.

---

Key Discussion Points & Insights

Retrospective Formats: Going Beyond the Standard

• Natalia’s Favorite Format:
  • Natalia recommends the Spotify Health Check, a team review event held every 2–3 months, which is more extensive than a regular retrospective ([01:40]).
  • The Health Check covers 11 dimensions, including psychological safety, fun, boredom, learning opportunities, quality, and empowerment.
  • Insight: This format consistently brings “aha moments” for both Natalia and her teams:

    “Every single time when I run this with a new team, I have some kind of aha moments, some kind of discoveries.” (Natalia, [02:43])

  • She appreciates how this exercise uncovers both expected and surprising team dynamics and uses it with every team she supports.

Defining Scrum Master Success: Leaving Things Better

• Natalia’s Core Principle:
  • Success is about making organizations and teams better—and fostering an environment that continues improving after the Scrum Master leaves.

    “We need to aim to let the place where we work a little bit better than it was and to make sure that this place could improve itself.” (Natalia, [04:35])

• Replaceability vs. Irreplaceability:
  • Good Scrum Masters make themselves replaceable, building teams that could function well even in their absence.

    “We don’t make ourselves irreplaceable, we make ourselves as replaceable as possible.” (Natalia, [05:09])

  • She contrasts this with a management mindset that equates dependence with leadership.

The Role and Risks of Metrics

• Initial Recommendations:
  • Natalia notes the necessity for agreed-upon metrics with leadership. Suggested metrics include predictability, team morale, customer satisfaction, and technical excellence.
  • She firmly rejects velocity as a meaningful measure:

    “Velocity is not a metric from the start. I even don't want to mention that.” (Natalia, [06:17])

• Gameability Warning:
  • Metrics are easily "gamed" by smart teams; velocity is especially susceptible.

    “They can game what we measure in kind of couple of weeks. They already know how to do that.” (Natalia, [07:11])

Gemba Walk: The Power of Observation

• Gemba Walk (Go See):
  • Natalia advocates for the Gemba Walk, from Toyota’s Lean approach, as a way for Scrum Masters—and leaders—to ground their understanding in real work and context.

    “They need to go out where the work is done and to see how that work is done, to see if that metrics are realistic or not.” (Natalia, [07:44])

Vasco’s Reflection: Soft Metrics and Conversations

• Vasco introduces the idea of measuring the quality of conversations in and around the team, referencing business philosopher Esko Kilpi.
• He sees collaborative, decision-making conversations as fundamental to knowledge work.

  “The quality of the conversations inside the team and between the team and its stakeholders…” (Vasco, [09:54])

What Makes a “Quality Conversation”?

• Contextual Quality:
  • Natalia points out that quality depends on context (pressure, personalities, timing).

    “The quality conversation could be very different depending on the context, depending what happened, …what is your personality type, and so on.” (Natalia, [10:51])

• Indicators of Quality (Vasco):
  • For tough deadlines: Good conversations surface impediments, generate agreements, and foster collective action.
  • Early in projects: Openness, exploration, and “better questions” matter more than decisions.

    “Are we discussing several options—or are we converging too early? Converging too early means there’s probably a power imbalance in the team.” (Vasco, [12:17])

Realism Check: Not Every Day is a Good Day

• Natalia emphasizes that not every team, especially new ones, is capable of consistently high-quality conversations.

  “You really need to assume that people are humans. ...Sometimes you need to step out and say, okay, you are not prepared today for that conversation...” (Natalia, [13:19])

• High-performing teams are rare and even they have off days. Sustained poor interactions, though, are a red flag.

Soft Metrics Matter

• Vasco suggests Scrum Masters often overlook “soft metrics,” like the team’s ability to communicate and adapt.

  “There’s a lot of soft metrics that help us understand the ability of the team to make decisions, to adapt, to deliver, that we very often don’t consider.” (Vasco, [14:38])

• Encourages listeners to look beyond the “hard” numbers.

---

Notable Quotes & Memorable Moments

• [02:43] Natalia: “Every single time when I run this with a new team, I have some kind of aha moments, some kind of discoveries.”
• [04:35] Natalia: “We need to aim to let the place where we work a little bit better than it was and to make sure that this place could improve itself.”
• [05:09] Natalia: “We don’t make ourselves irreplaceable, we make ourselves as replaceable as possible.”
• [06:17] Natalia: “Velocity is not a metric from the start. I even don’t want to mention that.”
• [07:44] Natalia: “They need to go out where the work is done and to see how that metrics are realistic or not.”
• [10:51] Natalia: “The quality conversation could be very different depending on the context, depending what happened, …what is your personality type, and so on.”
• [12:17] Vasco: “Are we discussing several options—or are we converging too early? Converging too early means there’s probably a power imbalance in the team.”
• [13:19] Natalia: “You really need to assume that people are humans. ...Sometimes you need to step out and say, okay, you are not prepared today for that conversation...”
• [14:38] Vasco: “There’s a lot of soft metrics that help us understand the ability of the team to make decisions, to adapt, to deliver, that we very often don’t consider.”

---

Timestamps for Key Segments

• [01:40] – Natalia describes the Spotify Health Check as a retrospective format
• [04:35] – Natalia’s definition of Scrum Master success
• [06:00] – Role and selection of metrics; velocity rejected as useful
• [07:44] – Gemba Walk: Importance of observation over pure measurement
• [09:54] – Vasco introduces measuring the “quality of conversations”
• [13:01] – Natalia challenges the assumption of “ideal” teams; the human reality
• [14:38] – Reflection on soft vs. hard metrics

---

Conclusion

This episode urges Scrum Masters to redefine success through the lens of team resilience and self-improvement, to rely less on gameable hard metrics (like velocity), and to pay more attention to qualitative indicators—such as conversational quality, team morale, and organizational learning. Both hard and soft metrics have their place, but fostering an environment of honest reflection, adaptability, and trust is the true mark of success.

---

Transcript

[A]

Hey there, agile adventurer, just a quick question. What if, for the price of a fancy coffee or half a pizza, you could unlock over 700 hours of the best agile content on the planet? That's audio, video, E courses, books, presentations, all that you can think of. But you can also join live calls with world class practitioners and hang out in a flame war free and AI slop clean slack with the sharpest minds in the game. Oh, and yes, you get direct access to me, Vasko, your Scrum Master Toolbox podcast. No, this is not a drill. It's this Scrum Master Toolbox membership. And it's your unfair advantage in the agile world.

[B]

So if you want to know more.

[A]

Go check out scrummastertoolbox.org membership, that's scrummastertoolbox.org Membership. And check out all the goodies we have for you. Do it now. But if you're not doing it now, let's listen to the podcast.

[B]

Hello everybody. Welcome to our success Thursday. This week we have with us Natalia Kurusi. Hey Natalia, welcome back.

[C]

Hello Vasco, nice to meet you again.

[B]

Likewise, likewise. So yesterday we talked about how important it is to have some way to convey success to stakeholders of the work that we do. We'll talk more about that today. But before we jump into that, do share with us what's your favorite agile retrospective format and why?

[C]

Yeah, that's a good question. As you know, you can go online to find retrospectives and to find a lot of them. Actually I have one specific one that maybe this is not a retrospective, but more like an Inspect and Adapt event. And many of our Scrum masters, they know this kind of format. This is Spotify Health Check. That's actually not retrospective. This is more like a review of the team once in two to three months. I like to run this not very often because Spotify Retrospective or Spotify Health Check is a big exercise. You usually need about one hour and a half, two hours for that. And we don't expect that the teams are going to change so often. So I run this once in two to three months. What I like about this activity is actually this is a retro and this is a multi dimensional retro. From my point of view, I have 12, not 12, 11 different dimensions. So they look at the psychological safety of the team, of how much fun the team has, how boring the work is, how much learning possibilities they have. They look also at the quality of work, at the level of empowerment and other stuff. So that's retrospective of that Inspect and adapt event is giving us a possibility to look at the teams from different perspectives and then to understand what you really need to do in what direction we need to change and so on. What is interesting about this kind of activity is that every single time when I run this with a new team, I have some kind of aha moments, I have some kind of discoveries. So sometimes I think, okay, I think that this team is not empowered. And then after that I understand. No, they are very empowered. And then sometimes I think, oh, that team is having a lot of fun together. They are learning a lot. And then I understand after the retrospective that their scope of work actually is very boring for them. They are not learning a lot. So that's all the time. Some discoveries for me as a coach, as a Scrum Master all the time. That's why I like this format. I think in every single team, when I work with them, I propose this format. And that worked very, very well.

[B]

Yeah, absolutely. I really like how you phrase it, that no matter whether you know or don't know the team, when you run this type of retrospective, you get insights, you get those aha moments. And I think that's really important because very often we talk about the retrospective being important for the team, which it is, but it's also important for us because we are there to serve the team and those insights are very important for the work that we do. And talking about the work that we do. Natalia, of course, Thursday Success Thursday here on the podcast. So we want to hear from you. When you think about Scrum Master, the role, how do you define success for yourself?

[C]

Yeah, that's an interesting one. I think yesterday in our discussion we a little bit touch base how a good Scrum Master look like. And as I said yesterday, I think from my point of view, we as a Scrum Master, we need to put a scope for us. We need to aim to let the place where we work a little bit better than it was and to make sure that this place could improve itself. So that's, I think that's one of the biggest responsibility of Scrum Master and the most inner Scrum Master is if you become a coach and then enterprise agile coach, you move that responsibility at the level of organization already, so you make sure organization becomes better and it becomes better and better. So you have that kind of continuous feedback, continuous improvement cycle in place. And I also mentioned in some of the series, so we need to make sure as a Scrum Master, we don't make ourselves irreplaceable, we make ourselves as replaceable as possible. So we grow our people in such a way that every single moment somebody can step in and take our responsibilities. So if you go into vacation and the team cannot succeed without us, you are not very good scrum master. This is kind of management style. Like managers, they think, okay, oh my God, I'm such a good manager. If a team cannot talk about me, this means that I'm a very good good manager or very good leader. From my point of view, these kind of people are not good leaders. Because if a team is not independent enough, you are not a good scrum master. You are not not a good leader. Another thing that probably we touched base a little bit yesterday. This is metrics. As I say, even if you are coach, if you are scrum master, you need to make sure when you enter into organization that you can put some metrics in place and you agree that metrics with your leadership, the persons that are giving you reviews, performance review of a person that is deciding if you are doing great or not. And that metrics could be some of the metrics. Like I would say velocity is not a metric from the start. I even don't want to mention that. But from my point of view, that could be even predictability, that could be team morale, that could be customer satisfaction, that could be technical excellence of a team. So what I recommend to the scrum masters is to pick up some of these metrics and to agree them with the organization and then to measure themselves across that metrics. Yeah, so that's important. But another more important thing is that the team, they are very, very smart and they can game their metrics very easy. We all know that. So that's why I say that velocity is not a metric because the velocity could be gamed very, very easy. So we can just inflate the story points and that's it. Our team members are very smart. So they can measure, they can game what we measure in kind of couple of weeks. They already know how to do that. And that's why I'm a very big adept of the Gamba Walk or go see. You know, this is Toyota Systems approach. Because from my point of view, a scrum master not only need to measure, but he also needs to feel the room to feel that measurements are really good. And I recommend that to do that Gemba World could go see not only to the scrum masters, but to the leaders of organizations as well. So they don't need to stay in their room and just look at their screen and look at the metrics. They need to go out where the work is done and to see how that work is done, to see if that metrics are realistic or not. So I probably went a little bit out of.

[B]

I think you defined quite a lot of things that are really, really important. And we started talking about metrics yesterday. And I think it's important for us to just dive into that a little bit more. Because one of the things that you said, which is very clear, some metrics are very easy to game, velocity being one of them. So although they may be informative, they are not really anything to do with our success as Scrum Masters. Other metrics are not easy to game by the team, for example, 360 feedback from other people outside in the organization. But they are only indicative if you don't have the context. They don't really mean anything. But one of the things that you said, which is incredibly important, is that we need metrics, but we need them in a different scope for us as Scrum Masters. We need them so that they trigger reflection and they trigger adaptation from our part. Because the really important part is, as you said at the end, the Gemba walk, right? Like, it's talking to people, it's seeing what's happening, it's understanding what's going on. Because without that, we can't make up our minds. Metrics are, as you said, only either gameable or only a very narrow perspective on reality. There's a metric that I wanted to suggest and hear your thoughts on it. There's a metric that I've been thinking about for several years, based on the work by Esko Kilpi. Esko Kilpi, may he rest in peace. He passed away a few years ago, but he's a business philosopher here out of Finland, and he had developed this idea of the conversation. Conversation meaning that the knowledge works, the work gets done through conversation, the information we exchange, how we make decisions together, what kind of topics come up and are prioritized, all of that. I was thinking about this idea of the quality of the conversations, and then we can discuss what the quality might mean, because that's also entirely contextual. That would need another deep dive. But I was thinking about this. The quality of the conversations inside the team and between the team and its stakeholders. What do you think about that, Natalia?

[C]

Yeah, the quality is very important. It depends on how we define the quality. So, for example, if me and you now stay in discuss and we have an observer that observe our conversation.

[B]

Only a few thousand at the moment, right?

[C]

Yeah. Okay, good. So I think we are speaking like friends, right? This is very relaxed, friendly conversation. Non formal, you know, so yeah, we are good. But if, for example, we fail to give our delivery right now, and there is kind of $20 million on the con right now on the age, and you have very angry client that just called you and say, like Vasco, you need to discuss with Natalia because she just failed my release just now in production, probably will not have that kind of friendly conversation, right? So as you say, the quality of conversation. But the quality conversation could be very different depending on the context, depending what happened, depending what, what kind of day you had, what is your personality type, and so on, right? If you are a person from Latin America, every conversation will look like you are fighting. So it's what do you understand by quality of conversation, Vasco?

[B]

Yeah, that's a very good question and one that is very important for us to answer in context. So I give two completely different contexts. Just try to illustrate that, for example, if we are about to deliver, right, like it's, you know, next week or a couple of days from now, the quality of conversations for me signifies that we are able to discuss finding whatever is impeding us from delivering, come to an agreement. So deciding the next steps and then acting together, right? Like it's not about acting against each other, but acting together, right? I might not agree with you, but I don't have a better idea. So I commit to what you decide and that you can see in the quality of the conversation, right? Like for example, are we discussing several options? That would be a question in my mind. Are we discussing several options or are we converging too early? Converging too early means that there's probably a power imbalance in the team. Somebody has more power than others and that person decides everything. That reduces diversity and thinking through the options. So that would be the case if we need to deliver in the very short term, if we are in the beginning, if we are in the beginning of the delivery, where we need to explore, we need to think about customer, we need to think about stakeholders. I want to see conversations that are more open ended. The result of those conversations, instead of being decisions, is more like better questions that we then need to go out and answer. So that's what I'm thinking about, the quality of the conversations. What do you think about that, Natalia?

[C]

Yeah, yeah, I understand. Like what? Everything that you just said, it works perfectly. If you have a, well, GLA team, if you have a team that already passed that forming, storming and they are in norming or they are at least in performing in norming somewhere there. But what about if that team is just formed now, what kind of quality of conversation can you expect from them? So from my point of view, you don't have a team. You have a group of individuals. And yeah, it's very hard because some of the teams doesn't pass that storming. Never. They pass the storming. And then once again, I never have seen a team that didn't have that kind of negative emotions to blame each other at some point and things like that. So you are what you are describing. This is a deal team that I have never seen in my life. So what I understood as a scrum master, that you really need to assume that people are humans. They have emotions some of the time, they have very negative emotions some of the times they are very frustrated, they are very angry. Sometimes you need to step out and to say, okay, you are not prepared today for that conversation. Let's discuss tomorrow. Let's come back to this conversation. And that happens all the time. So we cannot expect that quality of the conversation. Need. It's good when you have them, but you will not have them every single day. This is what I'm trying to.

[B]

Yeah, absolutely. And that's a very good point. I would only counter that if you have it no day, right. Instead of having it every day, which is, I guess you have a very important point there. You can't always be at the best level of performance. No one can. Right. Not even high performance athletes. But if you never have a good day, I think that's something that we should be aware of.

[A]

Right.

[B]

And perhaps the point about conversations is that very often we focus on metrics that have to do with hard metrics, like even velocity. Although it is very easy to game, it's still a hard metric, right? Like, how many story points did you deliver? But there's a lot of soft metrics that help us understand the ability of the team to make decisions, to adapt, to deliver that we very often don't consider. So that's also kind of perhaps a tip for everybody listening out there to think about what are the soft, let's call it, metrics that I don't usually look at and could I learn something from that?

[C]

Yeah. Yeah, Right. Agree.

[B]

Thank you very much for that conversation. It was an awesome reflection, Natalia. Thank you.

[C]

Thank you, Vasco.

[A]

All right, I hope you liked this episode. But before you hit next episode, here's the deal. This podcast is powered by people like you, the members who wanted more than just inspiration. They wanted real tools and real connection to people who are practicing agile every day. We're talking access to over 700 hours of agile Gold CTO level Strategy talks, Summit keynotes, live workshops, E courses, Deep Dive interviews, books, and if you're into no Estimates, we got the pioneers of.

[B]

No Estimates in those Deep Dive interviews as well.

[A]

Agile Business Intelligence, creating product visions, coaching your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners, plus a private Slack community which is free of all of that AI slob you see everywhere. And of course without the flame wars. It's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together. So if this podcast has helped you before, imagine what you will get from this podcast membership. So head on over to Scrum Master toolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the [e-mail] membership because listening is great. It's important. But doing it together, that's next level.

[B]

I'll see you in the community Slack we really hope you liked our show, and if you did, why not rate this podcast on Stitcher or itunes. Share this podcast and let other Scrum Masters know about this valuable resource for their work. Remember that sharing is caring.