Aligning Metrics and Mindsets in Agile Teams | Karthiga Seturaj

Summary

Scrum Master Toolbox Podcast: Agile Storytelling from the Trenches
Episode: Aligning Metrics and Mindsets in Agile Teams | Karthiga Seturaj
Host: Vasco Duarte
Guest: Karthiga Seturaj
Release Date: December 17, 2024

---

Introduction

In this insightful episode of the Scrum Master Toolbox Podcast, host Vasco Duarte engages in a compelling conversation with Karthiga Seturaj, an experienced Scrum Master. The episode delves deep into the intricate relationship between metrics and team mindsets within Agile frameworks. Karthiga shares her experiences, challenges, and strategies for aligning metrics in a way that fosters growth and continuous improvement without stifling team morale.

Inspiration from "Project to Product"

Karthiga begins by highlighting a pivotal book that significantly influenced her approach as a Scrum Master:

"Project to Product" by Mick Kirsten
[03:13]

Recommended by her Vice President of Engineering, Karthiga immersed herself in the book to implement the Flow Framework in her organization. She underscores the book's impact on her understanding of value stream management and bridging the gap between business stakeholders and technical teams. The insights gained from the book reshaped her perspective on metrics, emphasizing the importance of flow and its alignment with business value.

Key Metrics for Scrum Masters

When discussing critical metrics for Scrum Masters, Karthiga emphasizes the importance of identifying and understanding bottlenecks within the workflow:

"I would say the bottlenecks, right? So personally, flow efficiency is the one that I like the most... If you focus on flow efficiency, that's a good place to start understanding where the delays are."
[05:02]

She advocates for Flow Efficiency as a foundational metric. By analyzing how much time a team actively works on their backlog versus waiting, Scrum Masters can identify areas for improvement. This approach aligns with Lean principles, focusing on reducing wait times and optimizing overall flow time.

Challenges with Introducing Metrics

Despite thorough training and a clear understanding of metrics’ purposes, Karthiga encountered resistance from team members who felt pressured by the metrics:

"Sometimes or many times I would notice developers or POs saying things like, oh no, if I do this, our metrics would be impacted... Can we do something about the ticket which is blocked so that our numbers won't look bad."
[06:15]

This behavior highlighted a common challenge: metrics can inadvertently become targets, leading to counterproductive actions. Karthiga reflects on the difficulty of managing team perceptions and ensuring that metrics serve as tools for improvement rather than punitive measures.

Strategies for Thoughtful Metric Implementation

To navigate the complexities of introducing metrics, Karthiga outlines several strategies aimed at fostering a positive relationship between teams and their metrics:

1. Establishing Feedback Loops

"Get feedback from the teams to understand whether this is helpful... Involve them when you are putting that metrics together."
[09:43]

Karthiga emphasizes the importance of creating continuous feedback channels with teams. By involving team members in defining and selecting metrics, Scrum Masters can ensure that the metrics are relevant and supportive of the team’s objectives. This collaborative approach fosters ownership and reduces apprehension around metric tracking.

2. Distinguishing Between Internal and External Metrics

Drawing from Vasco’s insights, Karthiga agrees on the necessity of differentiating metrics based on their audience and purpose:

"I think that would definitely help. And also giving them more clarity on why leadership wants to look at it."
[13:38]

She highlights the distinction between internal metrics (used by the team for self-improvement) and external metrics (used by leadership to assess broader performance). By clearly defining the purpose and audience of each metric, teams can maintain focus on meaningful improvements without succumbing to the pressures of external evaluation.

3. Addressing Goodhart’s Law

Vasco introduces Goodhart’s Law, which posits that “when a metric becomes a target, it ceases to be a useful metric.” Karthiga concurs, recognizing the pitfalls of metric-target conflation:

"Yeah, I think that's a good way to put it... and helps team understand where they should be focusing on."
[13:38]

To mitigate this, Karthiga advocates for using metrics as indicators rather than strict targets. This approach encourages teams to interpret metrics contextually, focusing on the underlying processes rather than merely achieving numerical goals.

Balancing Leadership and Team Needs

Karthiga discusses the delicate balance Scrum Masters must strike between fulfilling leadership’s requirements and addressing the team’s needs:

"Leadership wants to see where they can help in a way. They can set an environment, build an environment where the teams can thrive and grow."
[15:08]

She provides an example of addressing a bottleneck in testing by using metric data to make a case for additional resources. This illustrates how metrics can empower teams to advocate for necessary support, ensuring that the focus remains on collective success rather than individual performance.

Key Takeaways

• Flow Efficiency: Prioritize understanding and improving flow efficiency to identify and eliminate bottlenecks.

• Collaborative Metric Definition: Involve team members in selecting and defining metrics to ensure relevance and buy-in.

• Internal vs. External Metrics: Clearly differentiate between metrics used internally for team development and those shared externally for broader assessment.

• Avoiding Metric Targets: Treat metrics as informative tools rather than targets to prevent counterproductive behaviors influenced by Goodhart’s Law.

• Empowering Teams: Use metrics data to advocate for necessary resources and support, fostering an environment conducive to team growth and success.

Conclusion

In this episode, Karthiga Seturaj offers a nuanced perspective on the role of metrics in Agile teams. By emphasizing collaboration, clear differentiation of metric purposes, and a focus on flow efficiency, she provides actionable strategies for Scrum Masters aiming to align metrics with positive team mindsets. This conversation serves as a valuable resource for Agile practitioners seeking to harness the power of metrics without undermining team dynamics.

---

Listen to the full episode here to gain deeper insights into aligning metrics and mindsets in your Agile teams.

Transcript

[Vasco Duarte]

Hey, how are you doing? I'm Vasco Duarte, your host on the Scrum Master Toolbox podcast. And I've got some exciting news. So right now, as I record this, I'm holding in my hand the signed contract for our very first global Agile Summit. We're all in and I couldn't wait to share this news with you. So mark your calendars. May 18th to 20th of 2025 in Tallinn Eston, we're going to have a transformative experience. We're putting together an event that it's all about real life Agile. It's not theory or buzzwords, it's practitioners sharing what's working, what's making an impact, and how they've overcome challenges that you too will have to face, or maybe even facing. Right now, we're bringing together the best stories in Agile. From product leaders to engineering wizards to business visionaries, these will be stories that will inspire you to action. This isn't just another conference. It's a chance to connect with the people that are shaping the future of Agile. And here's the best part. Right now we're in our super early bird phase. And that means you can grab tickets at just 25% of the final price. Look, that's not just half off, it's half off of the half off. It's an incredible deal for our dedicated community members, just like you listening to this right now. So at the summit, day one will be all about hands on workshops. And days two and three, we'll dive into leadership, product strategy, coding, testing, and everything that makes Agile thrive in organizations. Right now remember, these are all first person, real life stories. Now whether you're a leader, a developer, or part of a consulting company, this event is built to take your Agile game to the next level. So don't wait. Go to globalagilesummit.com and grab your ticket. Today, let's all make 2025, dear. Agile truly transforms your teams, your business and our industry. I'll see you all in Tallinn. And Remember, go to globalagilesummit.com and get your super early bird ticket right now. It will only be available until the agenda is announced, so don't wait. Grab it right now. Right now that that's out of the way, onto the episode. Hello, everybody. Welcome to our Team Tuesday where we talk about teams and how sometimes they create their own problems. This week we have with us Kartiga Seturaj. Hey Kartiga, welcome to the show.

[Kartiga Seturaj]

Hey Vasco, nice to be back.

[Vasco Duarte]

So if we think about teams, of course there's a lot of stories of how teams create their own problems, and we'll talk about one of those in a second. But before we go there, Kartige, share with us what was the book that most inspired you in your role as a Scrum Master?

[Kartiga Seturaj]

Sure. So there is this book. You might have already read it by now. It's called Project to Product by Mick Christian. So I read this book in 2019 and year after it was published. So it was recommended to me by my VP of engineering at that time. So he basically handed me the book and told me, go read it. You're going to help implement this new flow framework in our organization. So it was very new to me, the Flow framework concept I was aware about. I mean, I knew about Scrum Kanban XP at that time, but Flow Framework, what was, you know, pretty new. I had no idea what that was. So I worked from home two days just to focus on the book, Heads down. And it was a great read. It's an easy read too. It's a small book. You can easily get it done in a day or two. I learned a lot about value stream management from that book and how to bridge the gap between business folks, people who do not understand the technical details behind software delivery, and the technical teams themselves. So it changed my perspective on also how to look at metrics. Most importantly, why flow is important and what are the different ways to measure and improve flow within a value stream? And more than that, how do you map it to the business value that we deliver? That's where a lot of organizations struggle to quantify business values. So this work would be a great help for teams who are looking to implement flow metrics or implement flow within their organization.

[Vasco Duarte]

Absolutely. Great book and I'll put the link in the show Notes. Is there a particular metric that you think is critical for Scrum masters out there to be looking at even before they read the book? And of course the book will help with that.

[Kartiga Seturaj]

I would say the bottlenecks, right? So personally, flow efficiency is the one that I like the most out of those four or five flow metrics that's been recommended in the book. If you focus on flow efficienc, that's a good place to start understanding where the delays are and how much time the team is spending actively working on their backlog. And this is how much time is spent on waiting. So lean also comes into play here. It's all about the wait times, right? How do we reduce the wait times? How do we optimize the overall flow time so Understanding the delays would be a good place to start. So flow efficiency would be a good metric.

[Vasco Duarte]

Absolutely. So check it out. Project to product by. Was it Mick Kirsten?

[Kartiga Seturaj]

Yep. Yeah.

[Vasco Duarte]

Right. So now let's move on to the topic of teams and how sometimes they create their own problems. So tell us the story of a team, Kartiga. Tell us a little bit about the context but then walk us through how these small little behaviors developed over time and eventually became a problem for the team.

[Kartiga Seturaj]

Absolutely. So this kind of based on what we just talked about, Flow metrics. Right. So we developed Flow Metrics Dashboard after implementing Flow Framework and as a Scrum master, I would go over these metrics in the team retros. So we trained the teams pretty well on how to use metrics because metrics are a touchy subject. Each person can a different emotion behind metrics. I always made sure the teams understand why we track metrics, how we are supposed to use it and how we are not supposed to use it. Even leadership did a pretty good job of reiterating that these metrics are not punitive, they are supposed to help you. We need to focus on them so that we can help them help ourselves get better at what we are doing and also understand what are the areas to improve. In spite of all this training, in spite of all this pre work that we did before introducing metrics, sometimes or many times I would notice developers or POs saying things like, oh no, if I do this, our metrics would be impacted. Our numbers won't look good. They would even ask me directly in standups or refinements, can we do something about the sticket which is blocked so that our numbers won't look bad. My God, I would be like, no, no, no, that's not the intention behind tracking metrics. Right. Let's not worry, worry about the numbers, please. And it kind of helped me understand how, what, how the day to day actions, day to day decisions that the teams make was based on this metric. So how the behaviors would, you know, change based on what we are tracking. There's been a lot of research, a lot of books also in this space measuring what matters and things like that. Even in spite of doing those pre work, I felt kind of bad for the teams. I felt sorry for them because they're feeling this way. And then we did again another kind of retweeting, a refresher on metrics and how the team should use it and why it's not something that it's going to be punitive, it's not performance based it's all team metrics. So I'm very passionate about metrics. I love numbers because they don't lie. It's not about the numbers, it's about the story behind those numbers. That's what I would tell the teams. Again, again, focus on the story, whether it's a good story or a bad story. If it's a good story, let's celebrate. If it's not, then let's talk about what can be improved or change. That's about it. Don't think about the actions that you take and how it impact the numbers.

[Vasco Duarte]

Yeah, absolutely. One of the things that I worry about when introducing metrics is exactly the type of behavior that comes with it. Right. And it's imposs to avoid having a reaction to metrics, especially if we want metrics to be in a dashboard. And managing this expectation of how the metrics will be used is quite critical and especially if the metrics will be visible outside the team. Right. So for all of the Scrum masters and agile coaches out there that are perhaps right now at this moment introducing some metrics that will be visible outside the team, what kind of lessons you can share with them, what kind of tips you can share about doing that carefully and thoughtfully.

[Kartiga Seturaj]

Sure. So the concept of feedback, right. Getting feedback from the teams to understand whether this is helpful. So when it comes to metrics, it's being used at different levels. As you said, leadership needs to have a dashboard. They need to have something that they can look at to. Not just to track performance, just to make sure that, you know, things are on track. Is there anything that they can help with? So each based on who's looking at it, each person has a different use case. But when it comes to teams, it's always better to get feedback from them directly, even involve them when you are, you know, kind of putting that metrics together, if it's the first time you're doing it, help involve them in the metrics definition, make sure that they're part of it each and every step of the way so that they have a sense of ownership, they are building it for themselves. They do have a say on what they think we should be measuring that would be helpful for them. Leadership might have an idea which might be different than what teams need. So you can, as a scrum master, it's important for you to balance both giving leadership what they need, but at the same time also helping the teams understand. We are tracking this for so and some reason, but also getting their feedback on other metrics which they think would be helpful from a developer or a tester standpoint. So having this feedback loop put into place will definitely help.

[Vasco Duarte]

For me, one of the things that really helped is to use or kind of emphasize the difference between metrics for the outside and metrics for the inside. Metrics for the outside would be metrics the team would publish for others to give feedback on or to follow up or whatever. And then metrics for the inside are metrics that only the team cares about and only the team takes insights from. Right. And the reason why I've introduced this concept in some of the teams that I've worked with is because I saw this, you know, the pattern you described, which is called Goodhart's law, which says that basically when any metric becomes a target, it ceases to be a useful metric. Right. So if we look at a thermometer as a temperature target measurement. Right. Then of course everybody will be disappointed because the temperature will never be exactly what we want it to be. However, the thermometer doesn't care about it. It is just a way to measure the metric, the temperature metric. But people associate meaning and they associate consequences to those metrics. Right. For example, cycle time is a great example. Cycle time is a measure of a system. It's not a measure of performance of individuals or teams. It's a measure of a system. But when we give that metric to teams, they mistakenly consider that it's a target for them and it evaluates their own performance. Right. And we really need to be careful about that because as Good Heart Goodhart's law says. Right. When a metric becomes a target, it ceases to be a useful metric. And, and for me, this is how I try to manage this, how should I say it, informative metrics that don't really mean anything for the team. Like cycle time is a metric that is systemic. It's not dependent on only the team and then internal metrics. Right. Like we could look at specific things that happen within the team and for example, how long does a specific task stay waiting for someone to pick it up? Right. Like, for example, that's. That would be an internal metric within the best coverage. Yeah. For example, and the team could do something about it. Right. Because it is within their domain. But then other metrics, I wouldn't worry the team too much about it. What do you think about that, Kartika?

[Kartiga Seturaj]

Yeah, I think that's a good way to put it. I've never thought about in that way. That kind of helps set some guidelines around who's using this metric for what purpose. And helps team understand where they should be focusing on. So obviously they would want to focus on improving the test coverage for their code or reducing the wait time in their value stream. Yeah, I think that would definitely help. And also giving them more clarity on why leadership wants to look at it. Right. So there might be few things which is outside team's control. For example, say there is a bottleneck in testing which has happen and in one of my previous teams they consistently would have there was bottleneck in the testing call up at their Kanban board every iteration. So when the leadership is looking at that trend, it's going to be a trend. They're going to look at the trends and teams over time. They need to have more testers. So this is kind of making a case for that. Right. So if they need more resources, if you need more tester on your team, this is the data that leadership is going to use to make a case for hiring a new tester. So that might also help the teams understand how leadership uses it. I think that's the concern. Sometimes some team members might have that if this is directly tied to their performance review. No, that's not the case here. Leadership wants to see where they can help in a way. They can set an environment, build an environment where the teams can thrive and grow.

[Vasco Duarte]

Yeah, absolutely. And that's a good point as well. Metrics can be used to support certain decisions. Thank you for sharing that, Kartiga.

[Kartiga Seturaj]

Absolutely.

[Vasco Duarte]

We really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes, share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.