Building a Coaching Service Where Survey Scores Become Living Improvement | Scott Smith

Summary

Podcast Summary

Podcast: Scrum Master Toolbox Podcast: Agile storytelling from the trenches
Episode: Building a Coaching Service Where Survey Scores Become Living Improvement | Scott Smith
Host: Vasco Duarte
Guest: Scott Smith (Agile Coach)
Date: December 3, 2025

---

Episode Overview

The main theme of this episode is the practical journey of initiating a new coaching service within an "AI domain" organization of around 30 people. Scott Smith discusses early-stage experiments and strategic thinking behind launching a coaching offering aimed at both individual team members and leaders, tying this initiative to the organization's annual pulse survey results—essentially, making “survey scores become living improvement.” The conversation revolves around defining success, feedback collection approaches, anticipated challenges, and means to foster engagement, especially among those unfamiliar with coaching.

---

Key Discussion Points & Insights

1. The Motivation: Moving Survey Results into Continuous Improvement

[02:10]

• Scott Smith shares that the coaching service idea stems from a desire “to increase our survey results ... which measure the health of our teams, our leadership and our people.”
• The coaching service is envisioned as an experiment, piloted initially with graduate-level hires, with the hope that demand will expand to more senior staff and leadership.

“We decided we’ll go ahead and give it a go with a bit of a pilot … and hopefully see more demand from more senior members of the domain and even leadership, which would be a great point to get to.”
Scott Smith [03:35]

2. Defining Success for the Coaching Service

[04:08]

• The primary success metric is qualitative feedback from participants and their stakeholders.
  • Direct feedback from coaching recipients (graduates, senior team members, leadership).
  • Observational feedback from people interacting with coached individuals (are they displaying improved traits?).
• Success is not just uptake but visible, positive change perceived by the broader team.

“Feedback from them, if it’s positive markers, then that would be a great metric ... Even other metrics might include feedback from someone else who has dealt with someone who I’ve coached … that would also be fantastic feedback.”
Scott Smith [04:31]

3. Feedback Collection: Initial and Ongoing Approaches

[05:52]

• Vasco probes the strategy for feedback, emphasizing two distinct stages:
  1. Before coaching starts: Gauge interest and readiness in the domain.
  2. Once in motion: Collect feedback during and after coaching via existing routines (one-on-ones, retro feedback, etc.).
• Scott initially considered seeking explicit “expressions of interest,” but shifted toward a “lighter touch,” launching the service to see what happens organically.
• The approach will favor quick, low-overhead experiments over detailed preparation.

“I think a lighter touch, just roll with it and put out the service without, you know, too much overheads and too much preparation and see what the uptake is rather than going looking for expressions of interest.”
Scott Smith [07:44]

4. Anticipated Barriers: Awareness and Busyness

[08:49 - 09:47]

• Two key challenges identified by Vasco:
  • Many workers have never experienced coaching and may not understand its benefits (“they might not know what the word means”).
  • Senior staff are time-poor, often not setting aside time for their own professional improvement.
• Scott’s approach:
  • Provide clear, concise information explaining what coaching is—and isn’t.
  • Share FAQs and prepare guidelines about what a session might involve.
  • Clarify the spectrum of support (coaching, mentoring, possibly training) available.

“Provide some relevant upfront information ... FAQs and provide them a guideline as to what a coaching conversation might look like, what they may or may not want to bring up.”
Scott Smith [09:47]

• Offering transparency about session formats and possible outcomes can help demystify coaching and attract initial interest.

5. Incremental Experimentation and Learning

[11:06]

• Both agree that treating these initial communication and awareness efforts as feedback experiments themselves is critical.
• The first feedback loop occurs even before formal coaching sessions, revealing team openness and understanding of coaching.

“That will be already a first feedback collection point ... you start to understand what people’s perspective is towards coaching and also how open they are to actually engaging with a process like that.”
Vasco Duarte [11:06]

---

Notable Quotes & Memorable Moments

• On Launching the Pilot:
  “We decided, we have decided we’ll go ahead and give it a go with a bit of a pilot ... and hopefully see more demand from more senior members of the domain and even leadership, which would be a great point to get to.”
  Scott Smith [03:35]

• On the Value of Feedback:
  “If that feedback is positive and they perhaps have been displaying some traits that they had come to me for coaching of, then that would also be fantastic feedback to see that, you know, the coaching services that are providing are of value and has been noticed not only with the people who are coming, but others who are interacting with those.”
  Scott Smith [04:31]

• On Experimenting Without Overhead:
  “A lighter touch, just roll with it and put out the service without too much overheads and too much preparation and see what the uptake is rather than going looking for expressions of interest.”
  Scott Smith [07:44]

• On Helping People Understand Coaching:
  “We might be moving more into this mentoring during the conversation as opposed to coaching based on how the conversation is going to and things that might come out of it. It could be training for example, if they so desire.”
  Scott Smith [10:04]

---

Timestamps of Key Segments

| Timestamp | Segment/Event | |-----------|------------------------------------------------------------------| | 02:10 | Scott introduces the coaching service and its context. | | 04:08 | Discussion about how to define and measure “success.” | | 05:52 | Vasco asks about feedback strategy and staging. | | 07:44 | Scott describes experimenting with a “lighter touch” rollout. | | 08:49 | Challenges: coaching unfamiliarity, senior staff time constraints| | 09:47 | Scott’s information-sharing strategies for onboarding. | | 11:06 | Vasco highlights feedback even before coaching starts. |

---

Tone & Style

The conversation is candid, collaborative, and experiment-oriented—typical of practitioner-to-practitioner Agile exchanges. Both Scott and Vasco emphasize learning by doing, iteration, and the value of small, practical experiments.

---

Summary Takeaway

This episode offers a hands-on look at starting an internal coaching service meant to drive genuine, observable team improvement—making annual survey scores an active part of daily work. Key lessons include focusing feedback on real behavioral change, lowering the bar for initial engagement via lightweight experiments, and communicating clearly to demystify coaching for all levels of staff. The dialogue delivers actionable advice for any Agile coach or team lead seeking to transform culture and outcomes with small, empirical steps.

Transcript

[A]

Hey there, agile adventurer, just a quick question. What if for the price of a fancy coffee or half a pizza, you could unlock over 700 hours of the best agile content on the planet? That's audio, video, E courses, books, presentations, all that you can think of. But you can also join live calls with world class practitioners and hang out in a flame war free and AI slop clean slack with the sharpest minds in the game. Oh, and yes, you get direct access to me, Vasko, your Scrum Master Toolbox podcast. No, this is not a drill. It's this Scrum Master Toolbox membership. And it's your unfair advantage in the agile world. So if you want to know more, go check out scrummastertoolbox.org membership, that's scrummastertoolbox.org Membership. And check out all the goodies we have for you. Do it now. But if you're not doing it now, let's listen to the podcast. Hello everybody. Welcome to our Wednesday, the coaching day here on the podcast this week. Joining us from Australia is Scott Smith. Hey Scott, welcome back.

[B]

Thanks Vasco. Great to be here.

[A]

Absolutely. So Wednesday is the coaching question because as Scrum masters we need to reflect constantly. We need to think about the teams, the stakeholders, the organization and how they are changing. And that means we're facing evolving challenges all the time, new challenges. And Wednesday is about picking one of those and kind of working together in a coaching conversation, exploring the topic, understanding what is there, what might be, things that are leading to that situation, and then maybe defining, creating, brainstorming, at least a few experiments to run. So let's start with the topic. Scott, what topic do you bring to us this week?

[B]

Yes, great, thanks Vasco. So the topic I had was around this setting up of a new coaching service within my current workplace. So I'm working in an AI domain at the moment, which is relatively new. There's maybe about 30 people or so in the domain at this point in time and have been exploring ideas with leadership around continuous improvement for the domain. And how do we, you know, do things like increase our survey results are, which are, is our pulse survey results that are done once a year which really measure the health of our teams, our leadership and our people. So one of the experiments which we've recently come up with, that I would like to share here, was the idea of perhaps sharing a new coaching service which I can share with the domain with myself, being available to provide one on one coaching or perhaps team coaching or even leadership coaching. So yeah, I've shared this idea and we decided, we have decided we'll go ahead and give it a go with a bit of a pilot perhaps thinking first of all that some of the younger graduate level people in the domain may take the opportunity to have some one on one coaching with us and then take it from there and hopefully see more demands from more senior members of the domain and even leadership, which would be a great point to get to.

[A]

When you think about creating this coaching service for people in that domain, or I guess we could call it in that organization department kind of set of teams at least. How would you, Scott, how would you define success for that coaching service, as you called it, that you want to set up?

[B]

For me, the, the things that I would be measuring would be the feedback, you know, from clients, from people who are taking the service up. So that could be, as I said, graduates, it might be senior members within the team or even leadership, but feedback from them, if it's positive markers, then that would be a great metric to see increasing the service and taking it further. Even other metrics might include feedback from someone else who has dealt with someone who I've coached. If that feedback is positive and they perhaps have been displaying some traits that they had come to me for coaching of, then that would also be fantastic feedback to see that, you know, the coaching services that are providing are, are of value and has been noticed not only with the people who are coming, but others who are interacting with those.

[A]

So when you think about that, so success being the type of feedback that you get from the people that are either engaged in the coaching or they are stakeholders for the people engaged in the coaching. When you think about that, what kind of system or kind of routine or approach do you intend to put in place in order to start collecting that feedback? Because when I think about that feedback, I imagine that we need to have two steps in the feedback collection. The first is the obvious one, the one that you collect within the coaching process, like ideas, thoughts, feedback that you collect from the person that you're doing coaching with? A second one, which is also quite obvious would be the discussions with their stakeholders, right? Like, and this could be, you know, regular one on ones or whatever that context might be that already exists. But I'm thinking even before that, how do you start collecting feedback before the coaching is actually in place? Right, because there's a point in time where the coaching will be announced and it will be available, but it will not yet be in practice. So how about that phase? Like how are you thinking about collecting feedback in that phase? And then of course later in any different than what I just described?

[B]

Yeah, it's a good point and it was something that I have thought about. My ideas originally were to put the feelers out really into the department in terms of what is the demand like you know, from the various levels of seniority across the department, get an understanding from that group of who may have an interest. So expressions of interest for coaching services. But after thinking that through a bit further, I felt like we want to be doing these short sharp experiments and I think a lighter touch, just roll with it and put out the service without, you know, too much overheads and too much preparation and see what the uptake is rather than going looking for expressions of interest. My assumption is that it's a greater likelihood that some of the younger, maybe graduate level people within the department will be the ones first putting their hands up for those coaching services. But who knows? That is only an assumption we've yet to find out. So I could be surprised and find that there are others out there who are of higher seniority.

[A]

I think of two challenges here and the first challenge is that many people in today's working age, so let's say 23, 25 and upwards, will not have ever experienced coaching. So they might not know what the word means. And then the second thing is.

[B]

Of.

[A]

Course in this case mostly the more senior people will be so busy with meetings and calls and tasks that they really have no time. And many of us don't have the habit of saving time for our own improvement and learning. How are you intending to deal with this potential struggle there? And I don't know the domain, I don't know the people. Maybe this doesn't make sense in your context, but what I just described certainly made sense in other areas where I've been part of this setting up of a coaching approach and support through coaching.

[B]

Yeah, yeah, for sure that first one. Then I think around people perhaps not on understanding what coaching is the approach there was to, you know, provide some relevant upfront information, FYI, URGs, you know, some questions and stuff that people may ask FAQs and provide them a guideline as to what a coaching conversation might look like, what they may or may not want to bring up as a, at a coaching session and really, you know, sharing as much of information as possible around what coaching is and what it isn't as well also explaining, you know, things like, you know, we, we might be moving more into this mentoring during the conversation as opposed to coaching based on how the conversation is going to and things that might come out of it. It could be training for example, if they so desire. So these were the types of things I was hoping to do in advance to prepare for sharing.

[A]

Those are great ideas, I think, and great experiments to run. Right, because that will be already a first feedback collection point where you start to understand what people's perspective is towards coaching and also how much how open they are to actually engaging with a process like that. Those sound like great examples and also very practical way to start collecting that feedback before the coaching process actually starts with the first few people that might want to take advantage of that. That's a great thought. Thank you for sharing that, Scott.

[B]

Yeah, no problem.

[A]

Alright, I hope you liked this episode, but before you hit next episode, here's the deal. This podcast is powered by people like you, the members who wanted more than just inspiration. They wanted real tools and real connection to people who are practicing Agile every day. We're talking accent access to over 700 hours of agile Gold, CTO level strategy talks, Summit keynotes, live workshops, E courses, Deep Dive interviews, books, and if you're into no estimates, we got the pioneers of no estimates in those Deep Dive interviews as well. Agile Business Intelligence, creating product visions, coaching your product, product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners, plus a private Slack community which is free of all of that AI slop you see everywhere. And of course without the flame wars. It's a community of practitioners that want to learn and thrive together. The best place to connect with community and learn together. So if this podcast has helped you before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the details at school scrummastertoolbox.org membership because listening is great, it's important, but doing it together, that's next level. I'll see you in the community.

[B]

Slack.

[A]

We really hope you liked our show and if you did, why not rate this podcast on Stitcher or itunes. Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.