Keeping Score: Measuring Impact in Development | The Development Podcast

Summary

World Bank | The Development Podcast

Episode: Keeping Score: Measuring Impact in Development

Date: September 28, 2024
Host: Samuel Ouzabafi

---

Overview

This episode explores the crucial question of how impact in development is measured—and why it matters—focusing on the World Bank Group’s new streamlined scorecard. Through real project stories from Mozambique and Jordan and conversations with development leaders at the World Bank and Asian Development Bank, the episode dives deep into the shift from simply tracking dollars spent to measuring lives transformed. The episode emphasizes transparency, accountability, and collaboration across development institutions to maximize real-world impact.

---

Key Discussion Points & Insights

1. What Does Measuring Impact Mean in Development?

• Personal Stories Illustrate Impact:
  • Mozambique’s Climate-Resilient Classrooms:
    After Cyclone Freddy, schools with retrofitted, storm-resistant classrooms provided continuous, safe education for children.
    • Katerina (Student): “When it rained, we didn’t even sit in the classroom. We put our books in plastic sheets and we ran home. Now the school is excellent.” (00:34, 04:00)
  • Jordan’s Cash Transfer Program for COVID-19 Relief:
    Cash support helped families meet basic needs and offered skills training for better livelihoods.
    • Fadia El Hanawe (Mother): “Before receiving the supplementary support, many things were lacking for my children like paying rent, school expenses, water and electricity. Now, thank God, it covers my needs.” (04:17)
    • Zaher Zayadna (Trainee): “It helped prepare me to find jobs. They gave me a certificate for working in this field so I can rely on myself and improve my family’s living conditions.” (05:07)
    • Fadia, about skills training: “When we receive training at the National Aid Fund, it helps us with finding job opportunities because the support will not be provided for the long term. My dream is to establish my own business in the future.” (06:20)

2. The World Bank Group’s New Scorecard: A Shift in Mindset

• From “Dollars to Lives Transformed”:
  • Ed Mountainfield (VP, Operations, Policy & Country Services):
    “The World Bank Group Scorecard is our yardstick for measuring the results of the development work that we do all over the world ... It’s about shifting from dollars to lives transformed.” (00:48, 07:57, 13:51)
• Fewer, More Focused Metrics:
  • Reduced from over 150 indicators to just 22 critical results indicators.
  • Designed to hone in on outcomes that matter most—such as food security, climate change adaptation, gender equality, and learning poverty. (08:08)
• Radical Transparency & Data Disaggregation:
  • Methodologies are clarified, published, and data is made accessible, allowing anyone to drill down by gender, youth, disability, country, or region.
    • Ed Mountainfield: “It’s much more transparent and systematic ... publish those methodologies, and also to make the data itself much more transparent so you can disaggregate it ... working towards radical transparency as part of this scorecard.” (09:22)
• Tracking Real People, Real Change:
  • Example: Bank’s food insecurity initiatives aim to reach almost 1/3 billion people, with a focus on Africa and South Asia. (10:36)
  • The scorecard enables both accountability and the ability to question and redirect efforts as needed.

3. The Importance of Keeping Score

• Accountability & Motivating Focus:
  • The scorecard is akin to a school report card, helping teams understand where to double down and where good news can be celebrated.
  • Ed Mountainfield: “If you make everything a priority, you kind of make nothing a priority ... really zeroing in on the things that we think are really the most impactful things.” (09:58)
• Communication & Storytelling:
  • New “results narratives” will be included so data is augmented by contextual stories.
    • Ed Mountainfield: “Data is very powerful, but it’s only one way of telling the story ... we’re also going to include in the scorecard what we’re calling results narratives.” (12:43)
• Future Transparency:
  • At upcoming meetings, 20 of 22 indicators will be published on a new interactive website for stakeholders and the public. (12:17)

4. A Broader Development Shift: MDB Alignment & Practical Challenges

• Pan-Development Bank Collaboration:
  • Lu Shen (Director, Results Management, ADB): "It’s about improving people’s lives at the end of the day ... this is at the fundamental basis for everything that we do.” (01:05, 15:19)
• Setting and Missing Targets:
  • Setting targets like SDGs is important, but so is harmonizing how metrics are created and tracked across diverse countries and agencies.
  • Lu Shen: “Ultimately, in order to have any kind of impactful reporting, you need to have the same metrics, people need to measure it in the same way. And everybody has to buy into the same system.” (16:05)
• Project Example – Behavioral Change:
  • Water, sanitation, and hygiene (WASH) project involving handwashing in schools—teaching that impact goes beyond infrastructure to changing daily behaviors as a ripple effect through communities. (17:49)
• Alignment is Key for Credibility:
  • On climate and other global challenges, MDBs (Multilateral Development Banks) are working toward standardized methods to credibly pool and report achievements.
    • Lu Shen: "I think it's extremely important for us to put our heads together to come up with a common methodology … so we can jointly report on what is our collective contributions. It’s not easy ... outcomes are difficult, right? Because we talk about attribution versus contribution all the time.” (19:24)

---

Notable Quotes & Memorable Moments

• “It’s about shifting from dollars to lives transformed.”
  Ed Mountainfield, 13:51
• “Now the school is excellent. I really like learning. I like to be a teacher when I grow up.”
  Katerina (Mozambique Student), 04:00
• “It’s about improving people’s lives at the end of the day. … This is at the fundamental basis for everything that we do.”
  Lu Shen, 15:19
• “If you make everything a priority, you kind of make nothing a priority.”
  Ed Mountainfield, 09:58
• “Data is very powerful, but it’s only one way of telling the story … we’re also going to include what we’re calling results narratives.”
  Ed Mountainfield, 12:43

---

Timestamps for Important Segments

• Real-life impact stories from Mozambique & Jordan:
  (02:10–06:42)
• Introduction and explanation of the new World Bank Scorecard:
  (07:56–14:46)
• Discussion with Lu Shen (Asian Development Bank) on impact, targets, and measuring challenge:
  (15:06–20:10)

---

Conclusion

The episode powerfully unpacks how measuring impact is central to effective development—it's not enough to tally dollars spent; the true score comes from lives improved. With the World Bank’s new, transparent scorecard and initiatives to align global metrics, development agencies are moving toward more accountable, people-focused, and resilient development solutions. Collaboration, clarity, and a relentless focus on meaningful outcomes are key.

---

Transcript

[Samuel Ouzabafi]

Foreign.

Hello and welcome to the Development Podcast from the World Bank Group. I'm Samuel ouzabafi in Washington D.C. today's episode is all about achieving impact in development. We're asking what it means, why it matters and how it is measured. We'll get a snapshot from a World bank project in Mozambique, where climate resilient classrooms are providing children with a safe and secure environment to learn.

[Katerina]

When it rained, we didn't even sit in the classroom. We put our books in plastic sheets and we ran home. Now the school is excellent.

[Samuel Ouzabafi]

We'll do a deep dive into the World Bank Group's approach to measuring impact through its new scorecard.

[Ed Mountainfield]

It's about shifting from dollars to lives transformed, going under the bonnet a little bit and recognizing there's good news and there's areas where we need to double.

[Samuel Ouzabafi]

And we'll get the view from the Asian Development bank on why impact is so crucial in development.

[Lu Shen]

It's about improving people's lives at the end of the day.

[Samuel Ouzabafi]

All that and more coming up in the Development podcast. Keeping Score Matters. Think of a sports game, a bingo hall, or even a political debate. Throughout our working lives, we are appraised and many of us will still remember what it felt like to get our school report detailing our strengths, areas to be improved, and maybe some unexpected achievements. When it comes to measuring impact and development, the principle stays the same. Keeping score is critical. We'll be unpacking the World Bank Group's new scorecard and exploring how it'll steer.

The organization's approach to impact.

But first, let's learn from a couple of projects that really help explain what.

[Ed Mountainfield]

We mean by impact.

[Samuel Ouzabafi]

Our producer Sarah Trenor found out.

[Narrator/Producer Sarah Trenor]

The sound of children at their school in Mozambique. These children, like many others, can find their education disrupted during extreme weather, so things like cyclones and floods. In 2023, Cyclone Freddie hit the region, causing widespread devastation. But climate resilient classrooms retrofitted with reinforced structures can help keep schools safe and open. Let's hear now From Katerina, a 10 year old pupil, about the changes.

[Katerina]

When it rained, we didn't even sit in the classroom. We put our books in plastic sheets and we ran home.

[Narrator/Producer Sarah Trenor]

Augustino Macbriga, director of Rosina Mackel Primary School, says this.

[Ed Mountainfield]

There was so much water coming in.

[Samuel Ouzabafi]

That the children sometimes had to flee their classrooms.

[Narrator/Producer Sarah Trenor]

The Resilient Classroom initiative, supported by the World bank, has so far transformed more than 700 classrooms in Mozambique. In Katarina's school, this meant reinforcing the roof structure, removing the old one and replacing it with a roof able to withstand extreme weather and prevent the school from flooding. When Cyclone Freddie hit the country last year, the school remained dry and unharmed. Around 1.2 million people have directly benefited from this project. Over 58,000 children have gained access to improved education infrastructure. For children like Katarina, it means they don't have to worry about a disrupted education and can look to their own bright futures.

[Katerina]

Now the school is excellent. I really like learning. I like to be a teacher when I grow up.

[Narrator/Producer Sarah Trenor]

Now let's travel to Jordan.

[Fadia El Hanawe]

I benefited from the supplementary support in that it covered my children's needs. Before receiving the supplementary support, many things were lacking for my children like paying rent, school expenses, water and electricity. Now, thank God, it covers my needs.

[Lu Shen]

And those of my children.

[Narrator/Producer Sarah Trenor]

Nada Elramahe is just one of the 1.2 million people affected by COVID 19 that the National Aid Fund's Cash Transfer program supported by the World bank has reached. Her experience echoes that of Zaher Zayadna. He is training in solar maintenance. Part of the fund's aim is to provide support for people gaining skills, moving from the informal to the formal sector.

[Zaher Zayadna]

I registered for the program through my training in solar energy at the National Aid Fund. They told us about the opportunity and better my living condition through the monthly payments. It helped prepare me to find jobs. They gave me a certificate for working in this field so I can rely on myself and improve my family's living conditions.

[Narrator/Producer Sarah Trenor]

The program is also a model in terms of digitized and transparent practices. It's had a big impact. According to analysis from the World Bank. Jordan's Unified Cash Transfer program in 2021, for example, is estimated to have reduced inequality by 0.7% and overall poverty by 1.4%. Jordan's National Aid Fund cash Transfer program has provided a lifeline of cash support for the poorest, most vulnerable households. It's now the largest program of its type in the region. Fadia El Hanawe has also gained new skills and is training to be a chocolate maker. She tells us more.

[Fadia El Hanawe]

I received training in chocolate making, how to produce it and market it. When we receive training at the National Aid Fund, it helps us with finding job opportunities because the support will not be provided for the long term. My dream is to establish my own business in the future.

[Samuel Ouzabafi]

Those are some pretty incredible personal stories and in a way they get to the heart of what the new scorecard is measuring. Impact on people as we know, the world faces many crises from food insecurity, climate change, inequality, conflict and biodiversity loss. The World Bank Group is working to address these global crises with a mission of creating a world free of poverty on a livable planet. But how does the bank, its partners, shareholders, and really anyone with an interest in or involvement in the institution know how it is going to. How do we know we are on track? It's a great question. Well, earlier this year, a revamped system for keeping score and measuring progress was introduced. The new scorecard streamlined the number of metrics it uses from over 150 indicators to just 22. It also moves towards radical transparency and measures impact across all different parts of the organization. The scorecard is kind of like the World Bank Group's report card. I wanted to learn more, so I asked Ed Mountainfield, the Vice President of Operations, Policy and Country Services, to join me in the DC studio to explain how it works.

Hi, Ed.

[Ed Mountainfield]

Hi, Sam.

[Samuel Ouzabafi]

It's a pleasure to have you here with us today on the development podcast.

[Ed Mountainfield]

Great to join the podcast.

[Samuel Ouzabafi]

The Scorecard. We're just going to dive straight in. Can you tell us what exactly is the World Bank Group scorecard?

[Ed Mountainfield]

So the World Bank Group Scorecard is our yardstick for measuring the results of the development work that we do all over the world. We have a new mission for the World bank to end poverty and boost shared prosperity on a livable planet. And the scorecard is about really translating that into the key development outcomes that are necessary to achieve that bigger picture goal. And it's also about helping us be more accountable and ultimately to focus and strengthen the impact of our work. So the central piece of the scorecard is focused on monitoring our progress across 22 results indicators that we think are absolutely mission critical to getting to where we need to be to help people all over the world to deal with the climate emergency and to uplift their lives. And this scorecard provides a clear view of how we're doing on some of the most critical global challenges, from protecting the poorest, to adapting to climate change, achieving gender equality and ending learning poverty. So it's really about focusing on results, focusing our work around achieving those results and holding ourselves accountable. The other thing that's a big step forward with this scorecard is that it's much more transparent and systematic than it's been in the past. So we've really worked to clarify the methodologies by which we measure our impacts, to actually document them, publish those methodologies, and also to make the data itself much more transparent so you can disaggregate it. You can see all of the backend data. And so we're not where we want to be yet it's a journey. But we're working towards radical transparency as part of this scorecard.

[Samuel Ouzabafi]

I can only see that being a good thing. And what exactly is the scorecard designed to measure and how is it different from the previous iteration?

[Ed Mountainfield]

An insight from previous scorecards we've had in the past is that we probably tried to measure and target too many things. And if you make everything a priority, kind of make nothing a priority. So really zeroing in on the things that we think are really the most impactful things. So it's designed to measure the outcomes of the support that we provide to developing countries. We use a simplified set of indicators that are trying to emphasize quality as well as usage. And these indicators have been carefully selected through a broad consultation as part of our World Bank Group evolution process with shareholders and partners. There are all sorts of insights that come from the data that are really starting to transform the way that we work. So just to give one example, a large part of the world's population currently faces moderate or severe food insecurity. In fact, over 60% of people in low income countries are in that category. So our current efforts are responding to this challenge with total results from our active portfolio of projects expected to reach nearly a third of a billion people and counting. And we're working to build that portfolio up and to ensure that it's delivering our support is aiming to target the most at risk populations, the majority of results reaching those in Africa and South Asia. And food insecurity will continue to remain a challenge. So we can use this to target our work, to hold ourselves accountable, but also to raise questions like is that enough? We've still got a huge gap to go, so how can we take it to the next level and what can we learn from from the experience? Also, the disaggregation of the data is something that's radically new. So in the past we've kind of published the headline numbers, but being able to drill that down and say, okay, this is what we're achieving in terms of tackling food insecurity, but how much of it is in Africa, how much is in the low income countries, Are we actually tackling the problem where the problem is most intense, or do we need to rebalance some of our work and shift our focus? So that's, I think a big part of the new approach.

[Samuel Ouzabafi]

Yeah, it sounds like this approach really puts people at the center of the World Bank's work. And in that same vein, I think transparency is also very important. So how will the results be published.

[Ed Mountainfield]

It's a work in progress. At the moment, we're covering only a portion of the results in our current portfolio. So as we work to make sure that we're capturing the results we achieve using these methodologies across all of our projects, we'll build these results up over time and reveal, I think, more of the impact that we're having. We've made a lot of progress and during the World Bank IMBIF annual meetings next month, the next set of the scorecards indicators will be released. We'll be releasing 20 of the 22 indicators on a new interactive and dynamic website, which will make the data accessible and reproducible and disaggregable for everyone. So all of the indicators, where possible, will be disaggregated by gender, by youth, by disability, by country, region. There's going to be a strong focus on the development of people in client countries. We know that data is very powerful, but it's only one way of telling the story of the impact that you have. So we're also going to include in the scorecard what we're calling results narratives. So these are a series of analytical notes that draw on the data, but also tell the story of how we achieve the results that we achieve and what some of the challenges are and getting a little bit behind beyond the data to tell a bigger picture story.

[Samuel Ouzabafi]

For somebody who works in communications, this is music to my ears, showing what that impact really means on the ground to the people that we, we, we support. We're going to cap this conversation off with a question that I think you've touched on in everything we've talked about. I want us to really just highlight this. Why is keeping score important?

[Ed Mountainfield]

The World bank group is ultimately not about the money it commits or disperses. It's about the difference it makes to people's lives on the ground. And that's what motivates all of us to do the jobs that we do all over the world. So I think it's about focusing on the thing that really matters, which is the impact we have on people's lives. So I think that's ultimately what it's about. It's about shifting from dollars to lives transformed. It's about shifting from just telling the headlines to going under the bonnet a little bit and recognizing there's good news and there's areas where we need to double down.

[Samuel Ouzabafi]

I love that it's a shift away from dollars to lives transformed. At Mountfield, it was a pleasure chatting with you and I look forward to a lot more Conversation coming up.

[Ed Mountainfield]

Great. Enjoyed it. Thanks a lot, Sam.

[Samuel Ouzabafi]

So we've heard lots about why and how the World bank is honing in on measuring impact. And in a way, it's part of a sea change throughout the development community. Lu Shen is the Director of the Results Management and Aid Effectiveness Division at the Asian Development Bank. So, in short, she knows a whole lot about measuring impact.

To someone who's coming into this topic cold, no idea what we're talking about, impact, development, etc. Someone who does not know what we do, how would you explain impact and why it matters?

[Lu Shen]

It's about improving people's lives at the end of the day. Right. We talk about results all the time. And, you know, we have different metrics and we have different targets, but at the end of the day, what is the purpose of anything that we do? I think back to when I joined adb. One of the cornerstones of ADB is alleviation of poverty. And so I think this is at the fundamental basis for everything that we do. How do we and our work improve people's lives? I think that's really at the core of all of the MDBs.

[Samuel Ouzabafi]

What happens when we miss targets? Right. Because a lot of time we set targets. One that always jumps to mind are the SDGs. What are your thoughts on that?

[Lu Shen]

Yeah, this is a tough one because I think it's important to set targets because it gives people something to work towards. But I think there are a lot of different things in the pathway that is challenging. Right. When you set targets, there is the whole idea of what is your target, how do you go about measuring it? And I think these are really tough things to do because when you think about it, conceptually, that's one thing, but ultimately, in order to have any kind of impactful reporting, you need to have the same metrics, people need to measure it in the same way. Right. And everybody has to buy into the same system. And I think that the challenge with working with our developing member countries is that everybody has their own way of doing it. That policy formulation is great. It's very academic and it's very idealistic.

[Samuel Ouzabafi]

Yeah.

[Lu Shen]

Target setting is important, but it's much more important to think through what are some of the things that we can do to make that consistency in terms of developing metrics, developing tools for people so that they can report in the same manner. When you do come to the end, for example, the SDGs by 2030, what are some of the achievements that we have versus what are some of the gaps?

[Samuel Ouzabafi]

Yeah, I think that's a very pragmatic way to think about it. Right. I think the people listening would be very interested in hearing an example of a project that you think has had a fundamental impact, whether short term or long term, on people's lives.

[Lu Shen]

I think one good example is there was a wash program. This was actually a unique project because they were looking at behavioral changes through a water sanitation project. And so about hat washing for kids. And so there's access to clean water, that's one thing. But if you actually don't wash your hands, ultimately how is that going to help improve your health? So I think this is a really, really interesting project because it really delves into the human factors of how do you actually get people to change behavior when you do mobilize, for example, civil society, when you actually have the right conversations with government as well as communities to make that kind of incremental changes so that they can actually start with a different behavioral pattern that's coming, in this case from the children in schools. And so there's that kind of ripple effect that we actually did not expect. Ultimately, when you do this, the correlation about how often do you fall sick and people will have that kind of realization, it's like, oh, well, maybe there's something to it.

[Samuel Ouzabafi]

This has been super eliminating, Lucien, but before I let you go, there's one thing I wanted to touch on. Now, the World bank has its new scorecard, but can you elaborate a little bit on how important it is that there's alignment in measuring impact across all the MDBs?

[Lu Shen]

I think it's extremely important for us to put our heads together to come up with a common methodology. And I know that the climate team has been working on this since the beginning of the year because we want to be credible in what we measure, especially when it comes to things like climate, where we can have the same methodology, we can use the same tools to jointly report on what is our collective contributions. I think this is incredibly important. It's not easy just because we all know outcomes are difficult, right? Because we talk about attribution versus contribution all the time. It's tough.

[Samuel Ouzabafi]

And that's about all for this episode of the Development Podcast. Join us for the World Bank Group IMF Annual meetings at the end of October to learn all about the new scorecard results. And we hope you really enjoyed this dive into the idea of impact and keeping score. It's a lot to think about. Be sure to tune in next time and if you'd like to get in touch, send us an [e-mail] Many thanks for listening. It's Samuel and we'll be back soon.

Sam.