How Will We Measure the SDGs? – Casey Dunning

Summary

The CGD Podcast: How Will We Measure the SDGs? – Casey Dunning

Date: March 22, 2016
Host: Rajesh Merchandani (Center for Global Development)
Guest: Casey Dunning (Senior Policy Analyst, CGD)
Main Theme:
A critical discussion of the adoption, complexity, and challenges surrounding the 230 indicators chosen to measure the Sustainable Development Goals (SDGs)—exploring their practicality, the dilemmas they introduce, and the implications for global development measurement.

---

Episode Overview

This episode dives into the final piece of the SDG puzzle: the 230 indicators agreed upon for measuring progress towards the 17 Sustainable Development Goals and their 169 targets. Host Rajesh Merchandani and guest Casey Dunning unpack the logic, challenges, and potential pitfalls of this ambitious framework, reflecting on its implications for global policy and data collection.

---

Key Discussion Points & Insights

The Numbers behind the SDGs

• 17 goals, 169 targets, 230+ indicators:
  • The SDGs consist of an expansive structure built for comprehensively addressing global challenges.
• Actual complexity is even higher:
  • While officially there are 230 indicators, Dunning notes, "if you look at the list, there's actually 241 indicators attached to targets. So some of the indicators are playing double duty." (Casey Dunning, 01:32)

Why Indicators Matter

• Indicators’ role:
  • They provide the measurement framework necessary to hold countries accountable and move beyond aspirational rhetoric.
  • "If you don't have something to measure the goals and targets, then there's simply high soaring rhetoric with no actual teeth.” (Casey Dunning, 01:02)

Variability in Indicators: From Clear to Confounding

• Straightforward indicators:
  • Some are simple and already tracked globally, e.g.,
    • “3.2.1 under 5 mortality rate”
    • "That one, through some research we have shown every single country except Liechtenstein has a data point for that indicator for every year looking back to 2000.” (Casey Dunning, 04:38)
• Problematic, vague, or complex indicators:
  • Others are convoluted and hard to measure, such as those relating to curricula or perceptions:
    • E.g., “Extent to which 1 global citizenship education and 2 education for sustainable development, including gender equality and human rights, are mainstreamed at all levels…” (Casey Dunning & CGD Colleagues, 03:32–03:56)
  • Dunning’s critique:
    • "They defy logic. You can't even define them in a single country context, much less apply it to a global context." (Casey Dunning, 08:08)
    • “How you would measure it, far from clear. …The prioritization of the indicators is lacking because they're all prioritized.” (Casey Dunning, 09:41)

Measurement Challenges: Universal Yet Impractical?

• Universal applicability is idealized but problematic:

  • Some indicators make little sense for certain countries (e.g., measuring extreme poverty in high-income nations like Belgium or Luxembourg).
  • Raises questions on relevance and efficient resource use.
  • “There are finite resources and time and personnel. …If you go with the latter logic, you then open up the whole agenda for a country to say, actually that indicator is not nationally appropriate… And you can use that same logic to apply to sexual and reproductive health and rights, or governance or democracy.” (Casey Dunning, 05:12–07:37)

• Data Gaps Remain:

  • Even for high-profile indicators:
    • “72 of the 193 UN member states have not reported a single data point on [extreme poverty] since 2000.” (Casey Dunning, 05:12)

Political vs. Technocratic Process

• How indicators were selected:
  • Intended to be technocratic, not political.
    • “One of the hallmarks of the indicator selection process...was meant to be a highly technocratic process, not a political one.” (Casey Dunning, 11:07)
• Consequences:
  • Some indicators are not ready or practical to implement.
  • “The group at the UN chose the path of having indicators that covered the full agenda, no matter if those indicators weren't ready for primetime.” (Casey Dunning, 11:07)

National Implementation: The Capacity Gap

• Developing countries face real constraints:
  • Not all have the capacity or resources to measure complex indicators.
• Case Study: Indonesia's feedback:
  • For one indicator (Labor Rights Compliance):
    • “We would absolutely love to measure and report this data. We don't know what it means, how to do it and don't have the capacities to do that.” (Casey Dunning, 13:25)
  • For another (Female genital mutilation):
    • “We can't report on this indicator because it's not nationally appropriate.”
  • Underscores practical dilemmas with universal measurement.

Possible Solutions: Prioritize and Tier

• Focusing on what can be reliably measured:
  • “Focus on a subset of indicators for which we have full country coverage and reliable reporting, are methodologically sound, and have regular updates…” (Casey Dunning, 15:15)
• UN’s tier system:
  • Tier I: Methodologically sound, data available
  • Tier II: Methodology set, data not universally available
  • Tier III: Methodology not established
  • "Helpfully, the interagency group on the indicators at the UN has classified these 230 indicators into three tiers." (Casey Dunning, 15:15)
• Risk of losing comprehensiveness:
  • Prioritizing only Tier I indicators risks reducing the holistic ambition of the SDGs to a “menu of options”—potentially neglecting difficult or controversial areas. (Casey Dunning, 16:30)

---

Notable Quotes & Memorable Moments

• On the challenge and ambiguity of some indicators:

  • "They defy logic. You can't even define them in a single country context, much less apply it to a global context."
    —Casey Dunning (08:08)

• On universality vs. practicality:

  • "There are finite resources and time and personnel... And so do you want people to spend time on this issue that is not at all relevant to them as policymakers?"
    —Casey Dunning (05:12)

• Indonesia’s candor:

  • “We would absolutely love to measure and report this data. We don't know what it means, how to do it and don't have the capacities to do that.”
    —Casey Dunning (13:25)

• On the potential downside of too many overlapping indicators:

  • “It makes me wonder, kind of, in having so many indicators, if we're not kind of shooting ourselves in the foot a little bit.”
    —Casey Dunning (09:41)

• On the tiered approach:

  • “The tier one indicators are the ones that would fit this description, the Tier 2 and Tier 3...still a lot of work to do to report on the indicator.”
    —Casey Dunning (16:18)

• Rajesh’s closing summary:

  • “Casey Dunning, thanks for making my brain hurt. Sorry, this is a conundrum."
    —Rajesh Merchandani (16:58)

---

Key Timestamps

• [00:06] – Introduction of the numbers: 17 goals, 169 targets, 230 indicators.
• [01:32] – The reality: more indicators than officially stated.
• [02:20] – What indicators are designed to do.
• [03:00–03:56] – Colleagues reading various indicators: from straightforward to convoluted.
• [04:38] – The exemplary under-5 mortality rate: how it’s universally measured.
• [05:12] – Data gaps in “proportion of population living in extreme poverty.”
• [08:08] – The critique of vague or impractical indicators.
• [09:41] – The challenge of prioritization and the risk of making the SDGs a “menu of options.”
• [11:07] – The technocratic nature of indicator selection.
• [13:25] – Indonesia’s real-world feedback on capacity constraints.
• [15:15] – Suggestion: start with the most measurable indicators (Tier I).
• [16:58] – Closing thoughts and appreciation.

---

Tone & Style

The conversation is analytical, occasionally wry, and constructively critical. Dunning is especially careful and diplomatic, even as she raises tough issues ("You've been so polite for the first few minutes… I think you've been champing at the bit to say that...").

---

Summary Takeaway

The SDG indicators, while a major achievement in scope and ambition, face massive challenges in practical measurement and relevance, particularly for countries with limited resources—threatening both the feasibility and universality of the global agenda. A focused, realistic approach—perhaps starting with the most actionable indicators—might be necessary to turn global aspirations into meaningful progress.

Transcript

[Casey Dunning]

Foreign.

[Rajesh Merchandani]

Thanks as ever for joining me for this edition of the CGD podcast. I'm going to give you some numbers. 17, 169 and 230. It's a numerical progression of sorts. But don't worry, I'm not about to ask you for the formula to work out what the next number is. Know what those numbers are? Well, they relate to the sustainable development goals, 17 goals, 169 targets, and as of recently, 230 indicators identified and agreed. And it's the indicators that we're going to talk about today, the final piece of the puzzle, the how after the what of the SDGs. So we have a complete picture of the global goals right now, don't we? Well, CGD Senior Policy Analyst Casey Dunning might beg to differ. She's been digging into the indicators we with forensic scrutiny and a rather large magnifying glass, and she joins me to talk about them now. Casey, welcome.

[Casey Dunning]

Thanks. It's great to be here.

[Rajesh Merchandani]

It is good that we have indicators, isn't it?

[Casey Dunning]

Absolutely, yes. If you don't have something to measure the goals and targets, then there's simply high soaring rhetoric with no actual teeth.

[Rajesh Merchandani]

There was a lot of talk about the number of sustainable development goals, 17, an unwieldy number, some people think. And then the 169 targets. And now we have 230 indicators. What's your take on that?

[Casey Dunning]

I can't say I'm surprised because once we knew we had 169 targets and each of the targets had to be credibly covered by at least one indicator, it kind of automatically lent itself to a large number of indicators. And in fact, if you look at the list, there's actually 241 indicators attached to targets. So some of the indicators are playing double duty, but I'm not surprised at the ever ballooning SDGs.

[Rajesh Merchandani]

So there's 241 indicators, but actually 230 different ones.

[Casey Dunning]

Individual ones. Yes.

[Rajesh Merchandani]

I see. All right, so it's even more complicated than we thought.

[Interjecting Colleague]

Indeed.

[Rajesh Merchandani]

Fancy that. Indeed. Generally, what are the indicators going to do?

[Casey Dunning]

So we have our goals and targets, and now the indicators will offer the support system for how we actually show if a country or a region or indeed the world is actually meeting that target. So in the case of extreme poverty, the indicator shows the proportion of the population that actually lives in extreme poverty.

[Rajesh Merchandani]

So they're pretty important.

[Interjecting Colleague]

Indeed.

[Casey Dunning]

Yes, very.

[Rajesh Merchandani]

So before we kind of delve into them a little bit more, let's actually hear a few of the SDG Indicators. We got some of our colleagues here at CGD to record some of them, so let's take a little listen to that now.

[Casey Dunning]

3.2.1 under 5 mortality rate 17.Proportion of.

[CGD Colleague 1]

Individuals using the Internet 16 unsentenced detainees as a proportion of overall prison population.

[Casey Dunning]

15.7.1 proportion of traded wildlife that was poached or illicitly trafficked 13.

[CGD Colleague 1]

Number of countries that have integrated mitigation, adaptation, impact reduction and early warning into primary, secondary.

[CGD Colleague 2]

And tertiary curricula 4.7.1 extent to which 1 global citizenship education and 2 education for sustainable development, including gender equality and human rights, are mainstreamed at all levels in A national education policies, B curricula C teacher education and D student assessment.

[Rajesh Merchandani]

So there you heard some of our CGD colleagues giving voice in fine form, I might say, to some of the. Of the 230 separate SDG indicators. Casey Dunning, senior Policy Analyst who's been digging into these is with me. What did you make of some of them? I mean, I guess some of them seem pretty reasonable. Some of them tend to go on forever really.

[Casey Dunning]

So you chose a nice mix of indicators here and I think it perfectly encapsulates what we have with these indicators. Some of them are straightforward measurements, are actually measurable and have full country coverage and are actually being measured as we speak.

[Rajesh Merchandani]

So. So things like 3.2.1 under 5 mortality rate.

[Interjecting Colleague]

Indeed.

[Casey Dunning]

And that one, through some research we have shown every single country except Liechtenstein has a data point for that indicator for every year looking back to 2000.

[Rajesh Merchandani]

So Liechtenstein, if you're listening.

[Casey Dunning]

Exactly.

[Rajesh Merchandani]

Come on, measure up, step up. But the point about that indicator, about under 5 mortality rates, I suppose, is that some of these indicators are not new. They've been measured for many years. Like for example, 1.1.1 the proportion of the population living in extreme poverty. I mean, we have pretty good data for that, don't we?

[Casey Dunning]

Actually, no. And that's what's kind of shocking about the indicator and the fact that so much has been made around the battle cry to end extreme poverty. The UN has endorsed this goal, the World Bank USAID here in in the U.S. however, this indicator, 72 of the 193 UN member states have not reported a single data point on this indicator since 2000. 72 of the 193. But what's interesting is once you dig into this cohort of countries, some are what you might expect. There are 10 conflicts and fragile countries. There are 19 small island states. However, in that group of countries, 45 of them are high income countries. And this raises interesting questions for me about the utility of having a Belgium or Luxembourg report on extreme poverty. Most likely there are no people living under $1.90 a day in these countries. And so is it worth having these countries build the data infrastructure to collect and report on this data? On the one hand, it's a universal agenda, and so every indicator and target and goal should apply to every country. On the other hand, there are finite resources and time and personnel. And so do you want people to spend time on this issue that is not at all relevant to them as policymakers? But if you go with the latter logic, you then open up the whole agenda for a country to say, actually that indicator is not nationally appropriate for being and therefore I won't report on it. And you can use that same logic to apply to sexual and reproductive health and rights, or governance or democracy. Choose your highly contentious issue. And so there's. It's a really interesting kind of conundrum that no one's really talking about in the public sphere because it is highly contentious.

[Rajesh Merchandani]

Excellent that you put it out there. Now they're talking about it.

[Casey Dunning]

We'll see.

[Rajesh Merchandani]

Okay, let's get back to the SDG and the SDG indicators. So that one we talked about under 5 mortality rates, that's pretty straightforward. As you said, almost every country in the world is measuring it already. It's understandable, it's actionable. And then they're slightly longer and longer winded ones as well that are a little bit confusing to understand. What ones really stand out to you?

[Casey Dunning]

Well, I think you chose the interesting ones that, and you were far more polite about it. They, they defy logic. You can't even define them in a single country context, much less apply it to a global context. And that's the definition of the indicator, not even how you would actually operationalize it and measure it. So having certain types of curricula in schooling or there's a lot of perceptions indicators in there, which the research have shown can be highly fraught in terms of what they actually yield. And the fact that you would have a perception survey for every country that probably couldn't be nationally specific to a given context, I think opens up a lot of questions in terms of the methodology of the indicator and what we'll actually get if it's applied.

[Rajesh Merchandani]

You've been so polite for the first few minutes of this podcast. I think you've been champing the bit to say that let's get into a little bit of that and let's talk about the one that you actually just mentioned about curricula 4.7.1. It might take me two minutes to read, please, though extent to which one, global citizenship education and two, education for sustainable development, including gender equality and human rights, are mainstreamed at all levels in A national education policies, B curricula, C teacher education, and D student student assessment. What could be clearer than that?

[Casey Dunning]

Exactly. And so this is the problem is that on the face of it, no one's going to disagree that this should happen. Young children and students should know these things. But how one would actually put that into a national or a local schooling plan, the implementation pathway is far from clear. How you would measure it, far from clear. And the fact that we have that indicator in there, along with indicators on literacy and numeracy that we also can't measure, it makes me wonder, kind of in having so many indicators, if we're not kind of shooting ourselves in the foot a little bit. Because that means that minds must focus on this indicator as they will also have to focus on literacy and numeracy. And so this, the prioritization of the indicators is lacking because they're all prioritized. Yes. It's meant to be a full and complete SDG agenda, not a menu of options. However, with 230 indicators, I don't see how it can't become a menu of options. It's simply too many.

[Rajesh Merchandani]

And if countries are struggling to measure numerous data, exactly how are they going to measure whether.

[Casey Dunning]

Exactly.

[Rajesh Merchandani]

Yeah.

[Casey Dunning]

So those who made this list are the UN's statistical commission. And one of the hallmarks of the indicator selection process, we'll call it How We Got this List, is that it was meant to be a highly technocratic process, not a political one. And so the thoughts of member states and the politics behind it in terms of how it can actually be implemented and used, were a bit divorced. And so the group at the UN chose the path of having indicators that covered the full agenda, no matter if those indicators weren't ready for primetime, whereas they could have taken the path of, okay, let's include what's ready right now and what countries can actually measure.

[Rajesh Merchandani]

And so that would have been more political.

[Interjecting Colleague]

Indeed.

[Rajesh Merchandani]

Inevitably.

[Interjecting Colleague]

Indeed.

[Rajesh Merchandani]

So these indicators are simply, this is how the world should measure this target.

[Casey Dunning]

Yes. And of course, there are countries and organizations that are even arguing that the indicators are reinterpreting the target or not covering it completely. So even with the list that we have now, there are still some who are saying actually we need more to fully encapsulate the agenda.

[Rajesh Merchandani]

So in any way. So at the beginning I said, you know, we have the what, which is the goals, the indicators of the how, but actually there's another how, which is okay, the indicators of the how, how do we do it?

[Casey Dunning]

Exactly. And so there's another group at the un, the high level group on, on these indicators who is tasked with the capacity building part of it. So their job is to focus on national statistical offices and, and discover the gaps there and work on helping to build the capacity there. So that's a separate process from the what of what the indicators are.

[Rajesh Merchandani]

One of the real issues with having 230 different indicators, I suppose, is that how are developing countries actually going to measure them? Not just for the reasons that we've said, that we don't physically know how to measure some of them, but of capacity, that they have so many other things to worry about.

[Casey Dunning]

So here's what's interesting and if you look through some of the countries in response to the approval of these indicators, issued country statements about their thoughts of the process and the outcome of the indicators. And I was reading through them and the statement from Indonesia especially struck me because it actually the country listed the indicators that they couldn't measure and record on and the reasons why. And I think it's a nice little example of the problems that policymakers and statisticians will see going forward. So for one indicator, the Labor Rights Compliance Indicator, they said we would absolutely love to measure and report this data. We don't know what it means, how to do it and don't have the capacities to do that. We would love for ilo, the International Labor Organization to come and help us and we'd happily report on it. We just can't do it right now. That seems fair. Then they list another indicator on the it's 5.3.2 on female genital mutilation and cutting. And they say we can't report on this indicator because it's not nationally appropriate. And so where you draw the lines for the reasons why a country or a region can't and or chooses not to report and measure an indicator? I think it's an open question and could have serious implications for the whole agenda.

[Rajesh Merchandani]

You know, we like to be constructively.

[Casey Dunning]

Critical here at cgd.

[Rajesh Merchandani]

So what should be done about.

[Casey Dunning]

So this, I mean, there are kind of many options right now. One is to focus on a subset of indicators for which we have full country coverage and reliable reporting, are methodologically sound, and have regular updates focus on these and make sure they're actually being reported. In all countries, even the most difficult ones, even the small island nations. And then once you have this subset of indicators ticking along, then you move to the next. And helpfully, the interagency group on the indicators at the UN has classified these 230 indicators into three tiers. So the tier one indicators are the ones that would fit this Description, the Tier 2 and Tier 3. Tier 2 has a agreed upon methodology for the indicator, but there's not full country coverage and there's still a lot of work to do to report on the indicator.

[Rajesh Merchandani]

So the tiers have been decided based on how much knowledge we have about how to do indicators?

[Casey Dunning]

Yeah, well, they're actually being decided right now.

[Rajesh Merchandani]

But what you're saying is that don't just do that, but actually deal with tier one first. Let's work our way through those first.

[Casey Dunning]

Yes, that would be one seemingly logical idea. However, the problem with that is then you open the agenda up to this kind of menu of options and you only focus on a certain part of the agenda and leave the arguably some of the most difficult areas of the agenda until later. And when that later will come is anyone's guess.

[Rajesh Merchandani]

Casey Dunning, thanks for making my brain hurt. Sorry, this is a conundrum.

[Casey Dunning]

It is.

[Rajesh Merchandani]

It's certainly one that you've done a lot of work on and will continue to do work on. And you can check out Casey's work on our website, cgdev.org Casey, great to have you on the podcast. Thanks for joining me.

[Casey Dunning]

Thank you, Rajesh.

[Rajesh Merchandani]

Thanks very much for listening. And please do remember to join me, Rajesh Merchandani, for the next podcast from the center for Global Development.