A (3:51)

In a way, this is the scorecard.

B (3:54)

System, the country scorecard system.

A (3:55)

So briefly explain that to us.

B (3:57)

Sure. So the country scorecards are basically a way that MCC gathers data from a number of independent third party institutions. That is, MCC doesn't create the data itself. The countries don't create the data itself. It comes from places like the un, the World bank, et cetera. And they assemble those data and they create country scorecards using them. And they look at a number of different indicators across the areas of, you know, democratic governance, corruption, health, education, economic governance, things like that. And they are able to compare a country's performance relatively across those indicators. And doing so gives a more objective way for the agency to pick out which countries should be funded with these large grant funding for mcc.

A (4:40)

Okay, let's think now about sort of areas perhaps where you both have concluded there is room for improvement. And one of these is this central pillar that results matter. Now that seems obvious, but when it comes to measuring and reporting results, what do we know about how well MCC has done in its first decade?

C (5:00)

Well, MCC, one of MCC's notable accomplishments is the establishment of what some people have called its framework for results, which requires that every investment has an explicit and transparent description of why these resources are being invested, for what purpose. And beyond that demonstration demonstrating that the impact, the expected results are larger than the costs that are going into the investment. That process is one which is meant to govern all of MCC's programs and reflect a little bit of the skepticism of aid effectiveness that Sarah mentioned earlier. The idea that if we're trying to improve the overall quality of the portfolio, we have to stop doing things that either we know are not going to be very impactful or to stop doing things about which we're not very sure. In other words, if there are things that we know will deliver substantial benefits to the people we care about, then we should focus our investments and our efforts in those things, rather than on speculative investments or on investments that, like I said, we know prior to making them are not worth the money they cost.

A (6:15)

And have there been such investments as speculative or made knowing that they're not going to pay off?

C (6:21)

So the Good news is MCC's portfolio, for the most part has stayed true to its original mission. We have found that something on the order of 90% of the investments were supported with economic analysis that suggested the anticipated results would justify the level investment. Now, that means that 10% did not. That is, 10% covered investments that either had analysis which showed they should not be made, that is, the results were not sufficient to cover the costs or were speculative in the sense where either the program itself, the design was unknown, or the benefits were unknown. And many observers would look at that and say that's actually a pretty good Track record. Only 10% of the portfolio doesn't fit the stated analytical framework.

A (7:17)

That's still about $120 million a year worth if you look at the annual budget.

C (7:21)

That's right. And something on the order of a billion dollars of the 10 plus billion that has been spent in the first decade.

A (7:27)

So a billion dollars wasted, the term.

C (7:29)

Wasted is something that some people would dispute. The question of how much good is provided is often an open question. And so if one spends $100 million on a project whose good value to the beneficiaries is only $50 million, then at some level you might argue that the $50 million is the foregone opportunity cost. The wasted amount, often it's unclear exactly how much the benefits will be generated at the end of the project. And so MCC's ex ante analysis, the estimates that they make up front, are really important because those that tend to have a high expected benefits are able to tolerate missing the mark by some amount. Those projects which are speculative or are known in advance to not be worth the investment then can be worth a lot less as well. And so it's really a question of it's not the glass being half full, the glass is 90% full. But I think it's fair for supporters like Sarah and myself who look at MCC as an institution whose mission statement is very explicit to then question two, what happened in those 10% of the cases? And also what's happening during implementation. And that's another thing that we focus on is that often what happens is even the projects that had good analysis, good estimates up front encounter problems during implementation. And when those problems are encountered, MCC does what they call a rescoping exercise, which means they have to shift money around, shift money into a project that needs more, shift money out of a project that doesn't need as much. And those decisions, unlike the decisions that are made up front, are not being made in a transparent way. And we know from our own experience that sometimes those decisions are not consistent with that focus on results.

A (9:29)

Okay, well, that's an interesting point. We're just going to take a little short pause and we will be back to discuss the recommendations that you guys have come up with, many of which revolve around transparency with the Millennium Challenge Corporation, the mcc. I'm with Sarah Rose and Frank Wiebe, and this is the CGT podcast. Stay with us. You're listening to the CGD podcast with me, Rajesh Merchandani. Today we're looking at 10 years of the MCC, the Millennium Challenge Corporation, the US agency tasked with reducing global poverty through economic growth. An innovative approach at the time. But how is it doing as it looks forward to its next decade in business? Senior policy analyst Sarah Rose and visiting fellow professor Frank Wiebe have been studying this topic. They're both with me now. Guys, you have done a lot of work on this. Three very substantial papers and each with a policy brief. And they're available to listeners on, on our website, cgdev.org going through these papers, I counted, I think, 31 recommendations. Is that right? 31 sounds about right. Yeah. And 15 I figured were about transparency or surrounding the area of transparency. So would you say that that is your biggest critique and recommendation?

C (10:55)

Well, I think there's the old adage that sunlight is the best disinfectant. Right. And MCC has built into its practices some of the best systems for driving an aid agency to deliver results. And that's why both Sarah and I feel strongly about continued support for the agency. That said, we understand that the senior managers within the institution face political pressures, just like other managers do in US Government agencies and, to be honest, in other multilateral development agencies as well. As they face decisions that put pressure on them to spend money rather than have unobligated funds on their ledger. They are forced to deliver certain levels of resources to country partners that those partners expect. And all of these pressures, the pressure on time and the pressure on amount, put downward pressure on the quality of the portfolio. And we see that in every development institution. And within that there are also then personal and professional incentives that are at times at odds with the overall quality of the program. That, again, we see in every institution. And so what we have seen at MCC is that these kinds of decisions are limited to a large extent by the structures that force transparency and allow outside observers to watch what kinds of decisions are being made. Oftentimes, it is where the transparency systems provide enough timely information to allow those external observers to hold managers accountable at the time. So even where MCC releases information, for example, on the quality of a compact, if it comes six months after the program has already been started, interest in the program has waned and the ability of anybody to raise questions about it has diminished considerably. Similarly, when decisions are made midstream, say two and a half years into a five year program, and that information only becomes available after the program is completed, obviously managers are susceptible to making decisions for these political pressures rather than the main focus of the institution, which is to deliver results with the resources they've been entrusted with. Right? And so in that sense, transparency and the, and the pressure to tell people what they're doing and why becomes a force to ensure that they're making decisions oriented around results rather than around these other factors that often influence decisions in.

A (13:42)

Your work, which, as I said, is available on our website, cgdev.org, another recommendation that you talk about is increasing flexibility or the ability to extend compact, which are the agreements with countries after the initial compact is concluded. Do you think some observers might be wondering why after, say, five years of a compact with mcc, a country isn't fixed in inverted commerce?

C (14:08)

One of the things that we found, especially in the early years, was that when MCC was created, there was both a lot of optimism about doing a differently, but there was a lot of naivete around the model as well. And I think a lot of old development workers would agree that whenever we hear the term transformational development, we become suspicious because rarely are aid agencies the agent of transformational change. Instead, development is a long term process that is not linear. There are backsliding moments, there are setbacks. And at some level, the best that an aid agency can hope to do is be a positive partner in that process. And so they said when MCC was created, there were some people involved in that process or observers watching that process who thought that MCC's engagement over a five year period with a substantial amount of grant resources, so MCC provides a five year and a large grant would be sufficient to, in some sense, Take these developing countries over the top and move them from developing to developed or at least middle income status. And what we have found, and not too surprisingly, is that instead where MCC has been most successful, you can see McCann establishing a mature bilateral partnership that does not end after five years and in fact could become more effective in a second compact and possibly even a third. The length of that relationship depends more on the country's starting point and the pattern of success and the success of the relationship between MCC and the country. And so those early years where people thought one compact would be enough was clearly naive, that view. Since then, MCC has negotiated with congressional supporters that, well, a second compact is okay, but again, there seems to be this sense of but we don't want to see a long term relationship. I think Sarah and I look at that and say, well, in some cases there shouldn't be a long term relationship. In some cases, especially for countries that start out as low or middle income countries, it's conceivable that over a 10 year period they would graduate to the point where a relationship between the US Government and the partner country through MCC would not be the most effective way of engaging. But for many countries that's not a 10 year process. And what we would like to see is that discussion around multiple compacts be toned down and make it more of a practical conversation about can the US Government use the MCC modality in a positive way that supports the development process and U.S. bilateral relationships with these well governed developing countries.

A (17:14)

So finally, just cast your eyes forward for the next 10 years. Sum up for me. What are the pitfalls that MCC should avoid or what should it carry on doing to do an even better job in the next 10 years? 10 years, because it's clear that you think it's done a good job in the first 10 years.

B (17:33)

MCC deserves big kudos for its largely transparent basis for selecting its partner countries based on independent metrics of good governance. It's very quantitative, it's largely objective. However, there's somewhat of a downside to the system as well, in that a system that is quantitative can look more prescriptive than it really is. And in some cases MCC and its board of directors, which is the entity that's responsible for making country eligibility decisions, has taken a more literal interpretation of the scorecard than the quantitative indicators can really support. These indicators are the best available to measure these kinds of good governance policies, but they're not perfect. They don't measure the absolute precise truth at any given moment in a particular country. And so countries may meet MCC's criteria and look like a reasonable development partner in one year and in the next year the scorecard might look slightly different and not pass the formal criteria for eligibility when there's been no change on the ground in policy performance by the country whatsoever. In some cases, situations like this have in the past led to decisions to change a relationship with a country, a country that's been entering into a good faith partnership with the United States government on the basis of a small statistical noisy change in an indicator. The US Government has made decisions to curtail a relationship to, to suggest that it won't continue a relationship in the future unless certain actions are taken. And this, I think is deeply problematic in terms of MCC's relationships with its countries, in terms of the way it makes decisions to work with its countries, et cetera. And so in the future, what I hope to see is that MCC and its stakeholders, including its board of directors, including all those who watch the selection system and are interested in it, are able to maintain a nuanced view of this system and allow MCC to make decisions that are not purely tied to the indicators, but are based more on what's the policy, the actual policy performance of what's happening on the ground in these countries.

A (20:09)

And Frank, one from you perhaps to kind of, you know, do this and you will succeed point for me, that's.

C (20:15)

Very easy and do this and you will succeed. You don't spend money on things that you know are not worth the cost. MCC has for the most part avoided that. But as you pointed out, even a small share of a large pot of money can become a large amount over time, and especially in countries where time constraints have forced decisions to move ahead even before projects weren't. Baked is a term that MCC staff often use. They don't know what's going to happen. In Indonesia, there was a $50 million procurement reform project that had virtually no design that was approved. Similarly, in that country, There was a 300 million plus project that's described as Green Prosperity, where only the broadest outlines were known in advance. That's a large amount of money to put opposite an undesigned, untested, unproven mechanism. There were a lot of pressures on that compact, there were a lot of pressures on other compacts as well that led to poor decisions. In terms of accountability, I would say MCC needs to emphasize its focus on accountability, not making bad decisions on investments. And when they determine that investments they thought were good turn out to not be good, they need to have the strength and the discipline to re engage with partners to shift the money to investments that make sense.

A (21:52)

Okay. Frank Wiebe, Sarah Rose, thank you very much for joining me for this week's CGD podcast. Looking at looking back at 10 years of the MCC and casting our eyes forward to the next 10 years as well. Frank and Sarah's work is available on our website. As I said, cgdev.org check it out. I'm Rajesh Merchandani and join us again for the next CGD podcast.