← Into Africa
Into Africa
The think tank making impact evaluation skills accessible to every Tanzanian
00:28:02
Catherine Nzuki is joined by Dr. Constantine Manda, the co-founder and inaugural Director of the IE Lab at ESRF; and Assistant Professor of Political Science at the University of California, Irvine. They discuss what impact evaluation is, why it’s important for every Tanzanian to be introduced to impact evaluation methods, and how the IE Lab tailors its training for policymakers and local authorities.
Loading summary
Transcript45 lines
- [00:02]
A
By 2050, one in four people in the world will be African. The choices that we make now across Africa will shape the world's collective future. These are the stories, the trends and the issues we face in the present that will define the coming decades. Welcome to the Afropolitan. I'm your host, Katherine Nzucki. Impact evaluation is something we traditionally think of as an academic or technical skill. The Impact Evaluation Lab at the Economic and Social Research Foundation, a think tank in Dar es Salaam, Tanzania, well, they're working to change that. The core mission of the Impact Evaluation Lab is to expose every Tanzanian to ie, even at its simplest idea, regardless of education levels. And so I'm joined today by the co founder and the inaugural director of the IE Lab, Constantine Manda, who is also an assistant professor in the Department of Political Science at the University of California, Irvine. Constantine, welcome.
- [01:15]
B
Thank you.
- [01:16]
A
To start off, could you explain what impact evaluation is?
- [01:21]
B
So impact evaluation is really a catch all kind of broad term on trying to causally identify whether a program or policy or a certain intervention in people's lives. Typically in developing country settings, but not exclusively, there are obviously extensive research done even in the advanced economies to try and be confident. It's really a way to say, you know, based on the methodologies that we're engaging with, based on the assumptions of these methodologies, how confident are we that we've been able to say for certain that the outcomes that we care about and that we're measuring are realized because of this program, because of a particular policy, because of a particular intervention, and not anything else? Human lives are very complicated, as you can imagine. It's impossible and actually in many cases constrained by the ethics of doing human subjects research, to be able to control every single aspect of a human being, especially those who are part of a human subjects research. And so trying to specify a set of assumptions that once valid, you would be very confident statistically to say that, you know, the outcomes that we are observing are coming from the program and not anything else about these people's lives.
- [02:45]
A
Please just tell us what motivated you to create the Impact Evaluation Lab, the IE lab.
- [02:51]
B
The motivation grew out of a fellowship program that I was a part of. This is a visiting fellowship in the Department of Economics at the University of California, Berkeley. So this was in the fall of 2012. Part of that program involved receiving funding that the fellows could then take back to their home country in East Africa. In my case, it was Tanzania. When I came back to Tanzania in the spring of 2013, I was very, very interested in trying to close the gap between the need for impact evaluation in Tanzania and the supply of that impact evaluation from local researchers. I think all of us, and really the impotence of that fellowship was to try to expand access to the training and methodologies and resources necessary for one to design and conduct an impact evaluation to try and make sure that it isn't just scholars based in North America, in Europe, who are able to successfully publish results from impact evaluation, especially evaluations of government programs in Tanzania.
- [04:08]
A
Yeah, I would love for you to help me, you know, paint a picture of the impact of your work. You, for example, and your colleagues evaluated behavioral nudges that would encourage people to be vaccinated for COVID 19. How did you go about that?
- [04:24]
B
Right. So during the pandemic, Tanzania had a government administration that was very skeptic of vaccines that later came out, but even before that of a lot of the mitigation measures. So social distancing, wearing masks and so on. Unfortunately, the president at the time passed away a few months after he was reelected into his second term, around the pandemic. And we got a new administration that was the first in Tanzania to procure these vaccines. So as researchers, we wanted to contribute to that enterprise. So we nevertheless thought, okay, we gotta identify a cost effective or very, you know, cheap intervention that could do it. So we concluded that SMS as a delivery method is quite cheap. You could buy, you know, bulk SMS bundles for less than $10 and reach hundreds of Tanzanians. And so that's what we did with a lot of my junior colleagues who really were brilliant at leading this in terms of designing it, the intervention itself, figuring out how to deliver these SMSs, tracking outcomes and eventually analyzing this. And two of these colleagues of mine, one a former colleague who's now a pre doc at the Economics Department at Zurich, and currently another colleague, Ms. Noella Ringo, who is at ESRF, will be presenting this work at the Cent for Studies of African Economies conference at Oxford in March.
- [05:46]
A
Fabulous.
- [05:47]
B
So this I think is really illustrative of. I think one of the greatest legacies that I hope to continue to have is the capacity building of young researchers in Tanzania. And a few of them have succeeded in getting really amazing admissions to doctoral programs in the us, in Europe, competitive fellowships at, you know, really the most reputable institutions around the world.
- [06:08]
A
Oh yeah, I got to meet, I think most of the crew last summer. And I have to say, I mean, it felt like I was in the presence of truly the next generation of like, leading scholars. And I mean, I just Want to underline that they are some of the smartest people I've ever met.
- [06:21]
B
Absolutely.
- [06:22]
A
And so with like an SMS trial like that, how do you go about randomizing or how do you go about testing the efficiency of one text versus another?
- [06:31]
B
Right. So the experiment was set up as a randomized placebo controlled trial, which is a mouthful, but what it means, Katherine, is that we had two treatments. So we initially had many, many ideas of to incentivize Tanzanians to go get vaccinated. Remember, this is just a very short message that we're sending through their mobile phones. We sent that multiple times over a course of six weeks. Nevertheless, we were very skeptic of anything hitting. So we kind of concluded on two aspects of Tanzanians lives. One was that we realized no matter how skeptic Tanzanians may be on the vaccines, hardly any Tanzanian would want to contribute to somebody's death. So we implemented an SMS campaign that highlighted empathy for others. So, you know, you may not be fearful of COVID 19, but why not reduce the chances that you're gonna contribute to somebody's illness or death by just getting vaccinated? So we call that the empathy treatment. So we sent a portion of our sample, what we call the empathy message. The second treatment, Catherine, was trying to make Tanzanians think a lot about their future. So the message was saying, well, you know, you might not worry about COVID 19 right now because maybe you got infected and survived, but you don't know the long term health implications. You don't know how much it's going to cost you to take care of yourself given multiple COVID 19 reinfections over the years. So why not just go get vaccinated? Because it'll reduce the probability that you'll have adverse health outcomes. And so we call that the time preference treatment. And then we sent a placebo treatment to, because we were worried that people might say, well, did they get vaccinated because they received an sms? And you're comparing against a pure control group that didn't get any sms, Right. And then you're going to call all the groups later on and you know, if you see that the group that received the SMS got vaccinated, was it because of the content of the SMS or the fact that they just got a fancy SMS from an ngo? So we sent an SMS trying to encourage Tanzanians not to support child marriages. And we figured it was not health related enough so that it would have the same effects, but it would still be something meaningful to also test because then you could Compare on the outcomes of child marriage as well and attitudes on child marriage in Tanzania by comparing those who got the placebo anti child marriage SMS against those that got the COVID 19 SMS. And then you basically randomly assign folks within these three groups, either the empathy the time or the child marriage SMS treatment.
- [09:00]
A
And between the empathy and time treatments, was there one that was distinctly better at encouraging people to get vaccinated?
- [09:09]
B
So statistically, no. The treatments were indistinguishable from each other. But qualitatively, the empathy treatment did slightly better than the time preference. So this suggests that Tanzanians care about others more than they care about their future self.
- [09:25]
A
Like Udyama, this sort of notion and it's, you know, I don't want to overstate the context. Yes, yeah. But when you said that, I was thinking about Ubuntuyama, all these sort of like communal. Communal, yeah. A sense of community, a sense of fraternity.
- [09:37]
B
Absolutely.
- [09:38]
A
Wow. And you touched on briefly about policy. I know that you're running specific trainings for government officials and local government authorities. What do their trainings look like? How is it distinct from maybe, you.
- [09:51]
B
Know, the regular training?
- [09:52]
A
Regular training, yeah, yeah.
- [09:53]
B
So the regular training is much longer. The regular training is about three weeks over three months. The government officials typically receive anywhere between one day training to five day, you know, a whole week's worth of training. The one day training is usually just targeting the executive branch of the bureaucracy. So this will typically be permanent secretaries, this could be ministers in government, folks who really, you know, they can barely spend a day learning these things. But the objective there is just to expose them at minimum to the ideas of bias in statistical research, but also allow them and really equip them with the abilities to then consume research so that they're able to very broadly adjudicate whether this is a research project that the government should put resources towards, or this is a project that has a lot of bias in it. And maybe we shouldn't put too much weight on that evidence in this literature. There is really a huge ongoing debate about whether policymakers prefer rigorous research information or that they prefer more contextual research information. So would they be willing to trade off a little bit more bias but for the fact that the research project was done in their own context? Now, the research is still ongoing on this, but for the most part there is a high demand for rigorous evidence by policymakers. There's experiments that are done with mayors in Brazil by colleagues at Berkeley and elsewhere that shows that, like, you know, if you do offer them information that comes out of rigorous evidence, policymakers take it up and they change their, their implementation day to day based on this new information. But context is also very important. And so the Impact Evaluation Lab at ESRF is trying to really give policymakers their cake and their abilities for them to eat it at the same time.
- [11:47]
A
That's so interesting. And because, yeah, you're absolutely right that these policymakers have such intense demands on their time that, you know, even if you get them for a 9 to 5, that's huge. I'm curious why you chose to hone in specifically on BIA versus maybe other tools to engage with rigorous research.
- [12:07]
B
No, absolutely. So, you know, I don't want to stack the deck here towards just saying that, you know, these methodologies are supreme. There are obviously many other ways. And qualitative methodologies as well are really important in understanding mechanisms and really understanding the context that the human subjects of the research are really experiencing. But you know, on average, you can imagine that, you know, a minister who is sitting there and receiving guests who are trying, trying to get them to think about various investments. They're receiving researchers from across the world who are telling them X is the panacea or Y is the panacea. It becomes a really high noise to signal ratio. Right? So you're getting a lot of noise and you don't know among this noise that I'm receiving, where can I detect the true signals of what actually works? And so, you know, I think that's the kind of key, important part. And understanding the inherent bias in a lot of these summary statistics or a lot of the kind of research information that they're consuming is going to be very important at the very minimum, to allow them to not invest in things that are clearly not going to work or things that, you know, are too expensive given the impact that they make. So I think that's really the critical aspect is to try to get the policymaker to think about bias more than anything else.
- [13:31]
A
That is so interesting you mentioned and these training seminars that are happening at the Economic and Social Research Foundation. So say I come to one of those trainings, what does the day long or week long training consist of for somebody with a casual interest in this skill? And what would be the main evaluation methods that you'd want to get across?
- [13:51]
B
Right. So, you know, many students who study these things more formally in university spend at least a semester. So a week is not anywhere close to being enough. This is why we articulate that these training sessions are really an introduction to these methods. And for the advanced participant that they should definitely try to get more formal training. So what that entails, Katherine, is an introduction on understanding causality, including the fundamental ideas of counterfactual. So what would have happened to your intended beneficiaries had you not intervened by providing a particular program or intervention? And so the kind of workhorse in that toolkit includes what we call a randomized experiment where you're able to say the probability that somebody receives or doesn't receive this program is going to be unrelated to anything about these individuals or anything about their potential outcomes from the program. And so that's really the major workhorse that we're able to introduce to students during that week. We do obviously cover very briefly other methodologies, including what is known as the difference in differences, which compares beneficiaries to non beneficiaries across time, and various other methodologies. But the randomized experiment is at the minimum, what's important that we want everybody at the end of that week to.
- [15:13]
A
Be able to know, yeah, I want to attend one of them. Hopefully they'll let me in at esrf.
- [15:19]
B
Absolutely.
- [15:20]
A
And I really want to get to the core mission of the IE lab to have every Tanzanian, regardless of education levels or their job sector, to be exposed to impact evaluation. Why is that important?
- [15:33]
B
I think it's very important because we have limited resources and unlimited ideas on how to spend those resources. For a country that is a lower middle income country such as Tanzania. One of the things that development economists worry about is what we call a middle income trap. So this is the idea that you move from low income to a middle income bracket, but you stay there and you don't eventually get to the point where you could be called a high income economy. Now impact evaluation is going to make that process slightly more efficient and slightly faster. So this is not to say that everything needs to be evaluated, but certainly knowledge on what works and what doesn't work is going to save enough money and resources, including people's time, such that we would be better able to inform the government and the private sector on what works for development. And so that I think is the most critical aspect of why it's important for folks to be exposed to impact evaluation. And you know, not everybody needs to be publishing in the top journals by understanding this impact evaluation methodologies, but at the very minimum, be good consumers of impact evaluation ideas and research and results. I think as somebody who grew up in Tanzania, seeing many wasteful programs being implemented, it's really frustrating, Frustrating. And this is really why I've become very passionate about Program evaluation.
- [16:57]
A
Yeah. I want to turn to just how creative you guys have been at the Impact Evaluation Lab. You told me an anecdote last summer about a radio play that you used. Could you just tell us more? I remember it being so fascinating.
- [17:14]
B
Yeah. So this is a series of projects with co authors Donald Green at Columbia University, John Marshall at Columbia, Dylan Groves at Lafayette College, and colleagues at esrf, including Noel Laringo, Zakayo Zakaria and others. And really is to try to use the power of radio programming to try and fight against misinformation and disinformation in the health sector. So this grew out of the concerns during the pandemic that people were getting the wrong information about the efficacy of masks, the efficacy of vaccines. And given that most Tanzanians access their news through radios, we figured this is an amazing infrastructure to deliver really rigorous information from medical professionals that is informed by rigorous research on what works and what doesn't work. And so there's a series of studies, some of which have already been published, some of which are also going to be presented. So another colleague of ours, Marco Kitundu, will be presenting at Oxford as well, along with Noella. Noella will be presenting the SMS experiment. And so there again, you know, it's really about using the power of both information from experts. And we're working very closely with the Ministry of Health, the National Institute for Medical Research, our co authors and collaborators in this work. And you know, there's a series of interventions using radio programming also doing on the ground mobilization of folks and having a kind of call in radioed set of treatments where you have individuals able to interact with a physician who understands the efficacy of vaccines, the efficacy of wearing masks and some of these other kind of health seeking behaviors that we would want people to adopt. And so it's really a great way of merging the abilities of research to tell us something novel and something powerful while still contextualizing it to the particular context that people experience.
- [19:07]
A
I mean, it's so clever on so many levels because one, with radio, you're meeting people where they are. You know, if you've been stuck in dark traffic, you're listening to the radio. If you're wherever you are, we have radio and I don't think it's exclusive to Tanzanians, but we do love some entertainment. Right. And to use those two like existing platforms as a way to disseminate public health information is super interesting. I want to just ask you for one more anecdote, which is a meal voucher program that you guys ran In Dar es Salaam, could you tell us more?
- [19:37]
B
Right. So we also have a randomized experiment as well, trying to understand if providing vouchers to households in urban Dar es Salaam, in one of the poorest neighborhoods of Dar es Salaam, to move away from unhealthy foods. So, as you know, Catherine, you know, coastal Tanzanian culture really permeates how we consume food. So, you know, some of the most unhealthy food items that we consume in the morning, chapatis, vitumbuas, which are kind of a donut, if you will, tend to be quite oily, and, you know, deep fried food items that are not very healthy, very savory, but not the healthiest of food options. This is not necessarily unique to Tencent. I mean, even in the United States among poor neighborhoods, there is a limited availability of healthy food items. Right? And so the goal there was if the healthy items are more difficult to get because they're further away, or that they're just slightly more expensive, their price is higher than the unhealthy options, or that the unhealthy options are easier to cook at home, or rather, you know, the unhealthy options are easier to procure nearby, but, you know, cooking and getting ingredients for the healthy options is harder. So we provided these vouchers, and then we served survey these households at various periods of time, including several months after the voucher program ended. And we are now actually looking at the preliminary information, and we are collaborating with researchers at IFBRI, based here in D.C. not too far from this building, as well as researchers from the center for Effective Global Action at Berkeley, along with our colleagues, George Temba, a young researcher at esrf, and there as well, we're seeing really great results. We're seeing that the households that received these vouchers were able to increase their consumption of healthy food items. And then not only that, but even months after the end of the program, we're still seeing that this behavior is relatively stuck. The other cool thing about this project is we innovated in how we deliver feedback to the human subjects of the research. So we dramatized the results of the research and hired a drama group at the University of Salaam. We presented their research, we presented our results, and then they formulated a bunch of plays that we physically then went back to these communities to inform everybody, right? Because we didn't want just this benefits to accrue to just the people who received the vouchers. We wanted to explain the results of this project and that it could be applicable across all other households. And so we are collaborating with other researchers in Berkeley who are interested in kind of doing the same thing in other research settings as well. Because a lot of times we just go in and we take from these communities, we collect data and so on. We publish at fancy top journals, but there's very little feedback. And, you know, sending them a policy brief or inviting a few of them to a workshop is not sufficient.
- [22:26]
A
Yeah. Could you give me examples of the types of healthy foods that people could purchase?
- [22:31]
B
Right. So we focused on two major food items. Eggs. Because Tanzanians typically eat eggs from free range chickens, they tend to be much healthier than some of the other alternatives that people, especially during breakfast, and unflavored milk and yogurt products as well. So those were the kind of major subsidies that people could receive. There was a fancy little voucher that we designed that, you know, somebody would then go to the shop and grab some eg. And then, you know, whatever their total amount that they were meant to pay, they would pay, however, the sticker price on that voucher less. Right. So if it was 10,000 voucher and they, you know, bought 15,000 shillings worth of items, then they would only pay 5,000.
- [23:11]
A
And you are a assistant professor at UC Irvine and I would love to hear more about your work there. But as a scholar, as a professor, I would love to hear your thoughts on that tension between people that are being studied and researchers. Right. Because they tend to be a habit of extraction. Right. Like we are going to study you, we're going to test out these different interventions and then, you know, folks then come to, you know, D.C. or London and they present it at a fancy conference. And the people that are part of these experiments don't really get to be part of the findings and the results. So it's amazing that you are not only coming back with your findings, but making them digestible because quite frankly, very few people have time to sit down and read a 10,000 word policy brief. Right. So what do you think about that tension? And just more broadly, I'd love to hear more about the work that you're doing at UC Irvine.
- [24:03]
B
Yeah, So, I mean, I try to tie both of those aspects together. I think the first thing is we're doing much better than we were a few decades ago. And in part because the actors, not all of them, but I think for the most part most of them are people with goodwill, people who want to do good, they want to improve the state of the world. And so they invested in diversifying who gets to do this research. So I know it may seem unpopular these days in D.C. to say this, but diversity is really, really the key in relaxing this tension. I think if we have people of all sorts, of all sorts of nationalities, racial persuasion, sex and so on, you are going to allow for the ability of people to have a much greater empathy for human subjects. Because if you were experimenting on Tanzanians and you yourself are not Tanzanian, and you may have some blind spots with respect to figuring out how to do that as ethically as possible. So I think that's the first thing. I think we're doing much better than we were a few decades ago, but we are still not close to the kind of optimal equilibrium here. And I think related to that, you know, this is the reason I'm, I'm in D.C. today. And this week is really some emergent work that I want to do at UCI, trying to really merge the literatures on studying black people in the continent and black people in the diaspora. And so the University of California provides for a grant called the University of California HBCU Initiative. So HBCU here stands for Historically Black Colleges and Universities. And so I'll be at Howard later today in the economics department to present some work, but really to try and deepen connections and collaborations with Howard University and Morehouse College and the University District of Columbia, who are partner institutions in a grant application to bring HBCU students to our campus in Irvine in California as a pipeline to applying to PhD programs across the University of California and the US more broadly. And really, you know, this is once again inspired by the success that I have seen in my whole life, in my whole career, and in the work and building capacity of young Tanzanian researchers. And as part of that grant application, Katherine, we're hoping to also take the students at the end of that summer for at least a couple of weeks to Tanzania to be a part of observing how data is collected on these experiments that we're involved in, but also to think deeply about community centered ethical research on human subjects. And so once again, you know, diversity, diversity, diversity. And really that's the key. If we get as many different people who reflect the amazing diversity of the world, we'll get more interesting questions and hopefully learn more about our ourselves as a result.
- [26:53]
A
If you guys need a spare podcaster, you know you always need something.
- [26:59]
B
Absolutely. Yeah.
- [27:00]
A
Oh, Konstantin, this has been the most fascinating conversation. Thank you so much. Thank you for your generosity of your time. I just want to underline the point that Konstantin has been making. Not only are you brilliant and one of the coolest, like, scholars I've met. The team at the Impact Evaluation Lab is filled with young, smart, hardworking Tanzanian researchers that are doing some incredible stuff. And I'll link some of these publications in the show notes. Thank you so much for your time.
- [27:29]
B
Thank you, Katherine, for the kind words. But really, the true brilliance are, as you said, these really young colleagues of mine who are leaps and bounds more brilliant than I can ever hope to be.
- [27:37]
A
Oh, no. Okay, you're downplaying it. And I know they're gonna listen to this and say, we heard Manda on the podcast downplaying how smart he is.
- [27:43]
B
Thank you, Kathleen.
- [27:44]
A
Thank you. Thank you. Thank you for tuning in to the Afropolitan. Please subscribe to the Afropolitan wherever you listen to your podcasts. This podcast is produced by Sarah Baker and our music is by Wonder Child.