A (3:34)

And, of course, we'll put a link up to your website. It's pretty easy to find givewell one word. Google it. I have, as you probably know, as a listener to econ talk, Ellie, that I have some unease with some aspects of the effective altruism movement, which is part of what motivates, I think, givewell's philosophy. But I want to start by saying that your website's spectacular. The transparency and the care and the openness, it's really unparalleled, not just in charitable organizations and most organizations, any kind of organization, if you're actually doing what you say you are, and I lean that way. So I think you are. And it's quite impressive, and it's really wonderful. And it's in many ways a model, I think, for not just, again, charities, but other organizations. And your openness of both expressing uncertainty and imperfect confidence. There's a lot of humility in your website. It's very, very, very impressive. But I want to start off with a different kind of question. We'll get to many of those issues you raised in your description. But thinking about you and Holden as individuals back in, say, 2006, and if most people sitting at home listening to this conversation, if I said to them, yeah, you want to spend your charitable money, well, what would you do? And I think there are all kinds of different answers people might give. One standard answer. We've had Dan Pauletta on the program. He hates this answer. But many people would look at overhead and what proportion of giving goes to the actual recipients, et cetera. There are all kinds of different ways you could attack the challenge. I want to do good with my money. I don't just want to feel virtuous. I want to be virtuous. And that's my favorite thing about the effective altruism movement and what you're doing at GiveWell, which is to actually make a difference, not just feel like you're making a difference. But I'm curious what you had in your head long time ago. Throw your mind back there and you think, well, I'd like to spend this money wisely. Where do you start? It's a tough question. How did you begin to tackle that question? And how did the answer to it evolve over time?

B (6:03)

All the way back at the beginning, I think we recognized the exact challenge that you're describing, that we could be out there essentially trying to boil the ocean. And so very early on we said, well, we're never going to be able to get the perfect answer. We let's pick an area that we can dig into. And that's why I chose clean water in Africa back, way back when, in 2000, in 2006. And it felt like a big and challenging area, but one that was tractable enough that I could learn something. As time moved on, and certainly when GiveWell started full time, we took a more serious look at how to address this question and started very broad. In GiveWell's first year, we looked both at international causes, but also organizations helping people in New York City, where we were living at the time. And it was in that process that I think we learned something that probably is obvious to many people, but was not obvious to me when I was 25, which was that the needs and then the opportunities to help people in low income countries are, are just so different than the possibilities for helping people in New York City. And that's not to say that it's not to diminish the problems that people face in New York City, but the magnitude of the problems overseas are so, so large. And so I think speaking personally or psychologically, we started with what seemed to me maybe even a different question than the one we're describing today. It was, I want to give some of my money to charity. I want to help people. How do I know that it's really helping anyone? How do I know that it's accomplishing a lot of good, never mind the most good. But as we dug into that question, we started to see how broad the differences were. And that to me was one of the most interesting questions I had ever been faced with. And eventually, over time led to more and more work to try and get a better and better answer to the question of how can we direct these funds where they'll do the most.

A (8:09)

And Right now you recommend four charities on your website, which is very unusual. Right. A lot of there are websites that they'll grade. Charities typically have things like overhead. Again, not efficiency or efficacy or impact, but just, you know, they seem to be fairly well run, whatever that means. But you actually GiveWell actually picks four charities that you hope have the biggest bang for the buck. And biggest bang for the buck is obviously a serious philosophical and statistical challenge, but you take it very seriously, which is spectacular. And tell us what those four charities are and give us an idea of what the bang for the buck is.

B (8:59)

Yeah, so let me describe them and let me just clarify one thing first, which is we do recommend these four organizations. Now we also have donors who give to something different. We call it an all grants fund, which gives money to things that don't qualify as these top charities. You can think about them as maybe the blue chip organizations. And I'm happy to talk about those other things too in a minute. But just what those four are. So those four two focus on malaria. One is Malaria Consortium and its Seasonal Malaria Chemo Prevention Program. One is Against Malaria foundation in its Insecticide Treated NET program. The former SMC is the acronym is Medical Treatment to Prevent Malaria. That's given to children during the rainy season when they're between the ages of 3 months and 5 years. Malaria nets are probably very well known. It's giving people nets to put over their beds while they're sleeping so they're not bitten by mosquitoes at night. And both of these programs have very strong evidence behind them that they reduce malaria cases and then malaria deaths. A third organization is Helen Keller International's Vitamin A Supplementation program. Giving vitamin A supplementation to again children under the age of five twice a year has been shown to significantly reduce child mortality, especially in populations that are vitamin A deficient. So this is sort of very different than what we think about in a high income country with respect to vitamin supplementation. These are people who have significant vitamin A deficiency. And this program reduces child mortality by about 25%. And then finally an organization called New Incentives. And it actually got started with some funding from this all grants fund that I mentioned. But today what it does is provides small cash transfers to caregivers to incentivize them to bring their children for routine immunization visits in northwest Nigeria. This is an area where immunization rates are among the lowest in the world. And so this program brings more people in just very roughly, we would estimate about $5,000 to avert the death of someone who would have otherwise died from malaria, lack of immunization, other childhood illness that vitamin A supplementation protects against or supports better health. And so, you know, about $5,000 per life saved is a way of just having a benchmark for what we're talking about here.

A (11:23)

And you qualify that. I mean, it's obviously your best estimate. It's obviously an estimate. You qualify because you're aware that. I'll give you an example given, give some examples of the, of the reasons you qualify. For example, one being, you know, that not everybody's going to use the nets correctly. Say, in the case of the malaria, what are some of the other issues that you try to take into account in that quantification?

B (11:49)

Yeah, so trying to take a lot of issues into account. So the first one is, well, let's just go through malaria nets as an example and show all the qualifiers. So the first one is we have data from randomized controlled trials that when you distribute nets, people use the nets, it reduces malaria cases. Randomized controlled trials are conducted very differently than national level malaria net distributions in randomized trials. At times researchers were trying to determine if someone using a net would prevent malaria and reduce death. And so they would at times go to people's homes daily and check if they were using the nets. This doesn't happen when you distribute nets nationwide in Democratic Republic of Congo. Similarly, the underlying levels of child mortality, of malaria, of poverty were much higher 25, 30 years ago when many of the malaria net RCT were conducted. So there have been changes in the underlying. The question is how valid are the results from those studies to today's experience. One particular way in which using nets has become more challenging and net manufacturers have adjusted is insecticide resistance. Mosquitoes are building up resistance to the insecticide used in the nets. They're the nets that were being used 10 years ago are not as effective, would not be as effective today as they were 10 years ago because of this resistance. In some cases we're still using older nets. We tried to take that into account in our analysis. And in other cases there are newer nets that have been proven more effective against mosquitoes in their current resistance. And I'll just say one last one though. There's sort of numerous considerations to take into account. But another question we have is we don't, I'll give a little bit of a lead in to help people understand this. You know, we don't care what givewell donors money, the sort of literal dollars that we direct accomplish. We care about ideally the causal impact that our work has on the world. So, for example, if we give a hundred dollars to the Against Malaria foundation, but that just causes some other donor to not give that money, well, they end up in the same place. We haven't actually changed what has happened in the world. And so we try to take this into account in our analysis. We call it a fungibility adjustment from the idea that money is fungible. And we try to have an estimate of the likelihood that our giving displaces money from someone else and an estimate of what that someone else would have done with those funds. And I think this probably gives an idea on one hand of the, I don't know, the intensity that we try to bring to the quantification, but also the admittedly inherent massive uncertainty in trying to pin down some of these parameters.

A (14:38)

Did you have any formal philosophical study that led you to think about these problems in a utilitarian way or any other way? Or were you just feeling your way in 2006 and going forward.

B (14:51)

Really just feeling our way? You know, no formal or advanced training. Took a philosophy class in college, but that doesn't count. I think that you asked earlier or you pointed out that givel's unusual in recommending such a small set of organizations. But I think the reason we've done what we've done, it's all driven by GiveWell serving. I don't know the person I was, or Holden, my co founder, was. So it's just constantly been a movement to try and get the answers we were looking for. And so it's coming more from a customer perspective rather than an expert philosopher or economist perspective.

A (15:34)

I mean, the reason I ask is that utilitarianism says we should try to do the greatest good for the greatest number of people. And I used to think that was a plausible idea. As I've gotten older, I've started to think that doesn't have a lot of content, but it gets at something we do care about, which is we don't just want to do good. We care about how much good, and we care about opportunity cost, what we might achieve if we hadn't chosen this outcome or that outcome, which you obviously are very aware of, but you are focused on saving lives of desperately poor people from certain causes. And my first thought would be, well, okay, they're not going to die of malaria, but they're so poor they're going to die of other things. And if you don't get at those underlying aspects of their lives, one, you're not really saving their life to ensure a longer lifespan of Any large amount. And secondly, the life that they have, the quality of it is still maybe quite low. Even if the quantity in terms of years, the thing we can measure, think we can't measure is the quality. And so your methodology pushes you toward those kind of outcomes. Comment on that.

B (16:56)

Yeah, I mean this is a great question. It's one that I think is still in our minds. That bothered us a lot early on. And so when we tried to look into it, we found two things, both of which were surprising to me at the time. The first is a huge proportion of deaths that occur in low income countries occur in those very early years. I mean a shocking amount in the first month, first year, first five years. Once people get past those years, they tend to live long lives. Not as long as people in high income countries, but still relatively long lives. And then separately, it is true that people's self reported life satisfaction is lower in low income countries than in high income countries on average. But it's also, I don't know, it's, it's not, I don't have the sort of way to describe it off the top of my head, but it's not, it's not terrible. You know, people are living lives that they're happy to be living. Their lives could be improved significantly. But you know, I would say that overall the people whose deaths we avert in childhood go on to like on average, live long, relatively happy lives. Averting deaths in childhood is not the only thing we do. We've supported organizations that aim to directly reduce poverty via could be direct cash transfers via an organization givedirectly that I know you've talked to in the past, but also other programs. When we've tried, and this is where the quantification becomes even more challenging, but when we've tried to look at what we get with those poverty reducing programs versus these childhood mortality averting programs, we've generally felt that the childhood mortality averting programs look better to us. You know, like we are getting more bang for our buck. Though, just to be clear, I mean this is one of the ways in which in the realm of philanthropic recommendations, transparency seems absolutely essential. Because I mean, there's no possible way that I would claim that we have the quote, true answer to the question of how to trade off between these things as much as in our view, this is what it is. And there are many donors who are big fans of GiveWell and use our work and also give a significant portion to other things either based on our work or not because they disagree with some of the underlying philosophical judgments that we're making. We would love to know how to give money to end poverty. Now if we could identify the sort of root cause of poverty and then direct money there, that would be amazing. And I think we would undoubtedly want to support that. I think unfortunately we just don't know the answer. We don't know how to do that as far as I know. And that's the reason. So I completely agree that it would in some sense would be better to alleviate the cause rather than the symptoms, but I don't know how to alleviate the cause. And then finally, just one quick final point is that I think that sometimes the disease reduction programs get short shrift. Reducing serious illness in childhood probably leads to better outcomes for that child itself by having a healthier young early childhood development, nevermind the effects on the family and the community, by reducing the deaths that are occurring and all the other illness that is not as severe, but is just removing people from their ability to do other things. These don't play a huge role in the way we actually calculate the benefits, but they're I think an important consideration with these disease reducing interventions.

A (20:38)

It's an interesting question about whether a thoughtful person who wants to spend their money wisely, and I'm thinking of a thoughtful person like me, meaning of comfortably financially but not wildly wealthy. I try to give about 10% of my income to charity. That is not going to fund a program to end poverty. It will not fund a program to figure out how to end poverty. So large donors and large organizations like yours have an opportunity. Somebody should have an opportunity to think big. Of course, that's happening in economics departments around the world that are trying desperately to solve these problems. They're not easily solved. Obviously our understanding of the process is imperfect or understand how to implement the process to improve the process is imperfect. So even if you have a simple answer that it might even be true, like more economic freedom or better education or whatever you think is the thing that might work, that's not enough. There's a implementation. It's sort of like the, you know, in innovation and entrepreneurship, having a good idea is not really the same as having a good product. And I think we don't have a lot of good products in the poverty fighting field and should probably spend some resources trying to answer that question.

B (22:21)

Yeah, and so I think we've tried to play two roles there. I mean, the first is, to use your analogy, when we see something that looks like a good product, even if it's not exactly the thing we normally look for we support it. And so just let me give one example. There's a center at Yale called Y Rise run by a professor Mushfiq Mubarak who I think I know fairly well from work at GiveWell and always found him specifically to be one of the people who brings together academic research and implementation in a very unique way. His center is focused on research into the scaling of programs. The so called Randomista movement, which started in economics 25 years ago, 30 years ago and was focused on randomized controlled trials to determine what programs work in economic development at sort of a micro level, has done a lot to push the field forward. And he's asking and what can we learn about the quote, science of bringing programs to scale? And this is not answering the whole question of how to end poverty, but it is one area where better answers would lead to better outcomes. And this is a case where some of this is not we supported this program, it's not a case where we had a quantified estimate of the good that would come from supporting this individual and their team at Yale. But it is a case where I think this person's track record makes it a good bet to contribute to the fight against global poverty. And so we've done that. I do think maybe secondly the niche GiveWell fills and why I think some of the we do what we do and not something else is we're really committed. And I think where we're good is setting things up so we can learn empirically about how things went. And that's not because that's the only approach and it's not because it's necessarily the best approach, but I think it's an approach that we're good at. It's what we do. And so we tend to, with the vast majority of the funds we direct, put them in places where we will be able to know if we were right or if we were wrong. We will try to learn and then we will try to improve. And that means that we tend to put less energy and less time and less attention on high risk opportunities. GiveWell would I think be very unlikely to fund think tanks in Africa to pride of identify pro growth policies in sub Saharan African countries. Not because it's obviously a bad idea by any means, but because it's really not what we're good at. We're very much, I think where we fit in the broad set of people and groups trying to improve outcomes for people living in low income countries is trying to give in a way that we can learn and improve over Time that we create feedback loops that we can learn from. And that is fairly unusual in the philanthropic sector. And so it's a niche I'm happy to try and fill.

A (25:43)

So you have four charities you're recommending right now. Why four? Why not 15? There's a million to choose from, probably more than a million. Curious how you narrowed it. What's the process by which a charity enters your space to be considered and then how does that decision get made? To those four? Why not six? Are there two that are right on the edge? You just didn't make it wise. In Shalem College in Jerusalem, one of the four were important for bringing leadership to a pivotal country in the Middle east that is often a source of conflict in the world. Just saying. Seriously, what, how do you get to four?

B (26:29)

Yeah, so the, the process essentially starts with a very wide funnel. And the way that an organization would get on our radar is one of a few ways. We, early on I literally read through thousands of charities, tax forms, went to thousands of websites and this was just to determine what do they do and you know, what questions should we ask them. We decided to focus that within low income countries. So the sort of starting point, and I'm happy to go back to that if you want, but the starting point was on some level, what are all the organizations that work to help people in low income countries and are large enough that they could plausibly be able to engage with us, answer some questions, have some data, and you're talking in the range of 1 to 2000 at that point. The main way though that we filter that list down is via academic research. There's a huge amount of research that has been done on what works to help people in low income countries. And so we start with the academic research on what programs work, how well do they work, what do we know? This is mostly relying on randomized trials, though certainly not entirely. There's a lot of them both in the field of economics and public health. And so we're looking for programs that, via that, that process programs that have strong evidence and could plausibly be implemented by a nonprofit organization. You know, there could be evidence for. Anyhow, so that's where we're starting and we're using that to narrow the list to the group of organizations that meet the following criteria. Number one, the evidence is strong enough that we believe it is significantly more likely than not that they're having a lot of effect. So we want to, in this top charities list, remove organizations where there's a, I don't know, a 25% chance of a very large effect. This is supposed to be high confidence in this short list. Number two, it's programs whose estimated impact per dollar exceeds our current threshold. So to explain that, right now we are raising approximately $500 million per year, and then we're just taking that. That money and essentially applying it to this. You could imagine a ranked order of all the possible programs we could give to. And so as we go down that list, one criterion that. That sets the bar for that top charities have to clear is how much money we got and how far down the ranked order list we would go. Right now, we put things in our model in terms of multiples of direct cash transfers. So we'll say we have an estimate for how good it is to just give a very poor person $1,000. That gets you a value of one. And then we're going to fund our list all the way down currently to 10. We think that's our. That's our estimate. You know, of course, very debatable, but that's. That's how we're doing it.

A (29:41)

Excuse me, Down. Down. Meaning. Meaning down to 10.

B (29:45)

So some down to 10, some are 50, some are 30, some are 20, some are 10. But when we find something that is 6, right now, we'd say, you know, that's not above our funding threshold, so it couldn't be on the top charity list. And then the third part, before it gets on the list, we want to have provided significant funding. Right now, for us, we say $10 million at least. For at least a year, just so we have some level of experience with the organization, we follow them, and we see what they do before they get on that list. Those are the. Those are the main. Those are the three criteria. And those four organizations account for approximately two thirds of the funds we direct. So this is not everything. There's. Obviously, you have to provide funding in order for an organization to even get to the stage of being a potential top charity, but the vast majority goes to organizations which are above our funding threshold. High confidence. And we've funded significantly for at least a year.

A (30:46)

And who makes that call? Who's making that call? Is it a vote? Is it a committee? Is it you? Is it who? How does that get done internally?

B (30:56)

Yeah, there's a lot of debate and discussion internally. And, you know, I think normally there's ultimately like, a fair amount of agreement among, let's say, the, you know, the three or four people who are senior members of the research team on a particular opportunity about, you know, how to, how to treat it. And so that, yeah, that's. It's not a formal vote or anything like that, but it tends to be, you know, via the analysis, debate and discussion, you know, reaching a conclusion about what we think about the opportunity, a consensus.

A (31:34)

A consensus emerges. What enters that conversation besides the hard numbers? I think the sort of, the general feeling people have is that it's just, it's all science. You just sit down and the numbers speak. Is that true for you? It's not true for other places, but for you, is that true?

B (31:59)

No, I don't. It's not true. I mean, look, the numbers do a lot of the work, but what else enters the conversation at the end? And, well, and let me explain this just concretely for a second. I'm talking about this funding threshold and I'm giving a number 10. Sometimes we reach the end of an investigation and the number in the spreadsheet is 7 or it's 13. 7 and 13 are not that different from 10, given all the uncertainty that's baked in. And so what do we do then and how do we move forward? I'd say there's a few different types of considerations that play into that analysis. Number one is to what extent is there upside or opportunity that we're not taking seriously here? You know, that could come from supporting a young person, a young organization with a small amount of funding that could lead to something bigger and better in the future. And we've done that in the past when the numbers have looked bad. And in hindsight, those were great decisions and happy to sort of talk about examples if you're interested. So one would be, I guess, like our term for it would be unmodeled. Upside is a consideration. Another one is. And again, you know this. I think on some level we feel fairly uncomfortable about all of these because they're so soft, but at the end of the day, they' important too. So another one might be to what extent do we have high confidence that the person that we're dealing with, the organization, or literally the individual, will be very open with us in the future about what's working and what's not? I said earlier that a major considering consideration for us is ability to learn. You know, we, we believe. I hope Even though we're GiveWell's 15 years old, we're still at the beginning of a long journey and an opportunity where we will learn is much better than one where we won't. And so having confidence that the person will be transparent about what's good, but importantly what is bad in the future is very important. And then finally, to some extent, I'd say track record of the person in the organization itself plays a bit of a qualitative role. So let's say we have two opportunities in front of us. One, the number is seven. The other number, the number is 13. But in one case, let's say the one with the lower score, the lower number. That person or organization has successfully delivered a wide variety of programs over the last five years. And we've seen them overcome challenges both programmatically, organizationally, for lack of a better word. We have confidence that they know what they're doing and we want to support them. That that's something that would push, push that up. And then on the other hand, let's say there's an organization we'd give funds to and we don't really feel like we understand the track record as well as we'd like. We're not, we're not as confident as we'd like to be that the programs have had the effect that the, the spreadsheet describes. The spreadsheet estimates. We estimate via the spreadsheet we'd be less likely to fund. And so, yeah, I think, I think those, those are some of the things, ability to learn, you know, track record, confidence in the, in the person or the individuals that we, you know, are taking into account at the end and that, you know, kind of thinking. And I think they all play into some of this unmodeled upside or maybe unmodeled downside that is, is important to consider.

A (35:44)

It's a strange analogy that comes to mind, which I think it's probably sometimes relevant for your decision making, is when a small business gets adopted by Walmart as a customer. So you've got a small business, you're making some product and you show up at Walmart and they like it a lot and they decide to buy X billion of that. And you're totally unprepared for that because you've never done anything like it. And it could actually just this great contract which, which has you celebrating and drinking champagne, could end your company because you're not gonna be able to live up to it and it's gonna be a devastating impact. And on top of that, you're now listening to them in a, at a level of, of detail and compliance that is extremely unusual for you. They're not just your largest client, customer, they're your largest customer by an enormous amount. So you've essentially strapped yourself to their, to their engine. So I'm Thinking about again coming. I was being facetious before, but we'll play with it for a little longer. Let's say Shalom College made your list. You know, we have a $12 million budget. There are things we're not doing, I'd like to do. If we had a little bit more. If we had a lot more, there's another set of things I do. But if all of a sudden I was on your list, we'd be flowing with money because there are a lot of people who trust you and you'd made that call. And how do you make sure that it's one thing to say, well, you know, we, we distributed a thousand bed nets last year. We think there's 4 million a demand for 4 million. So if we had more money, we'd distribute more bed nets. A lot of organizations aren't that straightforward. Many of yours are. They're literally, we need more capital to expand and cash to do more of what we already do. But how do you interact with them about strategic planning and their budget? And do you cut them off at some point because you feel like they've got plenty of money now? Let's take them off the list. How do you deal with that? Do you. Are you strategizing with them along the way, helping them grow? Are you giving them a pace at which they're growing?

B (38:06)

So there's a wide variety of ways that this goes. The first thing that we take very seriously is a question that we call or topic. We call room for more funding, which is how much funding do we think they could use? Well, and we wouldn't put an organization on our list that could only use a hundred thousand dollars. Well, if. Or we thought they could only use a hundred thousand dollars. Well, if that might lead them to get $10 million. We're also watching this over time. And so sometimes organizations have brought in a lot of money from the outside, either because we put them on or it happened independently and that affects what we'll give them. You know, we, we, you know, don't want to. We. We want to ensure that the funds are doing a lot and so that we watch it over time. We're tracking them, we're in touch with them, and that will cause us to give less. Mostly the type of support we're providing most programs is direct programmatic support. So in using one of the examples from before the organization, new incentives that does conditional cash transfers for immunizations in Nigeria, they might say, we work in this area of this state of Nigeria. We have succeeded. We want to Expand to this other area. Can we have. We would, you know, we need X dollars to expand. And there's, you know, that that type of engagement is the most common. We'll be looking at their past success, their track record in order to make a, have a sense of how well future growth will go. But it's fairly, I want to say, linear in its or organic in building upon past work for future work. There are some organizations and people where we've provided unrestricted support, which is really just kind of what you're talking about with Shalem College saying ultimately, we don't really know what you organization will do, but we have a lot of confidence in you. We want to give you the flexibility and the ability to see what needs you have, whether it's hire fundraisers or run a strategic planning process or, you know, open a new office in a new country, you know, without a program in mind. And so we've done that. That tends to be rarer and really happens when we both think organizations could really use it because they're constrained. And also we have an extremely high degree of confidence or trust that they are going to do great things with the funds we direct.

A (40:52)

I'm sure you've made mistakes and have regrets about things you did, you funded. You can talk about them specifically if you want, but if you don't, I'm curious how that, what you learned. How much is your process of evaluation that we're talking about, which is pretty subtle. Again, it's not a simple number. Oh, about like overhead or what proportion goes to salaries or whatever it is. What are some of the lessons you've learned that might apply, you know, generally to, to organizations and, and how have they changed over, over time? Like you said, you're still young, but it's, you've had, I'm sure you've learned a lot.

B (41:35)

Yeah, I mean, I think there's a ton of lessons and a ton of mistakes and happy to talk about them very specifically. Going, going all the way back to the beginning. You know, when we were starting, we were very frustrated by a lack of detailed information from charities about what they did. And so one of the first organizations that we recommended way back when was an organization called psi. I think they now are literally called psi, but it originally stood for Population Services International. And I should say as a caveat to this whole. All what follows, all of what I'm talking about is roughly now, you know, 10 to 15 years ago. So none of this is a statement about what PSI does today, but, you know, back then we were, when we, when we first came to them, they were able to provide a huge amount of data on what they had distributed, where it had gone, what the effects were. And we, we supported them in large part because I think we took those numbers at face value in a way that I think was overly enthusiastic about quantification as a path to getting the best answer. And over time, and this is 2007-2008-2009-2010, we started to ask, to what extent should we believe these numbers? Do we know that the numbers we're getting back if they say we distributed a thousand nets to this location in Zambia, well, how do we know those nets got there and that people use the nets? And as we dug in further, I think all the numbers that they reported were sort of true. They were not fraudulent. But also, I don't think they were gathered with the most rigorous method.

A (43:29)

Kind of like, kind of like GDP statistics in certain countries. It's, you know, an episode on that we'll link to. But yeah.

B (43:39)

And they've done a lot of work on this over time to improve. But I guess one, one mistake that we made early on that we've learned from is, I think, being overly enthusiastic about data in and of itself at the expense of, I don't know, critical evaluation of information to try to make good decisions. And so it, it moved us more towards, I think, a place we are today, which is, you know, certainly quantifying a lot, gathering the data we can, but, but then trying to go further. You know, another, another type of mistake that we made is we, we, we, we recommended an organization or no Lean Season and made it a top charity. Gosh, I don't know what year it was, but, you know, in the 2000 teens, at some point. And this was an organization that was actually set up by research that this professor Mushrik Mubarak had done at Yale. It focused on providing small incentives to largely young men to migrate from rural areas of Bangladesh to, to urban areas during the lean season. So people work in an agricultural economy. There's some time of year when there's a lot of food and there's a lot of work in agricultural locations. And then there's a time of year when there isn't. And people don't have work, they can't earn income. And he ran a small randomized trial where a very small incentive encouraged more people to migrate. They migrated to the cities, they earned a lot more money, they sent money back home. They were even more likely to migrate in subsequent years when they didn't receive the incentive and ran subsequent trials. We worked with others to sort of set up this program and I think we ended up making, ultimately we funded them. They operated for a couple of years alongside the funding we gave their operations. We also wanted to run another trial to see how well this worked at scale. And in fact the program did not work at scale when we, when we first funded it. We have different theories and by didn't work, I mean we didn't see that the, the treatment group in this randomized trial, they were not more likely to migrate than the control group. And there are a lot of potential explanations why anyhow won't go into those, happy to go into those if, if they're interesting. But I think we actually made two mistakes there. I think first, I mean, on some level, I think that's the process working. We made a grant, I think we made a good bet and we were wrong and we ended our support. Looking back, I think we didn't take seriously enough at the time the, the challenge of bringing this small scale, fairly complicated program to scale. You know, in the original trial it was, you know, people going and finding folks who were otherwise not going to migrate. And this happened at small scales, you know, a thousand, two thousand people. As it got larger, it had to go through large institutions and through microfinance banks in Bangladesh, which I think made it harder to deliver. So on one hand, I think we didn't realize how hard it was. But even looking back, I think in some ways, in hindsight, I wonder if we made a different mistake, which was we also exited too quickly. And I think that we went in and on some level, I don't think I would have said this at the time, but looking back, I think on some level we thought it would be easier than it was to deliver and that was a mistake. And then because we thought it would be easy when it didn't work, we were ready to end the program, as was the organization implementing it. And I think that was also a mistake. And I think in both ways the sort of better prediction to have made back then would have been if this is going to work, it's going to be hard. It's going to take some time and some iteration to get the model right at scale. Is that a bet worth taking? And then if it is, be ready, ready to stick with it over a longer timeframe. And so now we tend to commit to longer timeframes with programs that are more experimental to at least say, I don't know, make the initial decision with a Longer timeframe in mind, see if it's good enough on that basis and then be willing to stick with it over time, if it being willing to stick with it even if it doesn't look good at first, so it has the opportunity to improve.

A (48:23)

As an example of the kind of transparency we were talking about at the beginning that you ran a contest called Change Our Mind. Really bold thing to do. I really love that. Describe what it is and how you have used it.

B (48:38)

Yeah, so I mentioned that we do a lot of quantification for the programs we support. And all of this information about the quantification, the research, the analysis is on our website. It plays a major role in our decision making. And so because it's all public, we ran a contest where we asked people to try and tell us what we were getting wrong. And we said if you, you know, the, the winner would get, I don't remember what exactly $20,000. We had a lot of other sort of second and third prizes for folks and people went through our analysis to, you know, find things that, that we had gotten wrong. And there were places, I think on one hand I was. Well, a few things came out of that. First and foremost, I was somewhat surprised and gratified by the number of high quality submissions going into it. I sort of thought we got a lot of junk and we wouldn't get that many submissions. We got more than 50 really intensely considered submissions, including from people who have academic positions at prestigious institutions. I mean, people were really engaged in this project. People found errors. I'm grateful they weren't huge errors that show that we would have had to, you know, massively. We had made a massive error in so much of our funding, but certainly we had made small errors across programs. We recommend water deworming. I think most importantly, people called us out for being insufficiently transparent about the uncertainty inherent in our analysis. And I think that on some level we've tried to respond to this. I think we always have known that the numbers we're using are uncertain, but we didn't bring that wide uncertainty to the four in all of our analysis. So just to make it concrete, I said before that we take seriously the possibility that a dollar we give displaces a dollar that someone else has given. Or let's say it displaces some amount of money that someone else would give. But let's say a reasonable confidence interval around that is that maybe our median estimate is a dollar we give displaces 40 cents from someone else. But it could be as high as $0.80, as low as $0.10. And so we've tried to incorporate that more at the front of every single researcher as they're doing their work, is looking at that to remember the uncertainty inherent in the analysis that they're doing. The other thing that I'll just say we learned from that contest, which is more of a side note than a substantive critique of the research, is say about 20% of the people who submitted misunderstood what we were doing. So they said, you're making this error because you're doing X, but you should do Y. And in fact, we were doing Y. The spreadsheets were just convoluted. And so one of the things we've really been working on this year is improving our legibility. So, you know, we're very transparent. If you want to spend a lot of time, I mean, hundreds of hours, you can get the answer to the questions you have about our work, but most people don't have hundreds of hours to invest. And what we'd really like to do is make the reasoning legible enough that people can understand it, critique it, engage with it at a lower level of intensity. So we saw in this contest that people were struggling to do that, and so we're putting more energy, hopefully, into making our research easier to understand and critique.

A (52:11)

Yeah, it's a fascinating byproduct of that kind of experience. Right. What's the goal? But they helped you redo your website without intending to the phenomenon you're talking about of some other donors reducing their gift because yours goes up is called typically crowding out in economics. It's actually the first published paper I had in my career was on this question of crowding out. And it's a debated topic in economics, but there's also crowding in. And as now a fundraiser for an organization, I have a richer understanding than I had when I was a newly minted graduate student. If Gibwell puts a stamp of approval on an organization that could easily increase donations. Right. It could cause people to say, oh, I've been giving those people money, but I wasn't always sure about it. But if GiveWell, who looks at these things more carefully than I do, thinks it's a good idea, I'm going to increase my gift. Do you notice that ever?

B (53:19)

Yeah, I think so, for sure. I mean, I think one example of this, I mean, obviously our goal as an organization is to recommend programs and then raise money for them. And so we're very explicitly trying to get people to give more. And we've seen a lot of that directly and indirectly. So when we, you know, we might recommend an organization that then has more publicity around it because of the recommendation or it just builds on itself over time. But then I think there's another way this dynamic could work. And this is very hard to pin down, but, but I think it's at play. One of our top charities is Malaria Consortium, Seasonal Malaria Chemo Prevention Program. It goes by the acronym smc and it's a program that provides medicine to children to prevent malaria. And it's a relatively new program. And when we first supported it, it hadn't been rolled out widely. And I think one of the necessary conditions for large governmental funders. And so in the world of aid, this could be the Global Fund, which provides funding to Malaria, tuberculosis and HIV aids. But also the US Government and others is it's helpful to them to see that a program has been implemented successfully in order for them to have confidence in their technical bodies that decide what programs are in and out of their list of programs that they'll support to see that successful track record. But someone has to support some of that track record. So we were not the first to support this program by any means. It was an organization called Unit Aid, which provide some very early support to this program. But I think, and I can't verify but I think we played some role in growing the size of this program to the point where now we still support it. But also it is very much on the short list of programs that are funded by governments at the global level. And I think in the same way that we are worried about the crowding out phenomenon, I also think this is an example where there probably is to some extent a crowding in phenomenon of making something bigger so that it can get more support from others in the future.

A (55:27)

What keeps you up at night related to give? Well, you told me you have some kids. I'm sure they keep you up sometimes.

B (55:34)

But yeah, the kids, the kids definitely keep me up at night. There's a lot of things I think, look, there's certainly very specific decisions that are hard when we're weighing quantitative and qualitative factors and worrying that, and I'm happy to talk about examples if you want, but worrying that the personal bias for me or others is skewing our decision. But maybe more methodologically, I think a lot of what we're talking about is worrying. To me, it's very challenging. And what do I mean by that? When Gibble was very small, I think it was easier. And by small I mean a research team of Three or four people, five people. It was easier to say we have these numbers, we have this quantification. It's just a part of the story. It's a tool to think critically about the different considerations. But it's not the answer. And one of the things I've seen as we've grown as an organization where 70 people, about half of whom are working on research, so that's deciding where to give. So I see this tendency now that we're a larger organization for staff to want to rely more on the numbers. And I think I understand this. It's easier first and first to understand the rules. Maybe if the rules I give well, are I build a model, then I get a number, the number is the answer. That is straightforward. That's a problem that people can understand how to deal with. On the other hand, if it's do a lot of work with the numbers and then overlay your best judgment independent of personal bias to make a good judgment, that's hard to do. And so I really am. We're trying to find a way to maintain this good judgment, qualitative overlay and input to our analysis without getting too. While we grow. And I think in some ways that's what makes, in my opinion, one of the things that makes GiveWell special is that we're intensely interested in quantification while also highly aware of all the problems with quantification. And I think it's very hard to maintain that tension, those two considerations alongside each other, as we keep growing as an institution. So I have endless numbers of things that worry me, but that's one of them.

A (58:05)

I think that, I think that plagues every institution, I think charity and non charity. I think there's always a tension between what's measured and of course, what's measured gets managed. That's generally true, but it's not all you manage. You do bring other things into consideration. And there's no right answer as to how much one should weigh in and count more than the other. There is solace from certainty and there is fear and anxiety from uncertainty. And so we have a natural bias, I think, to move toward objective measures that as you say, people are. They find them comfortable. And certain types of people, the kind of people who do research, who are drawn to that kind of area, are going to be likely to be seduced or comforted or whatever word you want by an objective, precise, quantitative measure. A couple decimal points out, you know, a couple places after the decimal point. So it's, it's a really interesting, I think, challenge in terms of management generally, I don't think it's unique to you to give well, but I think it's probably pretty front center there. So I think in other organizations you can ignore it or miss it if you're not careful. You probably think about it more than most people do in most. Most places.

B (59:24)

Yeah. And I think, you know, there's just a research, an investigation that we've been working through, which is supporting an organization called Evidence Action in helping the Indian government roll out a chlorinated water program in various Indian states. And the background is that I think the Prime Minister Modi sees piped water, meaning bringing water to people's households and rural areas, as a major initiative that can, you know, a good initiative for the country. And the water ministry that they set up in India was looking for help bringing in chlorinated water. And so Evidence Action has done a lot of work on chlorinated water, and the Indian government solicited their input. And this was an investigation that. It's very hard. You can do all the work to quantify all the different aspects. How likely is it to work and how much money will you crowd in? How likely are you to crowd out? What is the effect of chlorinated water on mortality, et cetera, et cetera, et cetera. But it's so uncertain because that ultimately you have to overlay some degree, a significant degree of qualitative factors in thinking about how to make the decision. And this is a case where Evidence Action is an organization we've literally supported over the last 10 years, its entire history. It sort of comes out of Innovations for Poverty Action, the Yale group that focuses on randomized trials. Evans Action has a great track record of success. And ultimately for us, it's that. That really carries the day. In some sense. We can model all we want. When we make our best guesses, the numbers look good. But if we're excited to support it because of the organizational track record and in a different, you know, both in successful delivery of programs, but also honesty, transparency, what they've shared with us, if it had been a different organization, I think we wouldn't have supported. We wouldn't support the program because we wouldn't have that qualitative overlay. And I think substantively it's how we try to operate managerially. I think these sorts of examples are helpful to everyone internally, I hope externally too, and understanding what we do and why. That sounds great. Then good. Maybe some people say, I thought this was a math equation and they're out. But that's what we're trying to do to make good decisions about helping people.

A (61:53)

I mentioned earlier, I try to tithe roughly. I try to give 10% of my after tax income to things I care about and not just things I care about. Things I care about that I think the money will have an impact and an impact that I, that, that I, that I care about. Not just the cause generally, but the way that the cause is being supported. So I'm Jewish. I give a lot of Jewish. I give to a lot of Jewish organizations. I give to certain artistic activities, tax deductible stuff that, that speaks to me esthetically, that makes my heart sing. I don't measure anything, obviously. Am I making a mistake? Would you, if you had to make the case for why I ought to give to your top four, forget whether, which four, whether I could divide it equally. Should I stop supporting local charities here in Jerusalem? Institutions that I actually use, by the way, not just care about, but I get indirect services from them. Is that a mistake? If we were out late at night having a beer and heart to heart, you say, no, I like your us, but I can't respect you because you spent your money so poorly. Do you feel that way? And if you don't, why not? And if you do, how would you make the case to get me to do something different?

B (63:22)

Let me tell you how I think about it. You know, I'm also a member of a synagogue and I support the, their activities. The way I think about that is I don't bucket it into my mental charitable giving budget. I don't, I don't have a literal budget. I'm also trying to give about 10%. But I say, you know what? That's, that's external. That's not. When I think about my own charitable giving, I think about it as trying to help people to the greatest extent possible. I do that via GiveWell and everything else that I give to. I treat as, I don't know, something different. It's, it's when I, when I give to the synagogue or my school's pta, I see it as a shared responsibility to support a service that I use. And the reason that I give, that I think one should seriously consider giving to the organizations we recommend is, you know, just the magnitude of the impact that they have. You know, giving 10% means a lot of funding that can go to help people. And while it's, it's definitely imperfect, when I think about the impact of about $5,000 per death averted, that is weighty enough for me. That it causes me to want to give significantly to programs that are having that sort of effect.

A (64:51)

Man. Yeah, I'm pretty confident my charitable dollars don't save any lives in the sense that your organizations do. The thing I've been thinking about lately, and we can close on this. Get your thoughts. I've argued a number of times on the program that the argument for local giving, even though people here are better off than people, may be farther away, is that I have better information about a local charity. It's a Hayekian kind of argument. And that I have a principle of giving locally. It's more likely to be effective just simply because I have a better idea whether it works. I don't know if I've told the story before, but a reporter once asked me if I was in favor of giving money to education in Africa. And I said no. And they said no. How could you be against giving money to support education in Africa? I said, we haven't even figured out how to help education in the United States. I know how to help the budgets of schools of the United States. But to add actual knowledge that changes the lives of children, we struggle here in America to do that. And I know a reasonable amount about that set of issues. And to think that I could do good in Africa with that money, where I know nothing about it, is naive, foolish. I'm not being cruel. I think I'm being prudent. But what I love about what you're doing is it kind of takes away that argument. You've gone there and you've done the research. I may have quibbles with it or its accuracy, but certainly is, to the best of your knowledge, the organizations you give money to, the money achieves what it intends to achieve, and it does it in some level of effectiveness. And the other argument that I would come back to, I'm trying to rationalize my own habits here. Of course, I'm not. I don't want to pretend I'm just a truth seeker. Is part of my humanity, part of the essence of my being human is the connections I make with other human beings. And when I help a local organization, especially one that I can literally touch and observe directly, I get an emotional and it's part of flourishing, connecting to other people and helping them. Of course, there's an advantage to anonymity. Not suggesting that I need to see the people that I'm giving, say, food dollars to. But I do think that the connections we make, and I'm thinking now of sort of the extreme arguments of Peter Singer. And others that I should give to the bed nets in, in Africa because instead of, instead of throwing a birthday party for my kid, because birthday party for my kid, nobody's life is saved. And the bad debt saves a life. And to me, part of being a father is connecting to your child. And you do that through love and gifts and kindness. And that's part of being a fulfilled human being. So I'm thinking, having had this nice conversation, that maybe some portion of my charitable giving, as you suggest, could go toward the alleviation of extraordinary hardship and suffering that isn't related to quality of life, isn't related to my direct connections to other human beings. I'll never see those people. I can only imagine how transformative those resources are. But it's not unimportant. So maybe that's another way to think about it, that like you say, you can have some buckets. One bucket is for things you do that fulfill you as a human being and are not trivial, but they don't save lives and maybe should put some money aside for those activities that we think, at least we hope do save those lives and transform those people in ways that a really pleasant charitable dollar in a local place that's doing good work doesn't manage to match.

B (69:11)

Yeah, I mean, the reason I somewhat evaded the question of do I think you're making a mistake is I'm not going to say you're making a mistake because I make the same mistake. I spend money on all sorts of things that are not consistent with a Peter Singer oriented argument of maximizing global utility. You know, I want to be comfortable. I want my kids to be comfortable in ways that far surpass the quote, utility that would be offered to someone in, you know, Western Kenya if they received those funds. But I think that I also care a lot about my personal flourishing, my family's flourishing, and I put a lot of time and some money into those activities. But then I think that I don't think more money to those activities would lead to more flourishing. And I'm able to then direct the vast majority of the charitable resources I have to the opportunities that I think will help people the most. And so I do think that I'm trying to separate those, not trying to have the complete utilitarian argument and saying every dollar, every minute is intended to maximize global utility, but aiming to on some level do a lot on the side of personal and, you know, personal and familial and people close to me, their, their success and thriving. But then I have this opportunity to just decide where the vast majority of the dollars that I'm going to give go. And I could give that to something local and I don't know, be a big shot or something, or just give it to. Yeah, I don't know where I think it'll do the most good. That's what I do.

A (71:01)

My guest today has been Ellie Hassenfeld. Ellie, thanks for being part of Econ Talk.

B (71:05)

Yeah, thank you so much, Russ. Been great.

A (71:13)

This is Econ Talk, part of the Library of Economics and Liberty. For more Econ Talk, go to econtalk.org where you can also so comment on today's podcast and find links and readings related to today's conversation. The sound engineer for Econ Talk is Rich Goyet. I'm your host, Russ Roberts. Thanks for listening. Talk to you on Monday.