The following is a conversation between Dean Karlan, Professor of Economics at Yale University, and Denver Frederick, Host of The Business of Giving on AM 970 The Answer in New York City.
Denver: In the world of philanthropy, people are constantly debating which social innovations work and which ones don’t based on theories of change and stories and other such things. These conversations can go on and on and on, sometimes endlessly. But one person who quietly excused himself from that conversation, and instead went out into the field to test different approaches– to actually find out what worked and what didn’t– is Dean Karlan, a Professor of Economics at Yale University. He is the founder of not one, but two organizations: Innovations for Poverty Action, and more recently: Impact Matters. Good evening, Dean, and welcome to The Business of Giving!
Dean: Hi! It’s good to be here.
Denver: There are 2.7 billion people in the world who live on $2 a day or less. Getting them the right programs with the greatest impact is at the heart of Innovations for Poverty Action, or IPA. Tell us about IPA and exactly how you go about doing this work.
Dean: Sure. Let me tell you two things. You opened up with a perfect explanation of how we got to be doing what we’re doing– just long debates that are very far from the field…distant from people… about “Does aid work?” Those debates were very frustrating to listen to and also were very void of the hard data from just good, clean, simple tests on the ground to find out: “Does this particular policy in this country work?”
So what we did is, starting about 15 years ago, started going into the field and setting up randomized trials to just test very specific programs. That gets you very good answers, very clean answers, to very specific programs. And then when you see a collection of evidence from lots of places, then you can start making some grander statements about policy. But at no point are we ever going to get to an answer to the question: “Does aid work?” because the answer is really very simple. Sometimes it does, and sometimes it doesn’t. So the challenge is not answering that question, but just rather figuring out what are good policies to do.
I started Innovations for Poverty Action in 2002, and the basic goal of IPA was twofold. One was to help create those projects…to make those projects happen, and make them happen well. So we were the in-between – between: organizations that wanted to find out whether what they were doing was working; researchers who wanted to help answer that question; and donors who wanted to be able to fund the projects and find out whether they’re working. And so we were the organization that was on the ground helping make that collaboration take place.
The second part of what we do is communicate those results to the outside world. So the lessons aren’t just useful for the organization that’s doing the project, but actually helpful for other people elsewhere to learn from these ideas, and find out for themselves what might or might not be good ideas for them to try as well.
Denver: Yeah, if you don’t get this research into the hands of the policy makers who actually can act upon it, it really doesn’t do a heck of a lot of good.
Dean: Absolutely not.
Denver: So let’s talk about a few of the things that you’ve researched, and I’ll give you three or four examples, and let’s see how it goes. Suppose I am very passionate about education in Kenya, and I want children to spend more time in school. What’s going to be most effective? Build a new school? Offer scholarships? Buy school uniforms? Or perhaps purchase new textbooks? Which one works best?
Dean: This is a perfect example of why we set up these kinds of tests, because these all sound like good ideas to me. They all have some plausible, theoretical underpinning to it as to why that might unleash an obstacle that was preventing people from sending their kids to school. But how do we know which one is really actually the best one or not? We can go to the ground and set up some tests and find out.
Some of those exact studies you just named were started by Michael Kramer. He did this actually when I was in graduate school. He was really the pioneer that started taking these tests to very specific questions like this. What he found in that exact context was that a lot of children were not attending school because they were sick. And they were sick because of intestinal worms. So, a very simple pill for $0.50 which clears the child of intestinal worms was a very cost-effective approach for getting the child to go to school. Because once healthy, they then attended school.
So this program now is in some sense a great example of the process of doing a careful test– finding out that something works, and something is cost-effective, too. And not just that the benefits are there, but that they’re actually… for a dollar spent…creating more benefits than other alternatives. And then, we took that idea and ran with it to donors and governments to help show them: If this is a problem in your country–intestinal worms and low school attendance– then here is an approach to try to deal with that. And we’ve now seen over 100 million children dewormed through programs that we’ve coordinated. We’re very excited at the impact that this particular research has had on policy, and that our efforts in advocacy have had to help that bring that lesson to India and Kenya and other countries.
It’s actually much easier and lower cost to give things out for free than collect a tiny amount of money that doesn’t actually make too much of a dent in revenues anyhow.
Denver: Fantastic! Let me run this one by you, Dean. Charging a small fee– whether for a health project or a malaria net– will help assure that a person places a value on these items, and as a result, they will be more likely to use them. True or false?
Dean: Mostly false. The reason I add “mostly” is because this is a situation where there’s no doubt that there are some situations where that might hold some water and have some truth to it. But on average, the studies we found have very consistently shown that as you go from zero to a small price, you actually do much more harm than good for two reasons: The mere act of having to pay for something… and getting someone over that leap of paying anything… has usually just dropped demand by a lot. And the second is that: Actually collecting a tiny amount of money is a pain to do. It has all sorts of extra implications for your operations of your organization–collecting money, depositing it. It’s actually much easier and lower cost to give things out for free than collect a tiny amount of money that doesn’t actually make too much of a dent in revenues anyhow. It’s definitely better in the case of bed nets. There’s been very strong research, led by Jessica Cohen and Pascaline Dupas, in particular, that was coordinated by the IPA Kenya office, that found that giving away free bed nets was far and away the better policy than charging for them.
I do believe Muhammad Yunus deserves that Nobel Prize. What he accomplished was really impressive and has changed people’s lives around the world for the better. But it hasn’t achieved that full aspiration of lifting people up out of poverty, increasing average income and average levels of consumption.
Denver: Very interesting. Let’s talk about microcredit. Now, microcredit– which consists of small, uncollateralized loans to budding entrepreneurs– that was going to lift people out of poverty like nothing we had ever seen. So much so, that Muhammad Yunus of the Grameen Bank, who pioneered this concept in Bangladesh, went on to win a Nobel Peace Prize. Has microcredit lived up to its promise?
Dean: Yes and no. Mostly no, in that there’s clearly a world that promised too much, and it did not live up to that promise. It did not lift people up out of poverty on average. I do believe Muhammad Yunus deserves that Nobel Prize. What he accomplished was really impressive and has changed people’s lives around the world for the better. But it hasn’t achieved that full aspiration of lifting people up out of poverty, increasing average income and average levels of consumption. This has not happened.
What has happened is people have had some choices that they didn’t previously have. And that’s a good thing. They were able to buy something– like to build something for their roof, or put cement on a floor, or buy a TV, or even make a small investment and grow their business a little bit. But these were not transformational changes. They still had to pay back the loan and the interest on the loan. That was not that cheap. So their lives were better off, but not radically so.
The striking thing is that the microcredit world also had huge numbers of people advocating against microcredit, too, or aguing that it was actually leading to very deleterious impacts. The striking thing is when we look at the body of randomized trials now that have come out on microcredit, we don’t really find evidence for either side– either the lovers or the haters. We just find that people, on average, are doing OK. They’re doing better. They have choices. This is a good thing. We should be proud if we’re an investor in this space, and feel good that our investment money was helping people have more choice in their life. There is some evidence from India in one study that found that there was a small contingent of people that were able to take those loans and do really well with them. It’s very hard to predict who those people are, but there were some people. But on average, it was not finding big effects.
The graduation program is not an investment vehicle. This is not a: “Get your money bag, and be an investor and capitalist” approach.
Denver: Well one approach to poverty that has had some meaningful, positive results is a more holistic one that provides a range of support. I think you referred to this as a “graduation program.” Who is it directed by? What precisely does it entail? And what have the results been?
Dean: There’s a stark contrast between this and the microcredit model, because they both started off with the same goal: Let’s lift up income, on average, for the world’s poorest. Where microcredit failed on that, and like I said, it did succeed in some other dimensions. But where it failed was on two dimensions. One is lifting up average income for the world’s poorest, and on reaching the world’s poorest. It missed out on both of those dimensions– on who it was actually reaching, as well as the average impact.
So the graduation program is not an investment vehicle. This is not a: “Get your money bag, and be an investor and capitalist” approach. This actually says “No, maybe some of the problems of being really poor need to be solved through direct subsidy.” And maybe some of the problems, too, are integrated, and it’s not just one problem. There are maybe four or five different problems, and we need to have a more holistic approach in tackling all of them at once to help lift someone up out of the situation they’re in… out of the trap that they’re in… so that they can build a sustainable income for years to come.
So what this program does very concretely is: First, it provides an asset transfer, a productive asset transfer. Say, four goats, if they’re in Ghana; a cow in India; beekeeping in Ethiopia; guinea pigs in Peru – different things in different places. Very contact-specific. And households would get a choice over a set of a few different livelihoods. This is what they refer to as a livelihood– just being some sort of income-generating activity.
That was the heart of it. But then alongside that, they would get training on how to rear that animal… if it was animal… or attend to the business, if it was a business of some sort. They would get some coaching, which consisted of a household visit from a local NGO that would come and visit–somewhere between once a week and once a month for up to two years– to work with the household, identify the problems they’re facing, and help solve them. They would get access to some health services, access to a financial bank account for savings, and also some transfers of food for six months… so that they didn’t immediately feel the need to slaughter one of the goats in order to eat.
The collection of these things together, we found, where we did six randomized trials in six different countries: the average impact was actually strikingly big. The consumption went up; income went up. They went up far more than the cost of the program. Even though the program was expensive, the benefits were bigger. In one site– we now have seven-year results in India– and we’re seeing the results are even bigger.
Denver: Wow. So long term!
Dean: So they don’t dissipate. The long terms results are there. The six sites that I referred to were all measured at three years after the assets were transferred. And we see that everybody, on average, is doing better: higher income, higher consumption, higher happiness, higher power for women in the household, higher political engagement. The benefits were across the board on just about everything we measured.
The idea there is that the problem is deeper than just not having money. If you can give $100… and $25 worth of training… training that they can’t go out and buy, then you might actually have an even bigger impact than $125. That might have a $200 impact.
The right answer might very well be: don’t just give out cash, but give out cash in particular ways.
Denver: Great stuff all the way around. Several months ago, I had Paul Niehaus, the CEO of GiveDirectly on the show. They’re in the business of direct cash transfers to people in places like Uganda and Kenya. I know you’ve done a lot of work with them. Tell us what your research has shown on the impact that these unrestricted cash transfers have had, and I would be dying to know what you think about their latest initiative– which is giving 6,000 Kenyans sustained cash– guaranteed and unrestricted– to meet essential needs over the next 10 years.
Dean: I’m a big fan of what they’re doing. I was not one of the researchers, but IPA as an organization, did lead the randomized trial to evaluate GiveDirectly and found some really nice results… and nothing deleterious. I think one of the most important results from that study… not to sound too negative… but one of the most important results was actually beating down a negative story– that a lot of people were concerned that money would get spent on things like alcohol and tobacco. That’s simply not what was found. And they did do some very creative things to try to measure alcohol and tobacco use to deal with the fact that people might not admit these things.
In some sense, that was actually the most exciting part because you have to remember the philosophy behind GiveDirectly is: “Let’s let the people choose what to do with the money.” Once you take on that idea– whether someone uses it to build a roof, or to buy a goat so that they have higher income in the future, you could say, “Well, that’s their choice.” But we might want to say: “OK, but there are some things that are actually bad in terms of externalities. If they are buying alcohol and tobacco, that actually has negative impacts on the rest of society, whether it’s in their household, etc. We want to avoid a program that has people using the money to do things like that because of those negative effects on other people. And once we can observe that that’s not happening, well, let’s go and stick with the program that just says, ‘No. Let them do whatever they want.’”
Now, the counterargument to that, and this is where I go back to the way you opened up. At the end of the day, I don’t have a prior on this; I just want to see the tests to find out the answer. But the counterargument from a theoretical perspective would be, “Well, wait a second. If you give someone $100, for the most part, you know the benefits will be about $100.” They’ll get $100 of stuff that they’ll buy. Let’s just say, it’s that simple. Whereas if the problem is the way the “graduation program” thinks about it, that program says, “Well, wait. We think the problem is money. That’s why we transfer either cash, or the asset, either way.” It’s more or less very similar. “But we also do these other things about information and coaching.” The idea there is that the problem is deeper than just not having money. If you can give $100…. and $25 worth of training… training that they can’t go out and buy… then you might actually have an even bigger impact than $125. That might have a $200 impact. That’s the basic idea. But, ultimately, whether this was right or not… is the question to just go to the ground and test.
So that’s actually what we’re doing in Ghana right now. We’re just starting a study that is going to compare the graduation approach to cash. We will have a separate, distinct arm that is just giving out cash in a very GiveDirectly-style so that we can all learn which is the way to go. If giving out cash is actually better, then actually that’s great! And it makes life so much easier because it’s so much cheaper to do that. That would be an amazing find! The right answer might very well be: don’t just give out cash, but give out cash in particular ways. And that’s going to require more work to figure out how to frame it, how to time the cash, who to give it to within the household. These are all really important questions. If we can figure out those kinds of systematic answers to those kinds of questions, we can really help improve the efficiency of these types of programs.
Denver: I know GiveDirectly sometimes looks at these cash transfers as almost an index fund… the way we would look at the stock market to see how those people do. Then benchmark against that… the work of international aid organizations.
Much of what you do, Dean, is you’re trying to get people to change their behavior, which is going to result in some beneficial outcomes for themselves and their family and their community, and we know that is not easy. I think in this country, about 45% of us make New Year’s resolutions, and about 8% of us actually stick with them. So, you are the co-creator of a website called “StickK.” It helps people meet their goals through something called “commitment contracts.” How does that all work?
Dean: Sure. We’ve done this in developing countries, too. The website is targeted to the United States consumer. What you are able to do is go on the website. Say, there’s a goal you want to do – you want to lose weight. You want to run a marathon. You just want to run more, exercise more, exercise every day, do yoga, eat less chocolate, whatever you want. You can go online– or there’s an app– and you say what it is that you want to do. The kicker is that you say what the stakes are– what happens if you fail. A lot of people like putting down an anti-charity, something they really hate. So, you can put down the “super pacs” for instance on the other side. We take about four different hot political issues, and we have charities that are advocates on one side or the other. Gun control is a very popular one. NRA Foundation, or a gun control group. So, you say what the stakes are. A lot of people actually don’t put money though. Some people say, “You know what? The biggest stake for me is really the social shame. For that, I’m going to name friends. And they’re going to get told whether I succeed or fail.” And that might actually have more bite for them than the money.
Denver: Peer pressure.
Dean: And then they get to report whether they succeeded or failed. They can name someone to be a referee. That person gets to adjudicate whether they succeeded or failed, and there you go.
We also run this program as a positive – a carrot, not just a stick – when we work with corporations to do corporate wellness programs. They often want to help their employees exercise more, get diabetes screening, things like this. And they use the website as a tool for coordinating an incentive system for healthy choices for employees.
Denver: And in this particular case, your field research started with you trying to lose weight, right?
Dean: It did, actually. It did. I was my first data point on that one.
Denver: And it worked, after all. After a false start, or two.
Dean: Exactly.
We needed an entity that was in that business– that would map where there is strong evidence… and also reward charities for being honest brokers in a sense.
Denver: Late last year, you launched a website called “Impact Matters.” These are going to be impact audits of nonprofit organizations– kind of modeled after financial audits– to help donors make some better decisions on the charities they support. So, what specifically are you going to be looking for when you’re doing these impact audits of nonprofit organizations? And how is it going to be different than other organizations that rate charities like GuideStar, GiveWell and Charity Navigator?
Dean: Sure. This basically came out of a void that we see in the space where IPA is—I get confronted by people who say, “OK, that’s really interesting research. Take the graduation study. Now, where do I send my money?” And I’d always have to stumble a little on that and mutter something: “Well, you know, it’s not so easy. How do I know? All I can tell you is this program works, but…”
Denver: That’s not a satisfactory answer.
Dean: Yeah. In the case of the six sites we did, one of them was run by the Pakistan government. There’s nowhere to send your money if you just want to see that happen again. Two of them were run by Plan-International, but Plan-International is huge. So even if you said, “No, I want this money to go to that particular program,” it’s not so clear how that really works. So each one had their own story, but that’s not really very satisfying for the donor.
Whereas IPA’s research and our advocacy work–which we take very seriously– it’s hugely important to us, it’s not about identifying very specific charities that say, “This charity right here has really just zeroed in on this particular issue, and we should go support it.” We needed an entity that was in that business–that would map where there is strong evidence… and also reward charities for being honest brokers in a sense.
Let’s say a charity is really collecting good evidence and is not necessarily doing the program that has the absolute, most cost-effective result. But they’re doing a really good of job of learning, and they’re finding out that, “Actually, you know what? We didn’t do as good a job as we can, and now we’re going to morph and we’re going to try to do better.” And they’re documenting along the way so others can learn. I want to reward that charity. I want that charity to be supported because they’re really helping not just their own cause, their own programs, but helping others around them.
We wanted a rating system, a rating organization that helped make those kinds of assessments. What we saw in the world were two extremes, and it goes back now to your question about the other ones in the space. One really important dimension that was not in the space– and then something else where we just wanted to be a little bit in the middle– the one thing that was not in the space at all… is that none of the rating systems were there to help the charities. They were just there to service the donor. And that’s a good thing. I’m not critical of that. But we really wanted to be able to leave the review process for a moment, and where we can actually turn back to the charity and give them some actual advice, be able to be that guide to the data and to the evidence…
The problem is that it really doesn’t tell you much at all about what the program is actually doing and what they’re achieving… whether they’re choosing good ideas. And some things are just more expensive to run than others. This was punishing a lot of perfectly good charities and rewarding some bad ones.
Denver: Yeah, a little more active and not so passive.
Dean: To be able to say, “OK. You’re doing such and such over here, and there’s really actually good evidence on this.” We can make that link from where the data are… where the knowledge is… about what works in that particular cause and help get them access to that information to help improve what they’re doing. Or if it’s just a matter of giving them advice on how to better collect the data on their programs… that we can give them that kind of advice. That was not happening with any of the review systems that were out there, and we think that’s an important aspect of the process that can help to improve the sector, and not just tell donors where to give money.
On what people were doing to actually rank the charities. You basically have two worlds. You have Charity Navigator, which was using overhead data, a diminished rating of financial data. The problem is that that really doesn’t tell you much at all about what the program is actually doing and what they’re achieving…whether they’re choosing good ideas. And some things are just more expensive to run than others. This was punishing a lot of perfectly good charities and rewarding some bad ones.
We did some quick analysis and found that only 2% of charities were spending more than 50% of their money on overhead. So when you’re spending more than 50% of your money on overhead, I’ll agree that it’s probably fraud. But anything else, my honest reaction is to shrug my shoulders a little and say, “Well, I don’t know.”
We don’t see ourselves as a competitor. We see ourselves very much as a complement to what they’re doing.
Denver: Got to look a little further.
Dean: I don’t really care how the sausage is made. What went in… what came out. Tell me what the impact is, and then we can assess. One interesting exercise that we did a long time ago is go to the other end of the spectrum. We looked at GiveWell. So GiveWell does very rigorous analysis. I’ll admit I’m a little bit biased in that one of the things I like about them is that they use IPA’s research. I think every single one of their top recommendations is based on IPA’s research, so I’m actually quite excited about that. But they hold the bar really high, and they’re really trying to calculate: What is the best one? Truly being agnostic to geography and to cause! And what they’re saying is: We are going to compare across outcomes and across things that organizations are doing, and just declare one to be more cost-effective than another.
Whereas I think there is room in this world to say, “Wait a second. If you’re doing education in America, I’m not going to argue with you about whether it’s more cost-effective or not than doing health in a developing country. I’m just going to go ahead and look at the domestic education groups and decide: Which ones of these are evidence-based?” I think that’s actually a valuable role. It’s not one that GiveWell is trying to fill.
So we don’t see ourselves as a competitor. We see ourselves very much as a complement to what they’re doing. They’re really just doing it for that very top-end echelon and for people who want to see the decision making and say, “No, no, no. Just tell me the best one, and I truly want to be agnostic to where the organization is working and what the cause is.” But for someone who says, “No, no. My passion is education in America. My passion is homeless shelters. My passion is community development.” We want to be able to tell them which are the evidence-based organizations. So that’s one of the mantras and principles that is at the heart of Impact Matters.
Denver: Well, you sound like you found yourself a really good niche. Dean Karlan, Professor of Economics at Yale, and serial entrepreneur, thanks so much for being here this evening. And just by way of a quick review for our listeners, the website for Innovations for Poverty Action is?
Dean: Poverty-action.org.
Denver: StickK would be?
Dean: StickK.com. So that’s stick with an extra K.
Denver: And Impact Matters?
Dean: Is impactm.org
Denver: Great! Dean, it was a real pleasure having you on the program.
Dean: Great! Thanks a lot.
The Business of Giving can be heard every Sunday evening between 6 and 7 PM Eastern on AM 970 The Answer in New York and on I Heart Radio. You can follow us at bizofgive on twitter and at facebook.com/businessofgiving.