IPA and the cult of randomization

First of all, thank you so much to everyone for all the birthday wishes and phone calls on Sunday! I know I’m way behind on my thank-yous, but you guys mean the world to me and it was more than a little wonderful to hear from everyone. In fact, I’m more than a little intimidated by the sheer number of you that got in touch with me, because as frighteningly long as my list of people to write to was before, it’s now reaching dangerously unattainable levels.

For those wondering what a Kenyan birthday celebration looks like, I have just three words for you: I don’t know. According to my housekeeper Millicent (whose birthday, ironically, happens to be today), most Kenyans in this region don’t have money for parties or gifts, so birthdays just aren’t a big deal. For my part, the two best things about my birthday were (1) treating myself to a large bowl of Frosted Flakes, which I had managed to find in Nairobi the weekend before, and (2) my friends here surprising me with 5000 Kenyan shillings worth of cell phone airtime to call home, which at my current rate is equivalent to nearly 30 hours of international calling time (or 2500 international texts, take your pick). If you want to get really sick of hearing from me, give me your number.

Today, as a preface to the ever-delayed and long-overdue post on what I actually do every day here, I’d like to give a bit of an introduction to Innovations for Poverty Action as an organization and how it fits into the current trends of international development work. If you haven’t gotten around to reading the previous two posts yet, now would be a good time to quit this one and go read those first. Not because they’re necessary or good or any nonsense like that, it’s just that this one is really long and kind of nerdy and you might want to warm up a bit.

Innovations for Poverty Action is first and foremost a research organization (we have to be specific about that “research” part here in Kenya, because if we bill ourselves as a “development” organization, everyone will expect that we are just here to hand out money and medicine and things like that), and it subscribes to a very particular research methodology. When evaluating development programs of all stripes (microfinance, HIV education, community funding of public health interventions, controlling political corruption, getting farmers to invest in fertilizer and other forms of agricultural technology, helping rural families start savings accounts, vocational education vouchers, and community chlorine dispensers are just a few of our projects going on here in Kenya—check out IPA’s website for the full list of programs we’re evaluating around the world), the IPA model is to borrow a research method most often used in medicine and public health called a randomized controlled trial, RCT for short.

Those of you who are fellow RCT aficionados, indulge me a few minutes of explanation. The rest of you might be most familiar with this method for its use in evaluating pharmaceuticals or novel medical therapies: in clinical trials, patients are randomly assigned to a “treatment” or “control” group, and the effect of the new drug or therapy is measured by comparing the outcomes of the treatment group to those of the control group. It’s the scientific “gold standard” for evaluating the impact of a treatment in situations where we can’t know exactly what would have happened in the absence of that treatment.

IPA, however, is unique in the way that it takes this methodology and applies it to problems of economics, development, and foreign aid. One of the trickiest parts of evaluating development schemes is that you can never really know the counterfactual—that is, you can never know what the outcome would have been if your scheme had never been implemented. Maybe your elementary reading program showed that kids are doing 20% better on reading tests now than they were previously—but is that actually because of your program? Maybe a good harvest that year meant the kids in your program were getting better nutrition and their parents could afford to send them to school more often. Maybe new trade or pricing policies changed the average income in the area where you were working. Maybe the year the kids in your cohort were born, there was an especially good crop of green plants so their mothers ate more folic acid while pregnant, or maybe there was an iodine supplementation program going on then which resulted in improved cognitive development. It gets really messy trying to sort out all the possible variables that could have contributed to the gains you saw. RCTs offer us the best way of equalizing those variables across our experimental groups and isolating the true impact of a program. And right now in the world of economists (as opposed to the real world), randomized trials are like the shiny new toy that everyone wants to play with, despite the fact that it’s been sitting in their mixed-metaphorical toolbox of research instruments for years.

Consider that The Economist magazine, in its 2010 Innovation Awards, named "Randomized trials of aid and development schemes" as one of only seven nominees for the idea that will have the biggest impact on society in the next decade (the other nominees? 4G Networks, Geoengineering, Private space-launch services, Graphene electronics, Personal genomics, and Electric cars. Weird, right? Yeah, I don't know how "Jersey Shore" didn't make the cut either.) Esther Duflo, an MIT economist who helped found both IPA and its sister organization J-PAL (Jameel Poverty Action Lab), along with being one of the youngest tenured professors in MIT's history, was recently awarded the Clark Medal (second only to the Nobel in terms of prestigious economics prizes--former winners include Milton Friedman and Freakonomics co-author Steven Levitt) as the most accomplished economist under the age of 40. She has a Macarthur Genius Grant. She was invited to give one of the keynote addresses at this year's TED Conference. When the New York Times wants a quote on poverty and the Millennium Development Goals from someone that's not a one-named U2 singer, they call her. If all of that is too nerdy for you, fine, let's whip out the big guns: Oprah just named her one of the most powerful women in the world. Chew on that.

(Incidentally, she is also a charming if somewhat impatient woman with a low tolerance for poor French pronunciation. Her introduction to my colleague Thomas and me went something like this: "'Ello, I am Esther. You will find I am short, slightly left of center, and very French." We did, in fact, find that. Especially the French part.)

My own boss, Harvard's Michael Kremer, is practically pedestrian by comparison despite a Macarthur grant of his own, having managed to completely evade the Oprah radar with his paltry offers from USAID, his 2004 Arrow Award for the most influential economics paper of the year (based on de-worming research done right here in Busia), and the fact that he has essentially written the book on funding vaccine research for tropical diseases, finding markets to deliver needed immunizations to the world’s poorest consumers.

The point of all this isn't to impress you. You already know I’m a nerd, and honestly, I would feel weird if all of you--scratch that, if any of you—actually got excited at the mention of the Clark Medal or Arrow Award. The point is that after years and years of unaccountability and foreign aid based on ideology, politics, and Western perceptions of what developing countries “really need”, we feel like we’ve finally come up with some reliable tools to tell us what types of aid are most cost-effective and having the biggest impact, and the people developing those tools are getting a lot of attention. IPA is now routinely approached by governments and NGOs hoping to get their new programs evaluated, and designing many of these trials is a uniquely challenging process. Randomization in development trials isn’t as easy as it is in clinical trials of new prescription drugs, and Michael, Esther, and others have done a fabulous job of coming up with creative ways of measuring impacts, whether it’s randomly phasing in programs over several years, or randomizing at multiple levels, or rotating a program randomly between certain districts over time.

Once our research shows that something really does work, that a program is cost-effective, that a novel idea has a much bigger impact than anyone was expecting, IPA also has teams in place to help take these successful solutions to scale. We disseminate results to policymakers and officials, and we also push forward on our own partnerships to help fund and expand good programs. Perhaps the greatest success story so far has been IPA’s work with de-worming: research in primary schools in and around Busia about a decade ago was able to show that providing students with a single dose of de-worming medicine (an extremely cheap and easily administered intervention) had a huge impact on school attendance, as kids spent fewer days sick at home. That seems obvious. What wasn’t obvious was that this effect actually extends not just to the schools receiving the treatment, but to all the neighboring schools as well—de-worming at one school made it harder for worms to be transmitted from child to child and protected children in surrounding villages to a significant extent. IPA’s calculations found that the total benefit from these de-worming interventions, based on the total increased school attendance and improved health of all children involved, made it one of the single most cost-effective public health interventions in the world. The program is now being implemented in all high-risk primary schools in Kenya and it will continue to expand on a global scale. More info is available at Deworm the World. (If you want to read the award-winning paper on this research, you can wade through Michael's ridiculous publication list to find it here).

Although RCTs have done a lot of good for the world of development economics, reducing bias, isolating impacts, and saving money in the long run by isolating the most effective programs, there are a number of significant ethical and cultural concerns that come into play when implementing such trials in a developing context. They’re expensive and tough to carry out. It’s difficult to study rare or distant outcomes. They might not be easily generalized to outside contexts. And like all trials, they are subject to certain statistical limitations. The most important considerations, though, when implementing an RCT in a developing country, are those surrounding the consent and perceptions of the people who are actually enrolled in the trial. The subjects of our research are real people with real lives, people who often aren’t empowered economically or politically. Getting informed consent from these populations requires being really sensitive to what “consent” means in this context, and how people are going to perceive something like a randomized trial. A program which specifically enrolls some people who are not going to receive aid can be seen as incredibly unfair (even though we allocate as much of our budget as possible towards our actual interventions, and only enroll enough controls to ensure that we can make definitive statistical statements at the end of our study), and we need to be sensitive to that as well. It’s a fine line to walk sometimes, but in general my personal experience has been that people are grateful for our presence, regardless of how much aid we give their household individually.

We know aid isn't the be-all and end-all of development. In the long run, aid isn’t the sustainable or most successful means of improving people’s lives. We also know that aid has the potential to do a lot more good than it does, and we believe that it is still worthwhile to help struggling people as best as we can in the short-term while encouraging practices that support long-term growth. Even if it doesn’t prop up economies for years to come, aid really does change lives in the present, and we want to make sure governments, donors, and people like you are positive about the impact that aid money can have.

We are also huge nerds who go into the research business because we think it produces some really elegant experiments and gives us the opportunity to make randomization jokes all day long with people who won’t judge us. And we like the fact that research works. We've done it with deworming. We're doing it with chlorine dispensers. And Lord willing, there are a lot of other innovative programs out there that are going to find a lot more people because of research like this.