Is a randomized controlled trial right for your NGO?

SAN FRANCISCO — Last week, Village Enterprise released the results of its randomized controlled trial, which found that its holistic entrepreneurship program — which involved training and mentorship as well as funding — had a greater impact on extreme poverty reduction than cash transfers alone. In a meeting with Devex in San Francisco, Village Enterprise Chief Executive Officer Dianne Calvi said she wants to set an example in hopes that more nonprofit organizations become evidence based. But she acknowledged that RCTs are not the right fit for every organization. With growing interest from donors and practitioners in evidence, these randomized evaluations are just one possible tool in the toolkit. So, how do you know whether an RCT is right for your NGO? Innovations for Poverty Action, or IPA, is a research and policy nonprofit based at Yale University in New Haven, Connecticut, with a mission to create high-quality evidence and turn that evidence into better programs and policies for the poor, is known for its work on RCTs. Experts from the organization spoke with Devex about what organizations should consider before investing in what is considered the gold standard of impact evaluation design. IPA believes RCTs are just one of several monitoring and evaluation options that organizations have, and it recently launched a new program to help groups find the right fit for them. “RCTs are just a methodology at the end of the day,” said Annie Duflo, executive director of IPA. “It’s really exciting that the use of it has grown in the field, but it’s important to remember it’s not always the right tool or right time.” Nonprofit leaders should begin by thinking carefully about what questions they have, before determining the best way to answer those questions. For example, the path to answers looks different if you want to determine whether you are doing the right thing versus whether you are doing things right. If nonprofit leaders want to measure the impact of their program, an RCT is likely the best way to do it, she said, but they must ask a series of questions that looks like a decision tree. “The first question you need to ask is: Do I need to evaluate the impact?” she said. “If the answer is yes, the next question will be how confident are you about your program design? Have you piloted it enough? If the answer is yes, the next question will be whether your organization has the capacity to do this. Are you big enough?” When organizations decide to gather evidence without going through this series of questions, they can end up wasting their time and money. “It’s really exciting that the use of [RCTs] has grown in the field, but it’s important to remember it’s not always the right tool or right time.” --— Annie Duflo, executive director of Innovations for Poverty Action While RCTs are expensive, Duflo said she hopes that the cost in and of itself would not stand in the way of an evaluation. If done well, RCTs are a public good, she said, citing the investments the Bill & Melinda Gates Foundation has made in RCTs in the financial inclusion space as one example of donors building a body of evidence for a particular sector. Given that others benefit from the results, typically she does not advise organizations to pay for RCTs, but rather to partner with others in order to fund the evaluation. Still, the timing may not be right, the logistics or nature of the program could make a credible impact evaluation impossible, or the organization may not have the sample size or resources for an RCT. Responding to what the organization saw as a void of information on options beyond RCTs, IPA launched a new program focused on right fit monitoring and evaluation, or the Goldilocks Unit. Not all organizations are positioned to engage in RCTs, said Loïc Watine, who leads the program. The Goldilocks Toolkit features a set of principles identified by Mary Kay Gugerty and Dean Karlan, founder of IPA, who will soon publish a book on the same topic. These principles are Credible, Actionable, Responsible, and Transportable, or CART. ► Credible: “Collect high quality data and analyze the data accurately.” ► Actionable: “Collect data that can help you make choices, and commit to act on it.” ► Responsible: “Ensure the benefits of data collection outweighs the cost. ► Transportable: “Collect data that generate knowledge for other programs.” IPA’s new advisory unit will support organizations with the design and implementation of cost-effective and appropriately-sized monitoring, evaluation, and learning, Watine said, emphasizing that the real promise here is using data to inform decision-making. Karlan, the founder of IPA, has since stepped down as executive director, and launched a group called ImpactMatters. He remains a vocal advocate for RCTs as the gold standard of impact evaluation, but acknowledged they are not always affordable nor essential, and proposed a new set of metrics for what he calls “appropriate evidence.” ImpactMatters aims for the rigor of GiveWell, but the scale of Charity Navigator, and nonprofits can sign up for impact audits, as Village Enterprise has also done. “We want to make it very clear that not every program should have an RCT,” said Nathanael Goldberg, who oversaw the Village Enterprise RCT, and directs IPA’s Social Protection Program, which uses evaluations to measure, improve, and scale programs like Village Enterprise that focus on developing sustainable livelihoods. “RCTs should be done where there's an important unknown, something that's either important to a single implementer, or being done widely but not a lot of evidence behind it,” Goldberg added. The graduation model is the perfect example, he said, in a conversation focused on IPA’s work to evaluate ultra-poor graduation initiatives, which takes on several challenges of poverty at once, from health to savings to livelihoods. “You see the mistake with microcredit. Everyone got excited about what sounded like a very good idea. But they went on for decades scaling the intervention rather than rigorously testing it,” he said. On the other hand, the growing interest in graduation is more evidence based, with randomized evaluations built in from the beginning, and partners investing in the evidence before scaling. For example, the World Bank’s Consultative Group to Assist the Poor, and the Ford Foundation funded an effort to scale up BRAC’s work to other implementers in other contexts. In doing so, they gathered qualitative research, and worked with researchers IPA as well as the Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute of Technology, another high-profile partner in RCTs. Beyond universities, other groups that conduct RCTs include Development Impact Evaluation at the World Bank Group, which aims to increase the use of impact evaluation in the design and implementation of World Bank and other multilateral development bank-funded development projects. “The appetite for generating evidence definitely has grown among funders, among bilateral institutions, and among implementing organizations as well,” Duflo said. Once you decide whether an RCT is right for your NGO, other considerations include how to build the buy-in from the organization, through what means you will share the results, and of course how you will use that information. “Whenever you do an RCT, the most important thing is to really have learning in mind,” Duflo said. “RCTs should not be about looking backwards — like, ‘Did I do well?’ — It should always be forward looking.” Update, Feb. 23, 2018: This story was amended to clarify that Dean Karlan remains on the board of Innovations for Poverty Action.