DFID-backed RCTs take on 'new frontiers' in fragile countries

LONDON — Researchers funded by the United Kingdom are taking on “new frontiers” by using randomized controlled trials to test the effectiveness of governance, conflict, and corruption programs in some of the world’s most fragile countries, evaluation experts said.

Backed with £12 million ($16 million) in funding from the Department for International Development, two research centers — Innovations for Poverty Action and the Abdul Latif Jameel Poverty Action Lab, or J-PAL — have started piloting RCTs in countries including the Democratic Republic of the Congo, Pakistan, Liberia, and Uganda.

The Governance, Conflict, and Crime Initiative, launched last year, is breaking new ground, according to J-PAL’s senior policy manager Alison Fahey, who said implementing organizations may have been unaware that RCTs are possible in such contexts and on issues that include governance and conflict resolution. To date, they have generally been used to investigate less complex programs and contexts.

However, with donors such as DFID increasingly channeling funding into fragile and conflict-affected countries, there is an urgent need for robust research into the most effective interventions to inform policy, she said.

“Over the last several years, researchers have begun to use RCTs — traditionally used more in health and education — to address ever more complex and crosscutting questions, like those related to strengthening governance,” Fahey told Devex, adding that “now we're figuring out how to apply this methodology to even more challenging issues, like violence prevention or humanitarian programming.”

“Given the huge challenges involved in working in fragile contexts, it’s new and challenging, and researchers are entering new frontiers,” she said.

Although RCTs are increasingly being used to evaluate the impact of aid programs however, there is a debate over how far they can be used to inform policy. Practitioners also said donors and evaluators need to adapt the RCT design, including reducing their length and focusing on nonresource evaluation questions, to succeed in such complex settings.

Approximately 15 pilot evaluations of crime and conflict projects will be funded through GCCI, exploring issues such as why people join armed groups in DRC; how best to train local leaders to reduce gender-based violence in Peru; and community reconciliation in post-Islamic State Iraq. A number of RCTs will also look at programs working with refugees, Fahey said, such as evaluating the impact of mental health care on Syrian refugees living in Lebanon.

GCCI follows J-PAL’s Governance Initiative, launched in 2011 with funding from DFID and the William and Flora Hewlett Foundation to conduct RCTs of interventions designed to improve participation in the political process, reduce corruption, and improve state capacity. The recent £12 million grant from DFID will continue this work but add new research areas — namely the Crime and Violence Initiative, run by J-PAL, and the Peace and Recovery Program at IPA — to form GCCI.

Speaking at an event in London earlier this month, Diana Dalton, DFID’s deputy director of research and evidence, said GCCI is “critical” to inform DFID’s increasing work in fragile countries. In 2015, the U.K. government committed to spending half of the aid budget in fragile states.

“This is particularly a critical area for DFID because we are increasing our resources in tackling insecurity and improving justice over a wider range of countries,” Dalton said, adding that “GCCI is an important program to help underpin this challenging and complex work.”

She acknowledged skepticism around the “usefulness of RCTs and the value for money” they represent. Some have criticized their lack of applicability to other contexts or countries, limiting their utility when it comes to broader policymaking. However, she said GCCI was helping to counter such objections by “challenging those assumptions about what’s not possible and being ambitious about coverage, rigor, measuring outcomes, and importantly, about feeding knowledge back through the decision-making [process] to DFID decision and policymakers really effectively.”

Speaking to Devex, Rebecca Wolfe, Mercy Corps' director for evidence, peace and conflict, who has first-hand experience implementing RCTs in conflict settings, said, “it's pretty unheard of to do RCTs in a hot environment; we rarely see it in active conflict settings” due to challenges around security, as well as gathering and keeping track of people in the sample.  

These challenges can be overcome, she said, but it requires donors and practitioners to modify the type of RCT they roll out. She recommended shorter RCTs of up to one year, and sticking to nonresource-based programs, since randomizing who gets cash transfers in such settings, for example, is "emotionally too hard" for teams.

Wolfe, who is also a professor at Yale University, added that due to the high costs associated with RCTs, NGOs and researchers should make use of other evaluations that can be used to better inform future RCTs.

Peter Evans, who leads DFID’s governance, conflict and social development research team, said that in the past, he “would never have believed that you could do a rigorous randomized treatment anywhere, let alone in DRC or Pakistan,” adding that “it is fantastic that we’ve pushed those barriers.”

The London event on GCCI included presentations from seven researchers drawing on their experience conducting RCTs in governance, crime, and conflict, in Afghanistan, DRC, India, Liberia, and Pakistan.  

One speaker, Oriana Bandiera from the London School of Economics, said that RCTs provide a precise answer to a specific question, the findings of which are not always implemented by policymakers.

“There are cases in which the results uncover patterns which are not so convenient,” she said. That can impact whether or not recommendations are acted on. Talking about her work on a recent RCT assessing procurement reforms in Pakistan, which found that giving a drawing and disbursement officer more autonomy produced better results, she said the findings were only partially implemented.

“Policymakers see the bigger picture and albeit the intervention has benefit larger than cost within the experiment, costs might be higher once we factor in discontent from other parties, for example,” she added.