In high-risk contexts, higher stakes for impact evaluation

Impact evaluations often come with a steep price tag, but in settings where data can be limited and where respondents and researchers alike are potentially vulnerable to a constantly changing environment, the costs of conducting them could be even greater. Last month, a panel on impact evaluations in conflict areas, which was part of the “Making Impact Evaluation Matter” conference hosted by both the Asian Development Bank and the International Initiative for Impact Evaluation, tackled this subject. Organized by Jon Kurtz, director of research and learning for Mercy Corps, the four-person panel aimed to answer the question that looms large when evaluating impact in a risky environment: How does one effectively and ethically measure the impact of a program in a conflict-affected setting? Conducting an impact evaluation in itself is challenging, but running one in a conflict-affected setting complicates the picture even further, said Juliette Seban, research and evaluation adviser for economic interventions at the International Rescue Committee. In these contexts, it’s important to employ both quantitative and qualitative aspects to the research design and work with researchers who have a deep understanding of the conflict, Nassreena Sampaco-Baddiri, country director of Innovations for Poverty Action Philippines, emphasized. For Mike Duthie, senior director of impact evaluation at Social Impact, highly sensitive settings are not limited to war-affected countries. For instance, the U.S. Agency for International Development’s A Ganar program, which combines sports and education to help young participants turn away from high-risk behavior, is implemented in Central America, where there is a high level of crime. USAID commissioned Social Impact to evaluate the impact of the program. We caught up with the speakers from the panel and learned that the complexity of impact evaluations in conflict-affected settings requires an approach that enables evaluators to obtain accurate data and at the same time ensure the safety of respondents and researchers. A carefully constructed research design Unlike performance evaluations, impact evaluations go beyond measuring the output of a program. This requires establishing a sound counterfactual, which will enable researchers to properly link cause and effect between interventions and outcomes. “Impact is the difference between what happened with the program and what would have happened without the program, which can be estimated by having a comparison group to mimic the counterfactual,” Sampaco-Baddiri told Devex. In a risky environment, the task of setting up a counterfactual can be doubly difficult. For one, impact evaluation in a conflict setting may be prone to selection bias. Randomization can help researchers overcome this obstacle, but it’s no blanket solution to the task of determining a sample for the evaluation. “As such, it is important to build strong and committed partnerships on the ground in order to establish how randomized evaluations can help us understand and learn what programs work and which are the most cost-effective for the poor,” Sampaco-Baddiri said, noting that in Liberia, this meant that IPA researchers had to work with the government to evaluate a peace education program. Meanwhile, in Afghanistan, a randomized assignment to evaluate a vocational program for the youth was not possible because of the politicized nature of the selection process, Kurtz said. To address this, the team employed an improved version of the traditional propensity score matching technique, which approximates the impact of an intervention. A flexible research design — at least, one that is flexible enough to handle uncertainties but not too flexible that it ceases to be valuable to the purpose of the evaluation — is a must in unpredictable settings, which are often home to populations that can be particularly mobile and thus be harder to reach, Seban said. An IRC evaluation of a youth program in Kenya, for example, saw its sample size dwindle as the program beneficiaries, who are Somali refugees, were forced to relocate in the middle of the evaluation. “To avoid this, we need to plan for a larger sample size from the start and expect higher rates of attrition,” Seban told Devex. In the panel, Duthie also highlighted the importance of keeping in touch with highly mobile youth — which entailed getting as much contact information from them as possible — and designating a “sample size larger than what we thought we needed” for the evaluation of the A Ganar program. Sometimes, flexibility can also mean striking a balance between the interests of the implementing agency and the evaluators. While an implementing agency and impact evaluation researchers have one primary goal — “to find ways of improving people’s lives,” according to Seban — there can be differences in how they work toward this objective. The implementing agency, for instance, may want to focus on determining a certain impact of a particular intervention, while researchers may be more interested in measuring a wide range of outcomes. “I think the most important is to make sure that there are enough discussions between both sides to understand each other’s goals and constraints,” Seban said. “Allowing time for the discussion is critical but not always easy when program staff is already implementing the program.” In evaluating A Ganar, one challenge was in achieving an appropriate balance with implementers who were “rightfully wary of being seen to limit program participation,” according to Duthie. “We have used a lottery of eligible applicants, which has resulted in, at times, youth from the same households being placed into different groups,” Duthie said. “To permit the implementers the opportunity to ensure participation of particularly at-risk youth, we allow the implementers to select a small number of youth for each cohort to be automatically entered into the program — [in effect] bypassing the lottery — though these youth are not included in the evaluation sample.” The need for consent and confidentiality Working with minors in impact evaluations requires asking for their informed consent — a task that may not be as straightforward as it seems as it involves explaining the rationale behind the research to them and their parents or caregivers alike. The definition of a caregiver can also be different in each environment. “What if the minor is married? What if the minor lives with extended family and not with any legal caregiver?” Seban asked. “These questions are not specific to research in conflict settings but important to think about in relation to the specific norms and context.” While parental permission was not required for the age range of the participants that Social Impact worked with in Guatemala and Honduras, the team faced an equally important issue that, like the need for informed consent, is not limited to high-risk environments: ensuring the confidentiality of respondents. In both Guatemala and Honduras, where Social Impact aimed to measure the extent of illicit activities such as drug use and gang participation among the youth, a randomized response technique was used to get accurate responses and at the same time protect the respondents. “Each respondent is assigned an ID number, which is written on the survey cover page as well as the page with the risk questions,” Duthie said. “The randomized response technique is explained to the respondent, with multiple rounds of practice and verification, and then the risk question sheet is given to the respondent to complete on their own.” Meanwhile, in attempting to measure support for extremism in Afghanistan, the Mercy Corps impact evaluation team faced the same dilemma of getting data without compromising the individuals behind the responses. “We recognized that people are going to be guarded about sharing their feelings around these issues since it could be risky for them,” Kurtz told Devex. The team originally planned to use the endorsement question method developed by researchers who sought to measure population support for the International Security Assistance Force among Pashtun respondents. This technique, which asks benign Afghan national policy questions, produces data that will be analyzed similar to a list experiment — the method the team ended up using instead. Through the list experiments, the researchers were able to statistically determine sensitive beliefs without linking them to particular individuals. Both methods mask the identities of the respondents, but the team went with list experiments to protect its researchers. “Even though we knew this [the endorsement question method] to be the superior technique, we were unable to use it due to the risks posed to our program staff,” Kurtz said. “Specifically, there were major concerns that … respondents would associate our staff with other groups contracted by the U.S. Department of Defense, who had been using the endorsement questions in the area.” In high-risk settings, the physical safety of respondents and researchers can also be threatened. For instance, during the A Ganar evaluation, program implementers received death threats and extortion attempts, several youth participants were murdered, and a few of Social Impact’s data collection staff were robbed while working in the field, Duthie said. The team works closely with its local partners to understand and manage these threats, but USAID support has also helped Social Impact better protect respondents and researchers, according to Duthie. “We’re pushing to apply these approaches [impact evaluations] to challenging contexts where the evidence is even more limited.” --— Jon Kurtz, director of research and learning for Mercy Corps “For example, we had initially planned to conduct interviews at youths’ homes. However, when the lists of youth and their addresses were provided, we found, working closely with our data collection team and consulting with local police, that many of the youth lived in ‘red’ areas, where it is considered highly dangerous for any outsider to enter,” Duthie said. “Accordingly, we revised our approach to identify safe community spaces — such as a health center — outside of these red areas to conduct interviews.” Some other safety measures involve providing a stipend that youth participants could use for transportation to and from the interview center, requiring program staff to wear minimal accessories and vests that clearly identify them as associated with Social Impact’s local data collection partner while traveling, and traveling with all the windows open when passing through areas of questionable security to show the community that they are not a threat. But despite the dangers attached to impact evaluations in conflict situations, the reasons for doing them seem to outweigh the risks involved — at least, for researchers who continue to see the need for these studies. “If you look at where the world’s poverty is concentrated, it’s in fragile and conflict-affected states, and when you look at the vast majority of impact evaluations, they’re being done in stable contexts,” Kurtz said. “That’s a great development, that there’s more being done there, but we’re pushing to apply these approaches to challenging contexts where the evidence is even more limited.” Check out more insights and analysis provided to hundreds of Executive Members worldwide, and subscribe to the Development Insider to receive the latest news, trends and policies that influence your organization.