Improving the evidence base in the environment sector with RCTs

Rigorous impact evaluations of development programs in the environment sector are rare, in contrast to the relatively widespread use of this powerful tool to carefully measure the effectiveness of different strategies in, say, the health and education sectors. This limits our ability to learn conclusively which approaches work best to address some of the world’s most pressing development issues, including responses to global climate change.

Impact evaluations in the environment sector face many of the same challenges as in other sectors, including considerable upfront planning and close coordination across the implementation and evaluation teams. This is especially true when a randomized control trial design — the gold standard of rigorous, scientific impact evaluations — is used. Still, despite its importance, this kind of evaluation is especially difficult to implement in the environment sector, where theories of change typically depend on complex and difficult-to-anticipate changes in ecological systems and human behavior.

Simply put, creating a study that accurately measures how people, groups and environments change in response to specific programs is very difficult. As part of its commitment to evidence-based programming, the U.S. Agency for International Development has found a way to overcome these and other challenges to launch a pioneering RCT in Zambia that will contribute the strongest evidence to date on the relationship between property rights and farmers’ decisions to practice “climate smart” agricultural techniques, such as agroforestry.

The uptake of agroforestry — farmers planting and maintaining trees on their farms — as a climate change adaptation and mitigation strategy, particularly in sub-Saharan Africa, has been incredibly low, despite a plethora of studies touting its benefits to crop production and the environment. Researchers hypothesize that two of the major barriers to agroforestry adoption are insufficient access to quality agricultural extension services and insecure property rights. At the same time, the available evidence on the impact of land registration programs in sub-Saharan Africa is quite mixed, which may be due to the relative poverty of the areas targeted in Africa compared to other regions.

There are also very few RCTs — or rigorous impact evaluations in general — on land programs in Africa. In response, USAID designed what may be the first RCT to test whether improved access to extension services, more secure property rights, or both interventions together result in smallholder farmers successfully adopting agroforestry and other climate smart agriculture practices.

Conducting this impact evaluation has required extremely close collaboration across USAID, the program implementing partner, and the independent evaluation team. In addition to careful scrutiny of how the program and evaluation were designed, both the evaluation and program teams have benefitted from open lines of communication, including weekly updates on program implementation and progress on evaluation data collection. Still, along the way, we encountered a number of issues that might have derailed the impact evaluation, beginning with the need to build shared understanding with the program implementer and local community leaders of what impact evaluation is, why it is important and what implications it has for program implementation plans.

What sets impact evaluations apart from other kinds of assessments is that they measure the change in a development outcome, such as agroforestry adoption, that is attributable to a defined intervention. Impact evaluations rely on carefully defined models of cause and effect and require a credible counterfactual — for example, what would have happened without the program — to control for factors other than the program that might account for any change observed in the development outcome of interest.

In my experience, many attempts to conduct rigorous impact evaluations fall short because of the limited awareness and understanding of the methodology among program implementers, donors and local stakeholders — rather than from a lack of shared enthusiasm or commitment to learning. This is particularly true in the environment sector, which is still new to impact evaluations and where other kinds of “impact” studies are much more widely understood — namely, environmental and social impact assessments that seek to estimate future changes as a result of a policy change or development intervention.

As donors, if we want to see more impact evaluations, we need to spend more time educating our partners on how to define a credible counterfactual and plan program implementation timing and roll-out in ways that maximize the learning potential. The best way to model what would have happened without a program is for the evaluation team to randomly assign activities to treatment and control groups. This is different than the typical way of selecting beneficiaries, such as identifying areas that are the easiest to reach, or where the potential for success is deemed greatest. It requires that some people who are part of the evaluation do not receive benefits — the control group.

Impact evaluations also need to collect baseline data before the program begins its field activities, which means that the implementing partner has to wait until the independent evaluation team has completed baseline data collection before they start getting to know the communities with which they will be working. Ideally, the program also has to implement each set of activities across all communities at roughly the same time.

We have to build mutual understanding and find ways to meet the needs of both the implementation and evaluation teams. For our RCT in Zambia, the independent evaluation team was able to randomly assign the interventions and collect detailed baseline information on livelihoods, land holdings and land conflicts. The implementing partner could then use that in place of the typical participatory rural appraisal exercise they would normally use to inform their implementation approach. Such an approach required not just flexibility from our implementing partner, but also different analytical skills than we normally require for our environment programs. Luckily, we had an empathetic evaluation team who tailored the data collection process to allow the program to get started as quickly as possible, and we had a motivated implementing partner who clearly articulated the program methodology — including where there could (and could not) be flexibility — and proactively shared any unanticipated changes in the implementation plan with USAID and the evaluation team.

We also need to educate ourselves on what is required to conduct a rigorous impact evaluation and design our programs accordingly. When I took over as USAID manager of the program in Zambia, the program had been awarded and was in the early stages of work planning. Since we had only indicated the broad outlines of the activity in the award, we had to work with the implementing partner to quickly flesh out the program’s theory of change and implementation plan. It became clear that designing an impact evaluation requires a much higher level of upfront implementation detail than we usually ask of our implementing partners, at least in the land and environment sectors.

Donors have an important role to play in helping strike a balance between learning and implementation objectives. For example, even though randomizing land registration at the household level would have produced the strongest evidence of impacts on farmers’ land use behavior, this would have been absolutely unacceptable in the local governance context and could have even led to conflicts if households in the control group felt their land was being encroached upon by households in the treatment group. So, we reached an agreement to do land registration at the village-level after several discussions between the implementation and evaluation teams and with the chiefs, who are ultimately responsible for managing customary land and land-related conflicts in Zambia.    

Finally, it is important for donors and implementing partners to educate and incorporate expertise from local stakeholders, particularly in the design phase. To design our RCT in Zambia, we invited experts from local civil society and academia to help us understand the local barriers to agroforestry and security of property rights and to identify key interventions to overcome these issues. In Zambia, we also needed to build broad stakeholder buy-in for randomly assigning the program interventions, in particular with the chiefs. This meant breaking the impact evaluation methodology down into basic concepts and principles, such as why using a “lottery” approach to assign villages to different groups is both the most fair and most rigorous way to allocate limited program resources across their chiefdoms.

Ultimately, all of these collaborative efforts enabled us to design and implement a rigorous RCT. As I reflect on the many challenges we overcame in the process, it seems to me that the most important component of a rigorous impact evaluation is building strong relationships. It does not matter how accurate your power calculations are if the program is not designed and implemented according to plan. There are innumerable reasons why well-intentioned plans go awry in the real world. This is precisely why it is so important to develop clear and open lines of communication among all the stakeholders from the very start. At the end of the day, I think that a little trust and mutual understanding can go a long way toward improving the evidence base in the environment sector.

Follow Focus On: Devex World for more conversations emerging from the global development event of the year.