How M&E data can hide development impact

While Brendan Brown was dispatched to Ghana to conduct fieldwork for the Food and Agriculture Organization in 2012, he noticed an odd disconnect. The official statistics on conservation agriculture didn’t seem to match up with what he was finding on the ground. The data FAO had reported was binary: it categorized farmers as either an adopter of conservation practices, or not. In the field, Brown saw this as too simplistic. “Adoption is not an outcome, it is a process where a farmer learns, assesses, experiments and will decide how they are going to use that technology,” he explained. “If it is adopted, it might be in modified form, it might be at a lower threshold than what is totally beneficial, it might be on a few fields or they might embrace it totally.” Now a PhD student at the University of Adelaide, Brown was determined to capture a more nuanced picture. Using data collected by the Australian Centre for International Agricultural Research between 2010 and 2016, Brown was able to re-analyze a large dataset of over 6,500 households from 1,601 villages in Ethiopia, Kenya, Malawi, Mozambique and Tanzania to determine how applying categories of adoption would impact numbers. Using a binary adoption rate, most countries scored high — around 90 percent adoption. But the figures were misleading, he told Devex. “When we actually looked at how many of those farmers had truly embraced conservation agriculture — which means using conservation agriculture to the correct threshold on all of their land — we found that only 22 of the more than 6,500 farmers could be classified as total adopters.” Brown is now using his research to urge development organizations to think beyond binary measurements, which risk revealing only half-truths about development impact. He believes his work, which he presented at the Australian Agricultural and Resource Economics Society annual conference in Brisbane earlier this month, holds important lessons for the sector about how to tell if their interventions are working. Conflicting agendas Brown told Devex that self-reporting may be one reason for the discrepancies between data and reality on the ground. In the case of Ghana, organizations collecting the figures had an important stake in proving conservation agriculture was working. “The development sector tends to be self-reporting about the benefits that come,” he said. “FAO, for example, is the key driver of conservation agriculture in Africa and a lot of their stuff including journal article will claim large numbers. But at the end of the day they are using government agencies and NGOs to generate that data and there is vested interest in promoting the story of strong adoption of conservation agriculture.” Brown followed up directly with farmers conducting interviews to understand barriers to full adoption and use of conservation agriculture in farming practices. His interviews revealed a range of challenges, including lack of markets to buy the machinery required; competing uses of soil between agricultural production; and issues of security for farm equipment. “We found that even if farmers appreciate it is a beneficial technology, it does not match what they are able to do at the moment,” Brown said. In Ghana, the policies encouraging conservation agriculture focused on farmers being “all in.” That proved too transformational for many farmers to fully embrace. “We found that we would be much better off to be looking at how we can make pathways to greater implementation of conservation agriculture — start with growing legumes and find markets, and then farmers can have money behind them to invest in the next steps.” A stepped pathways approach would gradually increase financial stability of farmers and provide them with the support to gradually lead to full adoption, Brown contends. Understanding impact John Anderson, former deputy prime minister of Australia and board member for the Crawford Fund, said research such as this was “astonishingly important” and highlighted the difficulty in creating cultural shifts in agricultural practices. “In Africa in particular, people with enormous experience in this area of aid will tell you it is very, very difficult to get a generational change in attitudes,” he told Devex. “You’ll often even have quite extraordinary examples where a new technology has been trialled producing a surplus of agricultural stocks. But they then slip back into the old ways and don’t maintain a path forward that would help build surplus stocks and this is the lift off point for drawing many farms, their broader community and country out of poverty.” Based on his research, Brown is urging aid organizations to look beyond binary data in assessing outcomes. Short of that, he said, NGOs could build their programs around assumptions that aren’t correct — for example assuming farmers have already adapted certain practices, when they in fact lag behind. “These policies will be building on wrong foundations,” he said. “We will get into a grey area where 10 to 15 years down the road… farmers are going in one direction and policies in another with a big disconnect in between.” He acknowledges the challenge for NGOs, which are often under pressure to deliver fast results on a tight budget. “A lot of the time NGOs are judged on their ability to deliver impact, and funding can be dependent on demonstrating that impact. If they were to change their reporting structure to show subsidized versus true adoption, NGOs could open themselves to accusations of being ineffective in their programs.” Brown hopes to use his initial research to help build a framework for NGOs to use in monitoring impact. “The issue at the moment is having the right dataset and ensuring the right questions are being asked when people are asking surveys to populate these framework,” he said. With improved data, Brown said there is the potential to look at change over time and track the impact of different innovations with different subsets of the population including gender, age and disability. “Hopefully there will be an opportunity to collaborate with organizations including NGOs to build on this work.” Read more international development news online, and subscribe to The Development Newswire to receive the latest from the world’s leading donors and decision-makers — emailed to you free every business day.