Does the development sector need to think again about measurement?

The importance of data in achieving Sustainable Development Goals and having a significant impact is well-documented. But experts believe that many organizations lack the skills necessary to collate, monitor, and evaluate it for maximum progress. Good monitoring and evaluation, or M&E, can show what works and doesn’t work, how a project can be improved, and whether money is being spent on something that isn’t relevant, said Nathalia Quintiliano, a researcher and M&E consultant who has worked with UNOPs, Médecins Sans Frontières, and OCHA. But as it stands, most organizations “have got it absolutely wrong,” according to Joseph Sineka Limange, an M&E expert who has worked with the U.S. Agency for International Development, Grameen Foundation, and Johns Hopkins Bloomberg School of Public Health. “There are people who have handled M&E positions for several years and are not experts because what they were doing … was not based on systems.” --— Joseph Sineka Limange, M&E expert Right now, he said, many carry out “monitoring visits” instead of creating systems designed to understand in depth what went well and what could be improved upon. He and other experts talked to Devex about the barriers to using M&E to stop development spending from being wasted on ineffective interventions and how to ensure it's being used to its best effect. Not enough trained M&E personnel Some elements of NGO management view M&E as a donor requirement, rather than vital to project implementation, Limange said. As a result, they have seen M&E as merely a box-ticking exercise, which can mean there’s little desire to invest in M&E skills development for staff. Yet M&E systems certainly are an increasing requirement for some donors, said Quintiliano. Limange gave the example of USAID, which, he said, was advanced compared to other donors as it puts a lot of emphasis on the quality of data, quantitative data, and monitoring systems. Meanwhile the Millennium Challenge Corporation, designs programs that are set up for impact evaluation — a type of evaluation that pinpoints a particular effect of a program — according to Jeremy Danz, an international development M&E consultant who has worked with Partners In Health and World Education. For this reason, said Limange, the number of “credible M&E practitioners” is low. “There are people who have handled M&E positions for several years and are not experts because what they were doing … was not based on systems,” he said. Limange called for the creation of more degree courses within academic institutions that provide rigorous and practical M&E training. Currently, most courses, he said, tend to be only a few hours a week for a few months, which is not enough. The need to demonstrate value At the same time, existing M&E practitioners should focus on showing their value through the production of information that decision-makers will find useful, Limange added. This, he said, will lead to further appreciation of M&E — and perhaps investment in it as a development function. Information is likely to be more useful if researchers from the local context are involved in the evaluation, Santhosh Mathew, country lead on public policy and finance for India at the Bill & Melinda Gates Foundation, said in a statement sent to Devex. Mathew said their involvement would help to “identify more relevant questions, get more useful findings, and policymakers in developing countries are more likely to trust and use results when deciding how to spend limited resources.” “We’re past the days where it makes sense to design a development program in Washington, deploy it in Kenya, and assess it in Washington. We don’t see Kenyan researchers as the main authors of studies on United States policy. Why should we accept different trends elsewhere?” echoed Dr. Catherine Kyobutungi, executive director of the African Population and Health Research Center. The donor problem Aside from affecting the level of investment into M&E, the donor focus can also affect the quality of data collected, experts said. “Sometimes you can focus too much on getting numbers to justify to others that your project is working,” said Quintiliano. Alternatively, it can mean collecting one set of data for the donor and another for internal use. This is not ideal, she added, because it can lead to an emphasis on measurable outputs rather than a focus on the feedback of the targeted community. For example, there might be an emphasis on recording the number of dignity kits distributed in a refugee camp, or the number of self-help groups established, rather than on collecting data that could share how useful the kits and groups are and how they could improve. “The people who live in the places where the programs are taking place are really the experts for the work that’s taking place in their communities. … That’s the core piece of expertise that has to be rolled back up into a system and ideally back to funders,” Danz said. Organizations, he added, should consider how often they’re adapting implementation plans and strategies based on the ongoing information it receives from communities. For example, if a community is saying they don’t want a particular investment to be made in their community, how often is that feedback going to headquarters and plans being adapted accordingly? Danz asked. It’s the M&E officer’s responsibility, Quintiliano said, to find common ground with a donor and to “negotiate, explain, and decide which methodology, what indicators, and which criteria are more relevant to the project you're implementing.” After all, M&E, she said, should be “a learning experience” for all those involved in implementing a project.