How to improve your monitoring and evaluation function

At a time when development funding is under threat, it’s imperative that any money available be invested into effective interventions rather than wasted on those less effective. Robust monitoring and evaluation — or M&E — systems are key to ensuring that, say experts. “M&E is the sector that can push for changes because you have the evidence of what’s working and what’s not working,” said Nathalia Quintiliano, a researcher and M&E consultant who has worked with the United Nations Office for Project Services, Médecins Sans Frontières, and the U.N. Office for the Coordination of Humanitarian Affairs. M&E, as defined by impact measurement social enterprise Sopact, “is a combination of data collection and analysis (monitoring) and assessing to what extent a program or intervention has, or has not, met its objectives (evaluation).” Monitoring involves ensuring compliance with project goals while evaluation is a periodic assessment to understand if a project was effective and efficient, explained Quintiliano. According to the Food and Agriculture Organization, it is a process crucial to assessing whether progress is being achiev­ed, spotting implementation bottlenecks, and discovering whether there are any unintended effects of program or project. The U.N. Development Programme’s M&E handbook explains that it can help detect any issues at an earlier stage, reducing the likelihood of cost overruns or time delays. It is the only way to ensure the cost-efficiency and cost-effectiveness of policies and humanitarian programming once a project is being implemented, said Caitlin Tulloch, director of Best Use of Resources at the International Rescue Committee. In fact, Amanda Glassman, executive vice president at the Center for Global Development, said $1 million spent on what she called “proper impact evaluation” — a type of evaluation that pinpoints a particular effect of a program, perhaps via randomized trials or utilizing administrative data — could save hundreds of millions in mistargeted or ineffective spending. “COVID-19 has made it clearer than ever that making decisions based on incomplete or outdated evidence costs lives and livelihoods,” she said. Yet despite the potential of M&E to save the sector money and ensure more impactful programming, according to 2016 data, there is an annual $350 million funding gap when it comes to generating the data needed to monitor progress on the Sustainable Development Goals. At the same time, following the pandemic and amid the war in Ukraine, several countries — such as the United Kingdom and Norway — have reduced or repurposed their international development budgets. “The real role of M&E should be to improve people’s lives and make a change.” --— Nathalia Quintiliano, researcher and M&E consultant In this context, how can development organizations improve their M&E to ensure cost-efficiency? Devex asked the experts. 1. Set up consistent systems Good M&E needs to be a system of collecting, recording, and analyzing information, said Joseph Sineka Limange, an M&E expert who has worked with U.S. Agency for International Development, Grameen Foundation, and Johns Hopkins Bloomberg School of Public Health, rather than simply being an ad hoc project visit with write-up. That means taking a standardized approach to each project as much as possible, but also being careful to tailor indicators and targets to the specific context. This is sometimes not easy, Monica Peiró, director for knowledge strategy and innovation at DAI, said in an email. She said that it could be challenging to reconcile project-level M&E processes with the requirements for a whole organizational portfolio, especially when — as often in the development sector — this means “a diverse, worldwide, multiclient, multi-sector portfolio.” To remedy this, she suggested using a flexible reporting platform. Popular M&E tools include the KoBoToolbox, ActivityInfo, and Granity. But digital by itself is not enough. For M&E to work well, it needs to be supplemented with the right processes and the right roles. While most larger NGOs will have in-house M&E officers, Jeremy Danz, an international development M&E consultant who has worked with Partners In Health and World Education, said smaller organizations may draw upon external experts. And when it comes to data collection itself, it may not always be the officers collecting that but staff members in the field; health care workers, program officers, educators, and volunteers. This may make it difficult to collect and monitor information consistently. To help monitor consistently across an organization, the International Institute for Environment and Development has produced an M&E guide for non-professionals. It has enabled volunteers for the Ugandan NGO Conservation Through Public Health to better collect data on whether awareness-raising activities on conservation are helping to reduce pressure on natural resources and transmission of zoonotic diseases. 2. Begin monitoring before a project starts In order to have effective M&E, it is essential to do two things before you start the project — decide what success might look like, and take a baseline before you get going. “How can you understand if your project had a good impact if you don’t know how it was before?” Quintiliano asked. Typically, what you want to achieve is codified in a theory of change. Creating a theory of change involves identifying what difference you want to make, how you plan to make that difference, and what metrics you will use to measure whether you succeeded. Danz said it is a chance for an organization “to explain to themselves, their donors, and the participants in their programs how their activities are going to produce the changes they seek in the world.” A strong theory of change should, according to the National Council for Voluntary Organizations, be based on previous experiences or relevant research, be achievable, be supported by all stakeholders, and be testable. It can involve a lot of different activities and steps an organization might take to achieve its goal. “At each portion of their theory of change, they can then identify some key performance indicators, which should be time-bound, so they can assess the impact of their work at different levels and over time,” Danz advised. Even as early as the project proposal stage, organizations should have their M&E plan complete with a clear criteria for how the performance of a project will be assessed with quantifiable targets, said Limange. The Bill & Melinda Gates Foundation in fact requires partners to “define and agree on measurable outcomes and indicators of progress” and success early in the grant proposal process. While this may require resources smaller NGOs don’t have, there are online templates and guides, such as those offered by Intrac. 3. Prioritize beneficiary feedback When it comes to implementing an M&E plan, Quintiliano explained there can be a tendency to focus on monitoring the outputs of a project: any products and services being delivered. “People want to show their donors, and donors want to show the public that the project makes sense … but the real role of M&E should be to improve people’s lives and make a change,” she explained. That means that any good M&E system should also be collecting qualitative as well as quantitative data from the community in question. This might come in the form of surveys, focus groups, or interviews. “If you are in contact with the beneficiaries and stakeholders, M&E can show you that the strategy is wrong, where you can improve, and if you’re spending money on something that isn’t relevant,” Quintiliano said. Any feedback collated from project participants must then be assessed and considered regularly, Danz said, so that any adaptations to the intervention can be made as soon as possible. This will help to ensure a project is having the maximum impact it can. According to USAID’s Learning Lab, “establishing feedback loops increases the likelihood that evidence will inform decision-making.” 4. Consider the best approach for each project While there’s no dispute about whether or not M&E is necessary, “there’s plenty of discussions ongoing in terms of the best way to do M&E,” Danz said. He stressed the importance of assessing each project on an individual basis and deciding on an M&E approach that will work best for that particular intervention. For example, in an education program where children are the targeted beneficiaries it might be better to select a participatory method — when the stakeholders of a program are active participants in the monitoring rather than being observed by outside experts — of communicating through drawing rather than questionnaires, which may be better suited for a project working to improve mental health within a community. Simultaneously, other types of monitoring will likely be at play, including results monitoring, which tracks a project’s likelihood of meeting its goals; compliance, which ensures the project is meeting all regulations, contract requirements, laws and ethical standards; and financial, which tracks costs and adherence to the project budget. The method of data collection also needs to be appropriate for the evaluation method being implemented. “If you’re collecting information from program activities but you’re not collecting it in a way that it can be cohesively used later on in an evaluation then that data has perhaps less value moving forward,” Danz said. Impact evaluation, Danz said, can be useful when testing out a new type of intervention. “It answers the question: did the policy, program, or technology work against the object that was set out? So did this nutrition program improve nutrition?” Glassman said, explaining that it’s a good method if you’re comparing different alternatives versus the status quo but is less useful if the question is around how many people have been reached. “That can be answered with administrative data.” In Colombia, impact evaluation led to the reallocation of investments for better early child care, and in Uganda, it spurred on the adoption of a national food fortification policy. This type of evaluation typically requires a mixture of existing data — perhaps from standardized population-based surveys, official statistics, or program records — as well as new data that could come from questionnaires, interviews, or observations. IRC, as part of an impact evaluation, captures the costs of the interventions while tapping into existing finance systems data. “As someone who formerly worked as an external evaluator, I am gobsmacked by the wealth of data that development organizations have sitting in their accounting systems,” Tulloch said. “If we see finance systems as sources of programmatic data, not merely accounting tools, it opens up the possibility of doing cost-analysis on a much greater scale and with greater consistency.” Other evaluation types include process evaluations, which delve further into the data to look at why outcomes might not be achieved; thematic and cluster evaluations, which are larger scale and measure multiple projects pertaining to a theme; and joint evaluations where staff from another organization share an external perspective on the project. Other tips Quintiliano shared include disaggregating data as much as it can be to improve analysis, paying attention to cross-cutting issues, and considering how sustainable a project is, and whether it's efficient. “These are evaluation criteria that are very important for every evaluation at any point of a project.” Regardless of the types of M&E used, Danz said, all aspects should be constantly updated and reviewed. “M&E is much more like gardening than sculpting,” he said.