Are you current on your acronyms? What to know about M&E vs. MERL

Technology continues to disrupt the monitoring and evaluation operating environment, and the field itself is morphing to more readily embrace research on effectiveness and a focus on organizational learning. In other words, M&E as we know it is blurring with research and learning to hatch a new acronym: MERL. Positions posted on Devex are increasingly reflecting the need for “MERL” advisers, and practitioners in the space are focused on improving the quality, capacity, ethics and security of data collection — while everything from mobile phones to remote sensors are creating new ethical and privacy challenges, along with security risks. Organizations taking M&E seriously are often more inspired by a passion to improve their own work than by a global framework, and this means spending time to bring program staff and M&E staff together as part of a strategic thinking team, which was discussed heavily at last week’s MERL Tech event, hosted by FHI 360 in Washington, D.C. (Where speaker after speaker unknowingly provided a creative repertoire of MERL and big data similes). The transition from M&E to MERL “will entail as much change management as the switch to tech-enabled M&E has involved,” according to Kerry Bruce, chief measurement and impact evaluation officer at Global Fund to End Slavery. “It means being rigorous about reviewing data and committing to changing course, even when it is either difficult or politically wearisome to do so.” The same questions that apply to any organizational change, she added, apply to the transition from M&E to MERL: Is there urgency, is there communication, are their motivators? Here’s a look at some of the challenges facing MERL as characterized by an automobile hazard, a large mammal and a place you don’t want to eat, among a few other creative comparisons: 1. An unpopular restaurant What good is a restaurant that makes delicious food, yet has no customers to enjoy it? This is the question Vinisha Bhatia of Results Data Initiative, a project of Development Gateway, posed to the MERL Tech event audience. There are more than 20,000 separate indicators for health, which is just one example of the web of parallel data that exists within the development community. But results management is often tailored to the needs of specific projects, donors and implementing agencies — and this data is not readily shared or communicated between donors and with government and other stakeholders. The Results Data Initiative, Bhati explained, plans to investigate how development results information can be more effectively collected, managed, shared and used across agencies and implementers to inform better outcomes. To do this, breaking down the increasingly siloed "MERL" function from the "program" function is key. Often MERL officers have data that program people don't know exists — and sometimes vice versa. “The more program officers engage with, extract data from and demand more analysis from MERL specialists, the more the ‘wonderful food’ will actually be eaten,” Bruce said. 2. Teenage sex A quote about big data from Dan Ariely of Duke University has been circulating for the last few years, but it’s still very relevant, according to Ben Ramalingam, ‎leader of the Digital & Technology Cluster at the Institute of Development Studies, who gave a keynote at the MERL Tech event. The quote? “Big data is like teenage sex: Everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.” The development community used to have a “build it and they will come” ethos about systems, which led to lots of “information graveyards,” Ramalingam explained. “We have replaced that with 'collect it and they will come,' and I'm afraid that this is leading to an equal number of data graveyards,” he said, adding that data collection is often more about expert posturing, branding and dysfunctional competition than about sharing and standardization. Data can be likened to innovation, in that you “have to guard against buzzwords,” Joshua Kaufman, director of the Office of Evaluation and Impact Assessment in USAID’s U.S. Global Development Lab, told Devex. Thanks to the huge spread of mobile phones and Internet access, there is a low barrier of entry for collecting data, “but unless you do it right, it’s not going to lead to improvement in performance,” Kaufman said. “In fact it could lead to new barriers, as you may underinvest in more traditional structures.” 3. Share Bear The M&E community is not doing a good job of sharing, according to Jacob Korenblum, president and CEO of Souktel Digital Solutions, who began his presentation with a large photo of the ‘80s toy Share Bear. Korenblum applauded — but also poked holes in — several sharing examples, such as USAID’s Coastal Communication Adaption Project Facebook page, where routine updates of the project can be found, but only for those who use the social media site. Other groups are using radio, but it’s not a medium readily available or accessible on demand. Some implementers even set up booths for face-to-face interactions regarding the status of projects, but that’s not always the most efficient form of communication either. Ramalingam pointed to the effort to provide a technology-savvy response to the Ebola outbreak, but noted a basic problem: “Most operational agencies were simply unwilling to share their information openly with others, even at the levels of quarantine cases, cases treated and deaths,” he said. Korenblum provided several pieces of advice for sharing strategies, including understanding existing ecosystems, building feedback loops into project design, creating and tracking indicators related to sharing and putting privacy first. 4. Detective novel If this was a detective novel, now would be the time to introduce a man with a gun. But Rahel Dette, a research associate at the Global Public Policy Institute, said she hopes the situation doesn’t have to get that dire when it comes to the threat of a breach of constituent data. Even hackers themselves get hacked, she said, referencing major data breaches of banks, universities and health care institutions in 2015 alone. Dette introduced a threat model borrowed from software developers and encouraged conference attendees to consider assets, adversaries and vulnerabilities of data from the outset of collection and program design. The asset might be human resources information or sensitive data sets of vulnerable populations. The adversary might be intelligence actors, companies or militants, while vulnerability considerations should inspire questions like “Should we be collecting this data?” “Why?” 5. Deluge There is no real evidence that the “data deluge” is making development more evidence-based, according to Ramalingam. In fact, collection of data can create a “false sense of reality,” he said. No amount of tech-driven M&E can make up for bad management, and no amount of tech can make up for poor instruments and measurement frameworks in MERL, leaders told Devex at the event. “If you do not validate your questionnaires and your definitions, you may end up with a lot of poor quality data (collected with nifty smart phones!)” Bruce added. Many managers want a dashboard, for example, but aren’t clear on what data are available that will answer their questions. Another example is when managers commit themselves to being "data-driven" and then ignore the data presented and make decisions based on political exigencies, Bruce noted. 6. Flat tire Susan Davis, ‎founder and executive director of Improve International, is working to put the L in MERL. The WASH sector, Davis explained, has been producing the same “lessons learned” reports for 40 years. She recently took a look at the book “Watering White Elephants,” a collection of lessons from donor-funded WASH projects in Tanzania, published in 1988. The book’s jacket summarized just a few challenges in the WASH sector, like operation and maintenance issues, lack of professional staff and inappropriate technology. She then picked up Nicholas Kristof and Cheryl WuDunn’s “A Path Appears,” published nearly 30 years later, and found the WASH issues discussed nearly unchanged. We can reinvent the wheel, she said, “but let’s not reinvent the flat tire.” Maybe it’s time, she said, to add an A to MERL — for act. Davis suggested the creation of one independent monitoring system for evaluating effectiveness, and shared quality assurance organization PQASSO and nonprofit evaluator GiveWell as inspiration for the evaluation of programs, rather than autopsy. 7. Eight 100-pound gorillas. Several leaders noted throughout the event that M&E remains an “extractive industry,” meaning that MERL functions are often built with donors and accountabilities in mind that have little to do with what a local population actually needs or needs to know. Increased coordination among donors in their sectors of interest or perhaps around the SDGs would be good places to start in order to correct this, Bruce noted. It’s also a high priority for the U.S. Global Development Lab, which has recently launched Monitoring, Evaluation, Research and Learning Innovations, better known as the MERLIN Program, which seeks promising new innovations for MERL challenges. There is no one 800-pound gorilla for collaboration in the MERL space, USAID’s Kaufman told Devex. But if you could identify eight 100-pound gorillas and get them to agree on common platforms or a set of technical standards, it would go a long way. Devex Professional Membership means access to the latest buzz, innovations, and lifestyle tips for development, health, sustainability and humanitarian professionals like you. Our mission is to do more good for more people. If you think the right information can make a difference, we invite you to join us by making a small investment in Professional Membership.