Evaluating the evaluators: The role of independent evaluation in development

As aid donors tighten their financial belts and focus on value for money, monitoring and evaluation is becoming an increasingly important and scrutinized stage in the development project life cycle, affecting project selection, spending modalities and strategic planning. Structured, formal and transparent evaluation is often the best way for donors and beneficiaries to tell if a development project achieves its intended goals. While there are significant costs in both time and money, evaluation can also help donors learn from their successes and failures and improve future project execution. For these reasons, the world’s largest multilateral development institutions, including the World Bank, Asian Development Bank and the African Development Bank, have sanctioned and implemented independent evaluation departments with the mission to provide an objective assessment of their programs and overall operations and, according to the World Bank, “to identify and disseminate lessons learned from experience.” IE offices usually report to the institution’s board of directors and not to the president where — ideally — they can exert appropriate influence on operational decision-making. For instance, ADB stresses that IE provides “feedback on performance” while AfDB asserts that it also helps “deliver development activities to the highest possible standards.” But while IE offices are firmly established in these institutions, there are questions over their mandate and role and whether or not these departments are actually independent and effective. More specifically, do these departments have the “teeth” to inform and influence operational decision-making and behavior? Importance ADB’s current independent evaluation director general Vinod Thomas and World Bank senior economist Xubei Luo argue in a co-authored book that independent evaluation puts faces to names and impact to figures, and gives a more in-depth look into development project outcomes — not just output. “Evaluation can help improve development effectiveness … and promote accountability relating to actions taken by countries and international financial institutions, and contribute to learning about development effectiveness,” Thomas and Luo noted. Rakesh Nangia, AfDB’s operations evaluations director, echoed this sentiment, telling Devex that evaluation efforts for all the projects of a multilateral aid institution are crucial because not only does the effect reach internal operations, but the impact also extends to direct beneficiaries. “Measurable results are relevant for reform efforts, making a difference in getting better measures of performance, toward achieving the goals of increasing productivity at an internal organizational level and reducing poverty as a final impact,” he said, adding that the main emphasis is on how to do things better during — and even after — the implementation of a certain development project. Aside from improving development projects’ effectiveness and MDBs’ operations because of the “larger base of knowledge accumulated,” independent evaluations can also spur a sense of accountability to institutions, serve as a self-check as well as inform other wider high-impact decisions including policy, economic and even political. “Rigorous impact evaluation has caused so much excitement because it can provide clear evidence free of ideological or political bias,” Nangia explained. “IE can help strengthen the quality of evidence used to make program and policy decisions.” “Unfortunately, the two principal reasons for doing evaluations at present — a political accountability requirement and to prove we are accomplishing something — have driven out what should be the ultimate reason: to find better ways to achieve our objectives” --— Linn Hammergren, a former World Bank and USAID official A good example of IE’s ability to affect a development project’s design and implementation is ADB’s criticized $50 million loan to a dam project in Laos. Kurumi Fukaya, the project’s lead investment specialist, shared with Devex in August that, while contentions remain, the bank has done proper “prospective evaluation” by gathering and listening to the views of stakeholders resulting in changes to the project design, including the location of resettlements and other issues relating to the safeguard issues. Despite such examples of IE in practice, the jury is still out on how the discipline can best be operationalized within development finance institutions. According to an article by World Bank’s Daniela Gressani and Patrick Grasso, IE serves three primary purposes: inform and influence ongoing decisions, give light to what works (or does not) in development, and improve the performance of MDBs and other organizations. In an interview, Thomas explained that evaluation should serve as a platform for an organization to do better while also being transparent. “The beneficiaries would like to know whether the projects … are producing the results,” he told Devex. “The organization itself, any organization, would want to know how [they are] doing.” But some evaluation experts suggest that existing practices are commonly driven by the wrong motivations and turn out misguided. “Unfortunately, the two principal reasons for doing evaluations at present — a political accountability requirement and to prove we are accomplishing something — have driven out what should be the ultimate reason: to find better ways to achieve our objectives,” argued Linn Hammergren, a former official at the World Bank and the U.S. Agency for International Development. Independence Despite the perceived and actual gains that independent impact evaluations have accrued over the years, many still question the level of independence that these evaluation offices retain, especially those within MDBs. These perceptions are largely based on the potential conflicts of interest that may ensue from evaluation results that come out as negative or damaging to the image and interest of the multilateral agencies given that the departments are internally sanctioned and funded. This debate has fueled calls for the establishment of completely independent evaluation offices or the extensive use of third-party evaluators. “There are calls for making this evaluation even more independent … should there be not an outside agency doing truly independent evaluation, meaning not inside [an MDB] but outside,” Thomas told Devex. “If you’re part of [an MDB], are you really independent?” The IE chiefs acknowledge the validity of the conflict of interest argument, but counter that being part of the organization provides necessary credibility and access that external parties would have difficulty gaining. ADB even has its own public guidelines in avoiding conflict of interest in independent evaluations. “Independence [in evaluation] is not isolation.” --— former World Bank evaluation chief Roberto Picciotto “It’s a valid question but then it goes back to [the idea that] you have to have a balance of independence,” Thomas said. “So if you’re not really familiar with the organization, if you’re not embedded, you won’t have that effectiveness, so you can be very independent [but] sit somewhere very far away and have very little influence on the organization.” This is echoed by AfDB’s Nangia who indicates that there will be more challenges than gains if these offices are physically — and in principle — independent and separate from the MDBs, including the need for thorough embeddedness, familiarity with and presence in the organization’s structure and operations, as well as some instances where “data confidentiality” comes into play. “Conducting IEs also involves being present at [all] stages of the project cycle, where evaluators can influence and guide [the project],” he said. “There could also be issues of data confidentiality, access to restricted information, internal systems and also to staff.” External independent evaluation entities face the challenge of earning the trust and confidence of a bank’s board as well. “[The board of directors], to which evaluation offices report to, might not be as receptive or as willing to hear from a totally external entity that is not engaged with the daily work of the institution and which might be totally detached from its organizational philosophy,” Nangia said, clarifying that IE offices only collaborate with bank presidents, but do not report to them directly. Independent evaluators also argue that there should be a balance between autonomy and engagement. “Independence [in evaluation] is not isolation,” said former World Bank evaluation chief Roberto Picciotto, who is currently visiting professor at the International Development Institute of London’s King College. “[IE] enhances organizational learning by verifying the validity and effectiveness of self-evaluation products and processes … but they should be subjected to periodic review by external assessors in order to sustain the professional excellence of their work. Even independent evaluators need to be independently evaluated,” he said. Influence Irrespective of where an evaluation office sits within an MDB’s organizational structure, the most important factor is whether or not its assessments and recommendations will be listened to and acted on — especially if they are critical or negative. The evaluation chiefs agree that this depends on certain enabling conditions — something that should be looked at more objectively as the international development community, and not just MDBs, are becoming more aware of the importance of IE. Thomas shared from his experience that influence of independent evaluation largely depends on whether the leadership in a certain institution values evaluation — and the recommendations that come with it — seriously. “Independent evaluation offices are not entirely ‘toothless’ [but they] are not sufficient antidotes for inefficient resource allocation or policy mismanagement that can lead to poor development results.” --— Rakesh Nangia, AfDB’s operations evaluations director “You need the leadership of course who values evaluation [and] is confident that we’ll be stronger if there are comments [that] are pluses and minuses at the same time,” he said. “I think the two sides need to come together. It’s not always assured that this will be the case at any given time. Every time there’s a change in leadership, you have to weigh whether you are invited.” In terms of processes, on the other hand, Nangia shared that stakeholders should understand that independent evaluation is not a “be-all, end-all” solution when development programs fail or do not meet their targets. “Independent evaluation offices are not entirely ‘toothless’ as they are not totally powerless. [They] are not sufficient antidotes for inefficient resource allocation or policy mismanagement that can lead to poor development results,” he explained, asserting that IE offices already serve as “deterrents of potential misconduct.” IE officials agree that in order for their departments to accomplish their missions they must attract buy in from top management and other concerned stakeholders as well as necessary levels of investment in tools, personnel and other resources. “The true challenge is institutionalizing IE [and its influence] among MDBs and, above all, for it to remain relevant and be put to good use by all actors in the developing community,” Nangia said. Hammergren, who has served both inside MDBs and as an outside consultant, argued that IE offices within MDBs remain more influential than external ones and there are many hurdles facing evaluation practitioners. “I certainly felt some pressure from within, but this was nothing compared to the arm twisting I experienced once outside, both directly from those under evaluation and from the firm hired to do the work,” she said. Challenges and solutions The standing of IE within MDBs is changing as these organizations undergo dramatic restructuring and reform. Nangia cites several key challenges including disconnects in learning, internal usage of generated knowledge, institutional disincentives, learning and knowledge transfer, lack of trust between policymakers and evaluators, technical challenges and credibility, and believes there should be “new solutions” to “address [changing] issues,” which includes technological, organizational and management changes. Thomas also highlighted the importance of innovation in evaluation, saying that evaluators should be sensitive enough to be aware when and how to be innovative and regard risk-taking as central in furthering a development projects’ impact and targets. “[For a project] there’s a chance it will succeed … or fail, whereas if we just did the same thing again and again, the chances of failure is much less. But it may not make a big difference,” Thomas said. “You need to have evaluation with the mindset that says ‘we don’t want to reward failure,’ that’s not the point, but the risk-taking is a good thing.” Evaluation, just like development, is a multistakeholder undertaking. While IEs are responsible for holding MDBs to account, evaluators stress that the responsibility of making sure that development programs are effective and impactful while being environmentally and socially sensitive lies across their organization. “The role of the development community is to … ease the path of independent evaluation offices into areas where implementation did not do so well, and to help explore fields of knowledge where new ground is being broken,” Nangia concluded. “Changing society is for all of us. But we can only make a difference if we improve our knowledge of each other. We are in need of collective learning.” Check out more insights and analysis provided to hundreds of Executive Members worldwide, and subscribe to the Development Insider to receive the latest news, trends and policies that influence your organization.