Q&A: Summing up why evaluation matters

The Independent Evaluation Group acts as a learning and accountability mechanism, sharing evaluative feedback on the various projects and programs spanning the member institutions of the World Bank Group: the International Finance Corporation, the Multilateral Investment Guarantee Agency, the International Development Agency, and the International Bank for Reconstruction and Development.

Speaking to Devex, the IEG’s Director General Caroline Heider explained why evaluation of any kind is important, but emphasized why it matters so much in development.

“If I'm a customer of a company or a client of a big public sector utility and I buy services, assuming that the market functions, I can complain about it. I can purchase my goods somewhere else if I don't like the quality — so there is that direct feedback,” she said, adding that the provider can then consider that feedback.

When it comes to development institutions, relationships — whether between donors or beneficiaries — are indirect and feedback is less tangible.

“The money comes from other resources, so it's not the one who gets the service who pays for it,” she said. “You have many different actors in a more complicated system of accountability.”

Enter independent evaluation. By assessing the World Bank Group’s successes and errors, the feedback gap is being filled and driving the bank further toward its goal of accelerating poverty reduction and boosting shared prosperity. Heider gave a number of examples of how IEG’s evaluation has made a difference so far and explained how the process can contribute to a better understanding of development.

Below are more highlights from the conversation, edited for length and clarity.

Thinking of the development sector, what is the value of engaging in independent evaluation?

Independent evaluation has two elements that are valuable to the development sector. The first is the evaluation itself, the act of evaluating and of thinking evaluatively — what did we try to achieve and what have we actually achieved? The second is to reflect on whether we reached a better or worse place, and ask if we should repeat exactly the same or do something differently.

Independent evaluation is important because if I evaluate myself, I might be hesitant to reflect on problems, because I’m embarrassed or because peers or supervisors may embarrass me, not give me a promotion, or not recognize my efforts.

If you think of evaluation as a collective effort, as it is in organizations, that gets amplified and, collectively, groups of people would think reporting on certain problems could lead to negative repercussions. This is where independent evaluation can step in and actually talk about the things that institutionally are off the record or not really touched upon other than in one-on-one, informal conversations. Everybody knows the system is broken in certain places, but nobody wants to speak about it because the institutional norms forbid that. That's the space where independent evaluation can help solve problems.

Do you think enough development institutions and organizations are aware of the benefits of engaging in independent evaluation?

Evaluation has been expanding a lot over the last 15 years. Still, many institutions don’t have evaluation offices as independent as IEG. We report directly to the shareholders, which means even if we report something the operational side of the World Bank Group doesn’t like, they can’t really interfere.

The regional development banks have similar mechanisms. Then there are the U.N. agencies, but there the situation is more complicated. Many of them have evaluation offices that are much smaller and embedded deeper into management structures, which can facilitate and promote self-evaluation and learning from evaluation if it's taken seriously. But this gives management much stronger control over the message. If they don't like the evaluation message, they can try to manipulate it so that the member states don’t hear the more difficult, more critical messages. A number of U.N. agencies have moved to an arrangement where there is at least a dotted line to the executive board of member states so that there’s some degree of independence.

How has independent evaluation of the World Bank improved its systems and performance?

Two examples, one in which changes will take a long time, and another in which changes were implemented rapidly.

Our 2010 evaluation of safeguards recommended, among other things, the system of safeguards be reviewed and updated. Safeguards are policies that ensure projects don’t harm people or the environment. It took two years before the process to review them started, and then took another three or four years of review and consultations. The role of evaluation was to trigger the change process, and throughout provide feedback whether the revisions were addressing concerns we raised in the evaluation.  

The second example is an evaluation of the World Bank Group’s procurement policies, which really affect billions of dollars the World Bank loans and spends on goods and services procured. At the end of the evaluation, the World Bank had prepared a new policy and asked us to review whether it addressed the very detailed findings we’d presented. That's an example where the ink hadn’t yet dried and we saw an uptake of recommendations.

In both cases, and others, we are drawing on an entire system rather than a “one time delivery” of the message. If you provide a report and thereafter it's up to the policymakers or the readers to use it, you might not actually have as much of an effect; it is period follow-up, continuously feeding into the review process after the evaluation is completed, and messaging at various levels that you have to have to make something happen.

And, as a next step, it will be important for us to revisit our evaluation recommendations. For instance, if we do an evaluation again in a couple of years, it’ll be interesting to ask how well those recommendations worked, whether they were implemented and whether they’ve led to greater success? That's the one thing that we, as evaluators, have relatively little insight into at the moment.

What’s the next frontier for independent evaluation and how do you expect work in the sector will evolve over time?

I think the biggest challenge and opportunity is really around the question of complexity. When you look at the Sustainable Development Goals, many of them are very interrelated. They're not individual silos where you can look at one factor and solve the problem and then move on to the next. They’re all interconnected. For example, issues of climate change, forced displacement, or poverty are much more pervasive in the entire development agenda. What evaluation needs to do, as much as development, is understand how to operate in that interrelated context.

Evaluation has followed the project design approach of a logical framework with a results chain. This is linear so that you can trace input, outcomes, and impact, whereas in reality there are many factors that play a role and your inputs and activities might have effects other than producing your output. Likewise, your output might have other effects. Capturing that more dynamic nature of development and understanding how to apply modern, multiple, and different evaluation methodologies with data from different sources is going to be an interesting and important challenge. That’s where evaluation can contribute a lot to the dialogue of understanding development better.

Learn more about the role of the IEG within the World Bank and its examples of evaluation success here.