As an independent evaluator, Vinod Thomas has worked with five development bank presidents, three at the World Bank and two at the Asian Development Bank. All of them, according to him, see the quality of the work of independent evaluators as critical.
But how they use independent evaluation sets them apart. Some presidents would use it to implement reforms, others to tell those in operations, "Look, this what the independent people are saying; what are you going to do about it?"
Some presidents engage independent evaluators more than others.
"Some take you into the meetings, some don't; they keep a distance," he told Devex in a recent interview at his office near the ADB headquarters in Manila.
Thomas is currently preparing for a two-day gathering marking the 10 years of independent evaluation at the bank. The so-called learning event will feature the launching of a book detailing the challenges and approaches on the work of independent evaluation.
Here's an excerpt of my conversation with the ADB director-general of independent evaluation.
What were the biggest lessons learned about impact evaluation over the last 10 years?
In the 10 years, I think one of the lessons has been that an evaluation that draws on the lessons of the past has to have its eyes on the future because things change, so evaluation can't just be a history department. It can't be just backward-looking which is usually the case. So to be effective in an organization like the ADB, ADB needs to change with the changes in Asia, and evaluation needs to change and look forward, although it is drawing on the lessons of the past.
Read more on impact evalutations:
● Impact evaluations to guide policy
● How much do impact evaluations cost?
So for example, if we just look at the past, we would have some lessons of how Asia's growth rate can be maintained, right? But looking forward, you also have to ask not only that the growth is high but it is distributed better and also that it is environmentally sustainable because without that, you cannot have growth continue the same way. So evaluation needs to be more agile, more forward-looking, more dynamic and change with the times.
So that's one [lesson]. I could add one more: The independence of the evaluation has a great value. The self-assessment, the one who did a project has to give the first review of how things went. In an evaluation, the person can start with what do you think, how did you do last year? ... But independently, having an assessment on top of that is critical for an organization — some distance, some neutrality, some objectivity. So we see some differences between what the one who did the project thinks and what the independent evaluation would say. They're not exactly the same.
The ratings could be different. The one who does it thinks that "I did the best I can, so this must be extremely good" can differ from the one who's saying "we understand the constraints." But for the clients, for the beneficiaries, what matters is a better outcome and if that didn't happen, sorry, regardless of the fact that you did make some effort, the outcome is not good enough. So there can be a difference and we see that difference, you know, fairly systematically. So independently being able to say and call it like it is is good for the organization. It's not destructive criticism; it's objective, constructive criticism which outsiders including financiers of organizations like ADB take it as a sign of credibility that this organization is strong enough to say “we did very well in this area, we did less well in this area.”
What's the most rewarding aspect of your work?
You can be forward-looking and you can be independent and call it like it is, but what's most rewarding is to be able to influence the organization … That means you need to be effectively engaging with the management and the operations and the clients. So we need to be at the table of decision. We need to earn that place, so that means you need to have something to say; you can't just come in as independent evaluation and say "I want to come in and advise." You have to be invited.
So the most interesting is will an organization like the Asian Development Bank call on independent evaluation at a critical moment to say "how do you think we should go forward? What is your view?" And if you'd get a chance to say it, and that has some traction, whether it is to say "we need to stress inclusive growth much more," or "we need to stress climate action much more" or "we need to do more on governance" ... and if we can have the impact or influence on the organization's next steps, then it's rewarding.
I can say from the experience of nine years that it is not automatic that you're going to be invited. You have to earn it, one by the quality of the work, and two by the trust, the trust that you can be constructive, objective of doing the best for the organization and the clients, not to put out something that is interesting or even exciting but it is to make a contribution constructively. So the trust has to be there. And No. 3, I would say receptivity of operations.
I can say at the moment, at ADB, our experience has been, in the last two or three years, that evaluation is invited. The biggest example is we just did the midterm review of the 2020 plan of the ADB, and there's self-evaluation of management; we're invited to the table to produce an assessment and invited to the critical discussions where changes were made, to do a bit more on education and health, to do a bit more on climate adaptation. And so we had a voice. And it doesn't have to be necessarily big. The timing is important and the critical piece that you can contribute even if it's small is important.
Where do you think is the future of independent evaluation?
Looking forward, you've got to be much more effective in finding solutions to problems. And that I don't think we are quite there, partly the seat at the table business that I've mentioned. It's not automatic, that just because you're an evaluator it doesn't mean you get a chance to provide the lessons. Partly the methods and techniques especially to support innovation are still work in progress.
Partly I think we have to blend evaluation much more with research and policy, and not just say “this is evaluation and this is my conclusion” but blend it more with research and policy work to get solutions at the right time and at a right place. And with the invitation to be at the table, you have something much better to offer. It is happening to some extent, there are examples. But how to make those examples the norm, not the exception, that would be the challenge.
Join the Devex community and access more in-depth analysis, breaking news and business advice — and a host of other services — on international development, humanitarian aid and global health.
