Opinion: Measuring impact may never be as straightforward as measuring financial returns

We are in danger of promising something we cannot deliver. People increasingly want to know the impact return on their investments, alongside the financial return. We tell ourselves that, if our impact measurement efforts are sufficiently harmonized, we will get there. I think that’s a mistake.

By “we,” I mean that amalgam of impact investors and public development finance institutions that gathers in conference halls and breakout rooms and tries to coordinate our efforts through a bewildering array of frameworks, principles, and initiatives. I do not mean to denigrate the real progress that has been made — my point is merely that we remain a long way short of being able to measure impact in a common coin akin to financial returns, and we may never get there.

Once we know how to measure the effects of our investments, we must then interpret the results. The Impact Management Project has developed a framework, which CDC has adopted, that prompts investors to ask all the right questions: What difference are you making and for whom? One of the points that Clara Barby, chief executive officer of the Impact Management Project, always emphasizes is: It’s all about context.

I think we would all agree that adding $100 to the annual income of a subsistence farmer making $700 a year has more impact than adding the same amount to the income of someone making $3,000. It would be harder to agree on the right impact exchange rate between the two. It gets even harder when you are trying to weigh lives saved against contributions to economic transformation, children educated against carbon abated, and all manner of other tangible and intangible variables.

Private impact investors are not the only ones who would like an easily digestible return-on-investment number for impact — public development finance institutions and their shareholders would find it useful, too. Desirable though they may be, numbers like that are not found elsewhere in the world of development.

NGOs and aid agencies do not measure the impact of their grant-funded programs on a single scale — although ongoing efforts to benchmark development projects against cash transfers might deliver. The best of them quantify what they can, look at benchmarks, and interrogate value for money, but that does not take them all the way to an impact return on the dollar. Like DFIs, they are targeting a huge range of outcomes in a huge range of places, and nobody knows how much of one adds up to another.

Impact-led DFIs such as CDC have addressed the impact measurement problem in various ways, as befits their mission and size. CDC’s impact framework tries to strike the right balance between rigor, consistency, and flexibility, focusing on the question of what contribution we make to outcomes.

We do not presently put a numerical score on impact, although we are keeping an open mind. The biggest player, the World Bank’s International Finance Corporation, has developed a terrifically comprehensive scoring system, known as AIMM, which goes beyond project-level impacts to rate how investments affect markets and integrates performance monitoring after an investment is made. This and similar systems are decision-making tools intended to raise impact by influencing not only which investments are made, but how they are made.

These scoring systems function as a set of exchange rates that define how much of one thing is judged equivalent to another. Impact scoring produces a number to express how an institution ranks the impact of different investments, but it does not give external investors an answer to the question of how much impact bang they got for their buck, which they can take away and compare to other uses of it.

Some actors on the impact-investing scene are trying for that. The best known is probably Y Analytics, a spinoff from TPG, but there are others. The Global Innovation Fund, for example, puts a dollar value on its expected impact. One of the winners of this year’s Nobel Prize in economics, Michael Kremer, has applied a “social rate of return” methodology to USAID’s development innovation ventures program — he found it generated at least $3 in social benefits for every $1 invested.

These methods are not so far apart from traditional cost-benefit analyses undertaken by governments to evaluate public investments. On my bookshelf, I have a copy of “Project Appraisal and Planning for Developing Countries” by Little and Mirrlees from the 1970s. These are not new ideas.

Putting a dollar value on impact can be a very useful exercise for some investors. The experience of those at Y Analytics has been that the processes of arriving at the number regularly generate useful insights, and they are optimistic that “learning by doing” will cut the time and costs involved. But there are two reasons to question whether impact monetization will be the answer for everyone: It is hard to scale, and it is tricky to convince people to trust your outcomes-to-dollars conversion methodology. Simply put, economists’ attempts to put a price on life, liberty, and happiness are not universally welcomed. The problem with cost-benefit analysis has always been knowing how far to trust it.

Ten years from now, I could be looking back on this article with embarrassment as we all feed information about our investments into an AI system that spits out a dollar measure of impact. But assuming my caution is well-founded, what then? I think we need to start getting investors used to the idea that impact performance is always going to be disaggregated and it is going to be up to them to decide which outcomes they value more than others.

And perhaps we should keep in mind that while we would all love a single yardstick to answer the question of how much impact our money is having, it’s not something we demand from the rest of the development community.