Opinion: The coming evaluation revolution in corporate responsibility

If intentions were enough, the world would be a much better place. Most companies want their corporate responsibility programs to work, but few develop the evidence base to understand whether they actually do.

There's a pervasive belief in the good deed: That well-meaning investment will automatically produce positive results. That’s wrong. It's assumptive, not based on scientific insight, and means we pass up the opportunities to learn how to do better.

Too often, we measure outputs — pills distributed, lavatories constructed — without looking at whether people in the targeted area actually have more access to medicines or safe water as a result. We measure these things because they are easy to count. However, they can also be misleading: Who is taking the pills? Who is using the lavatories? Has better access to care or sanitation actually improved anyone’s health?

We should be measuring the impact on variables that we value as a society: Has a program improved health? Has it created new economic opportunities for women? Has it reduced environmental damage? Often, though, these hard outcomes seem too daunting to capture.

The more frequently companies can link social interventions and business models with their core strategic priorities, the more important it will be they understand the impact of these programs. Such measurements enable us to improve program management and resource allocation. Understanding the performance of an investment is not just good management practice: We need to understand exactly how social impacts and improved business performance fit together. And this will be how we can finally achieve social change at scale.

Measuring outcomes, not just outputs

For the past two years, Boston University researchers have been evaluating the impact of Novartis Access, a program against key chronic conditions, on Kenyan households where a resident has one of four specified noncommunicable diseases.

The Boston University baseline study on Novartis Access was conducted in eight counties in Kenya in 2016, asking patients diagnosed with 1 of the 4 chronic diseases whether they had their prescribed medicines available in their home. They also asked how much of their household health expenditure was spent on these medicines.

Researchers found that, as in several other African countries, awareness about NCDs remains low, with a lack of specialized doctors and medical training, despite a high mortality rate in Kenya due to NCDs.

Only 60 percent of people who had been prescribed a medicine had it in their home. And the amount they spent on treatment was high — some 75 percent of their overall household health spending.

They also found that while more than 50 percent of chronic diseases are diagnosed in the public sector, more than 40 percent of patients actually buy their medicines in the private, for-profit sector.

Let’s take the case of asthma: Half of those studied who knew they had asthma did not have any asthma medication at home. Of those who did have medicine, many had tablets, not the inhalers recommended by international treatment guidelines. Most surprising of all was that among patients who had a salbutamol asthma inhaler in their home, those in the poorest quintile were least likely to have received the medicine free of charge.

For each medicine researchers studied, it was some of the poorest patients who paid systematically more than wealthier patients; the poorest often come out worst even in accessing subsidized treatment.

The researchers returned for a mid-point evaluation in 2017, using a cluster-randomized trial design, and are conducting an end-point evaluation this year. This final evaluation will measure whether Novartis Access has improved the health status of patients with NCDs, not just whether it has improved access to diagnosis and affordable treatment. This detailed impact analysis will allow those running Novartis Access to adjust the program as it is rolled out in further counties.

This is, it seems to us, a sound example of academics and a pharmaceutical company collaborating to test whether a corporate responsibility project is having the impact it sets out to make. We now want to encourage other companies to be more systematic with their monitoring and evaluation as well.

Here are two ways this can be achieved:

1. New relationships between universities and the private sector

If companies provide open-source data on their social innovations to the public, they create a social good. They show other companies where to invest in future and they show society which corporate responsibility initiatives to nurture and encourage. These data must be independent and reliable, and it is here that academic institutions can play a central role.

Universities can be a neutral space to be a trusted anchor, with the legitimacy to act as guardians for such a database of open-source data. It is the role of academics to ensure that access to these data is offered transparently and in a form that can easily be used by other researchers.

There is a huge variance in academic institutions, and it will not necessarily prove a fit for them all. It will be important to think through all the long-term commitments involved for all parties.

2. Frameworks as tools for accountability and learning

Measurement and evaluation frameworks can also be a learning tool. As stakeholders with different priorities come and go, this means the frameworks need to be more agile. Social interventions and inclusive business models often operate in unstable health systems, so the exact conditions are not fully known at the point when an intervention or business model is designed. As a result, they need to be able to respond dynamically to different or changing conditions.

Measurement and evaluation provide the information and insights to enable these programs to adapt and respond in a fast and meaningful manner. But this also requires that the measurement frameworks are open to adaptation and respond to changes with a flexible but still rigorous design.  

When evaluation frameworks are robust, they force stakeholders to articulate and develop a shared language, and also a shared understanding. There needs to be disagreement, even conflict, to come to something that is really meaningful, instead of superficially agreeing on platitudes. This can be a bumpy ride, but a journey worth taking.

Evaluation needs to change

It is time for an evaluation revolution. This requires all of us, in all our different roles, stopping to think through the frameworks we use to monitor and evaluate interventions.

Painful as it is, we need to accept that our intentions are not good enough. That good deeds sometimes have unintended consequences, or even fail to change anything at all. It is better to know and change, rather than hope for the best and push forward blindly.

For more coverage of NCDs, visit the Taking the Pulse series here.