Opinion: Measuring what matters: Supporting systems level change

When a solution designed to improve lives scales to reach one million people, that is a milestone to be celebrated. Tackling global challenges such as poverty, access to quality education, and equitable health systems will require ambitious reach at scale.

That is why we are excited to highlight social innovators who have reached — or are poised to reach — that threshold, for instance through the Million Lives Club (MLC), recently announced at the Social Capital Markets conference. With the Skoll Foundation’s focus on supporting systemic change, the decision to participate in the MLC launch, with its core focus on lives reached, was not automatic. But when we looked at our portfolio of social entrepreneurs, we saw that more than half of them qualify for the MLC today — transformative systems-change tends to affect huge numbers of people.

Yet we know that counting people reached is, at best, a convenient proxy signal that you may be on your way to achieving your target impact, and as funders, our behavior and incentives can work to boost or impede that impact. What we ask for, what we reward, what we penalize, and what we count has a direct influence over how well systems can be transformed.

At Skoll, we look for organizations driving solutions that transform unjust, unsustainable systems holding back progress on social and environmental problems. In tracking the impact of our portfolio over time, we often see growing reach — either through direct scaling or through partnerships — as one indicator of success. Equally important is evidence that their solutions achieve meaningful social and environmental outcomes, and drive durable change for the better.

Systems change takes many years, but we look for shorter-term signals to spot it. For example, successful replication of a solution by government, the private sector, or civil society; leveraging a network of actors to achieve common impact goals; or contributing to changes in policies governing an ecosystem. Each of these expand impact beyond an organization’s direct reach.

Though systems change has been the North Star at Skoll for some time, we are still on a journey to understanding how to best measure it. Recognizing that others may face similar challenges, we’d like to share some ongoing questions we’re grappling with in measuring what matters.

Not everything that counts can be counted, and not everything that can be counted counts.

When looking across our portfolio, it can be tempting to anchor on countable impact metrics tied to direct service programming, e.g. the number of students reached, or patients treated. But even for organizations with highly countable interventions, the collaboration and influence that resist quantification matter too.

For example, the direct reach of Pratham's interventions in the Indian education system is impressive, but the impact of its partnership with state governments across India to implement its Teaching at the Right Level methodology is far greater.

Pratham’s commitment to measuring meaningful outcomes is one driver of its success in attracting government partners in India and select countries in sub-Saharan Africa. Its systems-level work has influenced a shift in the Indian government’s approach to education policy: from a focus on access to education to a focus on improving learning outcomes, for which the ripple effects are immeasurable.

In other cases, with solutions that focus squarely on a fundamental normative shift towards justice and equity, a focus on quantitative metrics could obscure the bigger picture of social change. Equal Justice Initiative takes a comprehensive approach to ending mass incarceration, excessive punishment, and racial inequality. It represents the wrongfully convicted and unfairly sentenced, trains others to carry out similar work, and uses strategic litigation to challenge the death penalty and expose injustices in the American legal system. EJI also launched an ambitious national effort to create public markers and memorials that address the legacy of slavery, lynching, and racial segregation. Its fundamental aim is no less than to change the narrative about race in America.

Just looking at these two examples shows the difficulty in settling on a common measurement approach that would do justice to capturing their true impact. In our aspiration to act as a trust-based funder with a focus on learning and adaptation rather than accountability, we let our program partners define their own metrics of success and don’t require reporting on precisely what our dollars fund compared with those given by others. Still, we acknowledge that we have more work to do to fully live these trust-based principles.

Transformative change is not linear

Developing a theory of change can be a critical tool for guiding strategic thinking and capturing impact goals and assumptions, but linear logic models neatly linking program activities, outputs, and outcomes can oversimplify the complexity of systems change efforts. There’s increasing momentum in the funder and evaluation community around a variety of systemic change evaluation approaches and methodologies — a promising indicator of increased comfort with ambiguity in the process of assessing impact. No single recipe or gold-standard methodology, like the randomized controlled trial, can fully capture impact when working in complex, adaptive systems.

When thinking about our portfolio’s impacts, we try to open the aperture and look for the signals that the world is moving towards peace, prosperity, and sustainability. To see that a third-grader has started reading at a third-grade level, or that even one million of them have reached that level, is a valuable indicator of change.

Still, the space between reading at a grade-appropriate level and economic transformation, personal empowerment, and dignity is packed with many factors that aren’t directly attributable to the program that produced it. We can’t truly understand this unless we continue to track progress and downstream ripples long after our funding term has ended, which we as a funding community don’t often provide the resources to do.

Transformative change is often not attributable

For some organizations in our portfolio, a very intentional avoidance of public attribution is core to their effective operation. Crisis Action acts as a behind-the-scenes catalyst and coordinator for organizations working together in opt-in coalitions to protect civilians from armed conflict. Its approach with coalition partners contributed to major systems-level influence, such as lifting a blockade on the port of Hodeidah in Yemen, enabling the delivery of almost half a million tons of desperately needed aid for 22 million people.

Its Handbook for Change — a tool for unlocking effective collective action — is now being deployed by a new organization, Digital Action, to address digital threats to democracy. What portion of the “credit” would we assign to Crisis Action if Digital Action is amazingly successful, or if it fails?

Can we shift away from a system that rewards assigning and taking credit, and towards one that rewards contribution and collaboration? Can we anticipate that roadblocks will be part of the journey to systems change and continue to incentivize learning from failures, as much as successes?

Power and privilege are a reality in the work we do

As we think through the next phase of evaluation practice at Skoll, we’re exploring equitable evaluation principles that inform how measuring and evaluating impact can advance progress towards equity; answer critical questions about how historical and structural decisions have contributed to the social change being addressed; and foster community participation in defining success and shaping how evaluation happens.

When counting the number of people reached, the agency and dignity of those people can be lost. How are they engaged with the solution? How much do they value the solution? Do they have suggestions for improvement? Did anyone ask?

Several organizations in our portfolio — Crisis Text Line among them — use net promoter scores as a key metric for success by asking how likely their users are to recommend their solution to a friend or family member. This metric is commonly used in the business world and finds a good home in the social sector when we position people served as customers rather than passive beneficiaries. What more can we do as funders to incentivize and value the customer voice?

Our learning journey on how the Skoll Foundation can measure what matters, in a way that accounts for the complexities of systems change, is far from complete. We hope that recognizing these challenges and continuing to inspect our own processes and principles will improve our efforts to support those creating the most transformative change.