This organization aims to change how impact is measured — and funded

SAN FRANCISCO — ImpactMatters, an organization that helps donors identify high-impact nonprofits, has unveiled a new nonprofit ratings service. The team says its five-star rating system, which it has already used to evaluate 1,078 nonprofits, will help donors understand how to achieve the most impact per dollar at scale by identifying the highest performing nonprofits across a range of sectors. ImpactMatters first launched in 2015 with the aim of encouraging the production and use of what it called “appropriate evidence.” The startup relied on a more involved process using impact audits, or consults, with nonprofits to help them measure their impact and make improvements based on evidence. But the team soon realized that nonprofits did not see the return on investment in meeting its criteria, which included cost-effectiveness, quality of evidence and monitoring systems, and demonstration of learning and iteration. “The reality is that there are few incentives for nonprofits to share accurate impact data about their work.” --— Elijah Goldberg, executive director, ImpactMatters “The reality is that there are few incentives for nonprofits to share accurate impact data about their work,” said Elijah Goldberg, executive director of ImpactMatters. ImpactMatters has changed its approach to rating nonprofits, but its goal remains the same: to create a marketplace for impact in the nonprofit sector. From 1.0 to 2.0 There is an important equation in calculating impact that too many nonprofits miss, said Michael Weinstein, president of ImpactMatters. “Observed success minus counterfactual success equals net impact,” he said. “If you assume zero for counterfactual success, you have unambiguously slapped yourself on the back for something you didn’t accomplish at all.” The downfall of most impact analysis is that it does not come to grips with the fact that not every success is due to the nonprofit under review, Weinstein added. “What separates us from the pack is that we spend our time giving as sophisticated an answer as possible to how much counterfactual success would there have been anyway and calculating difference between observed success and counterfactual success,” he said. When ImpactMatters first launched with ratings of four organizations, it was aiming for both rigor and scale by conducting nonprofit impact audits one by one. Was it really possible to achieve the rigor of GiveWell, which recommends eight top charities that meet its criteria for programmatic effectiveness, and the scale of Charity Navigator, which collects basic financial transparency data for more than 9,000 charitable organizations? Not exactly, the ImpactMatters team discovered. “The issue with 1.0 was a basic chicken-and-egg,” said Dean Karlan, who co-founded ImpactMatters with Goldberg, a student of his at Yale University. “2.0 overcomes that.” Nonprofits did not see how the impact audit would help them raise more money, due in part to the fact that not enough donors were visiting the website. Plus, the team began to see that its model of individual impact audits could scale by hundreds — but never by thousands. The group then tried to develop a series of tools that would make it easier for nonprofits to input their own data, but that also failed to take off. “The way we’ve short-circuited that with the current approach is we’ve gone where the data is,” Goldberg said. The team built on the methods it had developed in its 1.0 version and found ways to apply them at greater scale in the 2.0 version, he said. Its new methodology is based on gathering publicly available data and designing algorithms that estimate the cost-effectiveness of a nonprofit. So far, the organization has evaluated nonprofits focused on poverty, health, education, hunger, clean water, homelessness, veterans, and climate change. Moving forward, it plans to expand the number of causes it covers. Creating a marketplace for impact Dianne Calvi, CEO of Village Enterprise, which supports entrepreneurship and innovation in Africa, went through both the original impact audit approach and the new five-star rating system with ImpactMatters. Calvi said she saw a lot of value in the impact audits and hopes that“ImpactMatters will continue to focus their rankings not only on cost-effectiveness, but also consider the underlying quality of evidence in their ranking of nonprofits.” ImpactMatters is one of a number of charity rating sites that go beyond conventional measures, such as overhead costs, to calculate impact. Ultimately, it shouldn’t matter which organization succeeds in creating a marketplace for impact, as long as that is the ultimate outcome, said Nick Fitz, co-founder and CEO of Momentum, which launched an app last week that connects donors with nonprofits. Momentum is exploring a partnership with ImpactMatters, and both organizations are trying to help donors verify that their dollars are making a difference. “No one organization is going to solve all these problems alone,” said Jacob Harold, executive vice president of Candid, a merger between Foundation Center and GuideStar. “We need — at the very least — alignment, maybe cooperation, maybe data-sharing, maybe all the way to mergers.” The Bill & Melinda Gates Foundation, which supports ImpactMatters, is interested in seeing more collaboration between the growing number of organizations that are rating nonprofits in order to enable not just more giving, but smarter giving. Still, any organization that aims to go beyond collecting nonprofit tax returns and asking for effort on the part of nonprofits will face the challenge of incentives. “Right now, there is no easy evidence to nonprofits that providing good information on impact will help dollars flow their way,” said Victoria Vrana, who sits on the philanthropic partnerships team at the Gates Foundation. “You hear donors talk about it, saying, ‘We care about results’ — but nonprofits can’t really see it.” Chasing metrics Not everyone welcomes the growing emphasis on cost-effectiveness in philanthropy. Megan Tompkins-Stange, an assistant professor of public policy at the University of Michigan’s Ford School, said on Twitter that rating nonprofits in this way can be “destructive.” Metrics to gauge efficiency can divert the attention of nonprofits away from their mission-critical work, she said. “Perhaps most importantly, they become an end in and of themselves, instead of an indicator of impact,” Tompkins-Stange wrote on Twitter. “The core work of nonprofits can change, chasing the metric.” Cost-effectiveness measures are one important factor for donors, but they should not make their decisions based solely on those numbers, Weinstein said. He added that ImpactMatters is interested in developing a methodology to analyze advocacy and research nonprofits, whose impact may take longer to appear and may have many causes. Weinstein said he hopes that ImpactMatters will direct more money to charities with evidence of impact, and that groups scoring below five stars will work to improve. “It’s certainly a hope bordering on expectation,” he said.