With increasing attention to effective altruism, a movement to ensure philanthropic donations achieve as much impact as possible, there is also growing demand for information to guide those donations. That is where charity raters come in.
But these tend to fall into two groups: large charity raters such as Charity Navigator survey thousands of organizations, collecting basic financial transparency data, such as the percentage of an NGOs budget that goes to overhead expenses. Smaller charity raters such as GiveWell do deep dives into questions of programmatic effectiveness, then recommend a small group of the charities as worthy of funding. For GiveWell, only four make the mark of top charities.
These organizations typically face a trade-off between the investment of time and money per assessment and the number of organizations they evaluate. As a number of them rethink their ratings to account for a shift toward “results based development,” some of the sector’s brightest minds are asking: Might there be a middle ground?
In 2002, Dean Karlan launched Innovations for Poverty Action, an organization that has convened 400 academics to conduct 500 evaluations in 51 countries and counting as part of its mission to discover and promote the most effective solutions to global poverty. But while Karlan remains a vocal advocate for randomized control trials as the gold standard of impact evaluation, he acknowledged they are not always affordable nor essential, and proposed a new set of metrics to encourage the production and use of what he calls “appropriate evidence.”
“What should donors demand and what should organizations collect?” Karlan asked the audience at the Frontiers of Social Innovation conference in Palo Alto, California, last week.
Over the past few years, nonprofit information providers — or charity raters — have made an effort to break away from the reliance on overhead ratios. Devex took a look at three organizations currently engaged in vetting charities to find out what the future holds for results-based evaluations.
Karlan is the force behind a new organization called ImpactMatters, which aims for the rigor of GiveWell but the scale of Charity Navigator. ImpactMatters conducts impact audits, which are modeled after financial audits, to enable nonprofits and donors alike to make smarter decisions about how to spend their dollars, he said.
“Charity Navigator is by far the most popular rater and they don’t rate by impact at all. They rate solely on administrative costs and that’s just not impact,” Paul Brest, the former president of the William and Flora Hewlett Foundation and a board member at ImpactMatters, told Devex. “If an organization thinks it has impact and wants to be rated based on that, and more importantly if its funders want its grantees to be rated on that, you’re not going to get that through Charity Navigator.”
Brest said part of what drew him to the opportunity to advise ImpactMatters was the way its unique rating scheme, built on short-term engagements that assess groups across four dimensions, provides more flexibility than randomized control trials. ImpactMatters currently evaluates organizations on cost effectiveness, comparing the average impact and average cost per beneficiary. The organization also rates organizations from zero to 3 stars for “quality of impact evidence,” “quality of monitoring systems” and “learning and iteration.”
Elijah Goldberg, operations director at ImpactMatters who also leads an organization working to provide quality care for severely ill patients in Ugandan hospitals, told Devex these standards will be evolving as the team solicits input from the field.
Because much of this data is self reported, in the form of interviews and internal documents, some might question its quality and validity. Jacob Harold, the CEO of the nonprofit database GuideStar, said the key is to start by gathering whatever data is available and then determining the accuracy of that evidence. “Many organizations are lying to themselves, and that’s where having an external third party such as ImpactMatters to push you on your assumptions is really valuable,” he said.
ImpactMatters is not the first attempt to engage a bigger and broader audience than the small though influential audience of effective altruists. At the Hewlett Foundation, Brest and Harold worked together on the Nonprofit Marketplace Initiative, which was phased out because of what Harold called the lack of “a decent information system in philanthropy.” Now, Harold is working with Philanthropedia, which has its own unique approach, basing their charity ratings on opinions from expert surveys. And yesterday, GuideStar launched a new tool called Platinum, which allows nonprofits to share qualitative data or, as Harold puts it, more meaningful metrics.
GiveWell has acknowledged there are limits to the strict criteria that lead to the same results year after year and the team is making changes as a result. But part of why the same organizations rise to the top is that not all nonprofits and donors are convinced that gathering data, let alone learning from and acting on it, will garner more funding.
As ImpactMatters, which was incubated at Innovations for Poverty Action, moves from its public beta release of four organizations in December 2015 to the next iteration, its success will depend on demand for impact audits. The business model is to have foundations commission these impact audits so they can identify nonprofits they might fund and provide more funding back to the nonprofits they already support.
Kevin Starr, managing director of the Mulago Foundation and a board member at ImpactMatters, said that will only happen once more funders reward results based development by funding for impact or paying for success.
“What we look for really is a culture of measurement,” he said of the approach he takes as part of a group of funders called Big Bang Philanthropy. “Not are they doing a good job measuring for donors, but can they do a good job measuring for themselves, so they can get better?”
Read more international development news online, and subscribe to The Development Newswire to receive the latest from the world’s leading donors and decision-makers — emailed to you FREE every business day.
About the author
Catherine Cheneycatherinecheney
Catherine Cheney is a Senior Reporter for Devex. She covers the West Coast of the U.S., focusing on the role of technology and innovation in achieving the Sustainable Development Goals. And she frequently represents Devex as a speaker and moderator. Prior to joining Devex, Catherine earned her bachelor’s and master’s degrees from Yale University, worked as a web producer for POLITICO and reporter for World Politics Review, and helped to launch NationSwell. Catherine has reported from all over the world, and freelanced for outlets including the Atlantic and the Washington Post. She is also the West Coast ambassador for the Solutions Journalism Network, a nonprofit that trains and connects journalists to cover responses to problems.
