Nonprofit evaluator Charity Navigator launched in 2002 by asking the largest charities in the United States for their three most recent Form 990s — an informational tax form required of nonprofit organizations. Only about a third of the charities they asked handed them over.
Fast forward almost 15 years, and most charities willingly publish their 990s — which give the Internal Revenue Service an overview of activities, governance and detailed financial information — on their websites, Sandra Miniutti, Charity Navigator’s vice president of marketing, told Devex.
The 990 requires organizations to outline their previous year’s accomplishments to justify maintaining their tax-exempt status. Charity raters have historically honed in on financial transparency and what percentage of the budget goes to overhead expenses when they evaluate organizations.
Effective altruism: Are you doing good, better?
Where can a small set of individuals make the biggest difference? This is the question aspiring effective altruists are asking themselves. Devex took a look at the latest in the evidence- and reason-driven movement.
But as the tide increasingly shifts away from tracking outputs and activities, and toward “results-based development,” the information these forms contain is often inadequate to evaluate an organization’s broader impact. Charity evaluators too have started to rethink their rankings, seeking ways to rate organizations based on their results — like fewer deaths from malaria instead of the number of bed nets distributed, or how well children are learning instead of the number of schools built or teachers trained.
Several nonprofit information providers like Charity Navigator and GuideStar have tried to break away from over reliance on the 990 and its limited financial picture and broaden their perspectives. The two organizations partnered with the BBB Wise Giving Alliance to launch the Overhead Myth Campaign in 2013, an initiative to educate donors about the danger of relying solely on the comparison of operating expenses and total income — known as the overhead ratio — to evaluate nonprofits.
Both groups have also been working toward defining appropriate metrics for a new kind of rating — one that captures impact.
Others, like GiveWell, founded in 2007, have operated with a different strategy from the start by thoroughly investigating a much smaller number of charities and making a few choice recommendations each year based on deep research into program areas.
“It used to be ‘hey, should charities be rated? That’s not right.’ But today the conversation is more like ‘donors are demanding this information, what’s the best way to share it?’” Miniutti told Devex.
There’s been a lot of growth, but “measuring on impact is something the sector is maturing into,” she added. Devex took a look at three organizations currently engaged in vetting charities to find out what the current process looks like and what the future holds for results-based evaluations.
Charity Navigator: A stab at results reporting
Quick tip:
Interested in upping your rating? Many accountability and transparency metrics can be adjusted by double checking for policies you may have forgotten to include in the 990. If a charity with a conflict of interest policy failed to account for that in its paperwork, for example, just “show it to us and we can give you credit for it,” Miniutti said. Another thing charities may not realize is that they can claim their profile on the site and send any additional information to Charity Navigator analysts, she added.
Charity Navigator has continuously refined its rating system since its inception in 2002, but it currently rates charities by evaluating two broad areas of performance: financial health and accountability and transparency. Its zero to four star rating system as it exists today embodies those two dimensions, though the evaluator is working on a multiphase, multiyear project to add results reporting metrics to their star system.
When they began, they sought to assess how well charities were doing in informing donors about their own performance, but Charity Navigator was not rating that performance independently, Miniutti explained.
In 2013, the evaluator launched the first phase of its results reporting effort, piloting metrics by collected from a survey of charities with a variety of missions. This effort wrapped up in July 2015 after Charity Rater collected data on 3,000 organizations. Now, they’ve “put a pause” on additional data collection, Miniutti said, in order to analyze what they have collected and review feedback from various stakeholders — but also because the response rate for the current metrics wasn’t stellar.
The survey asked questions like: Does the organization have a logic model? Do the things they are measuring and reporting align with their mission? Are they doing periodic evaluation reports with an outside firm?
Maybe those weren’t the right questions, or maybe they were too advanced, Minutti said, adding that perhaps that information is “still in someone’s head and not on paper yet.”
In the meantime, Charity Navigator is looking to change its financial metrics — which help donors see how efficiently and responsibly a charity functions day to day and how well it is positioned to sustain its programs over time — and is planning to make several tweaks to the rating system in June or July. One of the metrics on the chopping block is revenue growth, which “too often sees big swings,” Miniutti said. Instead, with years of data now available to Charity Rater that was not available in 2002, the group may go to averaging.
Some of the weighting of the financial metrics may also change, and charities and donors can expect to see a new metric to replace revenue growth, though it is still being finalized.
GuideStar: A redesigned profile with a qualitative emphasis
Quick tip:
The most common question GuideStar receives from charities is “How much does it cost to update my profile?” The answer? It’s free.
GuideStar collects, organizes and presents charity information for 2.4 million U.S-based tax-exempt organizations.
Finances are only one piece of a much larger picture when it comes to nonprofit impact, Gabe Cohen, GuideStar’s director of communications, told Devex. Profiles are populated with Forms 990 — information that is “meaningful, but not intended to prove the effectiveness of an organization,” he said.
GuideStar encourages nonprofits to update their own profiles, and operates under three seals of transparency that are awarded depending on how robust a group’s profile is. Bronze means a nonprofit has supplied basic information like the name of the organization leader and geographic areas served. An audited financial statement will allow a jump to silver, while qualitative results information around goals, strategies, capabilities, indicators and progress will earn an organization the gold seal.
The information provider redesigned the site’s profiles in January 2016 for the first time since 2009 to better emphasize answers to key questions about programs and performance. Qualitative results information is “literally at the top of the profile now,” Gabe told Devex. “You won’t find overhead ratios on our site. That’s not something we’re providing our consumers anymore.”
In the next two months, GuideStar will launch a platinum level, which organizations can earn by increasing reporting of their quantitative metrics — how many more people graduated from high school as a result of an education program, for example.
“In the past, there’s been a lack of tangible information that can be provided in the absence of overhead ratios,” he said. “But now when organizations are asked, they can say with confidence, ‘That’s not what you should be looking at. Judge us by our progress. Look at our results, and potentially compare that to peer organizations.’”
After hundreds of interviews with experts in various sectors, the charity information collector assembled a list of about 600 common metrics and is asking organizations to use them first in an attempt to create common vernacular, though they expect changes as they learn more.
Unlike the for profit industry, there’s no one metric that unites the nonprofit space, and it’s hard to relate an advocacy group to a homeless shelter in terms of impact, for example. But that’s what GuideStar wants to create — “a tool that donors and funders can use to compare apples and oranges,” Cohen said.
GiveWell: An eye on third-party monitoring
Fast fact:
Givewell staff identify anywhere from five to 10 charities per year that they’d like to research, though the actual number is smaller after initial conversations, Hollander said.
It takes hundreds of person-hours to understand even a single charity in-depth. GiveWell dedicates those hours willingly. The effective altruism-focused organization identifies top charities by assessing them along four criteria: effectiveness, cost-effectiveness, transparency and room for more funding.
The team is upfront that the process is a lengthy one for the charity being vetted, Catherine Hollander, GiveWell outreach associate, told Devex. A charity representative can expect to spend plenty of time on the phone going over data on monitoring and evaluation and budgetary information. They can also expect a site visit from GiveWell.
But in order to be eligible for "top charity" status, a nonprofit must be explicitly focused on one or more of GiveWell’s identified priority programs, a list of more than 20 program areas the team identifies by looking at independent academic evidence around a given intervention.
“We want to be doing what we can now to create programs and charities that will be top charities in the future,” Hollander told Devex.
A big focus for GiveWell now is expanding that top charities list — currently a list of four — primarily by investigating charities that already exist, but also by supporting activities that could eventually result in a larger set of evidence-backed programs. They would ultimately like to see more top-rated charities that implement GiveWell’s priority programs, like vitamin A supplementation, immunizations, conditional cash transfers and micronutrient fortification.
To get there, GiveWell is funding additional experimental field research about programs that have some evidence of being cost effective, but require more. The group is also providing seed funding for small charities doing promising program work that GiveWell hopes to recommend in the future, and investing in the idea of funding third-party monitoring for charities that lack the capacity themselves, Hollander added. The nonprofit is in talks with several groups about whether and how to set up an initial test version for charities they’ve already identified.
Read more international development news online, and subscribe to The Development Newswire to receive the latest from the world’s leading donors and decision-makers — emailed to you FREE every business day.
