World Bank data — are we using it?

In the early 90s, Bill Gates and his wife Melinda, who had just started to get engaged in philanthropy, were looking to channel their charity activities toward health issues.

Gates approached William Foege, an epidemiologist credited for helping eradicate smallpox in the 70s, who then gave him a reading list that included the World Bank’s 1993 World Development Report. The report, a controversial but widely cited document that discussed the health challenges in low- and middle-income countries, celebrated its 20th anniversary last year — and led Gates to donate $1 billion for vaccinations in the developing world when he finished reading it.

If one report could have that kind of impact, then it’s hard not to imagine the influence that subsequent reports could have. But many policy reports, the bank discovered, are never read.

A new World Bank policy research working paper published this month reveals that more than 31 percent of these documents are never downloaded, while almost 87 percent are never even cited. These are unsatisfactory figures, especially since the Washington, D.C.-based institution spent around a quarter of its country services budget on its core knowledge products — including economic and sector work, technical assistance, the annual World Development Report and other internal reports, among others — in 2012 alone.

In the study, the researchers defined “policy reports” to include ESW reports, which offer original research and analysis and represent the World Bank’s official view, and TA reports, which complement ESW reports and are more advisory in nature.

Through an assessment of downloads and Google Scholar citations, the paper also found that more expensive policy reports on populous middle-income countries and more complex policy reports, such as those that focus on several sectors, are more likely to be downloaded and cited.

Downloads are a convenient way to measure impact because, as a 2012 World Bank blog post points out, they are “an indicator of an intent to use.” The downside of this method, however, is that reports do not necessarily have to be downloaded to be considered read. Some reports, for instance, could sometimes be presented in a different format, such as a blog post that summarizes the findings and policy recommendations — but page views of these blog posts are not part of the measurement of the reach of these reports. Meanwhile, citation counts are a reliable indicator of influence in academia, but they are understandably less commonly used to assess the demand for policy reports, which are meant for policy makers, not researchers.

To properly determine the impact of World Bank policy reports, the researchers turned to Google Scholar, which — unlike other citation databases — includes non-scholarly citations of articles, making it a valuable tool for assessing the reach of World Bank reports, which are not meant for academic journals.

Google Scholar, nevertheless is not a perfect measure of impact — a limitation that the recent study itself acknowledges. Its automated software, for instance, does not detect double-counting, and a report with different versions uploaded to several sites could have separate citations within each paper, and thus inflate the citation count for these articles.

This is not the first time that the World Bank has assessed the impact of its publishing record. In an earlier paper, bank researchers assessed the scholarly influence — as opposed to mere visibility, which would involve an analysis of mentions in news articles — of its reports. They found that the annual World Development Report, the same title that spurred Bill Gates to eventually start the Bill and Melinda Gates Foundation, was the fifth most cited series among books published by the World Bank.

That the World Bank continues to make this analysis public instead of internal is a good sign — and a move that other development institutions could perhaps consider. It also reflects the rhetoric behind World Bank president Jim Yong Kim’s push for the institution to go beyond being a “knowledge bank” to a “solutions bank.”

At the very least, perhaps this recently released report could be one of the most downloaded documents in the next assessment.

Read more development aid news online, and subscribe to The Development Newswire to receive top international development headlines from the world’s leading donors, news sources and opinion leaders — emailed to you FREE every business day.