How evaluators are using AI to find out what works

With just seven years left to hit the Sustainable Development Goals, the work of monitoring and evaluation — making sure that we know what is and isn’t working — needs all hands on deck. As with other areas of global development, experts in evaluation have been experimenting with ChatGPT since it launched a year ago to see if it can help with their mission. With a growing focus on impact, “we need to start being more flexible and perhaps start using some of the innovative technologies that are out there,” explained Carmen Nonay, a director within the World Bank’s Independent Evaluation Group. She was among a group of experts who took part in a Devex Pro panel discussion on the current opportunities and challenges in the world of monitoring and evaluation. Experiments are still in the early stages but some early lessons about how evaluation teams could use AI-powered tools like ChatGPT are starting to emerge. IEG has tried various uses, said Nonay. Its forays into text classification and image segmentation were unsuccessful. “We were hoping that the technologies would help us to speed [up] the work we’re doing, to enhance the capabilities, to have new insights to improve quality, but we didn’t see that,” she said. With two other simple experiments, however, they saw better results. In the first case, they asked ChatGPT to provide an R code to replicate a multivaried regression analysis for an early-stage evaluation of the World Bank’s economic response to the COVID-19 pandemic. “We provided ChatGPT with very specific instructions and the code it generated actually performed quite well,” she said. “To us it seems that this was a good application of ChatGPT because it reduced the need for rewriting code, or at least gained significant time [savings] on that task.” They also tried using it to summarize text — specifically, a recently published in-depth evaluation. “ChatGPT produced an accurate, well-written, high-level summary of the document … so I think there is quite a bit of potential there for us,” said Nonay, especially given IEG’s need to provide quick evaluation summaries of crisis responses as they unfold. If artificial intelligence tools could handle most of that work, it would mean that valuable staff could be dedicated to other tasks. UNICEF has also been using ChatGPT to comb through long documents, according to director of evaluations Robert McCouch. “We have quite literally thousands of evaluations within UNICEF that have been done over the years,” he said. “Back in the day … we would pump everything through Nvivo and do content analysis that way. Even that is too time consuming now for us.” Instead, they’ve found that using AI tools to quickly comb through multiple evaluations can be effective. For example, when a new executive director joined the organization last year, they were able to run all their evaluations through AI in order to quickly summarize “choke points, recurring findings [and] recommendations that haven’t been acted on.” It has also allowed them to quickly collate insights on what has and hasn't worked when a crisis hits in order to provide early evidence for an emergency response, including in Gaza right now. Meanwhile, Action Against Hunger USA’s evaluation team has mainly used ChatGPT in “ideation processes” for developing theories of change, interventions, and projects, according to Apollo Nkwake, senior technical advisor for monitoring, evaluation and learning. “I think there’s a lot of untapped potential,” he said. Watch the full discussion, and hear the experts discuss other innovations changing the way that evaluations are done, why this kind of work is facing an “existential challenge,” and how to ensure that evaluations are used by leaders to inform their decisions.