LONDON — Rachel Glennerster, the new head economist at the United Kingdom Department for International Development, believes that the boisterous debate around the use of randomized control trials is finally, thankfully, beginning to sober up.
“I would say that among most economists, nobody’s saying [RCTs] are the only thing, and nobody’s saying you shouldn’t use them,” she told Devex.
The soon-to-be former executive director of the Abdul Latif Jameel Poverty Action Lab, Glennerster was among the early evangelists for randomizing evaluation of poverty reduction impact. Under her management, J-PAL has spent the last 14 years spreading the gospel of randomization far and wide, with a special focus on replicating RCTs’ underpinning drive for evidence across governments, institutions, and policymakers, especially in the “global south.”
In other words, a signal from Glennerster that RCTs may have finally reached peak hype, or the “peak of inflated expectations” as it is known in the tech sector’s hype cycle, will come as a huge relief to well-intentioned but stalwart randomization skeptics such as Angus Deaton and Lant Pritchett.
Too often, randomized control trials fail to produce data that can be used to create more effective policy across contexts. Devex looks at the initiatives working to change that.
“Economists have now settled down into RCTs as just one tool,” Glennerster told Devex. Among academics, the kind J-PAL works to connect with the world’s policymakers, she said, “the trend toward using RCTs is simply part of this bigger movement in economics to care more about where we can really pin down what is causing what we see.”
Still, Pritchett points out, it’s two more stages of grief to go — the “trough of disillusionment” and the “slope of enlightenment,” if you want to know — before RCTs reach the “plateau of productivity.” At the same time, Pritchett and others acknowledge the tech analogy has its limitations: Randomization has and is doing a lot of good in the sector, and has already changed the way development practitioners “ask questions about what we see,” Glennerster said.
“Where can we be clear that something is happening because of this intervention or policy or condition, and where are we just pointing out that these things move together?” she told Devex.
As head of J-PAL for 13 of the organization’s 14-year life, Glennerster has spent her tenure building up and building out its original function as an academic network for professors using randomization to evaluate impact toward poverty reduction. Now, J-PAL is a platform to help public and private entities and experts engage with the deep toolbox of methods for evaluating impact. The golden thread of those tools remains randomization, but Glennerster said the development community and the mainstream media covering aid still has a long way to go toward understanding randomization’s role in the wider spectrum of evidence.
“The thing people get a bit confused about, is they think about it simply as program evaluation, or testing a particular program, you learn about that program and you scale up that program,” she said. In fact, program evaluation is “only partly what we’re doing” at J-PAL, she said. “That’s not mainly what we think [randomized evaluations are] about — we think it’s mainly about understanding how people work, how systems work, and just being really careful about identifying causal pathway.”
J-PAL and Glennerster are also very focused now on making the tools for gathering and evaluating evidence more widely available to governments in the “global south.”
This is one of the things she’s proudest to leave behind at J-PAL, she said, specifically “the relationships that have been built with developing country governments to help them think about evidence, which actually goes much beyond RCTs.” By design, most of J-PAL’s 350-person staff live and work in developing countries, and Glennerster prides herself on growing that outward-facing capacity.
“They are building up those teams ... to support the governments in those regions to think about evidence, to help them even just with the data that’s coming in on a daily basis, to understand what the challenges are and help them think about how to incorporate evidence,” she said.
Glennerster was herself, until recently, plotting a way to make her own mark within J-PAL’s global footprint.
Only a few months ago, she was all but packed and ready to move to Kenya — to work more closely with the government there to incorporate evidence in policy making — when she got an “interesting” phone call from Nick Dyer, the interim permanent secretary at DFID.
Change of plans
“I had wanted a bit of a change so I organized this sabbatical to spend time on the African continent, working with policymakers there to help them incorporate the evidence,” she told Devex.
“Having organized myself to leave, I then got completely out of the blue a phone call from Nick Dyer asking, would I consider this job?”
See more Devex stories on the future of DFID:
Glennerster said it was not the first of such phone calls she’d received during her time at J-PAL, which works closely with DFID, as well as the United States Agency for International Development, the Swedish International Development Agency, the French Ministry for International Development, and others. But in this instance, both personally and professionally, the timing made sense.
“The more I thought about it, the more excited I got about the opportunity,” she said.
A London native, Glennerster previously worked as a senior economist at the International Monetary Fund, and before that, in the British Economic Service at Her Majesty’s Treasury, after receiving her doctorate in Economics from Birkbeck College, University of London. She said the prospect of coming home from the U.S., and returning to the British civil service, gave Dyer’s offer a lot of weight.
“I’ve spent the last 13 years pushing the evidence agenda but also the last five years thinking about this question of, ‘how do we help governments in developing countries absorb and utilize this evidence?’”
“Given that DFID has this expertise and a lot of really good economists, I think they could be an important part of helping governments use this evidence more effectively. You really need to be on the ground working with governments regularly, and with the network of economists and others across the world that DFID has they’re already starting to do that, but I really think I would like to help them do it more effectively.”
But before doing much of anything, Glennerster said she reached out to the other directors at J-PAL — which is based at the Massachusetts Institute of Technology in Boston — to get their thoughts on the change of plans. Typically, DFID chief economists take a temporary leave or sabbatical from their current positions — often as academics or professors — to take up the three-year post at DFID. With all the wheels already in motion, Glennerster’s team agreed it was a great idea.
Her deputy, Iqbal Dhaliwal, has taken the helm since Glennerster left at the end of 2017, and she said “indeed he will continue, after I return.”
At J-PAL, embedded at MIT — and also married to a professor in the field — Glennerster noted that she is “surrounded by academics all the time,” and is ready for a change of pace. Mainly though, she wants to use her experience to influence the policy-making apparatus.
“The exciting thing is, I started life as a policymaker, I went into academics, I’ve always been at the border of policy and research [but] I’ve been on the research side of things … and getting back into [the policy side] day to day was very exciting,” she said.
Glennerster said she “loved” working in the government economic service. Advising the government about “what the research means for what we should practically do, is exactly the kind of thing that excites me and what I’ve spent my career doing.”
The more she thought about it, “the more I thought, everything I’ve done up to now is going to be useful for this job,” she said.
“Not to mention I’m looking forward to living in London again,” she laughed.
The shift represents a significant change for Glennerster, leaving the U.S. and a 350-person staff to manage a “very small staff with not a large budget.”
“It’s really a position to provide advice, this is what’s been explained to me,” she said.
The current chief economist, Stefan Dercon, who is also professor of economic policy at Oxford University, offered Glennerster advice. “He said, ‘You really have to make yourself useful, you have to provide useful advice, and you will have influence if you have something important to say on these issues, so you can’t force people to do things.’”
“But on the other hand,” Glennerster said, “he said that’s so liberating, you’re not worrying too much about staffing and budget, you’re free to come in where you think economics has something to say.”
It will take time to adjust, she explained, from setting the organizational agenda to simply — or not so simply — “bringing economics to the questions of the day and explaining why you think that you have something to say, why policy should change in a certain way given what we know about economics.”
While the chief economist role exists in most departments across Whitehall, Glennerster said at DFID it differs significantly, because it “takes economics very seriously, both if you compare it within Whitehall and to other agencies, it thinks much more like an economist, and many of the senior staff have indeed been trained as economists, or those who haven’t by now know a lot of economics, so that makes it quite an influential advisory position.”
She wants to spend the first two months getting to grips with DFID’s policies and current operations, she said, “and asking around, I want to see what DFID is supporting, see if there are challenges that people are facing about how to think about it, to see how I can play a useful role.”
Some of the biggest policy shifts at DFID — namely its increased investment in CDC, the department’s development finance corporation, and an ostensibly greater focus on fragile states — are key areas Glennerster hopes to probe. On the CDC, she said the question of how aid agencies support growth in the private sector is “hugely important,” but that “even with what we know I’m sure we can do better, because that’s pretty much everywhere we can do better if we work hard and think about it carefully.”
“The whole point of working with the private sector is the efficiencies that it has are outside of government, so you have to think quite carefully about how you come in ... and make sure that what you’re doing is very focused on where the market failures are,” she said.
How much of that thinking is already happening at DFID? “I’m sure that’s something they’re already thinking a lot about,” she said, “so anything I can say about that will have to wait until I understand a lot more about what they’re currently doing.”
Finally, she said, she is excited to continue making the case for evidence to partner countries and governments. Now operating from DFID’s point of view, and with DFID’s own enthusiasm for evidence-building steam behind her, she hopes to manifest precisely the kind of connection between policy making and academia that J-PAL and DFID have pursued independently for so long.