OXFORD, United Kingdom — Education researchers trying to decipher what works when it comes to reforming failing school systems must be careful not to get lost in the “nitty gritty” of impact evaluations and academic jargon, and instead turn their findings into useful lessons for policymakers.
This was the overarching takeaway from the annual conference for RISE — a £35 million ($46.32 million) education research program across six countries — which brought together the world’s leading global education academics for two days of presentations and discussions last week. Sessions covered the latest research into how to support and motivate teachers; financing; and the private sector, as well as the role of testing and assessments.
Closing the conference in Oxford, United Kingdom, RISE’s Research Director Lant Pritchett stressed that evidence from randomized controlled trials looking at isolated or small-scale projects needs to be grounded in political realities and “implementation constraints” if it is to have impact.
“If you want to play the game of making a difference in the world, you’ve got to take into account all the constraints,” he said, explaining that for research programs such as RISE this means that “you’ve got to constantly be thinking about the nitty gritty … [and] the method and how you best establish the evidence, but you want to embed it in a big picture of ‘how can I replicate this, how can I scale this, how is this going to really change behavior, and lead to sustained changes and improvements.’”
The seven-year program was launched in 2015 and is funded largely by the U.K. Department for International Development, with contributions from the Australian government. RISE supports country evaluation teams in India, Ethiopia, Indonesia, Vietnam, Tanzania, and Pakistan to try to understand how to improve education systems.
While primary school enrollment rates have jumped to over 90 percent worldwide in recent years, research suggests that millions of children are nonetheless learning little at school. Pritchett’s 2013 book “The Rebirth of Education,” is credited with helping shift the education debate from schooling to learning, something which was also emphasized by the World Bank’s 2018 World Development Report, “LEARNING to Realize Education’s Promise.”
Pritchett’s message about the importance of linking research to policy was seconded by DFID’s chief economist Rachel Glennerster, who praised RISE evaluators for getting “stuck into the messy details” of issues such as how to train teachers or manage schools, but stressed that it is “really important to come up with quite practical and tangible things we can talk to policymakers about.”
“My plea to you as researchers is to do the messy details but then also bring out the links to ... what that’s telling you about the big systems,” she said.
The agreement between Pritchett and Glennerster surprised some attendees, since the two have previously espoused opposing views about RCTs.
Pritchett has argued that because RCTs only offer evidence about what works in a specific program in a particular context, they are of little use in formulating broader policies. This point was reiterated by researcher Rebecca Thornton, who presented a paper at the conference on the “acute sensitivity of education program effectiveness to input choices and outcome measures, using a randomized evaluation.”
But as former head of the Abdul Latif Jameel Poverty Action Lab, or J-PAL, Glennerster has been a long-time advocate of RCTs, although she recently told Devex they were “just one tool.”
Head of education at the World Bank Jaime Saavedra delivered a similar message during his keynote speech. Drawing on his experience as minister for education in Peru, a position he held until the end of 2016, Saavedra said that impact evaluations, and RCTs in particular, are “useless” to governments unless they also explained the “why” behind their findings.
“It is almost useless that you come at the end of two years [of an RCT] and say it worked or it didn’t work … Not knowing the reason why it failed makes the whole evaluation useless,” he said.
While RISE — which is co-led by development consultancy Oxford Policy Management, think tank Center for Global Development, and the Blavatnik School of Government at the University of Oxford — has been designed to bridge the gap between research and policy, some attendees said the program is still too research heavy, especially focused on measuring test performance in literacy and numeracy, as well as being siloed by country. As a result, it is yet to offer practical guidance to education ministers, some warned.
Peter Hinton, associate fellow of Oxford’s Saïd Business School, also highlighted the limitations of measuring academic attainment through tests, which he said "may not give a true picture of what quality education means from a recipient community's perspective.”
“It could make sense for RISE researchers to ask such questions and listen to what recipients of education feel to be important when assessing education quality,” said Hinton, who advises impact investors on financing low-cost schools in Africa.
Other attendees criticized some of the presentations for being overly technical and called for findings to be put in “plain language.”
To ensure that findings have impact, Devex has learned that the Bill & Melinda Gates Foundation is considering funding RISE to boost its nonresearch work, including writing synthesis papers and policy notes. Gates launched a global education strategy earlier this month, pledging a $68 million budget for the next four years. The foundation already works in education, but has so far largely been focused on the United States.
Girindre Beeharry, Gates’ director of global education, told Devex that while no funding agreement has yet been made, the foundation “would be super sympathetic to the aspiration RISE has to look at the education sector as a systems problem that doesn’t have easy solutions and not reduce the solutions … to a few simple tools you can just scale up [because] education doesn’t work that way.”
“We are quite keen to see how we can partner with a group that has a similar perspective on education,” he added.
Mark Henstridge, RISE’s co-director, said that RISE and Gates have been discussing ways of ensuring the program’s research findings become “more than the sum of the parts,” and were negotiating a “modest but catalytic” grant for RISE’s nonresearch activities. This could be used for synthesizing evidence around emerging themes and supporting teams in lower-income countries. The funding could also be used to carry out “really focused diagnostic analysis of education systems” in countries or states, said Henstridge, who is chief economist at Oxford Policy Management.
“When you’ve built a substantial number of findings … you need to make them make sense to the people who are wrestling with the reform,” he said. Nonresearch work is “not some kind of overhead; it’s actually what you need to do to make the most of the research.”
Update, June 26: This story was amended to clarify the countries in which RISE works, and to correct Rebecca Thornton’s title
About the author
Sophie Edwards
Sophie Edwards is a Devex Contributing Reporter covering global education, water and sanitation, and innovative financing, along with other topics. She has previously worked for NGOs, and the World Bank, and spent a number of years as a journalist for a regional newspaper in the U.K. She has a master's degree from the Institute of Development Studies and a bachelor's from Cambridge University.
