Liberia forges ahead with education experiment despite lukewarm evaluation

LONDON — A highly anticipated evaluation into a polarizing Liberian education program has shown modest overall learning gains but raised questions about trade-offs. At the same time, the Liberian government has hailed the program a success and announced plans to scale up the model.

On Monday, evaluators published the final results of a three-year randomized control trial to assess whether children who were part of the Liberian Education Advancement Partnership, or LEAP, learned more than those in regular schools.

The program was launched in 2016 as a public-private partnership, which saw the government hand over management of 93 public primary schools to eight private providers. These included a mix of charities and for-profit companies, two of which were Liberian and the rest international.

Billed as a bold education experiment to harness the efficiencies and additional finances of the private sector, LEAP’s aim was to improve Liberia’s ailing and underfunded education system.

But the program has been dogged by challenges, including an election that brought in a new government, and fierce opposition from teachers’ unions, civil society groups, and development professionals over what they saw as the privatization of education.

Then last year, an explosive ProPublica investigation revealed that girls at More Than Me Academies, one of LEAP’s operators, had been raped by the co-founder. Another operator — the Liberian Youth Network — was also implicated in a rape scandal. While both cases occurred before the operators were tied to the LEAP program, they raised questions about safety and accountability.

Three years on, evaluators from the Center for Global Development and research NGO Innovations for Poverty Action delivered a lukewarm verdict on the project, which they said produced “modest” gains that depended on the provider. “Some produced uniformly positive results, while others present stark trade-offs between learning gains and other outcomes,” according to the evaluation. The evaluation also found that despite the additional resources, the program “failed to reduce sexual abuse” in its schools.

This follows an earlier evaluation after year one by the same researchers, which reported modest learning gains, high costs, and some negative side effects including kids being expelled from some schools to reduce class sizes.

While for some, the results confirm long-held concerns about the program, Liberia’s Ministry of Education has taken a different view and is forging ahead with plans to further scale up the program — which doubled in size after year one — with the best performing operators, drawing on funding from the UBS Optimus Foundation and others.

The government has also been advising Sierra Leone about potentially replicating the model, Gbovadeh Gbilia, Liberia’s head of the education delivery unit, told Devex on the sidelines of a launch event for the report held at CGD in London on Tuesday.

Longer term, successful LEAP operators could be funded by the Education Outcomes Fund — a planned, results-based education fund for Africa and the Middle East — a source close to the matter told Devex.

Mixed results

The evaluation assesses schools in the program against four measures — learning in English and math, access, financial sustainability, and safety — the last of which was added in year three in response to the More Than Me sexual abuse revelations.

The overall results showed LEAP kids who started in first grade gained four words per minute additional reading fluency, increasing to 15 words per minute,compared to kids in government schools, but that the results plateaued after year one. These gains are “frankly, not huge,” according to Justin Sandefur, senior fellow at CGD, who led the evaluation. Experts estimate that children need to read approximately 90 words per minute in order to fully understand the text.

LEAP schools did show positive outcomes. Teachers in LEAP schools were more likely to be in school and teaching. LEAP students also spent longer in the classroom, which inspired the government of Liberia to extend the school day to last from 8 a.m. to 3 p.m. for all schools in 2018.

However, there were also negative impacts, including higher drop-out rates for LEAP children, mainly linked to Bridge schools, according to the report.

Gbilia said the modest learning gains were significant because Liberia is an “extremely tough” place to operate, made more difficult by the 2017 election and economic pressures.

Operators and the government accused the evaluators of “diluting” the results by presenting learning data based on tests from all children who were meant to be in LEAP schools, known as intention to treat, or ITT, and not just those who actually attended — two-thirds dropped out or were excluded before the program started. While the evaluation presents learning data for both groups, critics say that the evaluation should have emphasized learning gains for the kids who stayed in school.

The RCT used “ITT estimates to dilute LEAP’s treatment effects,” as part of a bid to “divert attention away from the program’s significant learning gains,” Steve Cantrell, Bridge’s head of measurement and evaluation, wrote in an article.

But others defended the RCT and evaluation experts maintain that measuring ITT is standard practice, and it highlights the program’s impacts on the wider education system. Measuring only the treated children could incentivize providers to kick out kids furthest behind and only teach those likely to get better test results in order to improve their results, another expert commented.

Still, Bridge is not alone in criticizing the RCT. Alain Guy Tanefo, head of Omega Schools — which ran 19 schools in year one — said the study was biased. Omega’s schools showed slight negative results, and Tanefo said the operator will probably pull out of the program.

“LEAP was a fantastic opportunity … I can certainly see that the students and teachers who participated really benefited … but the results are more positive than what is reflected in the report … but that could be to overshadow the shortcomings in the research,” he told Devex.

Beyond learning

On child safety — specifically measuring self-reported instances of corporal punishment and sexual abuse — the report finds that while there were fewer reports of beatings by LEAP teachers compared to government schools, children were as likely to report being sexually abused. Nearly 4% of students surveyed in 2019 reported sexual intercourse with a teacher, and 7.5% said they had some form of sexual contact with a teacher.

Some operators complained that measuring self-reported safety was not in the original RCT design, but Sandefur considers these findings to be significant, and believes the problem of sexual abuse in schools deserves more focus.

“These schools were spending between two and 10 times as much money with outside management, and didn’t manage to solve it,” he said during the CGD London event.

The More than Me sexual abuse scandal “cast a dark shadow over the whole program, raising questions about ... why this organization was given control of public schools,” Sandefur also told Devex.

The report also shows mixed results on financial sustainability. The original scheme was partially funded by the government — which supplied the teachers — and a pool of international philanthropists topped it up with an additional $50 stipend per child. Operators were free to also fundraise for the program.

LEAP was set up with the expectation that providers could operate better schools for $50 per pupil or less, at which time the government hoped to take over.

However, year one results showed that many operators were paying far more — on average $300 per pupil. These costs went down over the lifetime of the program, falling to $119 per pupil on average, with some operators hitting the target, and others still spending three times as much.

Tom Dannatt, founder and U.K. CEO of Street Child, pointed to unpredictable project funding as one of the biggest hurdles to the success of the program overall.

“We never really knew when the next paycheck was coming … so we basically ran it on a pay-as-you-go model … we absolutely could have done better without the constant stop [and] start,” he said.

Best performers

Street Child, Rising Academies, Bridge, YMCA, and More Than Me are the operators with the best learning outcomes, according to the report, which highlights Street Child as having the best overall results. Rising Academies also had strong overall outcomes, but costs more than the government target.

More Than Me, which has been renamed Hilltop Schools since the scandal, had the highest learning gains, but failed to protect children from abuse. Bridge also showed learning gains, but reduced access for some children. The Liberian NGO YMCA scored highly across the board, but was tainted by a sex abuse scandal.

Although the RCT was never intended to produce policy recommendations, Sandefur urged donors and the government to focus more on accountability, pointing out that providers were rewarded by being allocated more schools even after expelling students en masse or being implicated in sexual abuse cases.

“Sanctioning under-performing providers is an obvious policy recommendation of our results,” the report states.

But the Liberian government is focused squarely on learning outcomes, according to Gbilia, and plans to keep working with the providers that showed the best learning results.

Bridge aims to more than double the number of schools it operates to 170, while Rising Academies has been asked to triple its schools to 87. YMCA is expected to increase its cohort from four to 15 schools. BRAC, More Than Me, and Street Child are not expanding.

But despite the modest learning gains and high costs revealed in the latest evaluation, and the controversy that has dogged LEAP since it began, George Werner, Liberia’s former education minister and the architect behind LEAP, remains an enthusiastic supporter. Asked during Tuesday’s CGD event if he thought the program has been a success, he gave an emphatic yes.

"Liberia deserves an educational system it cannot afford," he said. "I don't believe Africa can forge ahead to develop its human capital by government schooling. We need meaningful partnerships.”