How did pandemic preparedness measures get it so wrong?

The global health community has two main measures for pandemic preparedness. The first is the World Health Organization’s joint external evaluation, or JEE — a tool it has used to help more than 110 countries measure progress related to the International Health Regulations. Introduced following the 2003 SARS outbreak, the regulations require countries to strengthen systems for detecting, managing, and reporting global health threats. The second is the much broader — and newer — global health security index, or GHS index, which draws on publicly available information to assess and rank the capability of countries to prevent, detect, and respond to outbreaks. But neither of them predicted how countries would fare during the COVID-19 outbreak. Countries that were thought to be well prepared, such as the United States, have in reality been badly affected, while others such as Vietnam exceeded expectations. “We tortured the data looking for a relationship [between these measures and real-world performance],” said Joseph Dieleman, an associate professor at the University of Washington’s Institute for Health Metrics and Evaluation, or IHME, and a co-author of a forthcoming study on the topic. “No one really wanted to believe that all of the existing measures of pandemic preparedness were failing us in the most relevant pandemic of the last century.” Dieleman added that “we want to believe that we know what we’re measuring and that what we’re measuring is valuable in preparing for a pandemic.” But, he told Devex, “COVID-19, at least, has shown maybe that’s not the case.” Unraveling the reasons behind this is important for understanding how to be better prepared next time — and reveals that it takes more than just hospital beds and doctors to fight a pandemic. “No one really wanted to believe that all of the existing measures of pandemic preparedness were failing us in the most relevant pandemic of the last century.” --— Joseph Dieleman, associate professor, University of Washington’s Institute for Health Metrics and Evaluation ‘A very political issue’ Measuring pandemic preparedness begins with defining what it means. For Dieleman, countries that successfully responded to COVID-19 were those that curbed the spread of infections. But curtailing new cases was not about country capacity alone; it was about whether they were able to use that capacity. “Take a country like the United States, which has a health system with a lot of ‘warts’ but … has a huge amount of resources available,” Dieleman said. “Slowing infections is more about public health, broader government, a sense of responsibility. … It uses a very different set of tools than, let’s say, just hospitals.” The JEE and GHS index focus on quantifiable health system or biosecurity aspects of pandemic preparedness — such as the number of hospital beds or sophisticated laboratories. Many experts say their weakness is that they have largely ignored the less tangible factors needed to harness those capacities, such as leadership, trust, and health equity. The GHS index ranked the U.S. as one of the countries most prepared for a pandemic, alongside a handful of European countries, Australia, Canada, Thailand, and South Korea. But its recorded death rate per capita has been higher than many countries with lower scores. Former U.S. President Donald Trump received widespread criticism for his administration’s initial handling of the pandemic, which included what IHME data analyst Sawyer Crosby described as downplaying the risk of the outbreak and undermining scientific guidance on platforms such as Twitter. Several studies have linked a distrust of government to vaccine hesitancy. Similarly, WHO has cautioned that when the public distrusts the government during outbreaks, suspicion of official information can undermine messaging around infection control and treatment. Dieleman said the JEE and GHS index have focused on documenting capacity for pandemic preparedness but not countries’ ability to implement it — which he admits can change quickly. “It’s a very political issue,” he said. “[Implementation] is a function of who’s in office, who their advisers are, and [the] public’s perception of trust in that person.” Both the GHS index and, to a lesser extent, the JEE briefly mention leadership and trust. But lessons from COVID-19 — particularly the early handling of the pandemic by the U.S. — have strengthened calls for these categories to be expanded, said Ben Oppenheim, an applied political scientist and a vice president at global health analytics organization Metabiota. In 2015, Metabiota began work on a national Epidemic Preparedness Index, which it now uses internally to help with modeling. As of 2018, Metabiota had not been able to include trust as a measure because data on institutional trust is fragmented and difficult to compare across countries. Still, Oppenheim said pandemic preparedness measures should be interrogating the role of trust in outbreaks — not just if it exists, but also between whom and how. He cautioned that tracking the role of political leadership — which is hard to define and locally contextual — within pandemic preparedness is likely to be even harder than measuring something like trust. “There's no variable that you can yank off the internet or some existing database that just has ‘leadership’ as a parameter” to plug into an analysis, he warned. “That's going to take care to build and build well. … But do I think it should be done? Hell yes.” Know the population Metabiota’s index was also recently used as a starting point to develop what the group says is “the first methodology to systematically quantify” pandemic preparedness at a subnational level. Both the JEE and GHS index are focused on assessing national capacity. However, Daniel Stowell and Richard Garfield — two advisers to the U.S. Centers for Disease Control and Prevention — have advocated for including subnational capacity and health equity measures into JEEs, given lessons learned from COVID-19. In many countries, outbreaks have disproportionately affected lower-income communities, as well as racial and ethnic minorities. In the U.S., for example, Black and Latino communities accounted for almost twice the number of reported COVID-19 cases as they should have by population size, according to a study published in February. In Canada, Margaret Haworth-Brockman and Claire Betker from the National Collaborating Centre for Infectious Diseases have developed what they call “equity prompts,” or statements that can be used alongside existing public health emergency preparedness metrics. They hope the prompts will push policymakers to consider how health equity might shape emergency responses. For example, a traditional pandemic preparedness indicator might measure a public health agency’s ability to use risk assessment results to inform emergency management planning. In Haworth-Brockman and Betker’s work, this indicator would be accompanied by an equity prompt that reads, “Risk assessments are explicit about disadvantaged populations and why there are additional risks for certain populations, including by sex, age, place of residence, race, and terms of employment.” In a review of the JEE earlier this year, WHO acknowledged that JEE scores alone are not good predictors of pandemic responses. The review did not highlight the necessity of including general health equity measures, although it did cite the need for more gender-disaggregated data to promote gender equality within pandemic preparedness. “National numbers are nice to have, but they don't really tell you where the risk is,” Haworth-Brockman said. “If you don't know who your population is and what their disadvantages are, how can you really plan?” Rethinking the metrics Meanwhile, makers of the GHS index say the tool was never meant to predict countries’ performance in a pandemic. Instead, Jessica Bell, a senior program officer at the Nuclear Threat Initiative, said the measure provides a time-bound snapshot of countries’ capacity to identify gaps and inform policy. The index was just 2 months old when China reported its first case of COVID-19, and both it and the JEE are likely to change as they integrate lessons from the world’s latest pandemic. Bell said she expects the next GHS index to be released in late 2021. It will include about 30 new questions around factors such as contact tracing, health equity, and the ability to implement nonpharmaceutical interventions like mask-wearing. It will also look to better gauge not just capacity but implementation. “Some of the feedback we got throughout the pandemic was around capacity versus functionality,” Bell said. “It’s one thing to say you have an emergency operation center, but it’s a different question to ask, ‘Do you exercise that center annually?’” The GHS index team is also looking to add questions about public trust and communication. Bell said it hopes to repeat the exercise every three years to refine questions about leadership gradually. As the world rethinks and redrafts its pandemic preparedness measures, Oppenheim is calling for a new intergovernmental panel to shepherd a new era in pandemic preparedness metrics. The body, he argues, could help the international community make sense of fast-evolving but fragmented research, quantify emerging risk factors linked to the climate crisis, and help shield scientific assessments from political interference. Bell agreed that a stand-alone body, working with the United Nations, could help convene high-level dialogues around pandemic preparedness with funders such as the World Bank to address financing gaps, which she characterized as the common denominator in many countries’ lack of preparedness. “It goes without saying, but COVID-19 has been a painful, brutal, horrific humanitarian catastrophe,” Oppenheim said. “We have to use this experience to learn how to improve the ways we measure preparedness so we can better track and improve it.”