• Public Health

The COVID-19 pandemic revealed that many countries are not equipped or able to effectively respond to emerging diseases. Doing better when the next dangerous pathogen emerges depends on understanding which capacities most effectively enable countries to prevent, detect and respond to epidemics and pandemics, and investing adequately in those capacities in advance. With that in mind, more than a dozen studies and reports have assessed the association between COVID-19 health outcomes and the leading country-level metrics of pandemic preparedness: States Parties Annual Report, the Joint External Evaluation (JEE) and the Global Health Security (GHS) Index. Most of the assessments have failed to find a clear association between COVID-19 mortality and infection rates and those pandemic preparedness metrics,1–8 including those studies that account for under-reporting and factors that influence COVID-19 fatality and infection rates such as age, sociodemographics, and key comorbidities.9

The recent article by Ledesma et al in this journal is the rare exception in finding several correlations between GHS indicators and COVID-19 excess mortality, but it has a critical error—it fails to account properly for gross domestic product (GDP) per capita.10 Doing so fundamentally changes the conclusions of the research as nearly all the paper’s significant findings are no longer statistically significant when this problem is addressed.

GDP per capita, a key control variable used in Ledesma et al’s paper, is highly right-skewed (figure 1A), meaning there are some countries’ values that are disproportionally larger than the mean and median values. Using a variable that is so heavily skewed often violates the necessary assumptions of a linear regression model—the model that Ledesma et al use—because the assumption of linearity is not met (figure 2A). In addition, a skewed variable can lead to heteroskedasticity, another violation of the standard linear regression model’s assumptions (figure 3A). Especially in smaller sample sizes (such as the one in Ledesma et al), skewed data can lead the estimator to be biased and inefficient, and the model results should not be trusted. Thus, it is the well-established standard to use a logarithmic transformation of GDP per capita.11 12 When GDP per capita is appropriately log-transformed, it leads to a distribution that is much closer to normal (figure 1B), has more of a linear relationship (figure 2B) with the dependent variable and has less heteroskedasticity (figure 3B). Critically, when GDP per capita is log-transformed the GHS composite index and 55 of its 56 subindices are no longer associated with lower excess COVID-19 mortality (table 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

The distribution of gross domestic product per capita: untransformed (A) and natural log transformed (B). Visual inspection suggests that the distribution of GDP per capita is closer to normal when it is natural log transformed. GDP = Gross domestic product.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

The relationship between GDP per capita and excess mortality with GDP per capita untransformed (A) and natural log transformed (B). Visual inspection suggests there is more of a linear relationship when GDP per capita is natural log transformed. GDP = Gross domestic product.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Testing for heteroskedasticity: Gross domestic product untransformed (A) and natural log transformed (B). Visual inspection suggests natural log transformation reduces heteroskedasticity.

In reviewing the article published online we contacted the lead author and editor-in-chief in August 2023 to advocate that the paper should be substantially revised or retracted in order to correct two critical errors: the failure to properly specify GDP and a previous failure to fully account for the risk of multiple hypothesis bias. In September, an acknowledgement was added to the paper post-publication that most of the coauthors in this assessment of the performance of the GHS Index were also involved in creating that Index.13 14 In November the paper was revised to add a Bonferroni adjustment to address the paper’s earlier failure to fully account for the risk of multiple hypothesis bias. Nevertheless, the failure to log transform income remains unaddressed and unacknowledged.15

We continue to believe that the failure to properly specify GDP is a critical error and that the paper needs to be substantially revised to address it. The stated intention of Ledesma et al is to inform the ‘unprecedented increases in development assistance towards pandemic preparedness in low-income to middle-income countries’.10 Yet, an analysis that does not properly account for GDP per capita cannot distinguish between wealth and preparedness (as measured by JEE or GHS Index, among others). There are many reasons why wealthy nations might on average have better COVID-19 outcomes that are not captured by pandemic preparedness indicators, including the ability to secure greater and earlier access to vaccines, therapeutics, tests and other essential medical supplies. If this analysis is meant to guide investments in poorer nations, it must be able to demonstrate that its results hold irrespective of income. Currently, it does not.

The JEE and GHS Index were intended to be tools for identifying gaps in national capacity to direct financial and political support appropriately. They were never intended to predict pandemic outcomes and many analyses suggest they were indeed not predictive in the COVID-19 crisis. Yet, many of the measures assessed by these metrics, such as laboratory capacity, might well prove critically important against future emerging disease threats.

Uncertainty over how to assess a country’s preparedness to prevent, detect and respond to pandemic threats is undermining efforts to optimally allocate investments in pandemic preparedness and thwarting efforts to convince donors and policymakers to invest more in initiatives such as the Pandemic Fund that are designed to help countries improve global health security. An essential step in building broad-based support and confidence in global health security is building trust and this starts with being as rigorous and transparent in our assessments as possible. Without that trust, we cannot hope to better prevent, detect and respond to future threats.