Original research

Evaluation of the Global Health Security Index as a predictor of COVID-19 excess mortality standardised for under-reporting and age structure

Abstract

Background Previous studies have observed that countries with the strongest levels of pandemic preparedness capacities experience the greatest levels of COVID-19 burden. However, these analyses have been limited by cross-country differentials in surveillance system quality and demographics. Here, we address limitations of previous comparisons by exploring country-level relationships between pandemic preparedness measures and comparative mortality ratios (CMRs), a form of indirect age standardisation, of excess COVID-19 mortality.

Methods We indirectly age standardised excess COVID-19 mortality, from the Institute for Health Metrics and Evaluation modelling database, by comparing observed total excess mortality to an expected age-specific COVID-19 mortality rate from a reference country to derive CMRs. We then linked CMRs with data on country-level measures of pandemic preparedness from the Global Health Security (GHS) Index. These data were used as input into multivariable linear regression analyses that included income as a covariate and adjusted for multiple comparisons. We conducted a sensitivity analysis using excess mortality estimates from WHO and The Economist.

Results The GHS Index was negatively associated with excess COVID-19 CMRs (β= −0.21, 95% CI=−0.33, −0.10). Greater capacities related to prevention (β= −0.11, 95% CI= −0.20 to −0.03), detection (β=−0.09, 95% CI= −0.17 to −0.02), response (β = −0.19, 95% CI= −0.32 to −0.05), international commitments (β= −0.17, 95% CI= −0.30 to −0.04) and risk environments (β= −0.30, 95% CI= −0.42 to −0.18) were each associated with lower CMRs. After adjustment for multiple hypotheses, the GHS Index (table 2; β= −0.21, adjusted 95% CI=−0.41, −0.02) and risk environment (β= −0.30, adjusted 95% CI=−0.50, −0.10) remained associated with excess deaths. Results were not replicated using excess mortality models that rely more heavily on reported COVID-19 deaths (eg, WHO and The Economist).

Conclusion The first direct comparison of COVID-19 excess mortality rates across countries accounting for under-reporting and age structure confirms that greater levels of preparedness were associated with lower excess COVID-19 mortality. Additional research is needed to confirm these relationships as more robust national-level data on COVID-19 impact become available.

What is already known on this topic

  • Early analyses found that preparedness assessment tools, such as the Global Health Security (GHS) Index and WHO’s Joint External Evaluation, are positively correlated with crude COVID-19 outcome measures.

  • These findings have raised significant debates about the contribution of pandemic preparedness capacities in supporting effective pandemic responses.

What this study adds

  • When we account for under-reporting and population age structure, our analysis of 183 countries confirms the expected relationship to preparedness illustrating that efforts to prepare for and respond to pandemics before they occur are effective in reducing mortality during global health emergencies.

How this study might affect research, practice or policy

  • With unprecedented increases in development assistance towards pandemic preparedness in low-income to middle-income countries, the results of these analyses provide countries with a list of capacities that can be further evaluated to directly modulate their vulnerability to the current pandemic and future public health emergencies.

Introduction

The COVID-19 pandemic has exposed the extent to which pandemic preparedness policies were inadequate and disjointed across the world.1 Over the first 2 years of the pandemic, COVID-19 has infected over 40% of the global population2 while becoming the leading infectious cause of death.3 However, country-level metrics of pandemic preparedness have been under increased scrutiny. Initial analyses of reported COVID-19 burden against measures such as the WHO’s Joint External Evaluation, States Parties Annual Report and the Global Health Security (GHS) Index have found that countries with the strongest levels of capacities experience the greatest levels of COVID-19 infection and mortality rates.4–8 These paradoxical results have raised major debates about the contribution of public health capacities in supporting effective pandemic responses.9–11

However, country-level comparisons of COVID-19 outcomes and their preparedness capacities are subject to limitations owing to a lack of consistent and standardised reporting. Factors such as surveillance system quality and varying age–sex structures contribute to observed variability in how countries report COVID-19 outcomes and respond to the pandemic. For example, countries differ in their capacities to diagnose COVID-19 cases, determine cause of death and to aggregate and report these data into national surveillance systems.12 13 Countries with more capacities to perform diagnostic tests, determine cause of death and publish mortality data may show higher COVID-19 case and death statistics than those with less ability to do so. Similarly, countries with less capacity may under-test, undercount and under-report COVID-19 cases and deaths. Previous studies have demonstrated that under-reporting of COVID-19 mortality by factors of 50-fold to 100-fold is common in countries with weaker testing and surveillance systems.3 14 For these reasons, excess deaths may be used as a proxy measure for COVID-19-related mortality, as this measure relies less on countries’ capacities to specifically diagnose COVID-19.15–17

Previous direct comparisons of countries’ outcomes during the pandemic also may not fully account for cross-country demographic differences that make some countries more vulnerable to COVID-19 deaths than others. For example, age has consistently ranked as the most important risk factor for COVID-19 mortality.18 19 Therefore, populations with a larger proportion of elderly people have increased vulnerability to severe COVID-19 disease.20–23 A country’s underlying risk for severe illness may also play a role in how likely infections are to be detected,24 as disease severity has been shown to influence whether diseases will be detected in surveillance efforts.25 Thus, in assessing country-level differences in mortality during the pandemic, it is critical to adjust for differences in age structures.

Here, we address limitations of previous comparisons by exploring country-level relationships between pandemic preparedness measures and indirectly age-standardised COVID-19 excess mortality among 183 countries. To adjust for differences in countries’ surveillance capacities and age structure, we calculated national-level comparative mortality ratios (CMRs), a form of indirect age standardisation, during the COVID-19 pandemic. We used the GHS Index as a measure of national preparedness, as it includes data on 195 countries’ capacities to carry out necessary functions for preventing, detecting and responding to infectious diseases.26 This analysis allows us to assess the relationship between pandemic preparedness and COVID-19 mortality, accounting for biases in national COVID-19 statistics due to under-reporting and age-structure. Results from this analysis will yield urgent insights on the role of health security capacities in mitigating the impact of the current pandemic and future public health emergencies.

Methods

Data sources

We collated data on country-level capacities and preparedness against biological threats from the GHS Index. The measurement quantifies country’s abilities or potential to carry out public health functions necessary for infectious disease outbreak prevention, detection and response. The index is comprised of six categories of preparedness (prevention, detection and reporting, rapid response, health system, compliance with international norms, risk environment), which are composed of various indicators and subindicators assessed by publicly available data for 195 countries. Data on the GHS Index, its 6 categories, all 37 indicators and a subset of subindicators identified a priori were extracted for analyses. Further details of input data and methodology of the index have previously been described in detail.26

We used country-level data on total COVID-19 excess mortality from the Institute for Health Metrics and Evaluation’s (IHME’s) modelled estimates covering COVID-19 excess deaths from 1 January 2020 to 31 December 2021. IHME has previously published their estimation strategy and input data sources in detail.3 Excess mortality is an important measure of the true mortality impact from the pandemic as it is the net difference between observed all-cause mortality during the pandemic and mortality expected under normal conditions.

To facilitate computation of CMRs, we extracted age-specific COVID-19 mortality data from the demography of COVID-19 deaths database.27 The database contains daily COVID-19 death counts by age, sex and time for 22 countries covering Europe, North America and Northeast Asia from April 2020 to April 2022. For this analysis, we extracted the most recent cumulative COVID-19 mortality counts in April 2022. We further obtained country-specific single age population counts from the United Nations (UN) for the most recent data available.

Outcome measurement

Using COVID-19 excess mortality estimates with indirect age standardisation methods, we can directly compare excess mortality rates across countries for the first time using CMRs. Direct age standardisation requires detailed data on COVID-19 mortality by age, which are currently unavailable for most countries. The CMR, however, is a form of indirect age standardisation that borrows an age structure of mortality from a reference country so that only the age distribution of the countries of interest is required. CMRs have been widely used in epidemiologic studies to compare mortality across countries, including in comparisons of COVID-19 outcomes.28 We computed the CMR using the following formula:

Display Formula

where c represents the country of interest, u is the COVID-19 mortality for the S standard country at i age group where A is the maximum age group and p is the population size for c country at i age group.

We used the USA as the reference country during computations of the CMR. Thus, we first computed age-specific cumulative mortality rates for the USA using age-specific COVID-19 death counts from the demography of COVID-19 deaths database and population sizes from the UN for the corresponding age ranges. Age-specific mortality rates for the USA were linked with age-specific population sizes of each country to derive expected mortality. We subsequently computed country-specific CMRs by dividing observed excess COVID-19 deaths from IHME and expected mortality. A CMR greater than one represents an increase in mortality relative to the reference population and CMR less than one represents a decrease in mortality relative to the reference.

CMR values were nearly identical (Pearson r values ranging from 0.98 to 1) when comparing CMRs that used the other 21 countries with available age-specific COVID-19 mortality data as the reference (online supplemental figure S1).

Statistical analyses

We employed Pearson r correlations to initially explore associations between GHS measures and COVID-19 CMRs among 183 countries. We then used multiple linear regression analyses to further evaluate the relationships. Because the GHS measures are highly correlated and to prevent unnecessary adjustment for variables that may potentially bias results,29 we used bivariate regressions to observe each relationship independent of the other indicators. However, in each linear regression, we included gross domestic product (GDP) per capita to account for potential confounding identified a priori.

To adjust for possible heteroscedasticity in our regressions, we constructed CIs with robust standard errors.30 We further adjusted our CIs to account for the issue of testing multiple hypotheses (n=57) by using a Bonferroni correction, which changed our desired α of 0.05 to a cut-off at 0.0009. The coefficients and the corresponding CIs represent differences in CMRs associated with 5-point differences in the GHS measures, where negative effect sizes represent lower CMR values associated with greater levels of GHS capacities. Since all measures are normalised, ranging from 0 to 100, the coefficients are directly comparable.

We conducted a series of one-way sensitivity analyses to assess the robustness of our results. We first performed a sensitivity analysis where we used the 2019 edition of the GHS Index as input into our regressions rather than the 2021 edition as in our primary analyses. While our primary focus was on assessing the relationship of preparedness (eg, public health and medical capacities) on excess mortality, countries’ COVID-19 response/mitigation strategies may impact excess deaths. To examine this, we conducted a sensitivity analysis where we included the Oxford Stringency Index (SI),31 as a country-level covariate quantifying COVID-19 responses, in our regressions. We conducted a final one-way sensitivity analysis where we used excess mortality estimates from the WHO32 and The Economist.33 In this sensitivity analysis, we included excess mortality data from all three sources representing two time periods including 2020 only and 2020 through 2021.

Patient and public involvement

Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Results

Descriptive statistics

The 2021 GHS Index ranged from 16.0 to 75.9 with a global population weighted average of 45.2 (online supplemental figure S2). When stratified by IHME GBD super-regions,34 the Sub-Saharan Africa region had the lowest GHS Score at 32.9 while the high-income region had the highest score at 65.8 (online supplemental table S1). The CMR for COVID-19 excess mortality ranged from −0.33 to 14.3 with a global population weighted average of 1.79 (online supplemental figure S3). Super-regions with the largest population weighted averages included Latin America and Caribbean (3.28), North Africa and Middle East (2.90) and Sub-Saharan Africa (2.85) (online supplemental table S1).

Correlations

Country-level correlations among the GHS Index, GHS Index categories and subindicators on COVID-19 CMRs are displayed in table 1. Prior to age standardisation, there was a weak positive correlation between the GHS Index and observed COVID-19 excess mortality rate (r=0.11, p value=0.14, online supplemental figure S4). After applying indirect standardisation with derivations of CMRs, we found a moderate correlation in the negative direction between the GHS Index and CMR (r= −0.39, p value≤0.0001; figure 1). Correlations remained moderate and in the negative direction when examining the six GHS categories (figure 2): prevention of the emergence of pathogens (r= −0.32, p value≤0.0001), early detection for epidemics (r= −0.25, p value=0.0007), rapid response to the spread of pathogens (r= −0.34, p value≤0.0001), health system capacity to treat (r= −0.29, p value=0.0001), commitments to improve national capacities (r= −0.21, p value=0.0046) and risk environment for biological threats (r= −0.59, p value≤0.0001).

Figure 1
Figure 1

Relationship between the 2021 Global Health Security Index and comparative COVID-19 excess mortality ratio. The blue points represent countries while the line represents the linear regression line for the relationship between the 2021 Global Health Security Index and comparative COVID-19 excess mortality ratios with the shaded area representing the corresponding 95% CI.

Figure 2
Figure 2

Relationships between the 2021 Global Health Security index categories and comparative COVID-19 excess mortality ratio. The black points represent countries while the blue lines represent linear regression lines for the relationships between 2021 Global Health Security Index categories and comparative COVID-19 excess mortality ratios with the shaded areas representing corresponding 95% CIs.

Table 1
|
Pearson r correlation coefficients between 2021 Global Health Security indicators and comparative COVID-19 excess mortality ratio

When examining the 37 indicators and 13 select subindicators, 30 of these capacities were negatively correlated with CMR with a p value below 0.05 but 18 of these were statistically significant when accounting for the Bonferroni correction. Indicators with the largest correlations included the following: political and security risks (r= −0.57), infrastructure adequacy (r= −0.52), socioeconomic resilience (r= −0.51), public health vulnerabilities (r= −0.48) and healthcare capacities (r= −0.41). Other notable indicators associated with CMR were surveillance data accessibility and transparency (r= −0.31), emergency preparedness plans (r= −0.27) and risk communication strategies (r= −0.23). Correlations were similar when assessing the government effectiveness (r= −0.60, p value≤0.0001), public confidence in government (r= −0.37, p value≤0.0001) and trust in health advice (r= −0.22, p value=0.0037) subindicators.

Multivariable analyses

After adjustment for GDP per capita, the GHS Index remained negatively associated with COVID-19 CMRs (online supplemental table S2; β= −0.21, 95% CI= −0.33 to −0.10). The results indicate that each 5-point increase in the GHS Index was associated with a 0.21 lower CMR. All GHS categories remained associated with COVID-19 CMRs after adjustment. That is, greater capacities related to prevention (β= −0.11, 95% CI= −0.20 to −0.03), detection (β= −0.09, 95% CI= −0.17 to −0.02), response (β= −0.19, 95% CI= −0.32 to −0.05), international commitments (β= −0.17, 95% CI= −0.30 to −0.04) and risk environments (β= −0.30, 95% CI= −0.42 to −0.18) were each associated with lower COVID-19 CMRs after holding income constant.

Indicators related to prevention capacities that remained associated with COVID-19 CMR were zoonotic disease (β= −0.09, 95% CI= −0.17 to −0.02) and immunisation (β= −0.08, 95% CI= −0.13 to −0.02) capacities. For detection, laboratory capacity for detecting priority diseases (β= −0.06, 95% CI= −0.10 to −0.01) and case-based investigation tools (β= −0.09, 95% CI= −0.15 to −0.03) remained negatively related to COVID-19 CMR. The indicators for response capacities that were negatively associated with COVID-19 CMR included emergency preparedness and response planning (β= −0.07, 95% CI= −0.14 to −0.01) and access to communications infrastructure (β= −0.17, 95% CI= −0.27 to −0.07). The health capacity in healthcare setting indicator was associated with COVID-19 CMR (β= −0.10, 95% CI= −0.18 to −0.01). For international norms, the cross-border agreement indicator was negatively related to the COVID-19 CMR (β= −0.07, 95% CI= −0.11 to −0.03).

The risk environment category had the largest effect size for the six categories such that each 5-point increase in the risk environment was associated with a 0.30 (95% CI= −0.42 to −0.18) lower CMR after holding income constant. Notable risk environment indicators that remained associated with COVID-19 CMR included government effectiveness (β=−0.21, 95% CI= −0.29 to −0.12), socioeconomic resilience (β= −0.23, 95% CI= −0.32 to −0.14), public confidence in government (β= −0.08, 95% CI= −0.12 to −0.03) and public health vulnerabilities (β= −0.21, 95% CI= −0.32 to −0.10).

After adjustment for multiple hypotheses, the GHS Index remained associated with COVID-19 CMRs (table 2; β= −0.21, adjusted 95% CI= −0.41 to −0.02). However, many of the individual GHS Index indicators and categories were no longer associated with CMRs after taking into account multiple hypotheses (at the strict 0.0009 significance-level). Indicators that remained associated with CMRs included access to communications infrastructure (β=−0.17, adjusted 95% CI= −0.33 to −0.01), risk environment (β=−0.30, adjusted 95% CI= −0.50 to −0.10), government effectiveness (β=−0.21, adjusted 95% CI= −0.35 to −0.06), social inclusion (β=−0.13, adjusted 95% CI= −0.24 to −0.02), and public confidence in government (β= −0.08, adjusted 95% CI= −0.15 to −0.00).

Table 2
|
Unadjusted and income-adjusted effect sizes and corresponding multiple hypothesis adjusted 95% CIs using robust standard errors of 2021 Global Health Security measures on comparative COVID-19 excess mortality ratio

One-way sensitivity analyses

The results were nearly identical when using the 2019 iteration of the GHS Index (online supplemental table S3). The results were also largely consistent when including the SI as a covariate in our regression as our effect sizes were only slightly larger when including the SI at −0.27 (−0.47 to −0.06) for the GHS Index (online supplemental table S4). However, in our third one-way sensitivity analysis where we used excess mortality estimates from the WHO and The Economist, the relationships between the GHS Index and CMR become null (online supplemental table S5). The income-adjusted effect sizes for the GHS Index on CMR were 0.07 (95% CI= −0.05 to 0.12) for the WHO and 0.01 (95% CI= −0.05 to 0.08) for The Economist. The risk environment category was the only capacity that remained negatively associated with CMR when using the other sources of excess mortality data.

Discussion

There are multiple factors that create a challenging environment for fully understanding the impact of COVID-19 relative to existing external assessments. Some factors include consistent generation of high-quality data, availability of and competition for scarce resources such as PPE and vaccines, and imperfect understandings of variation between and within populations. For example, the availability of comparable data is a persistent challenge with international comparisons of COVID-19 outcomes. Detailed age-specific mortality rates are currently only available for 22 countries. COVID-19 case counts are further affected by variable country-specific testing capacity, inclusion criteria and reporting. Data on COVID-19 deaths are similarly limited and under-reported due to differences in vital statistics performance across the world. Additionally, the effect of intense competition for vaccines clearly suggests a strong influence of national wealth on COVID-19 outcomes. However, higher-income countries also tend to have older populations. Thus, examinations of excess mortality that are adjusted for age provides urgent information for assessing the role health security capacities have in mitigating COVID-19 burden.

This analysis therefore represents the first direct comparison of COVID-19 excess mortality rates across countries that accounts for under-reporting and national age structure. We found that after adjustment for income, higher GHS Index scores were associated with lower CMRs for excess COVID-19 mortality. The adjusted analysis confirms the expected relationship to preparedness illustrating that efforts to prepare for and respond to pandemics before they occur are effective in reducing mortality during global health emergencies. It is noteworthy that the relationship for the GHS Index remained present even after accounting for the strict thresholds during multiple hypothesis adjustments. While many of the individual capacities that were significant under the traditional thresholds did not meet the strict threshold established by the multiple hypothesis adjustment, there still remains evidence that associations are present. In fact, there is significant debate regarding the use of this correction, particularly in the context of this study where there are correlated and inter-dependent preparedness capacities, a limited sample size, and already predetermined hypotheses. The conservative multiple hypothesis adjustment also raises concerns of Type II error (concluding that a capacity is not significant when it truly is). Due to these concerns, we therefore are providing, for transparency, interpretations of findings that were present both prior and after conservative adjustments of multiple hypotheses throughout.

Our findings underscore that the core pandemic preparedness capacities of infectious disease prevention, detection and response are each associated with lower excess COVID-19 deaths. For example, prevention capacities may have reduced excess COVID-19 deaths by impeding the emergence of other infectious disease outbreaks35 36 that may have further burdened health systems and contributed to more mortality during the pandemic. In this context, our finding that the prevention indicator of immunisation capacities and rates being associated with fewer excess deaths may have appeared as this capacity likely minimised the number of vaccine preventable deaths37–39 and provided an infrastructure for successful COVID-19 vaccination programmes.40 41

We further observed that detection capacities, specifically capacities related to laboratory systems for detection of priority diseases and case-based investigations, were associated with less excess COVID-19 deaths. These findings are aligned with previous work illustrating that these capacities allow for early identification of cases,42 43 which increase the likelihood of early access to treatment, isolation of cases to minimise disease transmission and supports the effectiveness of mitigation strategies.44–46 These early detection capacities therefore contribute to improved health outcomes and fewer excess deaths.14 In addition, the finding that case-based investigation capacities were associated with reductions in excess deaths is consistent with previous work illustrating that these strategies reduced COVID-19 transmission47 48 and case fatality rates.49

Our findings also show that capacities for rapid responses to mitigate disease spread are associated with reduced COVID-19 burden.50–52 In particular, our results indicate that having a framework for emergency preparedness and response, which includes having health emergency plans, non-pharmacological intervention plans and considerations of vulnerable populations, was associated with fewer excess COVID-19 deaths. We may have observed this relationship owing to previous investigations finding that a lack of health emergency plans may lead to ineffective implementation of mitigation strategies.53–55 Therefore, having a framework for emergency response may equip countries with existing strategies that they can draw on during emergencies. Another response capacity that was related to reduced excess deaths was access to communication infrastructure. A myriad of studies has indicated that communication of disease risks increases knowledge of the disease56–58 and adherence to interventions,59 60 with some studies suggesting that risk communication is one of the most effective COVID-19 mitigation strategies.61 62 We may have found a strong relationship for communication infrastructure, as this capacity may have been essential for implementation of risk communication strategies in populations.

However, we did not observe an association between the health system category, a metric of health systems’ abilities to successfully treat patients, and excess deaths after adjustment for multiple hypotheses. Though we did not find an association, there was an effect prior to adjustment for multiple hypotheses. Further, there are numerous studies confirming that greater health system capacities are indeed associated with less COVID-19 burden by improving treatment outcomes,63–65 and that stronger health systems can minimise disruptions to essential services66 67 and therefore subsequently advert excess deaths. Our findings also show that capacity in healthcare settings, a core indicator of health system performance assessing available human resources and hospital beds in countries, was negatively related with excess COVID-19 deaths. Recent evidence indicates that settings with fewer human resources in healthcare settings are more vulnerable to excess COVID-19 deaths due to greater disruptions to essential health services.68 Therefore, considering that there was an effect without correction for multiple hypotheses and prior research, there is some evidence that investments in health systems can modulate pandemic outcomes. It is important to amass timely and accurate global data to more fully measure the strength and resilience of health systems to respond to infectious disease emergencies, while also meeting countries’ full set of health needs. Future studies should re-evaluate the role of health systems in supporting effective pandemic responses as global metrics of health system capacities improve.

Furthermore, we observed that other core GHS capacities, adherence to global norms and risk environment, not regularly assessed by other measures of pandemic preparedness were associated with diminished mortality. In regard to adherence to international norms, our findings provide some evidence that cross-border agreements are beneficial during a pandemic. For example, countries in the European Union shared the burden of the pandemic, as countries accepted hospitalised patients from overwhelmed countries, borders remained open to healthcare workers and those seeking medical care, and they shared essential knowledge.69 While these cross-border agreements have been shown to be difficult to implement due to differing country-specific rules and priorities,70 our results provide quantitative evidence that these collaborations can play a critical role in adverting deaths and major disruptions in care.

Finally, the GHS Index category that had the strongest and consistent relationship with excess COVID-19 mortality was the risk environment. The risk environment category assesses the socioeconomic, political, regulatory and ecological factors that increase vulnerability to outbreaks.71 A notable risk environment indicator that was associated with excess deaths was government effectiveness, which captures governments’ abilities to efficiently formulate and implement policies and accountability of public officials. This indicator was likely an important factor in cross-country variation of excess deaths as this capacity provides a framework for proactive policies to ensure supply of medical equipment and rapid implementation of interventions.72 73 We also found that levels of inequalities and social exclusion were each associated with fewer excess deaths. Across various countries investigations have highlighted that COVID-19 disproportionately affects vulnerable populations, as they are the least protected and often face the greatest risk from COVID-19.74–77 These discrepancies further propagate the pandemic and serve to exacerbate existing inequalities.78 Countries with lower levels of inequality were likely able to craft equitable responses that contributed to lower excess deaths and thus future preparedness plans should include measures to reduce disparities.79

The risk environment may be a primary reason why the US response was disjointed compared with other high-income countries. Despite the US ranking the highest in the GHS Index, the USA had the 41st smallest CMR and 30th largest risk environment score among the 57 high-income countries included in this analysis. While countries such as Iceland, Australia and New Zealand had the top 4 lowest CMRs and in the top 20 in risk environment. Evidence suggests that New Zealand was able to mount a success response because of strong leadership coordinating with many institutions to implement response measures in real-time, prioritisation of vulnerable populations in responses, effective communication strategies that induced population-wide support of responses and swift institutional approval of pandemic tools.80 81 Responses in Australia82 and Iceland83 also benefited from similarly strong, rapid and coordinated responses. While the USA has a multitude of pandemic capacities, the US response was fragmented due to states implementing different control strategies,84 early institutional rules preventing rapid mobilisation of diagnostic equipment85 and mixed communication that potentially harmed compliance in response measures.86

Overall, our analysis confirms that after adjustment for population age distribution and under-reporting of deaths, there are the expected country-level relationships between pandemic preparedness capacities and COVID-19 outcomes. Even after adjustment for GDP per capita as a confounder, owing to countries with greater income potentially having more resources to augment capacities and to allocate to health services to advert mortality, many capacities remained associated with reduced COVID-19 mortality. Our findings were also confirmed in our sensitivity analysis, where we further adjust for country-level differences in COVID-19 mitigation policies. These findings reinforce that regardless of income levels and real-time pandemic response policies, existing pandemic preparedness capacities, measured with the GHS Index, provide countries with a directly modifiable tool that they can build to avert mortality in the context of an evolving pandemic.

While our findings confirm the expected relationships between pandemic preparedness and COVID-19 outcomes, we identified a few capacities that were not associated with excess deaths. For example, previous studies have identified that greater levels of trust are associated with reduced COVID-19 burden,23 87–89 but we did not observe this relationship. However, we did find a relationship for public confidence in government, an analogous form of intuitional support and cooperation but confidence differs from trust in that it is built off previous evidence and experience.87 Thus, our analyses still provide some evidence that social and governmental support are important factors for responses to the pandemic. Future studies should continue to explore the country-level relationship between trust and COVID-19, and other capacities that were not related to excess deaths in this study including healthcare access and intervention planning.

Lastly, we found that the relationships between preparedness capacities and excess mortality became null when using data from the WHO and The Economist. A major contributor to the change in relationships is due to substantial differences in estimates between the three groups. For example, in countries with low GHS scores (<40), excess mortality estimates are generally twofold to threefold greater when comparing IHME to WHO estimates. Since these locations generally do not have reliable cause of death data, all three modelling groups rely on statistical models with various covariates and assumptions. Our initial investigations have shown that CMRs from the WHO and The Economist are moderately correlated with reported COVID-19 deaths while there is no correlation for CMRs produced from IHME estimates. Since under-reporting of COVID-19 deaths is a common problem in countries with low GHS scores, with postmortem surveillance studies in Africa indicating that deaths are undercounted by a factor of 10,90 91 the potential greater reliance on reported COVID-19 deaths by WHO and The Economist may partially explain the different estimates in countries with low GHS scores. Besides varying reliance on reported deaths, all three modelling groups also use different sets of covariates to produce estimates in locations without data. Overall, this sensitivity analysis revealed that pandemic preparedness capacities are not associated with worse pandemic outcomes and that there is a critical need for improved and robust pandemic outcome measures.

Strengths and limitations

This study has several strengths, including the ability, for the first time, to directly compare country-level COVID-19 excess mortality adjusted for age structure using CMRs. Our analysis was also able to evaluate multiple indicators of pandemic preparedness. This provides health systems with a collection of specific capacities that can be further evaluated to potentially modulate their vulnerability to the current pandemic and future global health emergencies. The identification of specific capacities is particularly timely as recent estimates show unprecedented increases in development assistance towards pandemic preparedness in low-income to middle-income countries.92

However, the results from this investigation should be interpreted in the context of the following limitations. First, our outcome data, excess COVID-19 deaths, are subject to measurement error due to varying levels of reliable capacities for vital registration systems and ability to enumerate all-cause mortality across countries. Due to a lack of data in Sub-Saharan Africa and Asia, the quantification of excess COVID-19 deaths in almost all countries in these regions was estimated using a statistical model with various predictive covariates.3 This is a limitation that is consistent for all three modelling groups of excess mortality. Though the excess mortality data used in this analysis are best estimates, the substantial lack of data in Sub-Saharan Africa and Asia reinforces the need to strengthen detection capacities in these areas. The lack of data based on direct measurement resulted in varying estimates of excess mortality by differing modelling groups, which was reflected in our analyses. The observation that excess death models that relied more heavily on countries’ reported COVID-19 deaths generated different results in our analysis, underscores the potential for heterogeneity in national surveillance capacities to affect our ability to track deaths at the global level. The inability to fully enumerate disease-specific mortality is a critical gap in global pandemic surveillance. Efforts to improve national surveillance for infectious disease emergencies must also include efforts to bolster countries’ vital registration and all-cause mortality surveillance. In the interim, countries may consider improving their surveillance by employing survey methods such as postmortem surveillance studies. One such surveillance study in Zambia found that actual COVID-19 deaths are 10 times greater than reported deaths.90 These methods may assist in constructing more robust measures of COVID-19 impact and therefore assist future studies in providing more robust evaluations of the contributions of pandemic preparedness capacities.

Second, a similar limitation is that due to the lack of age-specific data on COVID-19 mortality in Sub-Saharan Africa and Asia, we were not able to conduct sensitivity analyses using countries from these regions as the reference in computations of CMRs. Some evidence suggests that the age pattern of COVID-19 mortality is steeper in the elderly age groups in high-income countries while flatter in non-high income,18 differences that may potentially yield differing distributions of CMRs. Third, there is potential measurement error in the GHS Index as the metric was constructed using data that was publicly available and therefore may not capture capacities that are not written up or published. Similarly, the country-level analyses may obscure important variation in pandemic preparedness capacities within countries as capacities may substantially vary within countries. Third, the GHS Index–COVID-19 relationship is likely to change as the pandemic progresses because the outcome is still developing with new reliable data becoming available. Fourth, our analytic approach assumed a linear relationship between the GHS Index measures and COVID-19 excess mortality. Fifth, the multiple hypothesis adjusted analysis was conservative and may have introduced Type II error. We therefore presented results prior and after adjustments for multiple hypotheses. Finally, the ecological nature of the data prevents us from making inferences regarding pandemic preparedness capacities and excess mortality at the individual-level.

Conclusion

The measures within the GHS Index were not intended to serve as a predictive model of how countries will respond in a crisis, but an inventory of the resources and plans available within each nation. This analysis demonstrates that having greater national level health security capacities, as measured by the GHS Index, is associated with lower excess COVID-19 mortality. An established and regularly exercised response infrastructure is critical to address a health crisis, but so are the preventive measures that provide day-to-day services to ensure an accessible, equitable and capable health system for outbreak detection. Continuing to build, maintain and measure health security capacities will be effective in mitigating the impacts of infectious disease threats. Our sensitivity analyses illustrate an urgent need for improved pandemic outcome measures that are unbiased by measurement and country demographics to improve our understanding of the role of pandemic preparedness capacities.