Article Text

Download PDFPDF

Authors’ reply to ‘assessments of the performance of pandemic preparedness measures must properly account for national income’
  1. Jorge Ricardo Ledesma1,
  2. Christopher Isaac2,
  3. Scott F Dowell3,
  4. David L Blazes3,
  5. Gabrielle V Essix2,
  6. Katherine Budeski4,
  7. Jessica Bell2,
  8. Jennifer B Nuzzo1,5
  1. 1Department of Epidemiology, Brown University School of Public Health, Providence, Rhode Island, USA
  2. 2Nuclear Threat Initiative, Washington, District of Columbia, USA
  3. 3Bill & Melinda Gates Foundation, Seattle, Washington, USA
  4. 4Department of Epidemiology, Harvard University T H Chan School of Public Health, Boston, Massachusetts, USA
  5. 5Pandemic Center, Brown University School of Public Health, Providence, Rhode Island, USA
  1. Correspondence to Mr Jorge Ricardo Ledesma; jorge_ledesma{at}brown.edu

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Summary box

  • Assessments of pandemic preparedness on mortality during the COVID-19 crises are challenged by cross-country gaps in data sources, reporting and demographics.

  • Even when adjusting for log income and country-specific COVID-19 mitigation strategies, there remained evidence of associations between pandemic preparedness and age-standardised excess mortality associated with the COVID-19 pandemic.

  • This study adds to the sizeable literature examining the role of preparedness and pandemic impacts, but by more completely adjusting for cross-country differences in age.

  • Additional collaborative studies are needed as more data on pandemic impacts emerge.

We appreciate the close review of our work shown by Dieleman et al1. Understanding the role of pandemic preparedness during the COVID-19 crisis is challenging considering substantial cross-country gaps in data sources, heterogeneity in reporting of COVID-19 outcomes and differences in population age structures and healthcare delivery systems. Our study of indirectly age-standardised excess deaths through comparative mortality ratios (CMR) was our attempt to overcome several of these limitations, and we appreciate the opportunity to address the statistical considerations in our study.

We agree that income is likely an important factor in the preparedness–mortality relationship as income is likely a common cause of both pandemic preparedness and COVID-19 mortality; however, we disagree that log-transforming income is most appropriate, as Dieleman et al argue. In studies with larger sample sizes (n>85 in the presence of skewness2 (our study had a sample size of 178)), the skewness in income that Dieleman et al identified primarily only leads to heteroscedasticity in model residuals, and therefore, violates the homoscedasticity (constant variance) assumption of ordinary least squares. Heteroscedasticity subsequently leads to biased standard errors (SEs). To minimise potential issues and biases in SEs related to heteroscedasticity, we used heteroscedastic-robust SEs throughout our study. The proposal by Dieleman et al to transform income actually does little to improve issues related to heteroscedasticity based on the studentised Breusch-Pagan test for heteroscedasticity in the multivariate Global Health Security (GHS) Index-CMR regression (studentised Breusch-Pagan with log-transformed income (BPT=6.58; p=0.037) and no transformation (BPT=6.74; p=0.034) both indicate rejection of the null hypothesis of homoscedasticity), underscoring the need for robust SEs.

Although the concerns of income specifications did not arise during the peer-review process, a concern that did emerge was a need to consider country-specific COVID-19 mitigation strategies in the preparedness–CMR association. This was suggested because differing mitigation strategies are likely to impact excess mortality levels. When both log income and COVID-19 mitigation strategies, as measured from the Oxford Stringency Index, are considered in the regression, there remain associations between preparedness and CMR (table 1). In fact, this may be the most well-specified model considering that the residuals are substantially closer to being constant (studentised Breusch-Pagan (BPT=6.91; p=0.075)). These findings are largely consistent with our secondary analyses in our study where we included mitigation strategies in our regressions, which also handled heteroscedasticity better than the model proposed by Dieleman et al ((BPT=7.29; p=0.063)). While these associations may not remain during multiple hypothesis adjustments, it is noteworthy that this adjustment is likely inappropriate in this context where there are inter-related and correlated pandemic preparedness capacities, limited sample size and predefined hypotheses.

Table 1

Country-level effect sizes of 2021 Global Health Security measures on comparative excess mortality ratio

After its publication, we received correspondence from Dieleman et al regarding our analysis. We promptly shared with his team our data and answered multiple queries regarding our methodological approach. This process was valuable, and it provided the opportunity to re-review our entire analytical pipeline and statistical decisions for the manuscript. Through this correspondence, we identified a missed digit in our code that pertains to our correction for multiple hypotheses (Bonferroni correction). Due to multiple hypotheses, we intended to derive 99.91% CIs. On reviewing our analytic pipeline, we realised we mistakenly missed a 9 and thus computed a 99.1% CI. We immediately contacted the journal regarding this typo and have since published a correction. We regret not catching this mistake earlier in the process. However, it should be noted that the typo in our analysis does not change the overall conclusion of our analysis. When using the even-stricter 99.91% CI, we found that there remains a relationship between the GHS index and excess COVID-19 mortality (−0.21 (99.91% CI −0.41 to −0.02)).

In regard to multiple hypothesis adjustments, we note that within the biostatistical literature there is mixed evidence regarding the appropriate use of multiple hypothesis testing corrections. Some evidence suggest that the test of multiple hypotheses performs poorly when there is a predetermined list of predictors that is selected a priori, when predictors are correlated or interdependent owing to shared variances and is too restrictive such that real effects go undetected in the context of smaller sample sizes. In our case, we have a mix of predetermined list of predictors (GHS Index subcategories) and added predictors (eg, country income, stringency score in the sensitivity analysis), which makes it less clear cut which approach is preferable.

The letter from Dieleman et al argues that the finding of our manuscript—that better-prepared countries largely experienced lower pandemic-associated mortality is ‘rare’. However, there is another study that also identified a negative relationship between preparedness and COVID-19 mortality early in the pandemic.3 We further disagree with the implication that our findings are spurious and unexpected. What our paper does is add to the sizeable literature examining the role of preparedness and pandemic impacts, but by more completely adjusting for the single biggest confounder in the relationship between COVID-19 infection and death: age. Given that countries’ demographics vary, a standard approach is needed to compare the impact of preparedness separate from the impact of age. In fact, taking into account age likely already adjusts for a substantial proportion of country-level income considering the very strong correlation between log income and fraction of population 65 years and greater in 2019 (Pearson’s r=0.74).

As mentioned in our published article, we believe that additional collaborative studies are needed to assess the relationship between pandemic preparedness and COVID-19 outcomes. There remain significant challenges in trying to understand the impact of preparedness on excess mortality. Our own analyses of the three most widely used excess mortality datasets (IHME, WHO, The Economist) indicate that estimates in low-income to middle-income countries vary between twofold and fivefold further compounding challenges in understanding the preparedness–CMR association. We agree that trust will be a critical step in fostering support in GHS and the complex challenges in these studies underscore that working collaboratively will be critical. We will welcome opportunities to work collaboratively to evaluate the role of pandemic preparedness as more data emerges.

Ethics statements

Patient consent for publication

References

Footnotes

  • X @jennifernuzzo

  • Contributors JRL and JBN wrote the first draft of this letter. All authors reviewed, edited and approved the version for submission.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests DLB is a current employee of the Bill & Melinda Gates Foundation, which partially funded the Global Health Security Index. Additionally, SFD and DLB are members of the international panel of experts that provides non-binding advice regarding the development of the Global Health Security Index. CI, GVE and JB are employees of NTI, which received prior grant funding from the BMGF, Open Philanthropy Foundation and the Rockefeller Foundation for the development of the 2021 Global Health Security Index. JBN contributed to the development of the 2021 Global Health Security Index, for which she received grant funding from NTI. The present analyses were conducted outside of the scope and without support of grant funding received for the Global Health Security Index.

  • Provenance and peer review Not commissioned; internally peer reviewed.