Challenges in epidemiological data

Data dissemination

Multiple agencies and reporting sources often generate public health statistics on the same epidemic. This can introduce obstacles, including discrepant estimates and varying data structures, which can be challenging to resolve, as many sources do not provide complete methodological documentation and metadata, or maintain consistent archiving procedures. These challenges can arise from staffing constraints, funding and other resource limitations, and gaps in technical infrastructure; such challenges may be particularly acute in low-income and middle-income countries (LMICs).6

Data structure

Public health data are disseminated via disparate structured, unstructured and semistructured sources. Data may be provided in spreadsheets, images or descriptive text. They can be written in numerous languages and presented via static webpage, interactive dashboards or application programming interface (API). Extracting data from text-based reports requires contextual understanding and interpretation. Tabular reports are easily digested when data elements are clearly labelled, though sometimes they lack the information needed for unambiguous interpretation. Data provided in graphical format can provide information in a readily interpretable format, though poor graphical design can also obscure details and lead to misinterpretation.8

Throughout the COVID-19 pandemic, data formats have not only differed across countries but also changed within the same reporting source over time. Many countries initially announced cases through text-based press releases. As the situation evolved, many sources incorporated tables and graphs into situation reports. Some reporting agencies have since transitioned to reporting COVID-19 data via dedicated web pages or interactive dashboards (figure 1), or maps which sometimes update at irregular intervals and overwrite previous data, making obtaining past data impossible. Even within some countries, such as the USA, standardisation across reporting agencies does not exist.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Screenshot of the New Jersey Department of Health interactive dashboard.32

Data permanence

Lack of data permanence poses significant challenges for retrospective analysis. Reporting sources often only report the current view of the event, though some present time-series data to reflect historical trends (figure 1). However, those sources do not provide details regarding data corrections, which enable analysts to extract and analyse data reported at a specific point in time, as well as post hoc corrected values. This is especially common for dashboards generated while monitoring events. As these dashboards generally do not provide archived data, frequent visits to the page are required to capture time series data. Lack of data permanence for a specific source makes validation and replication of the data nearly impossible.

Epidemiological challenges

In many cases, the very nature of the epidemiological data may pose reporting issues, including missing data, reporting lags, retrospective data corrections and changes to case definitions over time. It is critical to recognise these issues when they are present in the data and exercise due care when using the data for decision making.

Missing data

There is no universal standard in handling unknown and zero values in outbreak reporting. For example, the US CDC reports typically use a dash to indicate unknown case counts,11 but the Mexico Ministry of Health uses a dash to indicate zero cases.12 Gaps in the data may also occur due to reporting delays or other known or unknown reasons. This can result in reporting lags and catch-up values capturing multiple reports worth of data in a single report. For example, a country that typically reports case counts daily may not report cases for 3 days due to a holiday, and then report a large case increase on the fourth day to catch up the total. Data gaps can be substantial in resource-constrained settings, as health agencies may by necessity prioritise monitoring their constituent population over providing timely reports.2

Demographic and behavioural data are often collected as part of disease surveillance. This kind of information is valuable but is not always publicly accessible. Legal and regulatory frameworks governing personal health data pose one important constraint. Resource limitations pose another; rigorously anonymising data for wider dissemination require dedicated time and resources, which may be difficult to mobilise in areas facing high disease burdens, or with human resource or technical constraints, such as LMICs.7

Reporting lags

Due to the magnitude of the COVID-19 pandemic, many sources provide daily reports. However, reporting agencies often retrospectively add or remove cases following the completion of diagnostic testing and data cleaning. Documentation of these changes is often unavailable, as are data on the actual incidence date necessary to assign cases to historical dates, leading to spikes or drops in reported cases that do not reflect reality (figure 2). Sources also differ in whether they provide case counts by the date of symptom onset, sample collection or diagnosis. Most COVID-19 reports provide cases by date of diagnosis. This translates to time series lags when overtaxed laboratories have backlogs that lead to substantial delays between illness onset and test result. This challenge may be exacerbated in LMICs, with insufficient laboratory capacity to keep up with increased testing requirements during a pandemic.13

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Drop in cumulative cases on 2 July 2020 due to data cleaning.33

Case definition

A lack of uniform case definitions for public health surveillance can limit the usefulness of data.14 Some reports clearly state the relevant case definition (eg, Austria Bundesministerium), while others do not (eg, New Jersey COVID-19 Dashboard).

Additionally, case definitions can change during an outbreak, especially one caused by a novel pathogen. For example, an unexpected spike in COVID-19 cases in Hubei province, China, occurred on 12 February 2020 when clinical, unconfirmed cases were retrospectively added to incidence data (figure 3). Similarly, the US CDC added probable cases to its case definition in April 2020; however, states do not consistently report these data.15 Changes in the case definition can have a substantial effect on the proportion of all infections identified as cases as time progresses.16

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Time series of the number of cases reported by day in China.34

Composite sources for COVID-19

There has been a proliferation of composite data sources created as a result of the scale of the COVID-19 pandemic.17 18 Each composite source uses a different approach to compile data and address the issues previously discussed. Differences can be due to the specific use-cases intended for a dataset—for example, visualising cases/deaths temporally in near real-time, forecasting or disease modelling.18 The way compilers transfer and analyse the complex epidemiological data can also create differences between the compiled sources.5

Composite sources also differ in the spatial granularity of data provided, which may be due to the availability of data or the amount of labour required to compile such large datasets.19 No publicly available compilation dataset contains all available subnational data for the COVID-19 pandemic. In addition, location names may differ, depending on the source of standardisation. For example, Johns Hopkins University uses the US State Department country designations,20 and Metabiota uses Google Geolocation API for naming standardisation.21 More information on some composite sources is shown in online supplemental appendix 1.

Recommendations and best practices

The process of creating composite datasets is complex, with many potential pitfalls. Here we present recommended best practices, drawing on lessons learnt during COVID-19 as well as several prior outbreaks, including SARS-CoV starting in 2002, Zaire ebolavirus from West Africa starting in 2013, the emergence of Zika in the Americas starting in 2015, the Zaire ebolavirus outbreak originating in North Kivu, Democratic Republic of the Congo starting in 2018.22–24 Two types of best practice recommendations are made, those that should be used when creating a compilation dataset and those that should be applied by official reporting sources (recommendations are summarised in table 2).

Metabiota’s process for creating composite data sources starts with a team of digital surveillance experts with training in infectious disease epidemiology who rely on semiautomated approaches for event detection and prioritisation. Next, officially reported data in a variety of formats are curated, structured and validated into one consistent data format.

While automated data scraping of public health data is a useful tool to speed up data collection, it has many challenges. Many of the COVID-19 reporting websites have changed their data structure several times over the course of the pandemic. Even minor layout changes can cause an automated system to extract the incorrect information from a website.2 Best practice would be to include both automated and manual checks into data scraping to ensure the correct data has been acquired as automated checks cannot guard against all possible errors.

Once data from the initial source is structured, analysts must determine whether a single source is satisfactory or additional sources are required to obtain a comprehensive view of an epidemic. This determination depends largely on the intended use for the data. For example, the properties of a dataset intended to provide near-real-time awareness of an ongoing event may differ significantly from those of a dataset intended to provide accurate retrospective descriptions of an epidemic that has ended. When determining if multiple sources should be fused into a composite database, we consider multiple features of each potential component source. These include, among others, the frequency of updates; the timeliness of updates; the geographic granularity provided; the distinction between confirmed, probable, or suspected cases and deaths; and whether a source archives previous reports or overwrites its data entirely with each update.

When a composite data source becomes necessary, we recommend starting with the most comprehensive source available (eg, widest geographical coverage) to form the dataset’s base, then incorporating additional data provided in other sources to fill gaps. For COVID-19, we chose the WHO’s database of national totals as our primary source based on its ability to provide the broadest worldwide coverage, with the greatest degree of consistency and in the timeliest manner. We then replaced portions of the WHO database using national and/or subnational reports, which typically updated more rapidly and with greater geographical granularity than the WHO source. For some countries, such as the USA, multiple national and subnational reports were fused into the composite database to provide the best available information. In such situations, discrepancies such as date range overlaps or conflicts between national level and subnational level case counts must be reconciled in the composite dataset.

Some compiled datasets list other data aggregating websites and media reports in their sources making the original data source and its validity difficult or even impossible to ascertain. Best practices would be to use only official data primarily from authorities.

Since many dashboards are updated as soon as new data becomes available, the counts for a given location may be incomplete for a specific date, and due to the lack of date/time stamps associated with the data on the dashboard, it is often difficult to know the vintage of the data and how recently it has been updated. Data updated on Thursday for Brazil may be cumulative as of Thursday, but it could also be cumulative for Wednesday, Tuesday or a mixture for different administrative units. This is often the result of a delay in reporting of data from the original source rather than by the data compiler, but without more information, the data could be misinterpreted, leading to errors in calculations and decision making. A more consistent approach would be to update the data when the source has provided the final count for a given day, though it is a trade-off for having the most current data. At a minimum, the date associated with the cases, in addition to the date the data have been updated, is needed for accurate interpretation.

Although the epidemiological challenges discussed earlier cannot be easily addressed during data compilation, it is important to be aware of the issues. Ideally, public health agencies would attempt to reduce the occurrence of missing data, address reporting lags and use a clear, standardised case definition. The data should be transparent so researchers are aware of when data are missing, there is a report of the data cleaning process and the case definition being used is clearly stated.

As COVID-19 has progressed, the case definition has evolved,16 25 including the addition of probable cases and deaths (eg, US CDC). While not all sources or locations have broken incidence data into suspected, probable or confirmed, we recommend using that information whenever it is available. The inconsistent reporting of probable cases can lead to under-reporting of the disease impact and can impact models or the ability to properly respond to the pandemic.1

Moving forward

As the COVID-19 pandemic recedes, efforts to improve public health data production, collection, integration and dissemination will surge. Similar surges followed prior infectious disease crises, leading to substantial (yet clearly incomplete) improvements to the landscape of global health data. In this article, we have argued that varying standards and approaches to data production introduce challenges to linking data and building a common view of complex, fast-evolving epidemics. Table 2 contains an overview of these challenges and our recommendations for moving forward. At minimum, efforts to reform the global health data ecosystem should bear such challenges in mind; optimally, reform efforts should incorporate standards and best practices to yield more robust, transparent and interoperable data. Doing so will require investments in global public health infrastructure, both in terms of more funding for regional and global health organisations that set standards and provide technical support to national ministries of health, and directly to ministries of health in LMICs.26 As a starting point, we recommend the creation of a global task force or expert panel charged with defining a set of standards for epidemic data, metadata and documentation. This group should be independent, composed of leading scientists, but closely linked with WHO, national and regional health agencies, as well as non-governmental organisations and philanthropic organisations.

Reporting challenges are not confined to low-income countries nor are they solely the result of resource limitations. The US CDC, like many reporting agencies, has faced challenges in providing COVID-19 data to the public, and developing clearer standards could be of great benefit. Although the CDC has recommended case definitions for national use, these categories for confirmed, probable and suspected cases have not been universally adopted by individual states. Individual state health departments provide disparate information to the CDC, which then combines that data into a dashboard that is meant to give a clear picture of the situation in the USA. Unfortunately, the lack of interjurisdictional coordination negatively impacts the ability to visualise the situation and to control the spread of disease.27 Frequently, incidence data differ between state health department reports and those shared by the US CDC. While the COVID-19 pandemic has aggravated this situation, the agency previously addressed these issues in the National Notifiable Diseases Surveillance System, and lessons learnt for that schema could be applied to COVID-19 data collection and reporting. In addition to these structural issues, the CDC was impacted by political interference for much of the COVID-19 pandemic. This follows a pattern observed during the pandemic, in which governance challenges at the local and national levels have impeded epidemic reporting and response, for example, in China and Turkmenistan.28–31 More generally, evidence suggests that poor governance and institutional capacity gaps can negatively impact the ability of public health systems to respond to epidemics.26

Providing high-quality pandemic data has been a challenge for all reporting agencies, but some have performed better than others. The Hong Kong Centre for Health Protection (CHP) quickly used their influenza reporting system to provide COVID-19 reports during the initial stages of this pandemic. The CHP was one of the first authorities to implement a consistent definition for probable COVID-19 cases. Furthermore, the Public Health Agency of Canada, while slower to iterate from early reporting to their current system, provides clear daily updates, which include uniform and standardised data from each province and retrospectively address changes to previously reported values.

Modellers need high-quality data to train and validate models.2 Deficiencies in input data can cause predictive models to produce inaccurate results, and can hinder policymakers in identifying and choosing appropriate disease control strategies. Producers of infectious disease surveillance data should work within a common set of standards to improve the quality and comparability of disease statistics. With the absence of such standards, users of disease statistics should be aware of potential challenges and follow best practices in reconciling and harmonising discrepant and discordant data.