Commentary•

Evaluating diagnostic tests during outbreaks: challenges and lessons learnt from COVID-19

Summary box

  • Timely access to quality-assured diagnostic tests during a pandemic or outbreak is essential to support public health measures and limit the spread of disease; independent test evaluation studies are necessary to provide objective evidence on assay performance, but challenges in study implementation can contribute to delays.

  • Challenges encountered during implementation of evaluation studies for COVID-19 diagnostic tests included logistical and resource-related constraints, delays in building partnerships and obtaining ethical approvals, continuous changes in COVID-19 incidence, and travel restrictions impeding in-person training, monitoring visits and delivery of supplies.

  • Opportunities exist to address these challenges through ensuring access to resources and building networks of partners sharing harmonised practices that can be rapidly mobilised and activated in emergency situations.

  • Investment in the development of such systems is required in advance of the next public health emergency.

Introduction

Diagnostic tests have played a crucial role in COVID-19 management worldwide. In particular, antigen-detection rapid diagnostic tests (RDTs) have been used at an unprecedented scale, including outside of healthcare settings, to limit spread of SARS-CoV-2.1 These tests were developed rapidly, with availability initially driven by the time needed to generate data on the relevance of this detection method for this novel pathogen. Global access to quality-assured COVID-19 RDTs has been limited at times, hindered by disruptions in global supply chains and competing interests for initially limited supply, as well as challenges encountered during clinical performance evaluations leading to delayed availability of data. Earlier access to such tests may have further reduced disease spread, potentially saving lives.

The 100 Days Mission is an ambitious goal introduced by governments and industry leaders to improve response to future pandemic threats using lessons learnt from COVID-19 and previous epidemics.1 The aim is to provide safe, effective and affordable rapid diagnostics, therapeutics, and vaccines within 100 days of a declaration of a major outbreak. Building sustainable resources for clinical evaluation of diagnostics that can be rapidly activated, mobilised and implemented is essential to achieving this target.

Clinical performance evaluation studies assess the accuracy of a diagnostic assay in discriminating individuals with or without the target condition.2 Such studies are essential to support regulatory submissions and can provide evidence for policy-makers and procurers to select appropriate tests for specific use cases. For COVID-19, for which emergency regulatory authorisation mechanisms were instituted to promote timely availability, these data requirements were often minimal.

As the COVID-19 pandemic continued, more than 1000 commercial COVID-19 diagnostic tests were advertised.3 Foundation for Innovative New Diagnostics (FIND), the global alliance for diagnostics, developed a directory of COVID-19 tests to assist global stakeholders in identifying marketed products.4 However, as most performance data came directly from manufacturers, there was a clear need for independent clinical evaluation data to inform global and national procurers on assay performance and reduce the risk of deploying poorly performing diagnostics that could impede disease control.

In response, starting in March 2020, FIND and numerous global partners conducted clinical performance evaluations of COVID-19 diagnostic tests. Here, we present the unique perspective of the clinical, data and scientific teams involved in study implementation, summarising some of the challenges experienced, and highlight opportunities for rapid clinical evaluation of diagnostics during future outbreaks (table 1).

Table 1
|
Challenges and opportunities in conducting evaluation studies of diagnostic tests during outbreak situations and identified opportunities for improvement

COVID-19 diagnostic clinical evaluation studies

The WHO R&D Blueprint, developed based on lessons learnt from the 2013–2016 West Africa Ebola virus disease epidemic, is a global strategy designed to fast-track availability of tests, vaccines and medicines for outbreak-prone diseases during epidemics.5 Activation of the R&D Blueprint framework helped to prioritise key diagnostic response activities to the COVID-19 pandemic, including clinical evaluation studies.

The FIND evaluations were conducted through collaboration with a network of 20 clinical partner sites worldwide.6 Sites were selected based on their experience in conducting clinical trials (including adherence to good clinical and laboratory practice), ability to import tests that were not yet nationally authorised, involvement in routine COVID-19 diagnostic testing, and geographical representation. By end of 2022, 35 antigen rapid diagnostic tests had been assessed in prospective studies, and 51 antibody and 24 molecular tests in retrospective studies, across 20 study sites.

Challenges and opportunities

Study materials

Appropriately collected clinical samples linked to detailed clinical and epidemiological data are fundamental to high-quality diagnostic evaluation studies. Ensuring that ethics approvals are in place for the collection, study and storage of these specimens is critical, but can take time and contribute to delays. Generic informed consent and data collection forms should be developed to limit such delays in the future. There may also be benefit in enabling sharing or exchange of specimens across sites, which may be facilitated through biobanking networks.7 8 Mechanisms to improve access to reference and control materials, such as online catalogues,9 would be beneficial.

Difficulties in some countries with obtaining importation approvals for tests that were not nationally authorised delayed delivery for evaluation. Limited availability of personnel experienced in customs clearance processes and engagement with regulatory authorities to negotiate for rapid approval of import permits and customs clearances contributed to delays. Improving staffing at sites and enhancing these relationships are needed.

Study documentation

On the sponsor side, delays in drafting study documentation, such as protocols and contracts, held up study implementation. Development of ‘emergency mode’ procedures are needed to accelerate the development of case report forms (CRFs), electronic data capture (EDC) systems and databases, as well as agreement on a clinical trial master file system to collect minimal essential documentation. On the study site side, delays in reaching signed agreements with sponsors highlighted a need for advanced implementation of partner agreements ahead of future health emergencies to improve rapid reaction and scale-up. Delays from other key actors, including ethics and regulatory approval bodies, also affected timelines. To address this, regular EDC trainings (including online materials) for preselected partners should be conducted, and generic disease agnostic protocols, informed consent forms, and CRFs for diagnostic evaluations developed. These documents should be agreed on by the international community, local ethics committees and regulatory authorities to allow rapid activation once an outbreak is confirmed.

Study implementation

Conducting test evaluations during an outbreak proved to be particularly demanding for laboratories since resources and trained staff were already engaged in outbreak response activities. Staff absences due to COVID-19 further impacted workload. Conducting evaluation studies in a timely manner without affecting primary activities was therefore challenging. Establishing mechanisms to allow study sites to rapidly access dedicated supplemental contingency funding for additional human resources, as well as critical supplies such as personal protective equipment are needed to prevent workload bottlenecks during future outbreaks. This would require advance estimation of potential resource needs and plans for hiring and procurement when the time comes.

In some settings, prospective study recruitment was affected by individuals’ unwillingness to participate, either due to stigma linked to testing positive for COVID-19 or fear of negative impact on employment. Given that future disease outbreaks may be associated with similar concerns, integration of social sciences and engagement with local communities during the early stages of clinical study design is imperative.

Study recruitment for prospective evaluations is naturally more efficient when incidence is high but given the dynamic nature of SARS-CoV-2 transmission waves, similar to other outbreaks, timing was unpredictable; often by the time an evaluation was ready to start, caseloads would be declining. Changes to national testing strategies as the pandemic progressed also impacted recruitment. Prospective evaluations, therefore, need to incorporate adaptive elements, in which specific tests to be evaluated and recruitment practices can be flexible, while recognising that adjustment of sample size may affect statistical power. Ensuring that diagnostic tests are compatible with universal media and stored/frozen samples can also help address this challenge. Samples accumulated during narrow windows of high transmission can be used for future studies during periods of lower incidence.

Travel restrictions imposed during the pandemic prevented sponsor visits at study sites, including for training and monitoring. This was mitigated using teleconferencing tools and development of detailed training and assessment materials such as competency assessment questionnaires, monitoring checklists, training and laboratory assessment videos, and secured EDC systems and dashboards to track and verify data entry in real-time.

As expected, the independent evaluations revealed variable accuracy across test types and manufacturers, but even the performance of similar tests sometimes varied across evaluation sites. This variability may reflect site-specific differences among study populations or settings as well as limited harmonisation of methods across study sites. To minimise variability in study design, networks of study sites should be established prior to future outbreaks, with agreement and adoption of harmonised best practices and standards. Networks should include sites from a range of countries, including low-income and middle-income countries, to ensure that quality diagnostics appropriate for use across different settings are available.

Several measures were adopted to ensure impartiality of analysis and result interpretation. Diagnostic tests were purchased, rather than provided for free by the manufacturers, to limit manufacturer authority over the analysis process and publication of results. Confidence in the robustness and transparency of results was boosted using well-defined and transparent scoring processes for selection of tests to be evaluated, and a commitment to submitting all data and analysis scripts to an open-access platform, such as the WHO COVID-19 Data Repository,10 regardless of manufacturers’ opinions of results.

Conclusions

Despite lessons learnt from previous epidemics and the implementation of the Research and Development (R&D) Blueprint framework, evaluations of diagnostic tests during the COVID-19 pandemic proved challenging. While generating clinical evaluation data for diagnostics tests in an emergency context will always require a balance between timeliness and quality, numerous opportunities exist to introduce efficiencies, allowing for earlier access to quality-assured diagnostic tests during future outbreaks.

Implementation of these efficiencies will require substantial funding in advance of future health emergencies. Unfortunately, while preparedness initiatives relating to vaccines and therapeutics continue to attract investment, the funding gap for diagnostic preparedness, which has consistently been present, continues to widen. COVID-19 has clearly demonstrated that diagnostics are crucial to disease control, particularly in the early stages of an outbreak before vaccines and treatments can be deployed. Thus, this funding shortfall must be urgently addressed before the next public health threat emerges.

Article metrics
Altmetricopen-url
Dimensionsopen-url