Validating Joint External Evaluation reports with the quality of outbreak response in Ethiopia, Nigeria and Madagascar

Introduction

Structured evaluation of a country’s ability to respond to health security threats has garnered a great deal of attention and effort in the last 2 years with implementation of the Joint External Evaluation (JEE) system.1 At the time of this writing, 95 countries had engaged in the full JEE process involving a national self-study followed by a 5 day, on the ground review involving international experts.2 JEEs are intended to provide a thorough review and evaluation of a country’s capacities in 19 key areas of public health.3 The scores for each of 49 indicators in 19 domains is measured on a five-point scale combining quantitative and qualitative characteristics. The accompanying narratives summarise major strengths and limitations in each country’s public health systems, and recommendations for improvements are made.

The JEE scores are subsequently published as part of the JEE report and publicly available on several websites. Much effort has gone into developing and carrying out the JEEs; little validation of the JEE scores and recommendations are available to date.

We review a disease outbreak in each of three countries having undergone the JEE process. We compare scores and recommendations from the JEEs to conditions identified during postoutbreak reviews affecting that country’s response to the outbreak. Such a comparison provides a field-based validation in outbreak-related areas of the scores and recommendations from the JEE.

Scores and recommendations from each country’s JEE were drawn from the JEE summary documents published on the WHO’s website.4 Information on each outbreak was collected using a combination of sources and methods, including the following:

• Documented first and last reports from US Centers for Disease Control and prevention (CDC)’s Global Disease Detection Operations Center;

• On-line media reports, UN Situation Reports and journal articles;

• Interviews with staff following each outbreak at the CDC Operations Center;

• Interviews with international and national responders during the outbreaks. These responders included staff from CDC, other international agencies, and national Ministries of Health. Questions specific to that outbreak were elaborated for these interviews on the bases of the above sources. In some cases, follow-up questions were posed to these informants in an iterative process to probe further prior responses and in triangulating information from the various sources.

• Finally, preliminary conclusions were shared with field and headquarter staff to further refine an understanding of the collected responses.

Correspondence between the strengths and limitations in national systems relevant to an outbreak are summarised from review of text in the JEE document. Summarised information on each outbreak was compared with the relevant country’s JEE scores and text in these topical areas. The topical areas of relevance included IHR Coordination, National Laboratory Systems, Surveillance, Public Health Workforce, Preparedness, Emergency Operations, Medical Countermeasures and Risk Communication.

A subjective assessment of similarity and difference between these two sets of information was created independently as judged by each of the three authors on a three-level scale. Each of the authors is involved in global health security work professionally and has taken part in JEEs and postoutbreak reviews, though not in the countries evaluated. The three reviewers did not consult in creating their agreement scores. High correspondence existed if both the JEE and description of an outbreak raised a common concern. For example, if both described highly effective systems for laboratory diagnosis a ‘high’ level of correspondence was recorded. If in the outbreak, instead, an inadequate response from the laboratory system was reported, ‘low’ correspondence was recorded. Similarly, if the JEE reported poor surveillance capacity, and surveillance during the outbreak was considered poor, a ‘high’ correspondence was reported.

A Kappa statistic was generated to identify how likely the level of agreement among raters could have happened by chance. The MAGREE macro in SAS was used as it is a multiple-rater kappa statistic which omits missing values.

Background

The earliest JEE in this set was carried out in Ethiopia, during March/April 2016. The other two JEEs were carried out in June and July of 2017. The outbreak in Ethiopia was identified as beginning a little over a year after the JEE. In Madagascar and Nigeria, outbreaks began 2 and 6 months after the JEE, respectively. See table 1 and figure 1.

Request permission

Comparison of JEE reports and outbreak results

Tables 2–4 present the results of JEE and outbreak reviews in Ethiopia, Nigeria and Madagascar, respectively. In the final column of each of these tables, scores from the three raters on the level of correspondence between the JEE and outbreak is presented as low (L), medium (M), high (H) or no response (N/A).

Thirty-seven variables were compared between JEE scores and field operation levels, by three raters, among these three outbreaks. This created a total of 111 scores representing a low, medium or high level of correspondence between the JEE and the outbreak response review.

For 13 of the 37 variables, all three raters agreed that the correspondence was high. For additional 13 variables, two raters rated the correspondence a high, while one rater considered it to be only moderate. Only eight times did a rater consider the correspondence to be low, and for none of the 37 variables did more than one rater consider it low.

While 37 variables were evaluated, a reviewer occasionally chose not to respond with a ‘low, medium or high’ response. In total, 107 scores were recorded among the three reviewers. The Coefficient of Concordance produced via the MAGREE routine for a Kappa test was 0.457; this represents an F statistic of chance probability of 0.037. In simple terms, the level of agreement was high (SAS summary reference ofvarious measures of concordance are presented in https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm%23statug_freq_a0000000647.htm The SAS routine used for MAGREE is presented in http://support.sas.com/kb/25/006.html).14

The comparison did not show any consistent or dramatic conflicts between JEE and outbreak information. Thus, though interpretations may vary regarding the degree of agreement between JEE and outbreak information, JEEs overall appear to provide a very good guide to strengths and weaknesses in actual outbreaks.

The comparisons made here had two important limitations. First, response capacity at the end of the outbreak often had improved a great deal from the beginning. Thus, comparisons depend on when the comparison is made. Second, much of the action in the outbreak is local, so laboratory, social mobilisation or treatment characteristics in one area may be very different from another. JEE scores seldom took variable capacity in a country into account.

In summary, it appears that after-action reports can provide a strong check on information in the JEE and can provide a near real-time update on capacities of the public health system.

Lessons

Low JEE scores existed in critical areas across all three countries with implications on the ability of those countries to detect and respond to the outbreaks. Those low scores, however, were not always critical limiting factors. In the case of Ethiopia and Nigeria, even with limited skill and personnel, large country systems were able to mobilise an adequate number of skilled personnel.

Some of the inconsistencies found between the JEE and outbreak review can be explained by particular details of the outbreak. For example, though Nigeria has a low JEE score for border health, the national EOC mobilised teams to border areas to coordinate response. Because of the needs in that outbreak, one strong area of the JEE made up for weakness in another. In each of the three countries, international staff strengthened the response in areas rated low during the JEE. The quality of outbreak response, when inconsistent with JEE scores, was generally better than that predicted by the country’s JEE.

Where internal skill, equipment, training and personnel were lacking, in each of the three outbreaks, national resources were supplemented by international resources that to a large extent made up for national limitations. The quality and variety of those international personnel and supplies, and the ability of national systems to absorb them, was described to be important in strengthening the outbreak response; this could not be captured by the JEE. It appears to the authors that the JEE strongly sensitised national authorities to their areas of weakness and to opportunities to supplement national resources with international staff and equipment.

The JEE assessment identified systems capacity at the time of evaluation. What cannot be assessed from a single outbreak event is the influence of the JEE experience over time. It appears to us that for all three countries, the JEE created a framework to understand key roles and activities needed to respond to an outbreak more effectively.

Major outbreaks following a JEE may provide an opportunity for more rapid improvements in situation awareness of systematic weaknesses to be addressed than existed prior to the JEE. A fuller evaluation of this would require comparison of the review of several outbreak after a JEE, compared with several outbreaks before the JEE.

In Nigeria, Lassa outbreaks occur every year. In the outbreak prior to the JEE, an observer noted that authorities would send epidemiologists out to collect data and as we identify problems in the response we will address them. In the post-JEE Lassa outbreak, there was a much stronger focus on technical activities needed in the areas of laboratory, surveillance, reporting, Emergency Operations Centres, Medical Countermeasures, Points of Entry, communication and biosafety. Specific weaknesses in human/animal surveillance and laboratory systems triggered discussion to involve the Ministries of Agriculture and Environment. The phrase ‘supply chain’ had become part of the vocabulary. There was discussion about weaknesses in newly established legislation. The JEE experience provided an intense context with which to focus on these issues, raising the level of understanding and discourse and creating a shared vision, which otherwise would likely have been far less.

Although the JEE raised the level of understanding for key roles and actions that shared vision can be expected to deteriorate over time as staff rotation occurs and people not involved in the country’s JEE assume relevant posts. The need to refresh the reflection that occurred during the JEE can, in part, be met by improving After Action Reviews (AARs). AARs should consider the trend among several outbreaks over time. They should move from a focus on the specifics of the current outbreak to a more general reflection on the JEE indicators and levels.

What would the response to the outbreak have looked like if the JEE had not occurred? In Nigeria, weaknesses may not have been recognised as well or as quickly. Having recognised those weaknesses, prioritisation of critical functions occurred (in the case of Nigeria) which would have been far weaker and the coordination among state and national authorities would have been far less had not the JEE sensitised a large group of people to critical functions. It is less clear that this occurred in Madagascar, where leadership authority was not clearly established, or in Ethiopia, where political considerations limited the ability of health leaders to organise and mobilise.

Several key qualities of these outbreak responses were not captured by each country’s JEE:

• In federal system countries, such as Nigeria, the coordination of roles between national and state level authorities and the assessment of variable levels of capacity in various states.

• Quality of AARs and their integration into International Health Regulations and interim JEE internal country assessments.

• The level and timing of intersectorial participation in public health activities during an outbreak. Early large-scale mobilisation can greatly reduce transmission, and obviate the need for later panic-level participation. This goes beyond coordination with security authorities or risk communication to affected communities. It most closely tracks to the PREVENT indicator of ‘IHR Coordination, Communication, and Advocacy’, but is a key part of response that is easy to see but difficult to measure.

Opportunities to focus on these issues in JEEs, and in After Action Reviews, Simulation Exercise Evaluations and annual national IHR reporting can be used to improve post-JEE National Action Plans and outbreak response in the future.