Article Text
PDF
Abstract
The International Health Regulations Monitoring and Evaluation Framework (IHRMEF) includes four components regularly conducted by States Parties to measure the current status of International Health Regulations (IHR) 2005 core capacities and provide recommendations for strengthening these capacities. However, the four components are conducted independently of one another and have no systematic referral to each other before, during or after each process, despite being largely conducted by the same team, country and support organisations. This analysis sets out to identify ways in which IHRMEF components could work more synergistically to effectively measure the status of IHR core capacities, taking into account the country’s priority risks. We developed a methodology to allow these independent components to communicate with each other, including expert consultation, a qualitative crosswalk analysis and a country-level quantitative analysis. The demonstrated results act as a proof of concept and illustrate a methodology to provide benefits across all four components before, during and after implementation.
- Public Health
- Health systems evaluation
This is an open access article distributed under the terms of the Creative Commons Attribution IGO License (CC BY NC 3.0 IGO), which permits use, distribution,and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.
Statistics from Altmetric.com
SUMMARY BOX
The International Health Regulations Monitoring and Evaluation Framework supports States Parties to assess and further develop core capacities for preparing and responding to health emergencies.
To date, no methodology has been proposed on how to combine the monitoring and evaluation framework components for core capacities.
This analysis does not propose to replace separate components with one amalgamated score; it proposes a mechanism to increase the reliability and validity of the monitoring and evaluation framework by leveraging the strengths of each component before, during or after implementation.
Insights from this analysis provide an opportunity to further strengthen core capacities for the International Health Regulations by synergising efforts across these components.
Introduction
The International Health Regulations (IHR) 2005 were first adopted by the World Health Assembly in 1969 and most recently revised in 2005 with the aim ‘to prevent, protect against, control and provide a public health response to the international spread of disease in ways that are commensurate with and restricted to public health risks and which avoid unnecessary interference with international traffic and trade’.1 Through the regulations, signatory States Parties commit to develop minimum core capacities for public health priorities and to notify the World Health Organization (WHO) of events that may meet the defined criteria of a public health emergency of international concern. The development of core capacities regards States Parties ability to ‘detect, assess, notify and report events’ and capacity to promptly and effectively respond to public health emergencies.2 The IHR also requires both States Parties and the WHO Director-General to report yearly to the World Health Assembly on the implementation of the regulations.
To support IHR core capacity development and facilitate annual reporting, WHO developed the IHR Monitoring and Evaluation Framework (IHRMEF).3 This framework is intended to aid the accountability of both States Parties and the WHO IHR Secretariat in reporting and, critically, provide review mechanisms to evaluate the status of core capacities within countries. The IHRMEF consists of four tools: the States Parties self-assessment annual reporting (SPAR) tool and the three voluntary tools of joint external evaluation (JEE), simulation exercises (SimEx) and after/intra action reviews (AAR/IAR) (table 1).3
The four components of the IHR Monitoring and Evaluation Framework2 3 16 20 23
There is a discussion in the current literature regarding the reliability and validity of the four tools. The first version of a self-assessment tool was launched in 2010 and has since evolved into the IHRMEF.4 In particular, the JEE was developed to address concerns around the self-reported nature of SPAR, which was first implemented in 2016.4 A review of SPAR scores in comparison to JEE scores found JEE scores to be on average 18% less than SPAR scores in 2017.5 However, this difference reduced to an average of 0.5%, ranging from −5% to 6% in 2018, which the authors noted to reflect both revisions to SPAR scoring and countries having experience with JEE.5 In a recent evaluation of the association between SPAR scores and infectious disease outcomes, a positive correlation was found between scores and disease control outcomes, with low-scoring countries having a higher risk of worse disease control outcomes than countries with a high score.6 During COVID-19, various literature emerged questioning the ability of the IHRMEF tools to accurately represent a country’s capacity to respond to a health emergency. One study found that in the first 8 weeks of the pandemic, countries with higher SPAR scores had fewer reported COVID-19 cases and deaths.7 While other discourse suggested JEE scores were not indicative of COVID-19 outcomes.8 9 However, many of these papers agree on the need for more in-country and context-specific considerations to strengthen the JEE. A retrospective review of AAR and SimEx reports from 2016 to 2019 showed over 75% of IHR core capacity areas were reviewed through an AAR and 100% through a SimEx.10 However, an analysis of SimEx and AAR activities conducted by the European Commission between 2005 and 2018 found the SimEx conducted did not align with low-scoring JEE or SPAR areas, nor did recommendations in the activity reports.11
While IHRMEF components are designed to be complimentary and support capacity development for the IHR, to date, these components can only be interpreted as individual documents or scores, with no combined representation of scores available as a single reference point. This independence is mainly due to components measuring capacities across a differing number of areas; for example, the JEE (2022, third edition) has 19 SPAR (2021, second edition) has 15, while AAR and SimEx measure capacity in at least one area.2 12 However, the WHO IHRMEF recommends that each of the four components be used to inform and review the findings of the other components. Such as SPAR and JEE providing context for SimEx and AAR, while in return, SimEx and AAR can review SPAR and JEE findings to support the prioritisation of capacity-building activities.3 Overall, the IHRMEF recommends that each component ‘can be triangulated to evaluate the functional status of IHR capacities’.3 Juxtaposing all the IHRMEF components would facilitate better monitoring and evaluation of a country’s IHR core capacities.
Literature is emerging regarding how IHRMEF components can complement each other or how components can be viewed as synergistic within a specific capacity area.13 14 One approach presents a way to combine scores from JEE and SPAR through a mapping of JEE indicators to SPAR indicators.13 This approach found a high degree of matching between indicators, therefore demonstrating a high correlation between the two components.13 Specifically, the methodology was proposed as a way for countries to use SPAR to monitor progress on IHR compliance and identify gaps between JEE visits.
Previous studies have shown that alignment of IHRMEF components across capacity areas (and potentially scores) is feasible.3 10 11 13 14 However, none have proposed a methodology that combines all four IHRMEF components into one quantitative score. This analysis proposes a methodology to triangulate results from all IHRMEF components to develop a single metric that increases the reliability and validity of capacity assessments. To determine the best methodology to achieve this goal, three stages of analysis were completed: an expert consultation, a qualitative consolidation crosswalk of IHRMEF reports and a quantitative analysis of country-level IHRMEF scores. Throughout all stages, all four IHRMEF components were included in the analysis. Scores from JEE and SPAR are publicly available on the WHO Strategic Partnership for Health Security and Emergency Preparedness Portal, along with published SimEx and AAR reports.15
Expert consultation on IHRMEF components
To better understand gaps between IHRMEF components and areas for improvement, we consulted with experts in the WHO’s Health Security Preparedness Department. Experts from units that support each IHRMEF component at the country level were consulted to provide a comprehensive background to the analysis. All IHRMEF components are regularly conducted, with SPAR the most commonly reported, and variation experienced across WHO regions (figure 1, online supplemental table 1).
Supplemental material
Total number of International Health Regulations Monitoring and Evaluation Framework activities 2016–2022, by WHO region. AAR, after action review; IAR, intra action review; JEE, joint external evaluation; SimEx, simulation exercises; SPAR, States Parties self-assessment annual reporting; AFR, Regional Office for Africa; AMR, Regional Office for the Americas; EMR, Regional Office for the Eastern Mediterranean; EUR, Regional Office for Europe; SEAR, Regional Office for South-East Asia; WPR, Regional Office for the Western Pacific
Four main considerations for triangulating IHRMEF components were identified:
SimEx and AAR capture data, which is not always covered by SPAR and JEE, could be used to complement these components.
SimEx and AAR recommendations, which are not systematically used to complement JEE and SPAR, could be used to better inform IHRMEF approaches within countries.
SPAR and JEE emphasise standardised quantitative data, while SimEx and AAR emphasise qualitative data.
These insights confirm the need for a mechanism to capture coherence and synergism across IHRMEF components, to encourage reference between individual components and to support IHR requirements at the country level. Based on this expert input, both a qualitative crosswalk analysis and a country-level quantitative analysis were conducted. The crosswalk analysis expands on the recommendations provided during the expert consultation.
A crosswalk analysis
A qualitative crosswalk analysis of IHRMEF component reports was performed to identify linkages and gaps between the information present in each component report. This was conducted using IHRMEF component reports from 11 countries with multiple IHRMEF assessments available from 2016 to 2019: Bhutan, Burundi, Cambodia, Cameroon, India, the Lao People’s Democratic Republic, Nigeria, Pakistan, Rwanda, the United Republic of Tanzania and Uganda (see online supplemental table 2 for assessments reviewed and years).
The crosswalk analysis was performed by selecting one report and crosschecking it against the other three IHRMEF component reports available for that country. For example, selecting a SimEx report and reviewing it to see if it included information from the other reports (AAR, SPAR and JEE). This was then repeated for the remaining three reports. Linkages and gaps from each type of report were recorded based on commonalities seen across the countries analysed.
This revealed that there was no evidence of systematic use of information from any other IHRMEF report found across the four components. For example, the SimEx or AAR reports reviewed did not cite recommendations from recent JEE or SPAR reports, nor did JEE or SPAR reports integrate results from SimEx or AAR into assessments or recommendations. The IHRMEF recommends a triangulation between components to support the evaluation of capacities, which this analysis found not to be occurring.3 Additionally, the IHR AAR/SimEx minimum reporting template (included in Annex 2 of the Country Implementation Guidance for AAR and SimEx) further supports triangulation of findings to IHR core capacity areas using a standardised scoring methodology; this was also found to be not completed for the SimEx and AAR reports included in this review.16 This analysis, along with the expert consultation, provides an opportunity to strengthen linkages between the IHRMEF and the proposed modalities mapped in table 2.
Proposed modalities to better link all IHRMEF components, based on the crosswalk analysis performed
A country-level quantitative analysis of IHRMEF
As the qualitative crosswalk analysis indicated little linkage between the four components, a country-level quantitative triangulation of component scores was performed. The goal of this analysis was to develop a single metric that increases the reliability and validity of capacity assessments by triangulating results from all IHRMEF components. Uganda was selected based on the availability of recent reports for all four IHRMEF components.17 This quantitative analysis included the following reports: SPAR (2018)15, JEE (2017)15, AAR (2018),17 SimEx on Public Health Emergency Operations Centre (2017)18 and SimEx on Ebola Virus Disease (2017).19 Both AAR and SimEx were evaluated on a qualitative scoring scale with four rating levels: U (unable to be performed), M (performed with major challenges), S (performed with some challenges) and P (performed without challenges).10 Both SPAR and JEE were scored using a numerical scale, with levels 1–5 representing the level of capacity for each area. Currently, no mechanism exists to compare scores between these two approaches. Thus, to allow comparability across all IHRMEF components, both qualitative and quantitative scores were converted to an average quantitative score (0%–100%) for each capacity area based on the SPAR (2018)20 guidelines as shown in table 3.
AAR/SimEx scores converted to comparable SPAR and JEE levels and expressed as a percentage.10 16
In addition to differences in scoring technique, each of the four IHRMEF components assesses different capacity indicators. In this analysis, we used the second edition of the JEE, which had 50 indicators over 19 capacity areas and the first edition of the SPAR, which had 24 indicators over 13 capacity areas from Uganda.2 20 The SimEx and AAR both require that one or more (but not all) SPAR capacities be reviewed, validated or tested.10 To triangulate results from all IHRMEF components, each capacity area assessed in the JEE, AAR and SimEx reports was mapped to the 13 SPAR capacity areas and used as a baseline. This resulted in eight of the 13 SPAR capacities being assessed by at least two other IHRMEF components (figure 2). Once capacity areas were mapped, a single adjusted score was then calculated for each area using the arithmetic mean of each available IHRMEF component converted score. SPAR scores were selected as the baseline to compare adjusted scores based on IHR reporting requirements and the resultant availability of data. While SPAR is reported annually, JEE is recommended for completion every 5 years, and AAR and SimEx occur ad hoc, although more frequently than JEEs. To accommodate this scheduling, we recommend the adjusted score be calculated based on the most recent SPAR, JEE, AAR and/or SimEx (completed within the past 5 years, as relevant). Given the longer time frame needed to implement most capacities, the JEE, AAR and SimEx remain relevant assessments beyond the year completed. This methodology incorporates all available, recent capacity assessment information while also allowing comparability across years.
IHR capacity scores for Uganda by IHR Monitoring and Evaluation Framework components and adjusted average. AAR, after action review; EVD, Ebola Virus Disease; IAR, intra action review; IHR, International Health Regulations; JEE, joint external evaluation; NA, not assessed; PHEOC, Public Health Emergency Operations Center; SimEx, simulation exercises; SPAR, States Parties self-assessment annual reporting; NFP, National Focal Point. ∗Adjusted score calculated by arithmetic mean from all available components within each IHR capacity area.
When comparing the adjusted scores to the baseline SPAR scores, four of the eight IHR core capacities demonstrated a decreased capacity when accounting for all IHRMEF components: laboratory, surveillance, human resources and national health emergency framework (figure 2). Three capacities, however, demonstrated an increased capacity: legislation and financing, IHR coordination and National Focal Point (NFP) functions, and health service provision. The adjusted score for risk communication could not be compared with SPAR as the SPAR score for this capacity area was missing data. While for most capacities there was not much variation between the SPAR score and the adjusted score, the largest gap between scores was seen for laboratory capacity (66 adjusted score compared with 87 SPAR score). Specific findings from the AAR and SimEx reports noted challenges related to a lack of linkage between surveillance and healthcare systems as well as limited data and outbreak investigation training among healthcare workers. This finding emphasises the differences in SPAR scores reflecting the existence of capacities, while SimEx and AAR are aimed at detecting more functional capacities based on simulated and actual real-life events, respectively.
Contextualisation
We reviewed IHRMEF components based on expert consultation, a qualitative crosswalk to triangulate all components and a quantitative methodology for combining all components into a single adjusted score for each IHR core capacity within a country. The adjusted score proposed is not intended to replace IHRMEF component scores but to provide a comprehensive contextualised picture of specific IHR capacities, leverage the strengths of each component of IHRMEF, link IHRMEF component scores and present assessments for each IHR core capacity in one standardised format. This assembly of IHRMEF scores can increase the reliability and validity of IHR core capacity assessments through the combination and cross-referencing of scores from multiple tools.10 In addition, comparing self-assessed SPAR scores to externally assessed JEE scores and functional assessments from AAR and SimEx can aid in reliability and consistency. This method can give richer detail to identifying strengths or gaps for national health security planning (national action plans for health security (NAPHS)) and prioritised agenda setting.21 Each stage of this analysis becomes more specific in approach and the number of country reports used, from general consultation to the consolidation crosswalk reviewing reports from 11 countries, to the quantitative analysis using one example country.
The IHRMEF recommends that each component ‘can be triangulated to evaluate the functional status of IHR capacities’, but no guidelines or tools are provided by WHO on how to do so.3 While at least one IHR core capacity area is assessed within all IHRMEF components, our results from the qualitative crosswalk found that there was no evidence of systematic use of information across the IHRMEF components in reference to one another. This finding further underscores the need for a more consolidated approach to IHRMEF to increase linkages and learning across different assessments. As seen in table 2, we have proposed modalities to better link all IHRMEF components. Three modalities are recommended: to review information and recommendations from the other three components when planning or conducting an IHRMEF activity; to use minimum reporting template scores from recent AAR/SimEx and to compare scores for specific capacities across components when report writing (where appropriate). The national action planning process transforms recommendations from various evaluations into actions that can strengthen the ability of countries to prepare and be operationally ready to manage major public health risks or events.21 Through having stronger linkages and incorporating the three modalities outlined into standardised and systematically used activity report templates, these components can better strengthen and support NAPHS development and implementation.21 22
To further support the triangulation of IHRMEF components, we have presented a methodology to synthesise AAR, SimEx, JEE and SPAR results into one adjusted score for each capacity area. This adjusted score requires a mapping of capacity areas evaluated by each component and standardising all IHRMEF components onto the same scale. The completion of the minimum reporting template for IHR AAR/SimEx included in Annex 2 of the Country Implementation Guidance for AAR and SimEx would greatly facilitate this ability to link IHR core capacity areas across all IHRMEF components.16 It is important to note that systematically using IHRMEF component information across all activities would not facilitate one report replacing the reference to all. For implementation, the varied methodologies to assess different aspects of the core capacities and the level of detail in each individual report are essential for capacity building. Combining the metrics as above will ensure that the technical (JEE and/or SPAR) and operational (AAR and/or SimEx) components are combined to increase the validity of the IHRMEF capacity area assessments.
Limitations of this paper include that States Parties were not consulted on the expert consultation nor validated the crosswalk analysis. In addition, the crosswalk analysis was further limited by the possibility that additional reports may be available within countries that are not shared with WHO and cannot be included. The country-level quantitative analysis was limited to the availability of IHRMEF components and how frequently or recently they were conducted. Overall limitations include the recent changes to capacity areas in the current versions of SPAR and JEE, which present challenges to incorporating more recent IHRMEF assessments into the analysis, and that future inclusion of more recent IHRMEF reports may find differing results. In addition, the crosswalk analysis was limited to the review of 11 of the 196 States Parties with available IHRMEF components reported. This methodology serves as a proof of concept and has not been validated against health emergency outcomes and impact measures, for example, COVID-19. However, the strengths of this analysis include a novel presentation of cross-mapping of all IHRMEF components, inclusion of a quantitative scoring methodology for AAR and SimEx in line with JEE and SPAR scoring levels and proof of concept showing the utility of this methodology using real country reports and data. Further analyses to extend these methods are recommended to broaden the consultative process and test this methodology in different contexts.
Conclusion
This analysis developed a novel methodology to allow communication between the independent IHRMEF components to support findings across each assessment and provide synthesised scoring for IHR core capacity areas. This can support countries in developing NAPHS based on all IHRMEF components. The demonstrated results of this analysis act as a proof of concept to illustrate a methodology to provide benefits across all four IHRMEF components that could be used before, during and after implementation. Further exploration of IHRMEF data and validation against countries’ performance in health emergencies is needed to generate additional evidence for a single adjusted score and the functional role it may play in strengthening global health security.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Handling editor Seye Abimbola
X @kandelnirmal
Contributors RN and NK conceptualised, designed and conducted the analysis. RN, RG and CB prepared the draft manuscript. RN, LBTC, RG, CB, QH, SdlR, RS, LPS, LV, TM, DC, PNN, IP, DS, RS, MS, AT, JX, ST and NK participated in the interpretation of the analysis and writing of the manuscript. The final version of the manuscript was approved by all authors.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Disclaimer The author is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the World Health Organization.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.