The global pool of simulation exercise materials in health emergency preparedness and response: a scoping review with a health system perspective ================================================================================================================================================= * Geraldine McDarby * Lindy Reynolds * Zandile Zibwowa * Shams Syed * Ed Kelley * Sohel Saikat ## Abstract Simulation Exercises (SimEx) are an established tool in defence and allied security sectors, applied extensively in health security initiatives under national or international legislative requirements, particularly the International Health Regulations (2005). There is, however, a paucity of information on SimEx application to test the functionality of health systems alongside emergency preparedness, response and recovery. Given the important implications health services resilience has for the protection and improvement of human life, this scoping review was undertaken to determine how the publicly available body of existing global SimEx materials considers health systems, together with health security functions in the event of disruptive emergencies. The global review identified 668 articles from literature and 73 products from institutional sources. Relevant screening identified 51 materials suitable to examine from a health system lens using the six health system building blocks as per the WHO Health System Framework. Eight materials were identified for further examination of their ability to test health system functionality from a resilience perspective. SimEx are an effective approach used extensively within health security and emergency response sectors but is not yet adequately used to test health system resilience. Currently available SimEx materials lack an integrated health system perspective and have a limited focus on the quality of services delivered within the context of response to a public health emergency. The materials do not focus on the ability of systems to effectively maintain core services during response. Without adjustment of the scope and focus, currently available SimEx materials do not have the capacity to test health systems to support the development of resilient health systems. Dedicated SimEx materials are urgently needed to fill this gap and harness their potential as an operational tool to contribute to improvements in health systems. They can act as effective global goods to allow testing of different functional aspects of health systems and service delivery alongside emergency preparedness and response. The work was conducted within the scope of the Tackling Deadly Diseases in Africa Programme, funded by the UK Department for International Development, which seeks to strengthen collaboration between the health system and health security clusters to promote health security and build resilient health systems. * Simulation Exercise * health systems * health service resilience * health security * emergency preparedness * health systems strengthening ### Key questions #### What is already known? * Simulation Exercises (SimEx) are established, practised in defence and security sectors which are later recognised in national and international regulations (e.g International Health Regulations, 2005). * The importance of SimEx as an improvement tool has yet to be identified to test capacities for health system resilience alongside emergency preparedness. #### What are the new findings? * Currently available global pool of SimEx materials lack an integrated health system perspective with a limited focus on the maintenance of routine quality health services delivered during response to a public health emergency. * Without adjustment of the scope and focus, currently available global pool of publicly available SimEx materials do not have the capacity to test health system resilience. #### What do the new findings imply? * Participation of health system authorities or stakeholders in the development, conduct of SimEx and accountability of findings for improvement in health system strengthening is needed. * Dedicated SimEx materials are urgently needed to fill gaps identified in global resources to harness the potential of SimEx as an operational tool to contribute to improvements in health system strengthening. ## Introduction With 28 600 cases and 11 325 deaths, the West African outbreak of Ebola virus disease (EVD) from 2014 to 2016 focused global attention on the importance of resilient health systems by exposing their vulnerability to disease outbreaks.1 Health workers were significantly more likely to be infected than the general population, with this increased risk largely attributable to the poor quality of infection prevention and control (IPC) practices and emergency preparedness.1–3 Guinea, Liberia and Sierra Leone lost between 1.5% and 8.1% of their country’s doctors, nurses and midwives to EVD, translating into significant reductions in the healthcare provision.2 Ineffective surveillance systems enabled EVD to spread locally as well as across borders.4 5 During an outbreak, as with any public health emergency (PHE), the resilience of a country’s health system is tested in real time. The capacity to respond to an outbreak and maintain essential services creates a surge in demand for critical resources. Health system resilience has been defined as ‘the capacity of health actors, institutions, and populations to prepare for and effectively respond to crises; maintain core functions when a crisis hits; and, informed by lessons learned during the crisis, reorganize if conditions require it.’4 Resilient health systems protect human life and are linked with positive health outcomes during a PHE as well as in its aftermath.4 A lack of health system resilience is associated with excess morbidity and mortality due to the PHE as well as from other causes, which can be at least partially attributed to a reduction in access to quality health services.6–8 It has been estimated that a 50% reduction in access to healthcare services during the West African EVD resulted in increased child and maternal mortality as well as increased mortality from other infectious diseases.6 8 These indirect deaths—10 623—were not insignificant.6 Thus, it is critical that a mechanism exists to test and build resilient health systems in order to reduce excess morbidity and mortality from future PHEs. National and international preparedness for PHEs has long been discussed on a global scale, especially in health security forums that seek to address challenges and gaps in meeting core International Health Regulations (IHR) (2005) capacities. One component of the revised IHR (2005) monitoring and evaluation framework includes Simulation Exercises (SimEx), which are defined by the WHO as ‘forms of practice, training, monitoring or evaluation of capabilities involving the description or simulation of an emergency, to which a described or simulated response is made.’9 SimEx have historically been an established tool in defence and allied security sectors, with recent adaptation to health security efforts in the context of disruptive emergencies, natural and man-made. They are now being used as a tool to assess compliance with national—for example, the UK Civil Contingencies Act 2004—and international legislative requirements, notably IHR 2005.10 There is, however, a paucity of information on the application of these exercises to test the functionality of health systems alongside emergency preparedness, response and recovery. SimEx are also an ideal opportunity to test preparedness of the various functions of the health system, particularly health service delivery, in response to PHEs. This would contribute to the development of strong linkages between health systems and health security sectors supporting an integrated approach towards building resilient health systems. Given the important implications health services resilience has for the protection and improvement of human life, this scoping review was undertaken to determine how the body of existing global SimEx materials considers the testing of health system resilience, together with health security preparedness, and response functions in the event of acute and/or protracted PHEs. ## Methods Recognising the dispersion of SimEx materials, a scoping review of academic literature as well as an institutional search was undertaken.11 A wide search strategy was employed to identify all relevant materials. Searches of different hazard types, exercise types and subject areas were combined (online supplementary appendix 1). The initial search returned 668 articles for screening with an additional 17 identified through bibliography review. Titles and abstracts were reviewed against inclusion and exclusion criteria (table 1), reducing the number for full-text review to 32. ### Supplementary data [[bmjgh-2019-001687supp001.pdf]](pending:yes) View this table: [Table 1](http://gh.bmj.com/content/4/4/e001687/T1) Table 1 Inclusion and exclusion criteria For the institutional search, institutions involved in the development, implementation or evaluation of SimEx were identified by experts in the field, as well as through relevant articles. Their websites were searched for applicable materials. Ultimately, 31 institutions, ranging from academic to national response agencies, were reviewed (online supplementary appendix 2), identifying 73 materials which were reduced to 19 following the screening of aims and objectives (figure 1). ![Figure 1](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001687/F1.medium.gif) [Figure 1](http://gh.bmj.com/content/4/4/e001687/F1) Figure 1 Flow chart of literature and institutional search and screening. n/a, not applicable; PH, public health; SimEx, Simulation Exercise. The final materials identified (n=51; online supplementary appendix 3) (32 emerging from the literature and 19 emerging from institutional review) were analysed by two independent reviewers for their scope to test aspects of health systems using the WHO Health System Framework, consisting of six independent but inter-related building blocks (table 2). Resilience was considered to be addressed if materials demonstrated evidence of an integrated emergency response, maintenance of essential functions or reorganisation of services within the context of a PHE. A smaller number of materials (n=8) were identified for examination of their scope to test resilience from a functional perspective (figure 1 and box 1). Materials were excluded from this portion of the analysis if they did not report either an exercise or objectives in sufficient detail to support this analysis. This portion of the analysis (n=8) involved five journal articles and three guidance materials (online supplementary appendix 4). A library of publicly available materials to support the development of SimEx was compiled in parallel to the review process. View this table: [Table 2](http://gh.bmj.com/content/4/4/e001687/T2) Table 2 WHO health system building blocks—components and resilience attributes Box 1 ### Questions used to assess functionality of health system in the context of public health emergencies (PHE) * Does the material look at aspects of health service delivery in the event of a simulated event? * Does the material test a function of specific building blocks of a health system in consideration of other related building blocks? * Does the material test preparedness and response of healthcare facilities alongside national incident management system? * Does the material look at impact of emergencies on provision of routine essential health services? * Does the material look at how PHEs impact health service delivery at subnational and district-level facilities? * Does the material examine the standard of care during PHE? ### Patient and public involvement There was no involvement of patients or the public in this study, in either the design of the methods or in the conduct of the study. ## Results ### SimEx consideration of health system building blocks The majority (63%) of the materials (n=51) identified exclusively tested preparedness and response to a PHE, without considering them as typical functions of a resilient health system (figure 2). A limited number were identified that tested aspects of health system resilience or recovery (figure 2). The categorisation of the scope of material was based on the research team’s interpretation, that is, if the exercises dealt with preparing and responding to a PHE or if they dealt with aspects of recovery or response. A few materials self-identified as testing health security or testing for contingency, and so warranted extra categories. The materials identified looked at aspects of health security across different geographical levels (international, national, regional and facility); they more frequently examined the regional/district (43%) and facility levels (31.4%) (figure 3). The most frequent hazard type (37%) simulated was biological which included pandemic influenza and other emerging infections (figure 4). ![Figure 2](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001687/F2.medium.gif) [Figure 2](http://gh.bmj.com/content/4/4/e001687/F2) Figure 2 Simulation Exercise (SimEx) materials by scope (n=51). ![Figure 3](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001687/F3.medium.gif) [Figure 3](http://gh.bmj.com/content/4/4/e001687/F3) Figure 3 Simulation Exercise (SimEx) material by geographic or administrative level (n=51). ![Figure 4](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001687/F4.medium.gif) [Figure 4](http://gh.bmj.com/content/4/4/e001687/F4) Figure 4 Simulation Exercise (SimEx) materials by types of hazards (n=51). CBRNe, Chemical, Biological, Radiological, Nuclear and Explosive. Though often not the explicit focus, all reviewed materials (n=51) tested at least one building block or an aspect of a health system building block (figure 5). A functional approach was apparent, though functions were generally tested in a narrow way both within and across relevant building blocks, rather than integrated across all relevant health system building blocks. IPC was the most commonly tested service delivery function with adherence to case management guidelines and standard operating procedures (SOP) being a frequent focus of testing.8 The quality of health services provided in the context of the response or measures of the maintenance of essential health services were lacking. Figure 5 and table 3 highlight examples of building block functions and the specific functions identified as tested in the material. ![Figure 5](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001687/F5.medium.gif) [Figure 5](http://gh.bmj.com/content/4/4/e001687/F5) Figure 5 Simulation Exercise (SimEx) focusing on health system building blocks (n=51) (some materials cover more than one building block). View this table: [Table 3](http://gh.bmj.com/content/4/4/e001687/T3) Table 3 Examples of health system functions identified in SimEx material Surge capacity was the most frequently tested function of health workforce, though a narrow approach was often apparent. Exercises tested aspects of surge capacity such as staffing, however, this was not addressed within other building blocks in relation to the invariable increased demands on services and supplies. A narrow approach to health information systems was also apparent, with three materials (out of 51) focusing on surveillance systems and only limited aspects of risk communication tested. Only one identified material tested the triggering of a response plan using a surveillance system.12 Leadership and governance (from a security perspective) was the most frequently addressed building block, with many materials testing aspects of this building block such as response plans and roles and responsibilities of responders. However, alignment with national structures and guidelines was not apparent, nor was consideration given to decision in risk identification. While access to medicines and supplies was frequently alluded to, testing of access to mechanisms or supply chain resilience was infrequent. Financing was alluded to in only 17% of materials (n=51), but testing of mechanisms to access funding in the event of PHEs was not identified. Tabletop exercise/discussion was the most frequent approach identified (figure 6). They are less expensive and faster to execute, particularly when considering large groups of stakeholders. The limitation with this approach is limitation in scope to adequately simulate PHEs and test the individual and integrative aspects of health systems. ![Figure 6](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001687/F6.medium.gif) [Figure 6](http://gh.bmj.com/content/4/4/e001687/F6) Figure 6 Approaches in the conduct of Simulation Exercise (SimEx) (n=51). TTX, tabletop exercise. ### SimEx consideration of health system functionality and system underpinning Eight materials were identified for further analyses on the functionality/system aspect of the six building blocks, using a set of questions (box 1). Most materials tested activation of appropriate emergency response mechanisms and structures within the respective administrative levels tested, for example, a facility-level exercise testing activation of all appropriate response mechanisms within the health facility. Some materials tested activation across different levels within the system, for example, a health facility responding to an outbreak activating regional or national response systems. Emergency response systems were generally assumed to have been activated, without testing system triggering. As was apparent with the building block analysis, healthcare functions were generally not tested in an integrated way. For example, a mass dispensing drill had no regard for the parallel response structures with which they would have to integrate during a response.13 Similarly, a facility response to a sarin attack failed to integrate with national response agencies.13 14 Reporting of alignment with national command and control structures or response plans was limited as was evidence of consideration of the impact of the PHE on other health system-level facilities or management structures. Materials to test the impact of emergencies on primary healthcare (PHC) or its response were extremely limited and community resilience materials, where identified, failed to link with health systems.15–17 The materials were limited in their focus on testing the quality of services delivered in the context of response to a PHE. Where present, measures tended to focus on clinical aspects of care rather than system and process measures. Measures to test the maintenance of essential services from a quality perspective were not identified in any of the eight selected materials. ### Limitations of the review The application of the health system framework to review SimEx materials introduced a degree of subjectivity. In reality, it is the same health system that provides routine healthcare, emergency-specific healthcare and response to a shock impacting public health. This was addressed through discussion as well as with input from health system and security experts in WHO. Similarly, the approach to the analyses conducted required sufficient detail in the identified materials in relation to exercises or objectives, which led to the exclusion of a number of relevant materials as they were not present either with sufficient details or as a package (written narrative, scenario, injects and postexercise report). Both the academic literature and institutional materials were analysed using the same approach. This likely led to an underestimation of effect in relation to academic literature as not all that was tested may have been reported, and an overestimation in relation to institutional materials as objectives may not have been applied effectively within exercises. A publication bias also likely exists in relation to health security exercise reports and materials given the sensitive nature of findings as well as in relation to materials developed and delivered by private companies. While multiple institutions and organisations (n=31) known to be involved in SimEx were contacted to identify unpublished materials, it is likely that there remains a pool of materials not made available as they were considered proprietary and/or sensitive. However, the objectives of the exercise are based on publicly available materials that are accessible and can be used as global resources, as such access-restricted materials fall beyond the scope of this review. In general, materials identified were from countries with developed capacities for emergency preparedness and response—materials from other settings may not have percolated the literature and may not have an institutional home where materials are placed. However, the inclusion of institutions known to support SimEx within low income settings is likely to reduce the impact of the bias towards high-income countries within the findings of the work. The exclusion of non-English materials has the potential to introduce a cultural bias, although materials in the review included materials from most continents (North America, Europe, Africa, Asia and Australia). The extent of any cultural bias is therefore limited, and unlikely to affect the findings of the work. Despite these limitations, there is no reason to suspect that materials not included in this review differ systematically from those included in such a way as to negate the findings. ## Discussion While all materials identified tested aspects of health systems, there was limited evidence of an integrated health system approach. Health system building blocks were touched on from a preparedness perspective, and often tested in a fragmented and isolated way without addressing interlinkages. IPC was frequently tested in the context of response to a pandemic or other emerging infection, with a focus on governance structures and adherence to case management and SOPs. IPC is central to the response to any emerging infection as was highlighted within the EVD outbreak in West Africa where poor IPC practices contributed to significant health worker transmission, leading to reductions in response as well as essential healthcare delivery.2 In a similar instance, a lack of strict adherence to IPC guidelines was associated with ongoing healthcare facility transmission during the severe acute respiratory syndrome outbreak in both Toronto and Taiwan in 2003.18 An integrated approach to IPC as a critical function of quality health service delivery in these SimEx materials is thus required. The availability and timeliness of emergency financing is critical in determining the timeliness and effectiveness of coordinated efforts in any emergency response. Despite this, limited consideration was given to the supply of resources required to meet the surge in demand, including staff, diagnostics, medicines or personal protective equipment, and to accessing the financing necessary to meet these demands.19 It is therefore of utmost importance that the rapid mobilisation of financing should be regularly tested in SimEx, taking into account the response required as well as consideration of maintaining quality essential health services. Health information systems, though often alluded to, were rarely tested in a robust and integrated way, with a lack of focus on surveillance systems in particular. The only material identified that tested activation of an emergency plan using syndromic surveillance failed to trigger activation of the response plan.12 The pivotal role played by data and information systems in routine healthcare delivery is even greater in the context of emergencies. A strong and reliable health information system ensures understanding of the epidemiology of disease, and is critical in coordination, communication and management of response efforts. Testing this functionality should be an integral part of SimEx conducted at any level. Consideration of the integration of preparedness and response across health system levels was lacking, with services and facilities tending to test their own response capabilities in isolation. PHC, despite being the likely first point of care in many PHEs, was rarely considered in the SimEx materials reviewed either in terms of supporting preparedness capacity or in terms of the effect of the PHE on PHC.15 PHC plays a central role in surveillance which was highlighted within the EVD outbreak in West Africa, where failure to identify EVD when it first presented within the community in Guinea led to a significant delay in response with a concomitant lack of containment. Despite evidence from EVD West Africa that community linkages can support health system response in the face of PHEs, the current global pool of SimEx materials were unlikely to link with community aspects and where community preparedness materials existed they rarely linked with health systems.16 17 The importance of integrating private healthcare facilities into emergency preparedness and response capabilities was highlighted by the recent dengue outbreak in Khyber Pakhtunkhwa in 2017. A private health facility, unfamiliar with emergency response protocols, failed to activate emergency response mechanisms in a timely manner. This, alongside a lack of case management guidelines, contributed to prolonged community transmission.20 No evidence of integration of private healthcare facilities into health system preparedness and response was identified within the current pool of global SimEx materials. While there is international consensus of the need to focus on quality in healthcare, the focus on the quality of services delivered in the context of a PHE was lacking.21 22 Where materials did focus on service quality, they tended to focus on clinical aspects of care. System and process measures were only identified in drills that tested mass dispensing capabilities in the context of a pandemic or biological attack. These capabilities represented parallel response structures and their linkages with appropriate health system structures were not tested, nor was the communication required to divert individuals from normal health services tested.23 24 No materials identified included measures to test the maintenance of or quality of essential health services during the response to a PHE. Evidence from West Africa shows that the indirect mortality and morbidity associated with discontinuity of health services as well as poor quality health services was significant across the three countries, with the biggest impact on maternal and child health.6 The current global pool of SimEx materials are limited in its ability to test health service resilience alongside preparedness and response. This in turn limits the opportunity to practically bridge health security and health systems at different administrative levels. Special attention is required in using SimEx approaches to drive sustainability of investment in health security preparedness or disease-specific programmes to proactively position available scarce resources into sector-wide development of health systems for all public health hazards per IHR (2005). ## Conclusion SimExercises are a well-practised method of testing and promoting emergency preparedness and response for local, national and global health threats. While much experience exists globally in this area, the proprietary nature of some of the materials creates a missed opportunity for the sharing of knowledge across global health security and preparedness communities. It would be in the interest of the global community to develop a mechanism to support sharing of lessons learnt that respects the integrity of private organisations involved in SimEx development and delivery. The effectiveness of SimEx has been demonstrated in identifying gaps in emergency response plans, skills and associated resources.11 25 The lack of an integrated health system perspective in the current global pool of SimEx materials limits their ability to support health system functionality and strengthening in the context of PHEs. This, along with their lack of focus on the quality of response and the maintenance of quality essential health service functions, means that they do not have the capacity to support health system resilience. The incorporation of a health system perspective into SimEx materials has the potential to enhance health system strengthening and the development of resilience alongside emergency response and health security capabilities. An integrated approach to SimEx including health security, emergency preparedness and health systems is required to address the gap identified by this review. As a result of these findings, an off-the-shelf SimEx package that addresses health system aspects within the context of response to a PHE is being developed, which will be freely available for all countries, particularly low-income countries. This could be further supported through the cross involvement in ongoing exercises and after-action reviews to enhance connectivity and support the development of shared ownership of improvement recommendations. These new-generation SimEx materials could be collated in a global repository that could be accessed by national authorities. Such a collaborative approach would allow the leveraging of the considerable expertise in SimEx present within health security and emergency preparedness sectors. Lessons learnt from this integrated approach will allow health systems to be built better, function better, which will ultimately lead to the protection and improvement of human life. ## Acknowledgments This work was undertaken within the context of the UK Department for International Development (DFID) funded Tackling Deadly Diseases in Africa Programme (TDDAP), which seeks to strengthen the collaboration between the health system and health security clusters to promote health security and build resilient health systems. Appreciation goes to colleagues in WHO: Prosper Tumusiime, Fred Copper, Dirk Horemans, Reza Sasanto, Monica Lamonge, Andrew Black, Raj Sreedharan, Mary Stephens, Ali Yahaya, Yu Zhang and Louis Ako-Egbe. Special thanks to WHO partners: Public Health England (Dr Neil Squires, Paul Sutton and Dr Elena Skryabina), UK Department of Health (Helen Tomkys) and DFID (Lisha Lala) for their support and input into the conduct of the global mapping of SimEx. ## Footnotes * Handling editor Seye Abimbola * Contributors SSa conceptualised the study. SSa, GM and ZZ designed the methods. GM, LR and SSa conducted the literature search. GM and LR collected the data. GM, LR and ZZ developed the figures. GM, LR, SSa, ZZ, EK and SSy all contributed to data analysis, data interpretation, manuscript writing and reviewing. * Funding This study was funded by World Health Organization. * Competing interests None declared. * Patient consent for publication Not required. * Provenance and peer review Not commissioned; externally peer reviewed. * Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information. This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/). ## References 1. 1.World Health Organisation. Health worker Ebola infections in Guinea, Liberia and Sierra Leone: a preliminary report; 2015. 2. 2.Evans DK, Goldstein M, Popova A. Health-care worker mortality and the legacy of the Ebola epidemic. Lancet Glob Health2015;3:e439–40.[doi:10.1016/S2214-109X(15)00065-0](http://dx.doi.org/10.1016/S2214-109X(15)00065-0) 3. 3.Kilmarx PH, Clarke KR, Dietz PM, et al. Ebola virus disease in health care workers--Sierra Leone, 2014. MMWR Morb Mortal Wkly Rep2014;63:1168. 4. 4.Kruk ME, Myers M, Varpilah ST, et al. What is a resilient health system? Lessons from Ebola. The Lancet2015;385:1910–2.[doi:10.1016/S0140-6736(15)60755-3](http://dx.doi.org/10.1016/S0140-6736(15)60755-3) 5. 5.Kruk ME, Ling EJ, Bitton A, et al. Building resilient health systems: a proposal for a resilience index. BMJ2017;357.[doi:10.1136/bmj.j2323](http://dx.doi.org/10.1136/bmj.j2323) 6. 6.Parpia AS, Ndeffo-Mbah ML, Wenzel NS, et al. Effects of response to 2014–2015 Ebola outbreak on deaths from malaria, HIV/AIDS, and tuberculosis, West Africa. Emerg Infect Dis2016;22:433–41.[doi:10.3201/eid2203.150977](http://dx.doi.org/10.3201/eid2203.150977) [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.3201/eid2203.150977&link_type=DOI) [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=26886846&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001687.atom) 7. 7.Sylvester Squire J, Hann K, Denisiuk O, et al. The Ebola outbreak and staffing in public health facilities in rural Sierra Leone: who is left to do the job?Public Health Action2017;7:47–54.[doi:10.5588/pha.16.0089](http://dx.doi.org/10.5588/pha.16.0089) 8. 8.Sochas L, Channon AA, Nam S. Counting indirect crisis-related deaths in the context of a low-resilience health system: the case of maternal and neonatal health during the Ebola epidemic in Sierra Leone. Health Policy Plan2017;32(Suppl 3):iii32–9.[doi:10.1093/heapol/czx108](http://dx.doi.org/10.1093/heapol/czx108) 9. 9.World Health Organisation. Development, monitoring and evaluation of functional core capacity for implementing the International Health Regulations (2005): concept note [cited 2017 Oct 6]. Geneva, Switzerland: World Health Organisation, 2017. 10. 10.World Health Organisation. After Action Reviews and Simulation Exercises: Under the International Health Regulations 2005 Monitoring & Evaluation Framework (IHR MEF) [cited 2019 May 24], 2018. Available: [https://www.who.int/ihr/publications/WHO-WHE-CPI-2018.48/en/](https://www.who.int/ihr/publications/WHO-WHE-CPI-2018.48/en/) 11. 11.Skryabina E, Reedy G, Amlôt R, et al. What is the value of health emergency preparedness exercises? A scoping review study. Int J Disaster Risk Reduct2017;21:274–83.[doi:10.1016/j.ijdrr.2016.12.010](http://dx.doi.org/10.1016/j.ijdrr.2016.12.010) 12. 12.Wallace DJ, Arquilla B, Heffernan R, et al. A test of syndromic surveillance using a severe acute respiratory syndrome model. Am J Emerg Med2009;27:419–23.[doi:10.1016/j.ajem.2008.03.020](http://dx.doi.org/10.1016/j.ajem.2008.03.020) [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19555611&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001687.atom) 13. 13.Kilianski A, O'Rourke AT, Carlson CL, et al. The planning, execution, and evaluation of a mass prophylaxis full-scale exercise in cook county, IL. Biosecur Bioterror2014;12:106–16.[doi:10.1089/bsp.2013.0089](http://dx.doi.org/10.1089/bsp.2013.0089) 14. 14.Eason MP. Sarin exposure: a simulation case scenario. South Med J2013;106:55–62.[doi:10.1097/SMJ.0b013e31827cd12d](http://dx.doi.org/10.1097/SMJ.0b013e31827cd12d) 15. 15.McNeal MS, Daniels EC, Akintobi TH, et al. Emergency preparedness toolkit for primary care providers. Atlanta, GA, USA: Morehouse School of Medicine, 2012: 147. 16. 16.Burke RV, Goodhue CJ, Berg BM, et al. Academic-community partnership to develop a novel disaster training tool for school nurses: emergency triage drill kit. NASN Sch Nurse2015;30:265–8.[doi:10.1177/1942602X15582064](http://dx.doi.org/10.1177/1942602X15582064) [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1177/1942602X15582064&link_type=DOI) [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=25899521&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001687.atom) 17. 17.FEMA (US Federal Emergency Management Agency). Whole community: planning for the Unthinkable tabletop exercise; 2013. 18. 18.McDonald LC, Simor AE, Su I-J, et al. Sars in healthcare facilities, Toronto and Taiwan. Emerg Infect Dis2004;10:777–81.[doi:10.3201/eid1005.030791](http://dx.doi.org/10.3201/eid1005.030791) [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.3201/eid1005.030791&link_type=DOI) [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=15200808&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001687.atom) [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000221233500002&link_type=ISI) 19. 19.Hildreth WBartley. The financial logistics of disaster: the case of hurricane katrina. Public Perform Manag Rev2009;32:400–36.[doi:10.2753/PMR1530-9576320303](http://dx.doi.org/10.2753/PMR1530-9576320303) 20. 20.Ali M. WHO paints grim picture of anti-dengue preparedness. Reliefweb, 2017. 21. 21.Kruk ME, Gage AD, Arsenault C, et al. High-quality health systems in the Sustainable Development Goals era: time for a revolution. Lancet Glob Health2018;6:e1196–252.[doi:10.1016/S2214-109X(18)30386-3](http://dx.doi.org/10.1016/S2214-109X(18)30386-3) 22. 22.World health organisation. Delivering quality health services: a global imperative for universal health coverage. Geneva, Switzerland: World Health Organisation, 2018. 23. 23.Caum J, Alles S. Ready or not: analysis of a no-notice mass vaccination field response in Philadelphia. Biosecur Bioterror2013;11:262–70.[doi:10.1089/bsp.2013.0064](http://dx.doi.org/10.1089/bsp.2013.0064) 24. 24.Veltri K, Yaghdjian V, Morgan-Joseph T, et al. Hospital emergency preparedness: Push-POD operation and pharmacists as immunizers. J Am Pharm Assoc2012;52:81–5.[doi:10.1331/JAPhA.2012.11191](http://dx.doi.org/10.1331/JAPhA.2012.11191) [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1331/JAPhA.2012.11191&link_type=DOI) 25. 25.Skryabina E, Riley P, Reedy G, et al. A scoping review of evaluation methods for health emergency preparedness exercises. Am J Disaster Med2018;13:107–27.[doi:10.5055/ajdm.2018.0292](http://dx.doi.org/10.5055/ajdm.2018.0292)