You are here

Article Text

Article menu
Original research
Patient and public engagement in decision-making regarding infectious disease outbreak management: an integrative review
  1. Sophie Kemper1,2,
  2. MEJ Bongers1,
  3. ENE Slok1,
  4. LJ Schoonmade3,
  5. JFH Kupper2,
  6. A Timen1,2
  1. 1National Coordination Centre for Communicable Disease Control, National Institute for Public Health and the Environment, Bilthoven, The Netherlands
  2. 2Athena Institute, VU University Amsterdam, Amsterdam, The Netherlands
  3. 3Medical Library, VU University Amsterdam, Amsterdam, The Netherlands
  1. Correspondence to Sophie Kemper; sophie.kemper{at}rivm.nl

Abstract

Introduction Worldwide, people experience the effects of infectious disease outbreaks on a regular basis. These effects vary from direct impact of the virus on health, to indirect impact of control measures on day-to-day life. Yet, incorporating the experiences, views and ideas of patients and the public in decision-making in managing outbreaks does not take place on a structural basis. However, this might be beneficial. We examined the current incorporation of patient and public engagement (PPE) in decision-making regarding outbreak management (OM).

Methods A systematic search was executed in PubMed, Embase, APA PsycInfo, Web of Science, Scopus and other literature sources. Papers describing PPE in decision-making regarding OM on a collective level (group-level) were included. Relevant information about study characteristics, methods, impact and embedment of PPE in decision-making in OM was collected.

Results The search yielded 4186 papers of which 13 were included. The papers varied in study context and design. Remarkably, no substantial patient engagement was identified. Overall, public engagement (PE) in decision-making regarding OM was mostly executed by a mix of methods, for example, workshops, interviews and surveys. Knowledge and idea sharing between the public and experts was deemed beneficial for establishing well-informed discussions. The efforts resulted in either direct implications for practice or recommendations in policy papers. Most papers described their efforts as a first step. No structural embedment of collective PE in decision-making regarding OM was identified. Furthermore, the quality of most papers was low to moderate due to insufficient description.

Conclusion Overall, various practices for PE can be potentially valuable, but structural embedment in OM decision-making on a collective level was low. Before PPE can be permanently embedded in OM, more evidence on its impact needs to be collected. Furthermore, reporting on the engagement process and used terminology needs to be harmonised to ensure reproducibility and transparency.

  • review
  • infections
  • diseases
  • disorders
  • injuries
  • control strategies
  • health policy

Data availability statement

Data are available upon reasonable request. Data extracted from the papers are directly available from the published papers. Data extraction table for the summary and assessment is available upon request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjgh-2021-007340

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Key questions

    What is already known?

    • Outbreak management (OM) is an ongoing process of planning before an outbreak, responding to the outbreak and recovering after the outbreak. The goal of OM is to minimise the consequences for a society with respect to medical, economic, political and social aspects.

    • It is mainly healthcare experts and policy makers that contribute to decision-making in OM.

    • Not much knowledge is available on patient and public engagement (PPE) in decision-making in OM.

    What are the new findings?

    • Patients seem to be under-represented in current engagement efforts in OM on a collective level.

    • There seems to be low embedment of collective level PPE in OM as a structural activity.

    • The reporting in papers is in need of improvement, as well as the execution of engagement with regard to ethical aspects of qualitative methods.

    What do the new findings imply?

    • A paradigm shift is needed in order to integrate PPE in decision-making in OM and to create opportunities to execute more research on PPE in decision-making in OM.

    • More knowledge is needed about the impact of PPE in decision-making in OM, which could be accomplished by means of evaluations.

    Introduction

    On a daily basis, the health of millions of people is threatened by infectious disease outbreaks worldwide.1 2 Outbreaks of infectious diseases occur frequently, such as Zika virus disease, dengue, measles or viral haemorrhagic fevers, with the COVID-19 pandemic as most recent example.3–6 In order to control these outbreaks, outbreak management (OM) policies are formulated.7 8 The goal of OM is to reduce the consequences of an outbreak on society, from individual level to macro level.9 However, even with OM policies in place, the impact of an outbreak on the health and safety of societies remains real and often detrimental. There are various reasons why OM might fail, varying from insufficient resources to poor acceptance of control measures.10 11

    Within the ongoing process of OM, three key aspects are distinguished: risk assessment, risk management and risk communication. Together these form the risk analysis framework (figure 1). First, in risk assessment, the infection is identified and characterised, and the likelihood and severity of adverse health effects from exposure to the infection are estimated.12 Second, in risk management, all measures to minimise the outbreak are weighed, selected and implemented, in consultation with all relevant stakeholders.13 Finally, the information about the risk and measures is exchanged between policy makers, municipal health services, hospitals, general practitioners, other organisations and/or healthcare professionals, media, patients and the general public, known as risk communication.14 In summary, when an outbreak occurs, this framework is used to develop an estimate of the risk of the outbreak to human health and safety; to identify control measures and to communicate with stakeholders about the risk and measures.

    Figure 1
    Figure 1

    The risk analysis framework, adapted from the WHO,72 displays the three aspects of outbreak management and their interaction.

    The main parties contributing to decision-making in OM are healthcare experts and policy makers (hereafter referred to as experts). Decision-making in OM is defined as the step-by-step process of making choices in shaping risk assessment, risk management and/or risk communication. This process is often executed with minimal direct patient and public input, however, engaging patients and the public in the management of infectious disease outbreaks could be very beneficial.15 16 This benefit of patient and public engagement (PPE) is increasingly recognised in general healthcare and has evolved considerably in past decades. Starting with the Alma Ata Declaration by the WHO in 1978, which included the following statement: ‘The people have the right and duty to participate individually and collectively in the planning and implementation of their health care.’17 Nowadays, integrating the input of patients and the public in healthcare policy is central to numerous health reform agendas.18 For PPE in healthcare policy-making, various arguments can be given. First, the normative argument reflects the legitimacy of the decision-making process. Patients and the public are end-users of healthcare and should have a voice in shaping it. Second, the substantive argument focuses on the experiential knowledge of patients and the public, which can be complementary to expert knowledge. Finally, the instrumental argument refers to the accomplishment of knowledge sharing by engagement and the achievement of social acceptance and avoidance of conflicts.19

    In the context of this paper, PPE is broadly defined as the spectrum of activities to integrate the collective values and attitudes of groups of public members who are affected by (including patient groups) or at risk during an outbreak, into the process of decision-making regarding OM. More specifically, engagement can be executed in various levels, with various goals. The International Association for Public Participation’s (IAP2) Spectrum of Public Participation (figure 2) describes five levels of engagement: ‘inform’, ‘consult’, ‘involve’, ‘collaborate’ and ‘empower’. With each level, the impact on decision-making increases.20

    Figure 2
    Figure 2

    The IAP2 spectrum of public participation, displaying five levels of engagement with their goals and promise to the public. With each level, the impact on decision-making increases. From the International Association for Public Participation (IAP2), 2018.

    In the context of infectious disease outbreaks, the Ebola virus disease outbreak in West Africa in 2014 clearly demonstrated that the values and perspectives of the affected communities cannot be ignored in managing an outbreak.21 This has become even more apparent during the current COVID-19 pandemic, when public perspectives are necessary to identify and successfully implement feasible and effective OM.22–24 Until now, the public has been engaged in certain aspects of OM, but this has almost always been limited to only execution of measures and gauging knowledge to improve communication efforts.25 26 PPE in decision-making in OM proves to be uncharted territory.

    Overall, the integration of collective views from patients and the public in OM may ultimately lead to the development of an innovative, higher-quality policy that is tailored to meet the full range of perspectives present in the diverse population within a country.15 21 27 In this paper, we aim to gain insight into the state of the art of collective PPE in OM decision-making with regard to methods, impact and level of embedment. This study will explore the following research question: How has PPE been incorporated in the decision-making of OM according to available literature, and what are best practices and recommendations for the future?

    Methods

    An integrative review approach was applied, as it allows for the inclusion of papers with different types of methodology.28 Based on literature, the types of methodology that were expected to dominate this area of research are mainly qualitative or mixed-method approaches.

    A comprehensive systematic search was performed on 7 July 2020, in collaboration with a medical librarian (LJS) in the following bibliographic databases: PubMed, Embase, APA PsycInfo (via Ebsco), the Web of Science Core Collection and Scopus. The choice for these databases was based on their (bio) medical and social nature and therefore agreement with the review topic. In the search, controlled terms (MeSH in PubMed, Emtree in Embase and APA Thesaurus of Psychological Index Terms in APA PsycInfo), as well as free-text terms were applied. The following terms, including synonyms and closely related words, were used as index terms or free-text words: ‘Public participation’, ‘Patient participation’ and ‘Outbreak’. Search terms expressing ‘Patient or Public participation’ were combined with search terms comprising ‘Outbreak’. ‘Outbreak’ terms consisted of general outbreak terms and specific disease terms known for causing outbreaks, varying in frequency and scale, for example, ‘Zika’, ‘Ebola’ and ‘SARS’, to increase the quality of the search. The search was performed without restrictions on publication year, language or article type. Duplicates were excluded. Online supplemental file 1 contains the full search strategies.

    Supplemental material

    Title and abstract screening

    Records derived from the search were reviewed based on title and abstract using the following criteria:

    Inclusion criteria

    • Papers written in English or Dutch.

    • Papers describing an infectious disease outbreak, which is characterised by a sudden increase in the number of cases.29 Outbreak simulation scenarios based on real outbreak situations were included, as they likewise create a context to execute and learn from PPE in decision-making in OM (these findings can also impact practice/policy).

    • Papers describing any form of inform, consult, involve, collaborate and/or empower (figure 2).

    • Papers describing the process and/or methods of engagement of members of the public, patients and/or representatives of patients or the public.

    Exclusion criteria

    • Papers describing endemic infections, as they require a different OM approach.

    • Papers only recommending or shortly mentioning engagement without details of process and/or methods.

    • Papers representing any type of expert opinion, for example, editorials.

    To test the screening tool, 2% of the records were screened separately by two researchers (ENES, SK). After screening 50% of the records, a 6.4% conflict rate was achieved. The researchers discussed a number of conflicting records (n=23), especially on the inclusion criterion about the description of some kind of engagement, which was occasionally difficult to evaluate. After double-screening 55% of the records, a conflict rate of 4.4% was achieved. This rate was below the threshold of 5% conflict rate determined beforehand in order to continue with solo-screening of title and abstract (SK). Rayyan QCRI was used as a data management system.30

    Full-text screening

    The included records were screened based on full text. Many derived articles described PPE merely in the execution of control measures. As this review focuses specifically on decision-making, one extra criterion was formulated in order to focus on specifically having a voice in shaping decisions regarding OM:

    • Papers describe members or representatives of the public or patients having a voice in shaping decisions with regard to risk assessment, risk management and/or risk communication.

    Double-screening was applied to all records (ENES, SK). Disagreements or uncertainties were discussed and resolved. When the disagreement could not be resolved, a third screener (DdR) executed an independent full-text screening and discussed with the other two researchers (ENES, SK) to make a final decision. Endnote V.X9 was used as a data management system.31

    Other sources

    To identify additional records, Google scholar, INVOLVE, WHO, IAP2, the Dutch patient federation, European patient federation and the National Coordination Centre for Public Engagement were examined. Similar keywords to those of the systematic literature search were used on these websites to find additional relevant records. Title and abstract screening were done immediately after finding potential suitable records by one researcher (SK). If eligible, records were added to the full-text screening process.

    To update the review, the original search query and the screening of other sources was executed once more on 11 November 2020. Similar eligibility criteria of the first screening were used, however, screening was executed by one researcher (SK). No new papers were included.

    Patient and public involvement

    Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our study.

    Data extraction

    A data extraction form was set up with four categories. The first included general study characteristics, such as type of infectious disease, country, aim of the study and type of participant. The second category focused on methods of engagement. The risk analysis framework (figure 1) was used to categorise the subject of engagement, and descriptions of methods and evaluations were included. In the third category, the impact of engagement such as implications for outbreak course or practice and/or policy was assessed. In the fourth category, the current integration of PPE in OM was assessed by describing the level of embedment. For this, the definition of Fagotto and Fung was used: ‘Embeddedness is a habit of deliberation among citizens. When that habit is embedded in a community’s political institutions and social practices, people frequently make public decisions and take collective actions through processes that involve discussion, reasoning, and citizen participation rather than through the exercise of authority, expertise, status, political weight, or other such forms of power’.32 As entailed in this definition, the level of embedment is explored from a perspective that focuses on collective engagement of groups of patients or the general public. This corresponds with the broad engagement terms included in the search strategy. The structural embedment of individual-level engagement is beyond the scope of this review. Two researchers (ENES, SK) independently extracted data from four records. The similarities and differences in retrieved information were discussed and resolved. The remaining data extraction was executed by one researcher (SK).

    Critical appraisal

    Critical appraisal was executed after data extraction because quality was not viewed as a criterion for eligibility. The Critical Appraisal Skills Programme Checklist was used to systematically assess the quality per study.33 34 Overall quality was valued as low, low to moderate, moderate or high. One researcher (SK) executed the appraisals.

    Results

    Thirteen records were included, which is displayed in the flowchart (figure 3).

    Figure 3
    Figure 3

    Flowchart of the eligibility screening process.

    General study characteristics

    All included papers varied greatly in study characteristics, method, impact and level of embedment, displayed in more detail in tables 1 and 2. Twelve papers were found in the database search, and one in other sources.35 With regard to the general characteristics of the included papers, the type of infectious disease outbreak varied between dengue fever (n=4),36–39 Ebola virus disease (n=1),40 Zika-virus (n=1)41 and H1N1 influenza (n=4).42–45 In addition, three papers used scenario simulations about an influenza A pandemic instead of actual outbreaks.35 46 47 Five papers reported on outbreaks in North America,35 41 43 44 46 two in central America,36 39 four in Oceania,37 42 45 47 one in Africa40 and one in Asia.38 All papers clearly explained the goal of engagement, which was mostly focused on understanding the values and needs of the public in order to develop a more effective and culturally appropriate policy. In most papers (n=12), groups of members from the general public were included, described as the public or community.35–42 44–47 Commonly, the general public was defined as lay citizens living in a community with no background in OM or healthcare policy. Some papers (n=3) also included community stakeholders who represented the interests of the public.37 38 43 Five papers included healthcare workers or policymakers.35 37 40 43 46 One study mentioned engagement of patients without further details on setting or method.32 Due to this serious lack of information, the remainder of the results will discuss data on public engagement (PE) only. The terms; ‘engagement’, ‘involvement’ and ‘participation’ were all used in the included papers to describe PE in decision-making in OM.

    View this table:
    Table 1

    General characteristics of included papers: authors, type of infectious disease outbreak, country, goal of PPE and type of participant

    View this table:
    Table 2

    Characteristics of the engagement process of included papers including the method, aspects of outbreak management, impact, scale and evaluation. To classify the aspects of outbreak management in which participants are engaged, the risk analysis framework was used (figure 1)

    Methods of engagement

    Participants were engaged in decisions regarding various components of OM, such as control strategies, communication efforts, resource allocation (medication or treatment), surveillance of infections and health services (table 2). No justification for the inclusion of these particular components of OM was given. When categorising these components into the risk analysis framework (figure 1), engagement was executed in either risk communication and/or risk management, but not in risk assessment.

    One study performed a randomised controlled trial (RCT) to assess the added value of community engagement. This engagement intervention consisted of discussions with community members and their leaders on possible dengue prevention strategies, compared with no engagement.36 Other papers used a range of qualitative methods for engagement in the components of OM, displayed in table 2: deliberative forums (n=4); stakeholder meetings, workshops or working groups (n=5); focus group discussions (FGDs) (n=4); interviews (n=6) and surveys (n=3). These methods were frequently combined (table 2). In most papers, several similar components were used including (1) presentations to provide information, for example, about the virus, outbreak situation and OM; (2) group discussions or interviews to obtain the views of participants on the selected OM components; (3) group discussions with stakeholders to gather information on, for example, socio-political context or history of OM and (4) surveys to gauge further perspectives of the participants. Alonge et al (2019) questioned the general public and stakeholders about o.a. potential strategies for strengthening community resources, characteristics and factors limiting shocks in the context of Ebola outbreaks. After these interviews, a stakeholder meeting was held to corroborate all information and provide recommendations on strengthening community resilience in Liberia. McNaughton et al (2012) studied dengue control strategies, over a time span of 18 months. After FGDs and interviews, telephone interviews were conducted with the general public. During the telephone interviews, information was collected on lay knowledge of dengue and the acceptability, safety, engagement and authorisation of strategies for dengue management. Lastly, De Kraaij et al (2007) executed deliberative forums with simulation scenarios about influenza pandemics to deliberate control measures. Forums were executed with citizens and stakeholders separately. Presentations with information was given, small group discussions were executed on the pros and cons of various control measures, and electronical voting was implemented to gauge the overall support for measures. Afterwards, evaluations with citizens and stakeholders were executed by means of pre–post surveys, individual interviews, focus groups and document review.

    Few papers included evaluations, providing minimal but valuable information about how participants experienced the process of engagement.35 42 44 Participants stated that the engagement process served as a trust and knowledge building exercise for them and an opportunity to share ideas. Finally, some participants suggested that the process of engagement would increase the public’s support of the discussed decisions in OM.

    Impact of engagement

    The RCT provided a measurable impact of PE on the number of dengue infections in a specific area. Mean percentage of recent dengue infection was 11.3% in the intervention group, compared with 14.6% in the control group. Which means that in the intervention group, 14 out of 1000 households were spared having a dengue case. The impact of the qualitative methods is displayed in table 2. In general, the impact of these papers can be divided into two categories: (1) direct impact on practice and (2) long-term impact on policy. These categories are also reflected in the scale of the approaches. All papers with community approaches presented direct impact on practice, whereas national approaches resulted in long-term impact on policy. First, for the papers with a community approach, the views of the public were mostly immediately implemented into the response to the outbreak.36–41 43 45 For example, Brittain et al (2019) executed FGDs with the public on communication regarding Zika virus. Recommendations on the use of social media and local radio programmes to disseminate information were directly implemented in a community communication campaign about Zika virus.41 Second, for papers with a national approach, the views of the public were not directly used to change OM practice but were formulated as recommendations for future policy.35 42 44 46 47 This also applied for the studies using pandemic simulation scenarios, instead of an actual outbreak. Some papers35 42 44 stated that their outcomes were used in national OM-policy papers such as contingency plans, communication strategies and a pre-pandemic guidance report. To our knowledge, until this moment, only the recommendations of the paper of De Kraaij et al (2007) have been directly used in national policy papers.48

    Level of embedment

    The level of structural embedment of PE in decision-making in OM was assessed by identifying ongoing structures enabling PE on a collective level. No information could be subtracted about any ongoing PE platforms/processes, or other structural forms of PE on a collective level in any of the papers. Most papers described their efforts as a first step, which needed or could be used in future research.36 39 42 43 45 Furthermore, patient engagement was lacking throughout all papers but one. However, that paper provided no detailed description on patient engagement and its embedment.36 A number of papers (n=4) showed a slightly higher level of embedment as they were part of multiple-year projects in collaboration with governmental bodies.35 42 44 47 Examples were The Public Engagement Project, in collaboration with the US Centre for Disease Control and Prevention, and Fluviews, in collaboration with the Australian Government Department of Health. These projects aimed to obtain input from the public to develop future policy documents, which was indeed realised for The Public Engagement Project by De Kraaij et al (2007).

    Critical appraisal

    The quality of most papers was categorised as low or low to moderate. Two papers were qualified as high quality. More in-depth information can be found in online supplemental file 2. For most papers, it was not possible to extract all necessary data due to a lack of information. This was most apparent in the description of the engagement process and corresponding decisions and assumptions, key terminology and impact of the study. Furthermore, ethical considerations of qualitative research were mostly not properly described or taken into account (no clear distinction could be made due to lack of information). No details were given about considerations concerning researcher–participant relationships, information given to the participants about the goal and impact of the study and issues during the intervention that could have affected the results.

    Discussion

    This review describes how PPE in decision-making in OM has been reported in the literature, concluding with observations about best practices.

    Overall, there seems to be low embedment of collective level PPE in decision-making in OM as a structural activity. Nevertheless, we observed first and one-off efforts applying PE in decision-making on community practice and national policy. PE seemed useful in developing culturally appropriate strategies. In fact, most outcomes were used in direct community practice or in long-term national policy documents. This reflects, to some extent, the added value of PE in decision-making regarding OM. Moreover, in the included papers, the perspective of the public has been studied only in decision-making in risk management and communication, not in risk assessment. Their perspective was always obtained by a range of qualitative methods in various orders, in which patients were almost never engaged. Overall, we found a lack of rich in-depth description of methods and justification for certain terminology and decisions on methodology. These are very important factors in order to increase the reproducibility of PPE in decision-making in OM.

    We have identified a low structural embedment of PE in decision-making in OM on a collective level, while in other types of crises such as natural disasters, higher levels of embedment have been established, through the presence of ongoing engagement platforms with corresponding levels of trust and capacity. Linnel et al (2014) executed a review on successful approaches to public empowerment to enhance crises response abilities, and identified the significance of engagement platforms.49 Loewenson et al (2021) confirmed the importance of establishing social structures, before crises happen, in order to have more proactive and swift community responses. These results were found within context of the execution of the COVID-19 response, not in policy decision-making.50 One important aspect to progress structural PE embedment in decision-making in OM is improvement of both execution and reporting. In our review, relevant information on execution of the engagement process was missing, as well as ethical considerations. Moreover, clear definitions of terminology were lacking. Included papers used the terms ‘engagement’, ‘involvement’, ‘participation’, ‘public’ and ‘community’ interchangeably. This overlap in terms and meanings is apparent in ‘public engagement’ literature and points to the many different contexts, aims and approaches concerning PE (our term of preference).20 51–53 Awareness of this myriad of approaches to shape PE is important and should be considered carefully within the designated context of engagement. We suggest providing the term of preference and a clear definition. Overall, the lack of reporting on engagement process and terms impacts reproducibility and transparency. This subsequently impacts the progress of PE in OM.54

    Further investigation of community versus national approaches for PE in OM is necessary. First, the community approach to PE in decision-making in OM seems to have a direct beneficial impact, as outcomes were mostly directly used to shape local OM. This outcome is in line with findings from others, such as Gilmore et al (2020), who reviewed best practices for engagement during epidemic (and COVID-19) responses.50 55 In particular, engaging communities, described as: ‘groups within a parameter of a social boundary or catchment area’, resulted in context-specific and sustainable solutions. For the papers with a national approach, we were unable to identify the applicability of the recommendations. This leaves the question unanswered whether a community or a national approach, or may be a combination, would be best suitable.

    In the included papers, PE was incorporated into decision-making about risk management and risk communication of OM, and never in risk assessment. The explanation for under-representation of PE in risk assessment is unknown. It could be due to unsuitability of PE in decisions regarding risk assessment from the perspective of experts, patients and the public because of inaccessibility, competency and appropriateness. When considering the possibilities for PPE in risk assessment, parallels can be drawn to the field of natural disaster management. Crichton et al (1990) constructed the Risk Triangle, which explains the measurement, and therefore management, of a risk within the context of a natural hazard. The Risk Triangle depends on three elements: hazard, vulnerability and exposure.56 Vulnerability is defined by Schneiderbauer et al (2004) as ‘The characteristics of a person or a group in terms of their capacity to anticipate, cope with, resist and recover from the impact of a natural or man-made disaster—noting that vulnerability is made up of many political-institutional, economic and socio-cultural factors’.57 In general, various categories of vulnerable groups are already well known. For COVID-19, these are, for example, patients with a mental illness or children experiencing family violence.58 59 Nevertheless, vulnerability varies greatly based on the type of crisis and local conditions. Patients and the public can provide insight into their respective vulnerability such that risk assessment for a specific situation and population can be fine-tuned.60 Furthermore, views of patients and the public could result in more trustworthy and mutually acceptable risk-based decisions overall.61 To conclude on specific possibilities and impact of PPE in risk assessment, as well as risk communication and management, more research is needed.

    In all papers, multiple forms of methodologies were used that facilitated knowledge sharing between stakeholders. This knowledge sharing on OM seems imperative for successful engagement. A few evaluations on the PE process indicated an increased understanding by the participant of the subject at hand and a positive effect on acceptance of the policy. In a handbook of WHO on managing epidemics, establishing a dialogue between stakeholders is highlighted, in order to understand perception and beliefs as a basis for engagement.62 With regard to knowledge sharing and establishing dialogue, we want to highlight the usability of deliberative discussions.63–65 This type of method enriches the participants’ understanding of an issue at hand as well as their own and alternative perspectives. Moreover, there is room for different aspects such as presentations, plenary discussions, small-group discussions and voting rounds.66 67 Another possible method we want to highlight is participatory value evaluation (PVE), as discussed by Mouter et al (2020), which enables a greater reach. In this study (published after our search), PVE was executed during the COVID-19 pandemic in the Netherlands, reaching about 30 000 citizens. Preferences of citizens on pandemic management were gauged, and information about consequences of their theoretical decisions was provided.68

    We found that in all but one of the included papers, the patient perspective was missing. This might be unfortunate, as patients have experiential knowledge about the disease itself, its therapy and general restriction measures. Their input could be valuable to improve therapy and restriction measures, corresponding communication and information efforts, and patient behaviour and compliance.69 70

    The lack of patient perspective might have been caused by some of the methodological choices in this review. Our focus was on engagement on a collective level, whereas, for example, the WHO (2013) advocates for a focus on micro-level engagement with regard to patient engagement in healthcare.71 While recognising the importance of engagement on an individual level and possibilities here-in, this is beyond the scope of the current paper.

    This review has some limitations. Sometimes it was difficult during screening to assess whether engagement was executed in decision-making or only in execution of control measures or communication efforts. To minimise errors, this criterion was clarified with examples, and an extra independent check was done. Furthermore, for some papers, it was difficult to assess whether the outbreak situation corresponded with an endemic situation or not. Some parts of the screening, such as title/abstract and other sources, were done by one researcher. Bias by partly solo-screening could have resulted in exclusion of relevant records. Other limitations are expected with regard to the search strategy. We included specific infectious disease outbreaks in our search, chosen due to notoriety and scale. By including not all types of infections, engagement efforts might have been overlooked. Likewise for papers in other languages than English or Dutch. Furthermore, by only focusing on broad engagement terms or collective engagement, structural forms of engagement on an individual level could have been overlooked. As stated before, this would mostly have consequences for patient engagement. We have made clear that our conclusions on structural embedment only apply for the collective level. Moreover, it is possible that the structural forms of PPE are already existent in practice but merely not detectable in literature, or not yet available due to the ongoing status of some outbreaks, with as most obvious example the COVID-19 pandemic.

    Lastly, we want to acknowledge the complexity of PPE and myriad of terminology. The search was set up as broad as possible, in order to capture the maximum amount of relevant literature. However, we are aware that potentially relevant engagement efforts have been missed and accept that we may not have a complete picture of all approaches of PPE in decision-making regarding OM globally.

    Conclusion

    We want to formulate several recommendations for future efforts based on our results. The use of deliberative discussions, such as forums, seems promising, feasible and delivers interesting results which can be used in shaping decision-making in OM on a collective level. It is of great importance that evaluations are more frequently conducted. These evaluations not only gauge the perspective of the public on OM and the process of engagement, but also help to assess its impact. However, the exact impact of PE in OM remains difficult to assess, which makes it a valuable focus for future research. More research and clarity on the exact impact of PPE in decision-making in OM could translate into agenda-setting and ultimately even global guidelines. As in current guidelines, there is a focus on detection, mitigation and delivery or transfer of information during outbreaks. Besides, PPE is often viewed as a means to increase acceptance of measures, whereas it could be much more, as explained in the introduction.

    Furthermore, we want to highly recommend better reporting, and more awareness on ethical aspects. It would also be interesting to study the possibilities to develop long-term relationships with communities to possibly increase the level of structural embedment of engagement in OM.

    Overall, it has become apparent that PE in decision-making in OM on a collective level can be improved with respect to structural embedment, rigour of conduct and specification of terms and definitions. In this review, we have identified and reported these specific aspects of improvement as accurately as possible, in order to take a step forward and advocate for more PPE in decision-making in OM. Altogether, we strongly believe that it could result in improved legitimacy, quality and effectiveness of OM.

    Data availability statement

    Data are available upon reasonable request. Data extracted from the papers are directly available from the published papers. Data extraction table for the summary and assessment is available upon request.

    Ethics statements

    Patient consent for publication

    Ethics approval

    Approval for this integrative review research was not necessary as no confidential information from participants was collected on an individual level. Permission to use the IAP2 Spectrum of Public Participation was obtained from the International Association for Public Participation.

    Acknowledgments

    We would like to thank Doret de Rooij for her help with the discussions on eligibility of papers, and Dr Gea Dreschler for her help in editing the manuscript. Lastly, we want to acknowledge the inspiring work of the organisations, researchers, stakeholders and members of the public in the papers.

    References

    1. ↵
      1. World Health Organization
      . The top 10 causes of death 2020 [internet]. Available: https://www.who.int/news-room/fact-sheets/detail/the-top-10-causes-of-death [Accessed 18thDec 2020].
    2. ↵
      1. Smith KF,
      2. Goldberg M,
      3. Rosenthal S, et al
      . Global rise in human infectious disease outbreaks. J R Soc Interface 2014;11:20140950. doi:10.1098/rsif.2014.0950pmid:http://www.ncbi.nlm.nih.gov/pubmed/25401184
      OpenUrlCrossRefPubMed
    3. ↵
      1. Guo C,
      2. Zhou Z,
      3. Wen Z, et al
      . Global epidemiology of dengue outbreaks in 1990-2015: a systematic review and meta-analysis. Front Cell Infect Microbiol 2017;7:317. doi:10.3389/fcimb.2017.00317pmid:http://www.ncbi.nlm.nih.gov/pubmed/28748176
      OpenUrlPubMed
    4. ↵
      1. Suseelan Bhargavi B,
      2. Moa A
      . Global outbreaks of Zika infection by epidemic observatory (EpiWATCH), 2016-2019. Global Biosecurity 2020;2:None.
      OpenUrl
    5. ↵
      1. WHO
      . Ebola virus disease – Democratic Republic of the Congo, 2020. Available: https://www.who.int/csr/don/30-january-2020-ebola-drc/en/
    6. ↵
      1. Srivastava N,
      2. Baxi P,
      3. Ratho RK
      . Global trends in epidemiology of coronavirus disease 2019 (COVID-19). coronavirus disease 2019 (COVID-19, 2020: 9–21.
    7. ↵
      1. Erdelj M,
      2. Król M,
      3. Natalizio E
      . Wireless sensor networks and Multi-UAV systems for natural disaster management. Computer Networks 2017;124:72–86.doi:10.1016/j.comnet.2017.05.021
      OpenUrl
    8. ↵
      1. Mojtahedi M,
      2. Oo BL
      . Critical attributes for proactive engagement of stakeholders in disaster risk management. International Journal of Disaster Risk Reduction 2017;21:35–43.doi:10.1016/j.ijdrr.2016.10.017
      OpenUrl
    9. ↵
      1. Fakhruddin BS,
      2. Blanchard K,
      3. Ragupathy D
      . Are we there yet? the transition from response to recovery for the COVID-19 pandemic. Prog Disaster Sci 2020;7:100102. doi:10.1016/j.pdisas.2020.100102pmid:http://www.ncbi.nlm.nih.gov/pubmed/34171013
      OpenUrlPubMed
    10. ↵
      1. Philips M,
      2. Markham A
      . Ebola: a failure of international collective action. Lancet 2014;384:1181. doi:10.1016/S0140-6736(14)61606-8pmid:http://www.ncbi.nlm.nih.gov/pubmed/25218774
      OpenUrlPubMed
    11. ↵
      1. Ryschon TW
      . Ebola control measures and inadequate responses. Lancet 2014;384:1181–2.doi:10.1016/S0140-6736(14)61346-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/25217114
      OpenUrlPubMed
    12. ↵
      1. WHO
      . Risk assessment [internet]. Available: https://www.who.int/foodsafety/risk-analysis/riskassessment/en/ [Accessed 2ndFeb 2021].
    13. ↵
      1. WHO
      . Risk management [internet]. Available: https://www.who.int/foodsafety/risk-analysis/risk-management/en/ [Accessed 2ndFeb 2021].
    14. ↵
      1. WHO
      . Risk communication. Available: https://www.who.int/foodsafety/risk-analysis/riskcommunication/en/ [Accessed 2ndFeb 2021].
    15. ↵
      1. Marston C,
      2. Renedo A,
      3. Miles S
      . Community participation is crucial in a pandemic. Lancet 2020;395:1676–8.doi:10.1016/S0140-6736(20)31054-0pmid:http://www.ncbi.nlm.nih.gov/pubmed/32380042
      OpenUrlPubMed
    16. ↵
      1. Weizhen Dong D
      . Beyond SARS: ethnic community organization's role in public health -- a Toronto experience. Promot Educ 2008;15:53–5.doi:10.1177/1025382308097699pmid:http://www.ncbi.nlm.nih.gov/pubmed/19066240
      OpenUrlCrossRefPubMed
    17. ↵
      1. World Health Organization
      . Health for all series, number 1. Alma-Ata 1978. primary health care. Geneva: WHO, 1978.
    18. ↵
      1. Tritter JQ
      . Revolution or evolution: the challenges of conceptualizing patient and public involvement in a consumerist world. Health Expect 2009;12:275–87.doi:10.1111/j.1369-7625.2009.00564.xpmid:http://www.ncbi.nlm.nih.gov/pubmed/19754691
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Caron-Flinterman F
      . A new voice in science – patient participation in decision making on biomedical research, 2005.
    20. ↵
      1. International Association for Public Participation
      . IAP2’s Public Participation Spectrum. IAP2 Federation, 2014. Available: http://c.ymcdn.com/sites/www.iap2.org/resource/resmgr/foundations_course/IAP2_P2_Spectrum_FI NAL.pdf
    21. ↵
      1. Oluwatoyin Folayan M,
      2. Haire B
      . Communitarian societies and public engagement in public health. Crit Public Health 2017;27:6–13.doi:10.1080/09581596.2016.1252035
      OpenUrl
    22. ↵
      1. Mackworth-Young CR,
      2. Chingono R,
      3. Mavodza C, et al
      . Community perspectives on the COVID-19 response, Zimbabwe. Bull World Health Organ 2021;99:85–91.doi:10.2471/BLT.20.260224pmid:http://www.ncbi.nlm.nih.gov/pubmed/33551502
      OpenUrlPubMed
    23. ↵
      1. Bhatt N,
      2. Bhatt B,
      3. Gurung S, et al
      . Perceptions and experiences of the public regarding the COVID-19 pandemic in Nepal: a qualitative study using phenomenological analysis. BMJ Open 2020;10:e043312. doi:10.1136/bmjopen-2020-043312pmid:http://www.ncbi.nlm.nih.gov/pubmed/33310812
      OpenUrlPubMed
    24. ↵
      1. Williams SN,
      2. Armitage CJ,
      3. Tampe T, et al
      . Public perceptions and experiences of social distancing and social isolation during the COVID-19 pandemic: a UK-based focus group study. BMJ Open 2020;10:e039334-e. doi:10.1136/bmjopen-2020-039334pmid:http://www.ncbi.nlm.nih.gov/pubmed/32690752
      OpenUrlPubMed
    25. ↵
      1. World Health Organization
      . Risk communication and community engagement (RCCE) considerations: Ebola response in the Democratic Republic of the Congo.
    26. ↵
      1. European Center for Disease Control
      . Community engagement for public health events caused by communicable disease threats in the EU/EEA, 2020.
    27. ↵
      1. Gillespie AM,
      2. Obregon R,
      3. El Asawi R, et al
      . Social mobilization and community engagement central to the Ebola response in West Africa: lessons for future public health emergencies. Glob Health Sci Pract 2016;4:626–46.doi:10.9745/GHSP-D-16-00226pmid:http://www.ncbi.nlm.nih.gov/pubmed/28031301
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Whittemore R,
      2. Knafl K
      . The integrative review: updated methodology. J Adv Nurs 2005;52:546–53.doi:10.1111/j.1365-2648.2005.03621.xpmid:http://www.ncbi.nlm.nih.gov/pubmed/16268861
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Centers for Disease Control and Prevention
      . Introduction to epidemiology: epidemic disease occurrence, 2012.
    30. ↵
      1. Ouzzani M,
      2. Hammady H,
      3. Fedorowicz Z, et al
      . Rayyan-a web and mobile APP for systematic reviews. Syst Rev 2016;5:210. doi:10.1186/s13643-016-0384-4pmid:http://www.ncbi.nlm.nih.gov/pubmed/27919275
      OpenUrlCrossRefPubMed
    31. ↵
      1. The EndNote Team
      . Endnote. endnote X9 ED. Philadelphia, PA: Clarivate, 2013.
    32. ↵
      1. Elena Fagotto AF
      . Sustaining public engagement: embedded deliberation in local communities. everyday democracy, the Kettering Foundation, 2009.
    33. ↵
      1. Critical Appraisal Skills Programme
      . Casp qualitative checklist, 2018. Available: https://casp-uk.net/wp-content/uploads/2018/01/CASP-Qualitative-Checklist-2018.pd [Accessed 29th Nov 2020].
    34. ↵
      1. Critical Appraisal Skills Programme
      . Casp randomised controlled trial checklist, 2018. Available: https://casp-uk.net/wp-content/uploads/2018/01/CASP-Randomised-Controlled-Trial-Checklist-2018.pdf [Accessed 10th Dec 2020].
    35. ↵
      1. Publications of the University of Nebraska Public Policy Center
      . The public engagement project on community control measures for pandemic influenza findings and recommendations from citizen and Stakeholder deliberation days. Available: https://digitalcommons.unl.edu/publicpolicypublications/107
    36. ↵
      1. Andersson N,
      2. Nava-Aguilera E,
      3. Arosteguí J, et al
      . Evidence based community mobilization for dengue prevention in Nicaragua and Mexico (Camino Verde, the green way): cluster randomized controlled trial. BMJ 2015;351:h3267. doi:10.1136/bmj.h3267pmid:http://www.ncbi.nlm.nih.gov/pubmed/26156323
      OpenUrlAbstract/FREE Full Text
    37. ↵
      1. McNaughton D
      . The importance of long-term social research in enabling participation and developing engagement strategies for new dengue control technologies. PLoS Negl Trop Dis 2012;6:e1785. doi:10.1371/journal.pntd.0001785pmid:http://www.ncbi.nlm.nih.gov/pubmed/22953011
      OpenUrlPubMed
    38. ↵
      1. McNaughton D,
      2. Duong TTH
      . Designing a community engagement framework for a new dengue control method: a case study from central Vietnam. PLoS Negl Trop Dis 2014;8:e2794. doi:10.1371/journal.pntd.0002794pmid:http://www.ncbi.nlm.nih.gov/pubmed/24853391
      OpenUrlPubMed
    39. ↵
      1. Sanchez L,
      2. Perez D,
      3. Cruz G, et al
      . Intersectoral coordination, community empowerment and dengue prevention: six years of controlled interventions in Playa Municipality, havana, Cuba. Trop Med Int Health 2009;14:1356–64.doi:10.1111/j.1365-3156.2009.02379.xpmid:http://www.ncbi.nlm.nih.gov/pubmed/19840350
      OpenUrlCrossRefPubMedWeb of Science
    40. ↵
      1. Alonge O,
      2. Sonkarlay S,
      3. Gwaikolo W, et al
      . Understanding the role of community resilience in addressing the Ebola virus disease epidemic in Liberia: a qualitative study (community resilience in Liberia). Glob Health Action 2019;12:1662682. doi:10.1080/16549716.2019.1662682pmid:http://www.ncbi.nlm.nih.gov/pubmed/31507254
      OpenUrlPubMed
    41. ↵
      1. Brittain AW,
      2. August EM,
      3. Romero L, et al
      . Community perspectives on contraception in the context of the Zika virus in the U.S. virgin islands: implications for communication and messaging. Womens Health Issues 2019;29:245–51.doi:10.1016/j.whi.2019.01.007pmid:http://www.ncbi.nlm.nih.gov/pubmed/30878263
      OpenUrlPubMed
    42. ↵
      1. Braunack-Mayer AJ,
      2. Street JM,
      3. Rogers WA, et al
      . Including the public in pandemic planning: a deliberative approach. BMC Public Health 2010;10:501. doi:10.1186/1471-2458-10-501pmid:http://www.ncbi.nlm.nih.gov/pubmed/20718996
      OpenUrlPubMed
    43. ↵
      1. Charania NA,
      2. Tsuji LJS
      . A community-based participatory approach and engagement process creates culturally appropriate and community informed pandemic plans after the 2009 H1N1 influenza pandemic: remote and isolated first nations communities of sub-arctic Ontario, Canada. BMC Public Health 2012;12:268. doi:10.1186/1471-2458-12-268pmid:http://www.ncbi.nlm.nih.gov/pubmed/22472012
      OpenUrlCrossRefPubMed
    44. ↵
      1. Fain BA,
      2. Koonin LM,
      3. Stoto MA, et al
      . Facilitating access to antiviral medications and information during an influenza pandemic: engaging with the public on possible new strategies. Biosecur Bioterror 2014;12:8–19.doi:10.1089/bsp.2013.0058pmid:http://www.ncbi.nlm.nih.gov/pubmed/24552360
      OpenUrlPubMed
    45. ↵
      1. Massey PD,
      2. Miller A,
      3. Saggers S, et al
      . Australian Aboriginal and Torres Strait Islander communities and the development of pandemic influenza containment strategies: community voices and community control. Health Policy 2011;103:184–90.doi:10.1016/j.healthpol.2011.07.004pmid:http://www.ncbi.nlm.nih.gov/pubmed/21868121
      OpenUrlPubMed
    46. ↵
      1. Biddison ELD,
      2. Gwon HS,
      3. Schoch-Spana M, et al
      . Scarce resource allocation during disasters: a Mixed-Method community engagement study. Chest 2018;153:187–95.doi:10.1016/j.chest.2017.08.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/28802695
      OpenUrlCrossRefPubMed
    47. ↵
      1. Rogers WA,
      2. Street JM,
      3. Braunack-Mayer AJ, et al
      . Pandemic influenza communication: views from a deliberative forum. Health Expect 2009;12:331–42.doi:10.1111/j.1369-7625.2009.00562.xpmid:http://www.ncbi.nlm.nih.gov/pubmed/19754694
      OpenUrlPubMed
    48. ↵
      1. Centers for Disease Control and Prevention
      . Interim Pre-pandemic planning guidance: community strategy for pandemic influenza mitigation in the United States, 2007.
    49. ↵
      1. Linnell M,
      2. Johansson C
      . A literature review on community approaches that involve the public in crisis management: fostering community resilience through coproduction by response organisations and citizens, 2014.
    50. ↵
      1. Loewenson R,
      2. Colvin CJ,
      3. Szabzon F, et al
      . Beyond command and control: a rapid review of meaningful community-engaged responses to COVID-19. Glob Public Health 2021;16:1–15.doi:10.1080/17441692.2021.1900316pmid:http://www.ncbi.nlm.nih.gov/pubmed/33734007
      OpenUrlPubMed
    51. ↵
      1. Rowe G,
      2. Frewer LJ
      . A typology of public engagement mechanisms. Sci Technol Human Values 2005;30:251–90.doi:10.1177/0162243904271724
      OpenUrlCrossRefWeb of Science
    52. ↵
      1. Arnstein SR
      . A ladder of citizen participation. J Am Inst Plann 1969;35:216–24.doi:10.1080/01944366908977225
      OpenUrlCrossRef
    53. ↵
      1. National Co-ordinating Centre for Public Engagement
      . What is public engagement? [internet], 2020. Available: https://www.publicengagement.ac.uk/about-engagement/what-public-engagement [Accessed 11th Apr 2021].
    54. ↵
      1. Prager EM,
      2. Chambers KE,
      3. Plotkin JL, et al
      . Improving transparency and scientific rigor in academic publishing. Brain Behav 2019;9:e01141. doi:10.1002/brb3.1141pmid:http://www.ncbi.nlm.nih.gov/pubmed/30506879
      OpenUrlPubMed
    55. ↵
      1. Gilmore B,
      2. Ndejjo R,
      3. Tchetchia A, et al
      . Community engagement for COVID-19 prevention and control: a rapid evidence synthesis. BMJ Glob Health 2020;5:e003188. doi:10.1136/bmjgh-2020-003188pmid:http://www.ncbi.nlm.nih.gov/pubmed/33051285
      OpenUrlAbstract/FREE Full Text
    56. ↵
      1. Crichton D
      . The risk triangle. natural disaster management. Tudor rose, 1999: 102–3.
    57. ↵
      1. Schneiderbauer S,
      2. Ehrlich D
      . Risk, Hazard and People’s Vulnerability to Natural Hazards: a Review of Definitions, Concepts and Data, 2004.
    58. ↵
      1. Druss BG
      . Addressing the COVID-19 pandemic in populations with serious mental illness. JAMA Psychiatry 2020;77:891–2.doi:10.1001/jamapsychiatry.2020.0894pmid:http://www.ncbi.nlm.nih.gov/pubmed/32242888
      OpenUrlPubMed
    59. ↵
      1. Usher K,
      2. Bhullar N,
      3. Durkin J, et al
      . Family violence and COVID-19: increased vulnerability and reduced options for support. Int J Ment Health Nurs 2020;29:549–52.doi:10.1111/inm.12735pmid:http://www.ncbi.nlm.nih.gov/pubmed/32314526
      OpenUrlPubMed
    60. ↵
      1. Osborne J,
      2. Paget J,
      3. Napier D, et al
      . Addressing vulnerabilities in communities facing infectious disease threats: a need for social science-driven assessments. J Glob Health 2021;11:03003. doi:10.7189/jogh.11.03003pmid:http://www.ncbi.nlm.nih.gov/pubmed/33604031
      OpenUrlPubMed
    61. ↵
      1. Sexton K
      . Evolution of public participation in the assessment and management of environmental health risks: a brief history of developments in the United States. J Public Health Res 2013;2:e18. doi:10.4081/jphr.2013.e18pmid:http://www.ncbi.nlm.nih.gov/pubmed/25170489
      OpenUrlPubMed
    62. ↵
      1. World Health Organization
      . Managing epidemics: key facts about major deadly diseases. License: CC BY-NC-SA 3.0 IGO. World Health Organization, 2018. https://apps.who.int/iris/handle/10665/272442
    63. ↵
      1. Rothwell E,
      2. Anderson R,
      3. Botkin JR
      . Deliberative discussion focus groups. Qual Health Res 2016;26:734–40.doi:10.1177/1049732315591150pmid:http://www.ncbi.nlm.nih.gov/pubmed/26078330
      OpenUrlCrossRefPubMed
    64. ↵
      1. Setälä M
      . The role of deliberative mini-publics in representative democracy: lessons from the experience of referendums. Representation 2011;47:201–13.doi:10.1080/00344893.2011.581080
      OpenUrl
    65. ↵
      1. Jacquet V,
      2. van der Does R
      . Deliberation and Policy-Making: Three Ways to Think About Minipublics’ Consequences. Adm Soc 2021;53:468–87.doi:10.1177/0095399720964511
      OpenUrl
    66. ↵
      1. O'Doherty K,
      2. Gauvin F-P,
      3. Grogan C, et al
      . Implementing a public deliberative forum. Hastings Cent Rep 2012;42:20–3.doi:10.1002/hast.28pmid:http://www.ncbi.nlm.nih.gov/pubmed/22733326
      OpenUrlPubMed
    67. ↵
      1. Carnegie Mellon University
      . A Handbook for Deliberative community forums.
    68. ↵
      1. Mouter N,
      2. Hernandez JI,
      3. Itten AV
      . Public participation in crisis policymaking. How 30,000 Dutch citizens advised their government on relaxing COVID-19 lockdown measures. medRxiv 2020:2020.11.09.20228718.
    69. ↵
      1. Rawson TM,
      2. Moore LSP,
      3. Hernandez B, et al
      . Patient engagement with infection management in secondary care: a qualitative investigation of current experiences. BMJ Open 2016;6:e011040. doi:10.1136/bmjopen-2016-011040pmid:http://www.ncbi.nlm.nih.gov/pubmed/27799238
      OpenUrlAbstract/FREE Full Text
    70. ↵
      1. Di Giacomo D
      . Public health emergencies and quarantine: virtual patient engagement as challenge and opportunity for mental health strategy. Mediterranean Journal of Clinical Psychology 2020;8:2020.
      OpenUrl
    71. ↵
      1. World Health Organization. Regional Office for Europe
      . Exploring patient participation in reducing health-care-related safety risks. World Health organization. regional office for Europe, 2013. Available: https://apps.who.int/iris/handle/10665/326442
    72. ↵
      1. World Health Organization & Food and Agriculture Organization of the United Nations
      . Risk communication applied to food safety: Handbook. World Health organization, 2016. Available: https://apps.who.int/iris/handle/10665/250083

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Handling editor Seye Abimbola

    • Contributors SK contributed to study design, conducted the review and analysis and prepared the manuscript. SK is responsible for the overall content as guarantor. MEJB contributed to study design, analysis of the review and manuscript preparation. ENES helped with review analysis and advised on the manuscript. LJS helped with systematic search design, executed the search and advised on the manuscript. JFHK and AT contributed to study design, advised on data analysis and helped with manuscript preparation.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.