Article Text
PDF
Statistics from Altmetric.com
Introduction
The 2030 Agenda for Sustainable Development calls for real transformation, recognising that health goes beyond survival to include human rights, equity and the empowerment of vulnerable populations, including women and children.1 This Agenda demands strategies to address the underlying causes of ill health and inequity to achieve sustained improvements in health by ensuring healthy lives and promoting well-being for all at all ages. Within this context, governments and programmes struggle to make evidence-informed decisions to achieve these ambitious goals, while embracing these values.
Current processes for developing evidence-informed guidance in public health encompass scoping and formulation of key questions; evidence retrieval, synthesis and appraisal; and the formulation of recommendations. These methods were originally conceived for clinical interventions as part of the evidence-based medicine movement.2 In public health these processes are applied to a broad range of health interventions implemented across varied health systems and contexts where a myriad of factors act both directly and indirectly to impact health and broader societal outcomes. Importantly, policy-makers pose questions beyond those of efficacy and safety and need guidance on the best ways to deliver interventions. Thus developers of evidence-informed guidance often apply processes and methods designed originally for assessing the comparative effectiveness of clinical interventions that are ill-adapted to formulating recommendations on highly context-dependent public health and health system interventions.
A core function of World Health Organization (WHO) is to develop guidelines that set forth recommendations designed to support policy-makers and programme managers, particularly in low-income and middle-income countries, in making informed decisions about clinical practice or public health issues. WHO follows a transparent and rigorous process for developing evidence-informed guidelines.3 However, this process currently does not give adequate consideration to relevant aspects of complexity in health interventions or to interventions delivered in complex systems where outcomes occur at the individual, population and/or system levels. Addressing complexity creates challenges at all steps of the guideline development process, for example in framing the key questions, identifying relevant types of evidence, synthesising qualitative and quantitative data,4–7 assessing applicability of existing studies to both broader or more specific contexts,8 and formulating impactful recommendations. This suggests that standard guideline development methods may need to be enhanced to better address and reflect complexity.
Purpose and scope of the series
In order to address the challenges and realities of public health and health system interventions and to better meet the needs of decision-makers, in 2016 WHO initiated a project to strengthen its processes and methods for developing guidelines on complex health interventions and interventions delivered in complex systems. The work led to this series of papers which contribute to the broader conceptualisation of complexity and the implications for evidence synthesis and guideline development, whether at the global, national or health systems level. Ultimately, this work is intended to lead to more comprehensive and informative evidence syntheses and trustworthy and impactful guidelines that meet critical health needs for countries and for varied contexts, and to stimulate rigorous primary research on a broad range of policy-relevant questions.
This work is intended for use by both systematic reviewers and guideline developers where a complexity perspective is anticipated to be useful to decision-makers, programme planners and implementers. It will also be used to strengthen the WHO guideline development process and form the basis of more practical support for WHO staff who develop guidelines. This series is of particular relevance to public health and health systems interventions, but is also likely to add value to those clinical/healthcare interventions characterised by complexity.
Development of the series
A steering group at WHO, convened by AP and SLN, led the process, setting up work groups encompassing key topics and facilitating frequent consultations within and across work groups. The steering group identified leading international experts in a variety of relevant disciplines, including evidence synthesis, guideline development, knowledge translation and uptake, the social and behavioural sciences, and decision-making and programme implementation in public health, among others. Key organisations working in this area were contacted, and representatives from the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group, Cochrane, the Campbell Collaboration, and the US Agency for Healthcare Research and Quality participated in the work groups and in meetings.
An initial WHO internal meeting was held in February 2016, followed by a scoping meeting with external experts in Geneva, Switzerland (August 2016). The work plan evolved through a consensus-based and iterative process, and work groups were formed on key topic areas, each with a general scope and charged with developing one or several of the manuscripts in this series. Central concepts were discussed, such as the optimal structure of key questions underpinning systematic reviews (does a complexity perspective need a framework different from PICO (population, intervention, comparator, outcome)?) and whether the focus should be on complex (vs ‘simple’) interventions in contrast to interventions (of any type) situated within a given context (a complex system). External reviewers provided comments on each paper at face-to-face meetings of all work groups in Munich, Germany (January 2017)and Geneva, Switzerland (July 2017), ensuring that the focus of each paper was clear and messaging was consistent across papers. Draft manuscripts were extensively peer-reviewed prior to journal submission, including by individuals with diverse perspectives, backgrounds, affiliations and with global representation.
Series overview
The eight papers in this series are directed at the steps in the WHO guideline development process as summarised in figure 1. The first four papers explore the concepts and set the stage for the subsequent papers which address more specific methodological issues. Petticrew and colleagues9 summarise key concepts and distinguish between complex interventions and complex systems - defining this as a ‘complexity perspective’. The authors then describe how the systematic review or guideline team need to consider a theory of change, causal pathways, complex systems properties and context when conceptualising a review, and when interpreting the evidence and formulating recommendations. Based on the aspects of complexity that are of interest, the paper suggests how research questions might be framed and the types of evidence that can help answer these questions.
Papers in this series according to the guideline development process.
The paper by Rehfuess et al10 is depicted twice as the WHO-INTEGRATE evidence-to-decision framework plays a critical role both at the beginning of the guideline development process when scoping the guideline and formulating questions, and at the end of the process when integrating different types of evidence and formulating recommendations.
In the next paper, Booth et al11 expand on the importance of considering the context in which an intervention is implemented when synthesising evidence and formulating recommendations, the challenges faced when considering context, and the extent to which current systematic review and guideline development tools are helpful in meeting these challenges.
Rehfuess and colleagues10 propose a new evidence-to-decision framework rooted in WHO’s norms and values, reflecting the changing public and global health landscape and suitable for a broad range of health interventions and complexity features. This comprises six substantive criteria as well as the meta-criterion quality of evidence; each criterion is accompanied by a definition, example questions as well as suggestions for primary research and evidence synthesis and methods to assess the quality of the evidence. This paper further highlights the importance of considering criteria beyond benefits and harms of an intervention and is intended to ensure that guideline development groups reflect on all relevant evidence at the onset.
Focusing on two of the above-mentioned substantive criteria, Booth et al12 discuss formulating questions to address the acceptability and feasibility of complex interventions, particularly when qualitative evidence synthesis is likely to be informative. The authors review existing frameworks for formulating such questions from a complexity perspective and propose a comprehensive framework for further testing.
The next set of three papers in the series discusses evidence synthesis and the implications of taking a complexity perspective when synthesising quantitative, qualitative and mixed-methods evidence. In the first of these papers, Higgins and colleagues13 describe both the statistical and non-statistical approaches to tackling a complexity perspective in a synthesis of quantitative data in a systematic review, and provide practical guidance on how to select specific approaches. Flemming et al14 focus on qualitative evidence synthesis and the approaches available when developing guidelines in light of a complexity perspective. Noyes et al15 address the contribution of mixed-methods primary research and the integration of qualitative and quantitative data in research synthesis, as well as the use of such evidence to inform recommendations. Qualitative and quantitative research approaches frequently complement each other, particularly for questions of intervention feasibility and acceptability. This paper also examines the special considerations for systematic reviewers and guideline developers when deciding whether mixed-method reviews will add value.
The final paper in the series considers the implications of complexity for the assessment of the quality of the evidence. Montgomery and coauthors16 address the challenges inherent in taking a complexity perspective when using the GRADE system to assess the certainty of a body of evidence in a systematic review or guideline. Key issues include consideration of context and implementation when framing key questions; defining certainty such that it reflects context; and how coherence of evidence across a causal pathway can be incorporated into GRADE assessments.
This series in the context of prior work
The last few years have witnessed much activity in relation to the methods for assessing and synthesising evidence and formulating recommendations on complex interventions, which the current series builds on. A series of papers published in 2013 in the Journal of Clinical Epidemiology examined various aspects of complexity relevant to systematic reviews of interventions.17 The authors of papers in this series considered the conceptual, analytical and methodological aspects of complexity in systematic reviews, distinguishing between complexity of the intervention, implementation, the context and the participant responses. A more recent series published by the US Agency for Healthcare Research and Quality in 2017 on complex intervention systematic reviews18 provides practical guidance and tools for reviews of complex interventions, which complements the more conceptual papers presented in this series.
The Context and Implementation of Complex Interventions (CICI) framework is a useful tool for considering the context, implementation and setting of complex interventions in an integrated way to understand whether and how interventions work.19 Another recently published tool, the intervention Complexity Assessment Tool for Systematic Reviews (iCAT-SR),20 presents six dimensions which help reviewers to describe and categorise levels of intervention complexity and think about how complexity might be incorporated into each stage of the review process. This tool is considered in the two papers by Booth and colleagues.11 12
Recent developments in grading of evidence include the GRADE CERQual (Confidence in the Evidence from Reviews of Qualitative research) approach,21 which provides a standardised framework for assessing the level of confidence that decision-makers should have in a body of qualitative evidence. A series published in 2018 in Implementation Science provides detailed guidance on how to apply the approach.22 The GRADE Working Group is developing methods for assessing the certainty of evidence on complex interventions;23 the challenges and suggestions articulated in the paper in this series by Montgomery et al16 will be used to inform GRADE guidance in this area.
Implications for reviewers and guideline developers
This series advances the application of a complexity perspective to systematic reviews and to all stages of guideline development in the fields of public health and health systems. We focus on complex adaptive systems and on the diverse contexts in which a health intervention is implemented, rather than on the intervention itself, whether ‘simple’ or ‘complex’. This series delineates approaches and tools that can help reviewers and guideline developers take complexity into account, while outlining a research agenda.
In order to benefit from a complexity perspective in the evaluation of intervention effectiveness, we need continued close collaboration among social, organisational, implementation and systems scientists, along with epidemiologists, evaluation specialists, and qualitative and quantitative researchers. Epidemiology has been instrumental in developing methods used to evaluate and synthesise effects of interventions, yet further advances in making sense of complex interventions will require contributions from multiple disciplines. In addition, to really add value to systematic review and guideline development processes, a complexity perspective should reflect multiple stakeholders’ needs and perspectives, including healthcare providers, programme managers, communities and individuals, as they are critical to understanding and implementing interventions in complex systems.
In our call to better consider a complexity perspective when developing evidence-informed guidance on public health and health systems interventions, we recognise the need to be both strategic and pragmatic. While the principle of considering complexity is worthy, incorporating the full spectrum of conceptual and methodological considerations is not possible in every systematic review and guideline. If and when to address different dimensions of complexity for specific interventions needs further consideration, and the proposed methods to do so require practical guidance for their application. We do not yet know how a complexity perspective may benefit end-users of systematic reviews and guidelines, and ultimately the beneficiaries of evidence-informed recommendations. Novel approaches to considering complexity in systematic reviews and guidelines need to be evaluated and to evolve as evidence emerges. WHO as a producer of guidelines used across a broad range of settings and health systems is in a strong position to lead this work through its own processes, and in collaboration with partners can help advance the science of adopting a complexity perspective in systematic reviews and guideline development. We hope that this series of papers provides a starting point for future work in this area.
Acknowledgments
Numerous external reviewers were involved in the different meetings held including Robert Balster, Mauricio Beller Ferri, Ketan Chitnis, Karen Daniels, Matthias Egger, Maicon Falavigna, Asha George, Ansgar Gerhardus, Davina Ghersi, Jeanne-Marie Guise, Elizabeth Higgs, Steve Hodgins, Zohra Lassi, Cicely Marston, Melanie Marti, Lorenzo Moja, Pierre Ongolo-Zogo, Deepak Paudel, Amir Qaseem, Neena Raina, Nigel Rollins, David Ross, Jonathan Simon, Slim Slama and Chi-Chi Undie. Elizabeth Fox, Stephanie Levy and Marianne Monclair participated as observers. WHO gratefully acknowledges their time and dedication reviewing this series of papers; the series has greatly benefited from their thoughtful input. Figure 1 was designed by Lushomo Communications.
Footnotes
Handling editor Soumyadeep Bhaumik
Contributors SLN and AP drafted this manuscript. EAR, HS, ÖT, NPF and JMG provided comments on drafts prior to completion and submission. All authors approved the final manuscript. SLN and AP are the guarantors of this work.
Funding Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation.
Disclaimer SLN, AP, ÖT and NPF are staff members of WHO. The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions or policies of WHO.
Competing interests SLN, EAR and NPF are members of the GRADE Working Group, whose framework and methods for guideline development are used by the WHO staff developing guidelines. JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake. The authors have no other interests to disclose.
Patient consent Not required.
Provenance and peer review Not commissioned; internally peer reviewed.
Data sharing statement No additional data are available.