Table 1

Designs and methods and their use or applicability in guidelines and systematic reviews taking a complexity perspective

Case study examples and referencesComplexity-related questions of interest in the guidelineTypes of synthesis used in the guidelineMixed-method review design and integration mechanismsObservations, concerns and considerations
A. Mixed-method review designs used in WHO guideline development
Antenatal Care (ANC) guidelines (online supplementary file 1)
14 15 27 45 48 49
What do women in high-income, medium-income and low-income countries want and expect from antenatal care (ANC), based on their own accounts of their beliefs, views, expectations and experiences of pregnancy?Qualitative synthesis
Framework synthesis
Meta-ethnography
Segregated and contingent design and sequential synthesis
Quantitative and qualitative reviews undertaken separately (segregated), an initial scoping review of qualitative evidence established women’s preferences and outcomes for ANC, which informed design of the quantitative intervention review (contingent)
A second qualitative evidence synthesis was undertaken to look at implementation factors (sequential)
Integration: quantitative and qualitative findings were brought together in a series of DECIDE frameworks Tools included:
Psychological theory
SURE framework conceptual framework for implementing policy options
Conceptual framework for analysing integration of targeted health interventions into health systems to analyse contextual health system factors
An innovative approach to guideline development
No formal cross-study synthesis process and limited testing of theory. The hypothetical nature of meta-ethnography findings may be challenging for guideline panel members to process without additional training
See Flemming et al 14 for considerations when selecting meta-ethnography
What are the evidence-based practices during ANC that improved outcomes and lead to positive pregnancy experience and how should these practices be delivered?Quantitative review of trials
Factors that influence the uptake of routine antenatal services by pregnant women
Views and experiences of maternity care providers
Qualitative synthesis
Framework synthesis
Meta-ethnography
Task shifting guidelines (online supplementary file 2)15 24 27 45 54 What are the effects of lay health worker interventions in primary and community healthcare on maternal and child health and the management of infectious diseases?Quantitative review of trials Combination of a segregated design and sequential synthesis
Several published quantitative reviews were used (eg, Cochrane review of lay health worker interventions)
Additional new qualitative evidence syntheses were commissioned (segregated)

Integration: quantitative and qualitative review findings on lay health workers were brought together in several DECIDE frameworks. Tools included adapted SURE Framework and post hoc logic model
An innovative approach to guideline development
The post hoc logic model was developed after the guideline was completed
What factors affect the implementation of lay health worker programmes for maternal and child health?Qualitative evidence synthesis
Framework synthesis
Risk communication guideline (online supplementary file 3)15 27 45 55 Quantitative review of quantitative evidence (descriptive)
Qualitative using framework synthesis
Results based convergent synthesis
A knowledge map of studies was produced to identify the method, topic and geographical spread of evidence. Reviews first organised and synthesised evidence by method-specific streams and reported method-specific findings. Then similar findings across method-specific streams were grouped and further developed using all the relevant evidence
Integration: where possible, quantitative and qualitative evidence for the same intervention and question was mapped against core DECIDE domains. Tools included framework using public health emergency model and disaster phases
Very few trials were identified. Quantitative and qualitative evidence was used to construct a high level view of what appeared to work and what happened when similar broad groups of interventions or strategies were implemented in different contexts
Example of a fully integrated mixed-method synthesis.
Without evidence of effect, it was highly challenging to populate a DECIDE framework
B. Mixed-method review designs that can be used in guideline development
Factors influencing children’s optimal fruit and vegetable consumption30 56 Potential to explore theoretical, intervention and implementation complexity issues
New question(s) of interest are developed and tested in a cross-study synthesis
Mixed-methods synthesis
Each review typically has three syntheses:
Statistical meta-analysis
Qualitative thematic synthesis
Cross-study synthesis
Parallel-results convergent synthesis design or a segregated or sequential synthesis design
Aim is to generate and test theory from diverse body of literature
Integration: used integrative matrix based on programme theory
Can be used in a guideline process as it fits with the current model of conducting method specific reviews separately then bringing the review products together
C. Mixed-method review designs with the potential for use in guideline development
Interventions to promote smoke alarm ownership and function
57 58
Intervention effect and/or intervention implementation related questions within a systemNarrative synthesis (specifically Popay’s methodology)54 Can be accommodated within various review designs
Four stage approach to integrate quantitative (trials) with qualitative evidence
Integration: initial theory and logic model used to integrate evidence of effect with qualitative case summaries. Tools used included tabulation, groupings and clusters, transforming data: constructing a common rubric, vote-counting as a descriptive tool, moderator variables and subgroup analyses, idea webbing/conceptual mapping, creating qualitative case descriptions, visual representation of relationship between study characteristics and results
Few published examples with the exception of Rodgers, who reinterpreted a Cochrane review on the same topic with narrative synthesis methodology.
Methodology is complex. Most subsequent examples have only partially operationalised the methodology
An intervention effect review will still be required to feed into the guideline process
Factors affecting childhood immunisation
59
What factors explain complexity and causal pathways?Bayesian synthesis of qualitative and quantitative evidence Can be accommodated within various review designs
Aim is theory-testing by fusing findings from qualitative and quantitative research
Produces a set of weighted factors associated with/predicting the phenomenon under review
Not yet used in a guideline context.
Complex methodology.
Undergoing development and testing for a health context. The end product may not easily ‘fit’ into an evidence to decision framework and an effect review will still be required
Providing effective and preferred care closer to home: a realist review of intermediate care.3 60 61 Developing and testing theories of change underpinning complex policy interventions
What works for whom in what contexts and how?
Realist synthesis
NB. Other theory-informed synthesis methods follow similar processes
Can be accommodated within various review designs
Development of a theory from the literature, analysis of quantitative and qualitative evidence against the theory leads to development of context, mechanism and outcome chains that explain how outcomes come about
Integration: programme theory and assembling mixed-method evidence to create Context, Mechanism and Outcome (CMO) configurations
May be useful where there are few trials. The hypothetical nature of findings may be challenging for guideline panel members to process without additional training. The end product may not easily ‘fit’ into an evidence to decision framework and an effect review will still be required
Use of morphine to treat cancer-related pain62 Any aspect of complexity could potentially be explored
How does the context of morphine use affect the established effectiveness of morphine?
Critical interpretive synthesis Can be accommodated within various review designs
Aims to generate theory from large and diverse body of literature
Segregated sequential design
Integration: integrative grid
There are few examples and the methodology is complex.
The hypothetical nature of findings may be challenging for guideline panel members to process without additional training.
The end product would need to be designed to feed into an evidence to decision framework and an intervention effect review will still be required
Food sovereignty, food security and health equity63 64 Examples have examined health system complexity
To understand the state of knowledge on relationships between health equity—ie, health inequalities that are socially produced—and food systems, where the concepts of 'food security' and 'food sovereignty' are prominent
Focused on eight pathways to health (in)equity through the food system: (1) Multi-Scalar Environmental, Social Context; (2) Occupational Exposures; (3) Environmental Change; (4) Traditional Livelihoods, Cultural Continuity; (5) Intake of Contaminants; (6) Nutrition; (7) Social Determinants of Health; (8) Political, Economic and Regulatory context
Meta-narrativeAim is to review research on diffusion of innovation to inform healthcare policy
Which research (or epistemic) traditions have considered this broad topic area?; How has each tradition conceptualised the topic (for example, including assumptions about the nature of reality, preferred study designs and ways of knowing)?; What theoretical approaches and methods did they use?; What are the main empirical findings?; and What insights can be drawn by combining and comparing findings from different traditions?
Integration: analysis leads to production of a set of meta-narratives (‘storylines of research’)
Not yet used in a guideline context. The originators are calling for meta-narrative reviews to be used in a guideline process.
Potential to provide a contextual overview within which to interpret other types of reviews in a guideline process. The meta-narrative review findings may require tailoring to ‘fit’ into an evidence to decision framework and an intervention effect review will still be required
Few published examples and the methodology is complex