Table 1

How do aspects of complex systems map onto review questions and inclusion criteria?

Aspect of complexity of interestWhy this is relevantExamples of potential research question(s)What sort of evidence may answer this question? (note: non-exhaustive list)Types of study to search for (eg, study designs)Examples
What ‘is’ the
system? How can it be described?10 11
It is helpful to have a theoretical model of how the system works—what the main influences on outcomes are and their interconnections. This can help with scoping the review.What are the main influences on the health problem? How are they created and maintained? How do these influences interconnect? Where might one intervene in the system?Potentially any sort of evidence may be helpful: background theoretical literature; epidemiological and other evidence on the determinants (eg, of childhood obesity). This can be presented as a conceptual diagram of the system or part of the system.Theoretical papers; previous systematic reviews of the causes of the problem; epidemiological studies (eg, cohort studies examining risk factors of obesity); policy documents; network analysis studies showing the nature of social and other systems.47 Cochrane review of audit and feedback mechanisms (shows the use of a theory of change).
Tools: logic model 46 23 24
Interactions between components of complex interventions.1 Users may wish to know which components are essential for effectiveness (ie, to bring about system change) and which less so; some components may dampen intervention effects (through dissynergies).Effectiveness question: What is the independent and combined effect of the individual components? Process question: How do the components work along and in combination to produce effects? (How do they interact to produce outcomes?)Evidence of the independent effects of components of the intervention and synergistic/dissynergistic interactions between those components may be available in the form of either quantitative or qualitative data.47 Studies with multiple arms, for example, factorial designs (eg, randomised controlled trials of multicomponent interventions). Studies with different configurations of components, to permit indirect comparisons between studies47; ’Tracer studies’ to understand how the intervention works at various levels of the system; social network analyses to understand role of actors and/or networks; modelling studies, drawing on different types of data to understand how various domains have interacted.Changing prescribing practice involves interactions between pharmacists and the organisations in which they are located.1
Interactions of interventions with context and adaptation.48 49 Complex interventions can legitimately adapt to their context—the same intervention can look different in different contexts or it may need to be delivered in a context-specific manner.1. For a research question about implementation: (How and why) does the implementation of this intervention vary across contexts?
2. For an effectiveness review: Do the effects of the intervention appear to be context- dependent?
1. Process evaluations; studies which describe the implementation of the intervention.
2. Effectiveness studies from a range of contexts: individual studies conducted in a range of contexts.
1. For example, qualitative studies; case studies.
2. Trials or other effectiveness studies from different contexts; multicentre trials, with stratified reporting of findings; other quantitative studies that provide evidence of moderating effects of context.
Community-based interventions to address depression may legitimately vary between contexts—the form of the intervention varies, but the underlying theory and objectives remain the same.50
System adaptivity (how does the system change?).11 Systems may adapt to (accommodate or assimilate) new interventions, which may affect their effectiveness. (Note that systems are often embedded within other systems and can coevolve.)(How) does the system change when the intervention is introduced? Which aspects of the system are affected (see the CICI framework51)? Does this potentiate or dampen its effects?As above; process evaluations; possibly policy analysis analysing change in the system over time, depending on the intervention.1. Qualitative studies; case studies; quantitative longitudinal data; possibly historical data; effectiveness studies providing evidence of differential effects across different contexts; system modelling (eg, agent-based modelling).The introduction of a tax on sugar-sweetened beverages (SSBs) may affect individual consumption; manufacturers may reformulate SSBs to avoid the tax—and may also reformulate food products.52
Emergent properties.46 Where effects emerge from synergies within the system—such as from interactions between parts of the system or between individuals or groups within the system.What are the effects (anticipated and unanticipated) which follow from this system change?Qualitative research is often a source of evidence on unanticipated effects, including adverse effects; any quantitative evaluation may also produce such evidence.Prospective quantitative evaluations; qualitative studies; retrospective studies (eg, case–control studies, surveys) may also help identify less common effects; dose–response evaluations of impacts at aggregate level in individual studies or across studies included with systematic reviews (see suggested examples).Herd immunity in relation to vaccinating individuals53;population-health outcomes for sanitation interventions at household level (threshold effects); emergence of new social norms (eg, in smoking, following new tobacco control measures).54
Non-linearity and phase changes.53 1 55 Where the effect or the scale of the effect does not appear to be directly related to the cause. May explain why intervention effects suddenly appear or disappear.How do effects change over time? (Changes may be due to biological (genetic drift in virulence factors), ecological (changes in habitat creating or constraining new/different space for vectors or disease), epidemiological (changes in disease patterns by age, cause, location and so on) or social factors (changing social norms around gender and behaviours)).Longitudinal quantitative data (eg, Interrupted studies); qualitative data.Mainly prospective quantitative studies, including ITS studies; dose–response evaluations of impacts at aggregate level in individual studies or across studies included with systematic reviews (see above—might fit in either place).Use of quantitative time series methods alongside qualitative ‘story telling’ to identify phases/evolution in social care policy in England.56
Positive (reinforcing) and negative (balancing) feedback loops.20 These can potentiate or reduce the effects of interventions: for example, between behavioural and environmental features within the system, for example, when availability of healthy food promotes healthy diets, creating demand.What explains change in the effectiveness of the intervention over time? Are the effects of an intervention damped/suppressed by other aspects of the system (eg, contextual influences)?Potentially quantitative or qualitative data.Qualitative studies of factors that enable or inhibit implementation of interventions; quantitative studies of moderators of effectiveness; long-term longitudinal studies; development of conceptual diagrams to illustrate potential feedback loops, and to help identify ways in which they may be identified empirically.1. Provision of cycling lanes encourages more cycling; 57 in theory cyclists may feel safer because there are larger numbers of visible cyclists, so more people cycle (positive/reinforcing feedback).
Multiple (health and non-health) outcomes and dependencies.22 Changes in systems can produce a range of health and non-health outcomes—both anticipated and unanticipated, with no single ‘primary’ outcome.58 Outcomes from one individual (or level) may be affected by outcomes from another.What changes in processes and outcomes follow the introduction of this system change? At what levels in the system are they experienced?Quantitative and qualitative data (eg, qualitative data have been used to identify unanticipated adverse effects 59). Examination of specific pathways and many outcomes along the overall theory of change and over time (including lag effects across one or several decades) would be useful.Quantitative studies tracking change in the system over time; qualitative research exploring effects of the change in individuals, families, communities and so on.Many social programmes produce changes in health outcomes, but also non-health outcomes (eg, employment, education) at individual, family, community and city levels.60–63
  • CICI, Context and Implementation of Complex Interventions; ITS, Interrupted Time Series.