Article Text
PDF
Abstract
There is growing interest in the potential for complex systems perspectives in evaluation. This reflects a move away from interest in linear chains of cause-and-effect, towards considering health as an outcome of interlinked elements within a connected whole. Although systems-based approaches have a long history, their concrete implications for health decisions are still being assessed. Similarly, the implications of systems perspectives for the conduct of systematic reviews require further consideration. Such reviews underpin decisions about the implementation of effective interventions, and are a crucial part of the development of guidelines. Although they are tried and tested as a means of synthesising evidence on the effectiveness of interventions, their applicability to the synthesis of evidence about complex interventions and complex systems requires further investigation. This paper, one of a series of papers commissioned by the WHO, sets out the concrete methodological implications of a complexity perspective for the conduct of systematic reviews. It focuses on how review questions can be framed within a complexity perspective, and on the implications for the evidence that is reviewed. It proposes criteria which can be used to determine whether or not a complexity perspective will add value to a review or an evidence-based guideline, and describes how to operationalise key aspects of complexity as concrete research questions. Finally, it shows how these questions map onto specific types of evidence, with a focus on the role of qualitative and quantitative evidence, and other types of information.
- complex interventions
- systems
- systematic reviews
- public health
This is an open access article distributed under the terms of the Creative Commons Attribution IGO License (CC BY NC 3.0 IGO), which permits use, distribution,and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.Disclaimer: The author is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.
Statistics from Altmetric.com
Summary box
There is little guidance on the implications of complex systems for reviewing evidence or for developing guidelines as a basis for recommendations for practice and policy.
Key aspects of complex systems include interactions between interventions and the system itself; emergent properties; and positive and negative feedback loops.
These and other aspects of complexity can be framed as specific review questions, and evidence can be sought for each of them.
Systematic reviewers can use this new guidance to consider whether a systems perspective will be of value to them, and how it can be operationalised.
It is also important to note that a ‘full systems’ perspective is not necessarily appropriate for all reviews (or even many reviews).
Introduction
A complexity perspective
Recent years have seen a rapid rise in interest in complex interventions, perhaps because interventions themselves are becoming more complex, along with their evaluations.1 2 Complexity is a concept underpinned by a set of theories used to understand the dynamic nature of interventions and systems.3 Complexity theory has been increasingly used within the health sector to explore the ways in which interactions between component parts of an intervention or system give rise to dynamic and emergent behaviours.4 5 Interventions are often defined as ‘complex’ in terms of their being (1) multicomponent (ie, the intervention itself may comprise multiple components that may interact in synergistic or dissynergistic ways); (2) non-linear (they may not bring about their effects via simple linear causal pathways); and (3) context-dependent (they are not standardised, but may work best if tailored to local contexts).1 There is a range of methodological guidance on reviewing evidence on complex interventions,6–8 and new tools have emerged to help reviewers and guideline developers to deal with complexity—such as the Intervention Complexity Assessment Tool for Systematic Reviews (iCAT_SR) tool, which aims to help reviewers to categorise levels of intervention complexity.9
The academic focus is often on clearly described ‘interventions’—these are often sets of professional behaviours, or practices or ways of organising a service. However a different perspective has gained traction.10 This sees interventions not as discrete, bounded activities, but as interconnected ‘events in systems’.11 12 The current paper (one of a series, exploring the implications of complexity for systematic reviews and guideline development, commissioned by WHO) differentiates between these perspectives, focusing on how evidence may be synthesised. The aim is that the paper will be of relevance to guideline development at the global level but, like other papers in the series, will also be relevant to other contexts, such as developing evidence-based guidance at the national or subnational level.
Complex interventions and complex systems
The term ‘complex intervention’ is often used to describe both health service and public health interventions, including psychological, educational, behavioural and organisational interventions. Examples include health promotion interventions (eg, sexual health education),1 public health legislation (like Smokefree legislation)1 and organisational interventions (eg, stroke units, which involve multicomponent packages of care).13 The interventions have implicit conceptual boundaries, representing a flexible but common set of practices, often linked by an explicit or implicit theory about how they work.14 15
This ‘complex interventions perspective’ can be differentiated from a complex systems perspective—sometimes referred to as ‘systems thinking’.16 This has a long history in other fields.17 What differentiates the two perspectives is a move away from focusing on ‘packages’ of activities, with the idea that the intervention is external to the target population, towards (in Hawe et al’s words) ‘a focus on the dynamic properties of the context into which the intervention is introduced.’11 In a systems perspective, complexity arises from the relationships and interactions between a system’s agents (eg, people or groups that interact with each other and their environment) and its context. A system perspective conceives the intervention as being part of the system, and emphasises changes and interconnections within the system itself. It does not carry the implication of a separate intervention intervening—as if from outside the system. Thus, reviewing evidence from a systems perspective requires ‘consideration of the ways in which processes and outcomes at all points within a system drive change. Instead of asking whether an intervention works to fix a problem, researchers should aim to identify if and how it contributes to reshaping a system in favourable ways’.10 Notably, a systems perspective can be adopted in relation to individual-level (eg, interventions targeting individual eating behaviours), population-level (eg, an intervention delivered to a wider population, such as a mass media campaign) and/or system-level (eg, interventions designed to change food environments, such as high streets) interventions.10 In all of these cases, the focus of a systems perspective is on how the intervention interacts with and impacts on the system as a whole.
These differences are clarified in a later paper in this series (Rehfuess et al18).
We broadly [distinguish] between interventions targeting individuals (eg, diagnosis, treatment, or preventative measures addressed at individuals), interventions targeting populations, and interventions targeting the health system or context. Population-level interventions encompass those concerned with whole populations or population groups as defined by their age, sex, risk factor profile or other characteristic; they are often implemented in specific settings or organisations (eg, school health programmes). System-level interventions specifically re-design the context in which health-relevant behaviours occur; they are often implemented through geographical jurisdictions from national to local levels (eg, laws and regulations regarding the taxation, sale and use of tobacco products). Health system interventions represent a specific type of system-level intervention and often result in complex re-arrangements across multiple health system building blocks (eg, task shifting as a process of delegating specific health service tasks from medical doctors or nurses to less specialised health workers).
Hawe et al11 give the examples of schools, communities and worksites as complex ecological systems, which can be theorised in three dimensions: (1) their constituent activity settings (eg, clubs, festivals, assemblies, classrooms); (2) the social networks that connect the people and the settings; and (3) time. They also note the need to understand the dynamics of the whole system, not just the intervention or the individuals within it, and to understand that ‘the most significant aspect of the complexity possibly lies not in the intervention per se (multi-faceted as it might be), but in the context or setting into which the intervention is introduced and with which the intervention interacts’.11 Not all definitions of complex systems are in agreement, but box 1 identifies some key characteristics and points of difference. There is of course no distinct boundary between the two perspectives, and the choice of which perspective to adopt is (and should be) led by users’ needs. Sometimes it may be useful to analyse interventions as if they were packages of interconnecting components, acting externally upon a pre-existing system; at other times, it may be more productive (in terms of producing useful, actionable evidence) to conceive of them as ‘events in systems’; or to treat interventions as subsystems within a larger system (such as the Sure Start intervention in the UK which aimed to support families with young children in deprived communities1). In these instances, an intervention may be conceptualised as an ‘entry point’ into a system—a means by which to understand how a system adapts and changes in response to internal and external events.
Overview of a complex systems perspective
Interactions between components of complex interventions
The functioning of the whole system, rather than parts of the system, or solely on the interventions within it (as opposed to a focus on the characteristics of the intervention, such as interactions between its components, in the case of a complex intervention).11
Interactions of interventions with context
The interactions between an intervention and the system within which it takes effect.11
System adaptivity
How the system itself adapts to the introduction of an intervention.11 12 For example: ‘A complex system is one that is adaptive to changes in its local environment, is composed of other complex systems (for example, the human body), and behaves in a non-linear fashion (change in outcome is not proportional to change in input). Complex systems include primary care, hospitals, and schools. Interventions in these settings may be simple or complicated, but the complex systems approach makes us consider the wider ramifications of intervening and to be aware of the interaction that occurs between components of the intervention as well as between the intervention and the context in which it is implemented’.12
Non-linearity
Interactions between individuals, between levels (eg, interactions between effects at the individual, neighbourhood, community, societal level), and interactions between different parts of the system.12 ,44 By comparison, discussions of complex interventions tend to focus more on interactions between components of the intervention, and the levels or groups which the intervention is ‘targeted at’ (as opposed to interactions between the levels or groups).1
Emergent properties
These are properties or behaviours which arise from interactions between parts of a system. These properties are not seen in any one part of a complex system nor are they summations of individual parts (community empowerment,19 ,44 social exclusion and income inequality are noted emergent properties relevant to population health). Obesity has also been used as an example of emergence, with individual exercise patterns being linked to the risk of obesity, but obesity is also a determinant of individual exercise patterns.45 46 So outcomes should be measured at multiple levels within the complex system.12
Feedback loops
Mechanisms by which change is either amplified (positive or reinforcing feedback) or lessened (negative or balancing feedback).
Multiple outcomes and dependencies
When outcomes from one individual (or community) may be affected by outcomes from another (see handwashing example in the main text).
There are several implications of adopting a systems perspective. One implication noted by Shiell et al12 is that interventions, whatever their perceived level of complexity (simple or complex), can bring about wider changes in systems. For example, legislation (which may be conceptualised as either simple or complex) can bring about changes in social systems; in the case of the UK, banning smoking in public places resulted in changes in the pattern and nature of smoking, drinking and socialising, as well as changes in health outcomes. 19
There are many potential sources of complexity to be considered in both complex interventions and complex systems perspectives. Some of these are described in box 1 and table 1. Diez-Roux also notes that complex systems are characterised by dependencies: that is, outcomes from one individual (or community) may be affected by outcomes from another.20 One example comes from drinking water, sanitation and hand hygiene (ie, ‘WASH’) interventions, which are protective against enteric infections. In this case most of the protective effects come from ‘herd protection’ (ie, an emergent property of a system), which occurs when an infectious disease intervention provides indirect protection to non-recipients, due to the reduction in environmental contamination. 21
How do aspects of complex systems map onto review questions and inclusion criteria?
For those developing guidelines, the above issues will often be explored during the scoping stage (see the WHO Handbook for Guideline Development, section 2.7: http://www.who.int/publications/guidelines/handbook_2nd_ed.pdf). At this stage it may be useful to consider whether producing the guideline will involve summarising the evidence on a specific complex intervention, or will go beyond this to take a complex systems approach. If it restricts itself to consideration of a complex intervention, it will be necessary to decide which characteristics of complexity may be most relevant to this task. Scoping the guideline requires considering the interventions, and the individuals and/or populations, and the potential benefits and harms.
We conclude this section by emphasising that not every systematic review needs to consider all aspects of complexity. Even if complexity is taken account of in the review, it needs to be done pragmatically, by considering whether it will enhance the review’s usefulness to decision makers. We want to avoid simply encouraging every review to be as complex (and potentially confusing, and impractical) as possible. A pragmatic balance therefore needs to be struck between appropriately and accurately representing the complexity of the intervention and/or system being evaluated, and producing useful guidance or guidelines. Box 2 may help with striking this balance.
Will a complex systems perspective be useful for my systematic review?
To answer this, first consider the priority questions for the review.
What do the review users want to know about?
If users only want to know about the effects of the intervention on individual-level outcomes, they may not be interested in a wider system perspective (although that may simply be because they are not aware that this could be useful). If they only want to know about population-level effects, but are less interested in interactions between levels, or between the intervention and its context, then again a systems perspective may not be of interest (but that does not mean that it is not important; see other questions below).
Other questions to consider.
At what level(s) does the intervention have its effects?
If the intervention involves changes to wider structures or systems which affect health (eg, through regulation, healthcare reorganisation, the introduction of new policies or through the reorganisation of services), then a system perspective may be helpful. This could involve considering the outcomes of the intervention at different levels—for example, the individual level, the family level, the community level, the organisational level, the societal level. It could also consider how effects at each of these levels interact. A systems perspective may of be particular value in evaluating the implementation of an intervention.
Does the intervention affect the context into which it is introduced?
Public health interventions often interact with their context; a systematic review could explore the extent to which this is the case. For example, some interventions do not only change individual-level outcomes, but also social norms: Smokefree legislation affects smoking rates, but it also affects the wider acceptability of smoking in public places. This, in turn, may affect individual smoking rates.
Through which processes and mechanisms does the intervention bring about changes?
Users may want to know about the processes and mechanisms by which outcomes are produced by an intervention. From a systems perspective, this involves consideration of system-level mechanisms—in other words, by what means does the intervention change the wider system (its structures and processes) to bring about change?
Taking account of complexity: why we need to think about theory, system properties and context
These complementary perspectives have implications for developing appropriate, answerable research questions for systematic reviews. For example, if the focus is on the intervention, then research questions are more likely to focus on the individual and interactive effects of components of the intervention. The pathways between the intervention and those outcomes will also be of interest. However if the focus is on the system, or on the interaction between the intervention and the wider system, then the research questions may focus on whether and how the system adapts to the intervention (what has been referred to as ‘what happens’ questions, rather than ‘what works’ questions10 22); describing and analysing feedback loops between different parts of the system; and describing how effects are produced within different parts and at different levels of the system. Complex interventions adapt to the system within which they are introduced, and they may change the system itself. This may even be their purpose, for example in the case of health system interventions (note also that a health system can also be seen as a subsystem of a much wider social system).
The review process is likely to start by describing the boundaries of the system. This can be done using a graphical display of the various relationships between elements of the system. Such displays have variably been referred to as conceptual frameworks/diagrams or causal loop diagrams (figure 1 shows in simplified form the interactions between humans and their environments, and how these influence health outcomes). Here, we adopt the term conceptual frameworks as the most generic term—they can be thought of as being the logic model.
Causal loop diagram of human health and climate change (Proust et al44). GHG, Greenhouse gas.
Developing a conceptual framework like this can be done through a combination of literature reviews, stakeholder input and discussions within the review team. In one example, relating to the causal pathways linking crime, fear of crime and mental health, the conceptual framework was developed from a review of existing theory.23 Conceptual frameworks have been shown to be a useful means of (1) thinking through complexity upfront, (2) prioritising research questions and (3) making methodological choices in response to these decisions. Templates for such conceptual frameworks/diagrams can facilitate the development of a logic model.24 Certainly an initial illustration of factors and processes can help reviewers refine the research questions and the review’s inclusion criteria. This initial illustration may remain unchanged (an a priori logic model), or it may be subject to modifications as the evidence synthesis progresses (a staged or iterative logic model).25
From a guidelines developers’ perspective, it may be helpful to start the guideline scoping phase by considering the system boundaries. Systems are potentially huge, and for pragmatic reasons it may be best to focus on only part of the system. For example, in the case of childhood obesity, the focus may be restricted to marketing of unhealthy foods. The boundaries can/should be determined in consultation with stakeholders—and they should also be closely related to the review question.
The role of theory
The conceptual framework in figure 1 is similar to using ‘explanatory’ theory to depict a system. This is different from a process-orientated logic model or analytical framework which usually corresponds to the ‘theory of change’ (usually quite linear) of a complex intervention. Explanatory theory sheds light on the nature of the problem and helps to identify a range of factors that may be modifiable.26 Conceptual frameworks like this describe the inter-relationships within the wider system or subsystems. These graphical displays are themselves representations of initial hypotheses, or sets of linked hypotheses, about the processes involved. As such, they can be used to help generate specific research questions (see below).
System properties
The literature refers to a number of common properties of complexity. Some of the most frequently mentioned are defined in table 1. In producing a systems-oriented systematic review, the reviewer should first consider where the intervention of interest is located with respect to the wider system. To do this she/he does not have to analyse the whole system. Second, she/he should consider whether any system-level characteristics (such as feedback loops, non-linearities and interactions between intervention components) are of relevance and why. Not all these effects will be relevant to every systematic review; the effects may be small and/or the system-level effects in question may have limited explanatory power in some cases. Evidence-to-decision frameworks (such as the WHO-INTEGRATE framework described elsewhere in this series18) will be of particular value here, as they ensure that all factors or criteria of relevance in a given guideline development or other health decision-making process are considered in a systematic way.
The role of context
Complex health interventions are often characterised by their sensitivity to context, the fact that ‘one-size does not fit all’ and that such interventions often interact with and sometimes adapt to the context within which they are implemented, which may have implications for the effectiveness, acceptability and sustainability of the intervention itself. 27 28 In the case of health research, any aspect of an individual’s life could in principle be described as their ‘context’—such as their location within any social, spatial, physical or cultural space. Moreover, the same or similar contexts may affect people in quite different ways—think, for example, of studies of employment and health (such as the Whitehall studies in UK civil servants), which show that even in broadly similar contexts, employees’ health can be affected by subtly different employment grades and different levels of control over their working environment.29
However in defining a research question for a systematic review it is important to think pragmatically about context and to identify which aspects of context are likely to matter most—for example, which are likely to have a significant moderating effect on an intervention. This decision can be informed by existing theory, and by the use of conceptual diagrams and logic models to reveal potentially important contextual elements in different parts of a system. It can also be informed by users’ needs.27
Not all information on context will come from empirical studies. For example information on the political context within which a policy intervention is implemented may be found in policy documents and through media analyses.30 There is also a growing evidence base on implementation, including systematic reviews which examine how guidelines are implemented; this points to complexity as being an important barrier to implementation. 31 32
Context often acts as a moderator of the effects of complex interventions; however it can also be part of the intervention itself. For example, some public health interventions explicitly aim to change contexts, such as Smokefree legislation, which restricts smoking in public places such as bars and restaurants.33 In fact many policy interventions are like this, in that they involve changes over time in social, economic, health or other systems. Wells et al14 refer to this as a ‘blurred intervention’. The role of context is dealt with in detail in another paper in this series, which describes how context is currently managed within existing systematic review tools and methods, and describes good practice in terms of the use of context within systematic reviews and guidelines.34
A worked example: childhood obesity and system properties
Both the determinants and the consequences of childhood obesity are complex: there is an intergenerational passage of obesity risk, with obesity in adults being perpetuated into future generations through multiple mechanisms, social as well as biological. These pathways cover various stages of the life-cycle during childhood, from undernutrition or overnutrition in fetal development, childhood and through to adulthood, and incorporate unhealthy diets and inadequate physical activity. The physical and psychological consequences of childhood obesity are wide-ranging, likely to last into adulthood, and impact on social and health capital and the economy. A systems approach to childhood obesity sees it as embedded within the wider political, institutional and cultural system. It addresses upfront both life-course and environmental considerations, including appropriate infant and young child feeding, and a child’s daily food environment (eg, as affected by marketing and advertising of products, or the quality and access to school and preschool foods) and physical activity environment.
Refining the question could start, for example, with drawing a (or referring to an existing) model to synthesise the evidence on determinants, such as the one reproduced in figure 2. This can help visualise the underlying characteristics and relations of systems and show how they inter-relate to produce childhood obesity. Thus through this lens, childhood obesity can be conceptualised as an emergent property of a complex system, rather than the result of individual lifestyle choices. Visualising these relations can also help unpack feedback loops and how they might suppress or potentiate the effect of an intervention.
Conceptual framework: soft drinks consumption and childhood obesity in countries with limited access to safe drinking water.
Taking, for example, excessive soft drinks consumption as a well-established factor in childhood obesity: a systems perspective allows one to move away from a linear ‘cause (soft drink consumption) and effect (excess weight gain)’ approach, to understand the range of factors contributing to soft drink consumption and how they might interact to reinforce this behaviour. Crucially it also helps to define the boundaries of this complex problem (and therefore the boundaries of the system) by facilitating a thought process of ‘what else is happening in this picture?’—thus in this (incomplete, rough-sketch) conceptual framework (figure 2), issues such as water access and safety may come into play, and/or who is producing and marketing soft drinks (not always a soft drink company). Thus the boundaries move away from the individual child to include municipal actors and laws, corporate players, and even industries we would not automatically include (such as the alcohol industry) when thinking of causal pathways between soft drink consumption and childhood obesity. Thus understanding relations between components of a system requires acknowledging the system’s context and culture.
Figure 2 also illustrates an example of adaptivity or ‘self-organisation’, where the system finds ways to diversify and evolve. For example in response to a levy imposed by the government on soft drink industry, it will adapt in a number of ways, including by reformulating its products. Evidence shows, however, that this is most often not a question of substitution (actually removing high sugar drinks) but rather creating a low-sugar alternative, adding to the overall offer. This adaptation makes the system more resilient to external shocks (such as a soft drink levy on industry).
Implications of a systems perspective for framing the research question and types of evidence included
Assuming that complexity is a relevant concern for a systematic review, the next step is to turn that ‘concern about complexity’ (or a specific aspect of complexity) into a research question, or questions. Again, caution is necessary, because not all elements of complexity are necessarily the focus of research questions. For example, feedback loops may be relevant to how an intervention may/may not work, so they may be important to ‘bear in mind’ when reviewing evidence, rather than being a main focus of the review. It is likely that reviews of evidence adopting a complex systems perspective will be interested in a wide range of questions, relating to the characteristics of complexity summarised above. In particular questions about how an intervention works, the nature of the interaction between intervention components, and between intervention and context are likely to be of relevance. There may still be an overall ‘what works question’ to which the PICO framework (patient, population or problem; intervention; comparison; outcomes) can be applied,35 but there may be a need to frame other questions in addition which may require a wide range of evidence to answer them. It is likely that a series of different syntheses or reviews may be undertaken, each of which may draw on a method-specific question formulation framework.27 36
Lumping versus splitting
Squires et al37 note the importance of considering how broad the scope of a review should be, often known as ‘lumping’ versus ‘splitting’. ‘Splitters’ argue that it is only appropriate to combine highly similar studies; for example, studies should be comparable in terms of their design, population, interventions, outcomes and context. ‘Lumpers’ however argue that a systematic review aims to identify the common generalisable features within broadly similar interventions unless there are good grounds not to. Squires et al37 suggest taking a lumping approach whenever possible as it allows the assessment of generalisability and consistency of research findings to be assessed across a wider range of different settings, study populations and behaviours.37 It could also be argued that it is in the very nature of complex interventions, and interventions in complex systems, that individual studies will vary in terms of context, population and of course the intervention itself. It is therefore uncommon to have groups of very homogeneous studies which can be ‘split’. By comparison ‘lumping’ allows decision makers to see how findings have varied across different population groups and contexts. Describing context clearly can therefore help users assess the potential generalisability of research findings. (A later paper in this series describes meta-analytical approaches to lumping and splitting.38
A more ambitious perspective would go beyond thinking just of lumping and splitting, and would acknowledge that no single study—or methodological approach—is likely to contain the breadth of evidence required to model a complex system adequately. Mixed methods reviews and studies—which blend the power of statistical aggregation with qualitative explanation—are likely to be the most useful approach.36
Table 1 gives examples of how different aspects of complexity may be framed as research questions and the implications for systematic review inclusion criteria. In general, the reviewer or guideline developer needs to start by thinking about the scope of their review or guideline: Is the focus solely on effectiveness? Or implementation? Or exposures (does X cause Y)? Or is it a question about process/implementation? Are users likely to be interested in the adaptivity of the intervention and the system surrounding it? Are they interested in variations in effects across contexts? Which components of the intervention appear to matter, and which don’t?
Choices about what sort of evidence to include in order to answer these questions then require further decisions about how to synthesise and appraise that evidence.36 39 They will also influence how one might assess the overall confidence in a body of evidence (see the papers in this series on evidence-to-decision frameworks and considerations of complexity in rating certainty of evidence.18 39
In conclusion to this section, it should be noted that synthesising complex sets of evidence can be methodologically challenging and resource-intensive. Not every review—even if the review aims to take a systems perspective—will be able to address all the aspects of complexity in table 1. Reviewers may therefore need to prioritise which aspects are likely to be most important to users, and focus resources on these. As we noted earlier in the paper, not every systematic review needs to consider all aspects of complexity. In many cases—particularly where resources are limited—a more straightforward review approach will be appropriate. However where a systems perspective is likely to be of value, authors will need to consider how best to include relevant evidence as far as their resources permit. Considering users’ priorities alongside table 1 may be helpful in this regard. It is also possible that as part of the guideline development process several reviews may be conducted, addressing different aspects of complexity. For example, a systematic review of effectiveness might be conducted alongside a review exploring the processes and mechanisms by which the intervention brings about change within a particular system, and/or exploring issues of acceptability and feasibility.27
How users can help shape the review question(s)
It is standard practice to involve review users in defining the review question(s). However there are different types of decision maker (or ‘stakeholders’) to consider, and they may have different priorities. They may have different views, for example, about the primary/secondary outcomes and about what aspects of complexity matter most. Decision makers may also have particular biases; some may not want specific outcomes or phenomena of interest to be considered. In public health, for example, stakeholders with vested interests, and sometimes policymakers, may be keen for individual-level interventions and/or outcomes and/or populations to be addressed in a research project, but may be less interested in interventions that act at the population level. For example some unhealthy commodity industries are often most accepting of evidence about individual-level informational interventions (such as educational interventions and provision of information), which are known to be only weakly effective, but are less accepting of evidence about population-level structural interventions aimed at the whole population, such as marketing restrictions, which are generally more effective.40 Guidance can be found elsewhere on how to obtain user input into framing review questions.35 41
Having a specific decision-making context or a specific decision maker in mind can help focus the review question and the review itself (Booth et al27). This may be particularly true of reviews with a complexity focus, which themselves may risk becoming overly complex. One simple application of this is to acknowledge that in many cases there is imperfect evidence for a defined magnitude of effect, yet it is also possible (based on theory, observation, natural experiments and experience) to be confident that a proposed intervention, compared with doing nothing, will not have an effect in the wrong direction and will do some good (see also concept of quality of evidence for a ‘non-null effect’ in a later paper in this series (Montgomery et al39)). Consideration of this prior wider evidence base, alongside any effectiveness evidence, based on randomised controlled trials or quasi-experiments, allows the decision maker to make the appropriate decision, by considering the balance between the evidence on both (or all) sides of a decision.36 42
Value of information approaches
The concept of ‘Value of Information’ (VOI) may be helpful to focus the review. VOI frameworks describe the anticipated value of the new information which would be generated by conducting a new piece of research (such as a new systematic review)43 The potential value of new research is that it reduces some aspect of decision-maker uncertainty. The potential implication for systematic reviews is that by assessing in advance what new evidence would be needed to reduce that uncertainty, the review question can be more closely tailored to users’ needs. VOI approaches usually rely on formal (quantitative) estimation of the value of new evidence, but even in the absence of this quantitative approach, it is useful to consider (and find out) the main areas of uncertainty for different types of decision maker and to explore what sort of evidence would be needed to reduce it. In the case of reviews of evidence on complexity, it may show that producing new synthesised evidence about, say, feedback loops would have little impact on a decision—and so this aspect could be left out of the review. This can be a helpful way of focusing the development of new guidelines—by asking the questions: ‘What area of decision-maker uncertainty is the guideline aiming to reduce?’ and ‘What new evidence would be most useful to review, to reduce that uncertainty?’
The degree of consultation with users may depend on the review topic and its political or other sensitivities. For highly uncertain, politically sensitive topics with little clear evidence, more input would be necessary to involve all potential stakeholders (or their representatives) in the decisions about the framing of the question. For ‘simpler’ less contentious reviews, this may be less crucial. In addition, any recommendation about an intervention does not depend on the evidence alone; other criteria must be taken into consideration. A subsequent paper in this series will show how the WHO-INTEGRATE evidence-to-decision framework helps with this process, based on six structural criteria (Balance of health benefits and harms, Acceptability, Health equity, equality and non-discrimination, Societal impact, Financial and economic considerations and Feasibility and health system considerations). A seventh criterion, Quality of evidence, represents a meta-criterion that applies to each of the six structural criteria; all seven criteria influence the strength of a guideline recommendation. Each criterion may apply at the individual level, the population level, the system level or several of these.
Conclusions
In reviewing evidence and developing guidelines, it may be helpful at the beginning of the process to explicitly consider whether to take a ‘complex interventions’ or a ‘complex systems’ focus. Both may be of value at different stages of the review. The decision should be taken with reference to users’ needs and available resources. Box 2 and table 1 may help with making this decision and in assessing the implications for what type of evidence to include. It should also be noted that this is an evolving field and what is now needed are concrete examples of complex systems-oriented systematic reviews. These will help clarify the feasibility and resource requirements for such reviews. Subsequent papers in this series will consider in more detail the practical steps which are likely to be involved, and the contribution made by different types of qualitative and quantitative evidence.
Acknowledgments
We acknowledge the contributions through participation in discussions and meetings of Lorenzo Moja, Jonathon Simon, Asha George, Mauricio Bellerferri and Neena Raina.
References
Footnotes
Handling editor Soumyadeep Bhaumik
Contributors All authors contributed to discussions to decide on the content, and/or contributed examples and to revising and/or writing or reviewing the text, and approved the final version. MP is the guarantor.
Funding Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.