Formulating questions to explore complex interventions within qualitative evidence synthesis

Introduction

Many commentators seek to define complexity in connection with complex interventions.1 Rogers distinguishes between simple, complicated and complex: simple is encapsulated in following a recipe, complicated by sending a rocket to the moon and complex in bringing up a child.2 In the first paper of this WHO series on Complex Interventions, Petticrew and colleagues turn the emphasis away from the activity itself (intervention or exposure) and towards the perspective adopted by the evaluator (in this case an ‘intervention perspective’ or a ‘systems perspective’).1 These perspectives offer alternative evaluation ‘lenses’ to be adopted by reviewers or guideline developers even when examining the same phenomenon. For example, when examining use of a safety checklist within operating theatres3 one could either adopt an ‘intervention perspective’ or ‘lens’ to consider issues that relate to implementation within a controlled setting (the theatre) or adopt a ‘systems perspective’ or lens to explore the wider organisational or system culture within which the checklist is being implemented (eg, within a culture of blame or of improvement).

When making decisions about complex interventions, guideline development groups need to take account of the sociocultural acceptability of the intervention, as well as how feasible the intervention will be to implement. Complex interventions are inextricably linked to context; interventions interact with, and sometimes change, the context within which they are implemented.1 Recognition that complex interventions are context-dependent not only holds implications for the effect of the intervention, but also for its sustainability, acceptability and feasibility. This paper examines implications of adopting a ‘systems perspective’, as opposed to an ‘interventions perspective’, when formulating questions to be addressed by qualitative research. As with the first paper in the WHO Complex Interventions series,1 it focuses on the first part of the evidence synthesis process, defining the question. This paper reflects on frameworks for structuring systematic review questions, informed by a rapid review of existing frameworks, to evaluate their suitability when exploring complex interventions. The paper proposes an alternative framework, the PerSPEcTiF framework, for further testing.

It is increasingly recognised that systematic reviews of effects do not adequately capture how or why the effects of complex interventions differ according to context.4 5 Decision makers are demanding different types of synthesis to provide such evidence. Qualitative evidence synthesis (QES), for example, increasingly contributes to recommendations from WHO and other guideline development processes.6 7 QES can provide evidence for diverse questions beyond those that typically relate to the feasibility and acceptability of complex interventions (see Box 1)4 8 9 QES can potentially provide rich data relating to the context of interventions, policies or conditions and the lived experiences, views and beliefs of those involved. However, typical question frameworks for QES do not adequately account for a complexity perspective,10 11 in particular they do not account for the presence and assimilation of multiple stakeholder perspectives or for the importance of contextual variation; critical if QES findings are to support holistic decision-making and if guidelines are to be applied with contextual sensitivity. As Squire and colleagues emphasise:

Such complexity … makes the task of formulating a good review question both more important and more difficult. Furthermore, given the expected heterogeneity, systematic review questions should go beyond simple effectiveness questions (eg, ‘does X work?’) to consider under what circumstances X works.10

Guideline development organisations, such as WHO, the National Institute for Health and Care Excellence in the UK and other members of the Guidelines International Network (G-I-N), need to develop guideline recommendations that are feasible and acceptable to those planning, providing, implementing or receiving care. In turn, guideline development requires systematic review methodologies that explore the complexity of interventions, the context in which they are implemented and, the emphasis of this paper, the lens or evaluation frame through which they are evaluated.12

How do guideline questions currently recognise complexity?

Question formulation is critical to guideline development13 because it determines both the priorities to be addressed by the guideline and the types of evidence that will subsequently be admitted when addressing these priorities. Systematic reviews typically specify the elements of a question using an epidemiological design framework—population/patient; intervention; comparison; outcome(s); popularly known by the ‘PICO’ acronym14 or, replacing intervention with exposure, with the non-intervention formulation, PECO.15 Use of a PICO/PECO question within a guideline development process implicitly privileges an experimental/observational epidemiological model. Notwithstanding this inherent incompatibility the PICO model persists within many QES.16 17

A systematic review team should routinely consider the potential added value of a complexity perspective for their review topic and work with guideline developers to decide whether a simple, complex intervention or complex system perspective is most appropriate and feasible for the review. The stage of question formulation thus offers a unique opportunity to surface and resolve issues that relate to such a complexity perspective.

As the example of a safety checklist illustrates, one cannot choose an appropriate perspective simply on the basis of intervention characteristics. Increasingly, health technology assessments, which have traditionally adopted an ‘intervention perspective’, are being encouraged to adopt a broader societal (and hence ‘systems’) perspective within the frameworks they use.18 19 Nevertheless, broadly speaking, questions relating to health technology assessment typically gravitate towards an ‘intervention perspective’ while those that arise within a public health context often adopt a ‘systems perspective’. Decisions on which overarching perspective to adopt must be based on what is most useful and most practicable. A review team can only attain partial understanding of complexity and they may implicitly or explicitly overlook something which subsequently matters.

This choice between an ‘intervention perspective' and a ‘systems perspective’ holds clear implications for question formulation. Where an ‘intervention perspective’ is to be adopted, a QES typically explores barriers or facilitators to implementation, or the acceptability of the intervention to individuals and populations. A complex intervention perspective may still be accommodated within a population-based and intervention-based framework,14 or by use of frameworks that add either setting20 or context.18 However, a ‘complex systems perspective’ for the same intervention makes the literature search and review process logistically demanding. Questions may be specified within an extended framework to specify the complexity of the decision problem, or, increasingly, review teams may articulate the problem within an a priori logic model,1 which may be open to ongoing revision and refinement throughout the review.10 19

Formulating complexity-related questions to be addressed in a QES

As described elsewhere within this series,1 8 guideline development groups consider issues that extend beyond effectiveness (what works) and cost-effectiveness (the cost–benefit of each option). Questions relate to intervention components, interactions between them, how intervention components adapt to and are modified by the context, how the system adapts to change and the role of feedback loops and emergent changes on the overall system.1 Further questions align to the WHO-INTEGRATE framework.8 For example, concerns such as patient values in relation to health outcomes, safety, sociocultural acceptability, equity and equality and societal perspectives require different question frameworks and draw on diverse types of evidence. Both companion papers in this series1 2 rehearse why a complex systems perspective might be considered separately from a complex intervention perspective and why a systems perspective could add value to the intervention perspective. The resulting premise of this analysis is that review authors may consequently need to advance their complexity perspective beyond the typical PICO framework used in conventional reviews. Box 1 draws on earlier papers in this series to identify specific questions.4 8 The PICO framework does not easily accommodate a complexity perspective10 21 requiring an alternative framework for articulating the review question.

Previous commentators have identified the need to consider how a complex intervention is implemented (ie, feasibility) and the environment or context (ie, acceptability and meaningfulness) within which the intervention or programme is delivered. The GRADE Evidence to Decision (EtD) framework22 and the Joanna Briggs ‘FAME’ framework23 acknowledge that feasibility, appropriateness and meaningfulness must be recognised in the shift from ‘what works’ to ‘what happens’.21

A guideline development group should prioritise questions relating to contextual variation that cannot be addressed in a primary study or questions which could be asked in a primary study, but are addressed robustly across multiple contexts through synthesis (Box 1). Questions may relate to contextual variation (variation across study contexts or between study contexts and the target context) or to contextual sensitivity (how the intervention must be adapted or modified to recognise contextual factors present in the target context). Questions addressed by a QES mirror those that can be answered by primary qualitative research. Clearly, questions that focus on differences in acceptability, delivery and implementation between interventions or programmes are better addressed through synthesis rather than by a single-context primary study. In contrast, a primary study is better equipped to identify nuances within a specific context. Simple PICO variants are unsuited to handle either the complexity inherent in any single complex intervention review question1 or the wide variation in question types and evidence types required for guideline development (as illustrated by the WHO-INTEGRATE framework).8 The collective experience of the authors is that question formulation frameworks for qualitative evidence syntheses should include elements relating to both context and stakeholder perspective. Dissatisfaction with the suitability of PICO for constructing questions relating to feasibility or appropriateness has led to several alternatives being proposed. We therefore resolved to conduct a rapid review of existing question formulation frameworks to examine their suitability to accommodate a complexity perspective.

Reviewing existing question formulation frameworks

We performed a rapid review for currently available frameworks for formulating questions. This was not a comprehensive methodological review; we sought to map question variants and to examine their suitability for capturing a complexity perspective. In the interests of transparency, methods and results of the rapid review are reported separately in online supplementary file 1. However, the focus of this particular analysis is on specifying a candidate framework that accommodates complexity perspectives, not on the underpinning technical process by which individual items for possible inclusion were identified.

Following review of 1481 references and 113 full-text citations, we identified 38 question formulation frameworks (See online supplementary file 1 for full list of included references) which were evaluated for their suitability to accommodate a complexity perspective.

Based on the methodological literature relating to QES and complex interventions, the team considered that to be useful a question framework should recognise context (articulated as setting, environment or context)24; acknowledge the criticality of stakeholder perspective (differentiated from target population)25; accommodate elements of time/timing26 and place; be sensitive to qualitative data (in the sense of eliciting themes or findings, rather than quantitative outcomes).9 None of the identified frameworks satisfied all four criteria. We therefore concluded that none of the existing frameworks was entirely suited to capture a complexity perspective when addressing questions potentially answerable by qualitative evidence syntheses.

Based on our rapid review of the elements of question formulation frameworks, we concluded that an optimal framework for use by guideline development groups would need to better incorporate elements of ‘context’, currently missing from existing frameworks. ‘Context’ encompasses temporal, spatial and societal dimensions1 and offers an umbrella term beneath which environment and setting are subsumed. We propose that ‘environment’ should exploit well-established public health connotations of wider societal determinants and health service characteristics within which a service is delivered. In contrast, setting describes the point where interaction between service user and service provider takes place.⁶¹ Within these two overarching constructs, further granularity can be accommodated; so, for example, a narrative review of contextual factors influencing health committees in low-income and middle-income countries identified four overlapping conceptual spheres—community and society which would be nested under environment and health facilities and health administration which would be articulated under setting.

An optimal framework would also capture different stakeholder perspectives, such as the views of partners or carers or, in a health systems context, of wider societal stakeholders. Finally, the question framework should encourage guideline developers to consider qualitative data, using alternatives to ‘outcomes’, and acknowledge contextual variation in time and space. Because no existing framework accommodates these requirements, the authors devised an alternative, PerSPEcTiF. Table 1 outlines the elements proposed in the PerSPEcTiF framework, which consolidates elements from existing question formulation frameworks.

Table 2 provides a worked example of a PerSPEcTiF question framework compared with the popular PICO and SPICE variants. Clear omissions of the latter two frameworks are flagged in the column ‘missing’. So SPICE overlooks important features of the wider environment as well as temporal (time/timing) aspects while PICO omits not only these two elements but also the local context of setting.

Online supplementary table 1 illustrates the subsequent search strategy for this example. By prompting review authors to identify key areas of a complexity perspective beyond a typical PICO, the PerSPEcTiF framework assists team members to identify appropriate search terms. Articulating the review question in this way helps to identify how eligible studies differ in time, setting, stakeholder perspectives and surrounding environment.

Question formulation in guideline development

According to the WHO handbook for guideline development, question formulation requires an eight-step process (Box 2).13 Complex interventions generate numerous questions with implications for the scale and complexity of retrieval and synthesis. Below we use selected case studies to highlight where and how PerSPEcTiF could be used.

Protocol development for a QES is iterative, not linear, starting with ‘guiding review questions’.27 For the WHO smoking in pregnancy guidelines, a Technical Secretariat listed scoping questions and outcomes related to tobacco use and secondhand smoke exposure. This list was reviewed and prioritised by international stakeholders. Scoping establishes whether a review question is feasible (the logistics) and where conceptual boundaries (inclusion and exclusion criteria) lie.28 Question formulation helps a guideline development group to consider what is feasible and what is meaningful. Will the QES review team mirror the scope of an accompanying effectiveness review? If so, the team may use similar subject terms for the searches for both effectiveness and qualitative data. Alternatively, will a system perspective examine the experience of a condition,29 how the condition is viewed by health professionals or society at large and how implementation might occur30? While QES can accommodate either a complex system or a complex intervention perspective, health technology assessments and guidelines often favour the narrower, intervention perspective, mainly for pragmatic reasons.31

Types of questions addressed by QES include: ‘Does evidence show that an “effective” intervention is poorly accepted within its target context?’ ‘Do studies demonstrate that an experimental intervention works suboptimally in a “real world” context?’. At this stage, all questions are potentially valid. The guideline development group and stakeholders need to agree on terms used when articulating each question.10 Questions for the WHO smoking in pregnancy guidelines required several consultations with the guideline development group prior to identifying and synthesising the best available evidence for each key question within a systematic review.

Step 2 involves drafting the prioritised key questions using a relevant question formulation framework. Question formulation should ‘prompt review authors to identify the key components of the intervention/s and how these interact’.32 The PerSPEcTiF question formulation framework encourages this complexity perspective32 by acknowledging contextual variation of both time and space, and by using alternatives to ‘outcomes’. In addition, by including ‘perspective’ as an element, the PerSPEcTiF framework accommodates a ‘systems perspective’, with organisational levels and causal pathways10 (step 3), thus extending beyond the PICO format. Logic models offer an alternative way of thinking through such complexity1 and may be revised throughout the review process, either opportunistically or at preplanned stages, as new data emerges.16.

Stakeholders are key when refining questions and ‘lumping’ or ‘splitting’ interventions33 and outcomes10 (steps 4–6). In prioritising questions to be addressed by qualitative evidence, guideline development groups need to agree which contextual factors are meaningful. This may require multiple iterations.7 Our rapid review revealed that current question frameworks fail to accommodate context. By incorporating ‘environment’ (the context within which a health service is delivered) and ‘setting’ (the point where interaction between service user and service provider takes place), PerSPEcTiF seeks to address these limitations.

It is preferable to identify important contextual variables a priori, to inform study selection criteria. Alternatively, they may emerge when analysing included studies, determining how the data extraction form is structured. Ideally, a review team would construct a simple typology of important contextual factors. The reality is demonstrably different, for example, grouping African countries by socioeconomic factors or by predominant religion results in different groupings. By considering the extent of contextual variation early, a guideline development group not only shapes the final set of key questions but also informs data extraction and formulation of recommendations.

Having finalised the overarching guideline question (step 7), this is broken down into key questions to be addressed within the guideline. Key questions must be clearly formulated and the guideline development group should consider whether each key question is best addressed by quantitative or qualitative evidence or by a combination of both. The review team should decide whether these key questions are best addressed by evidence retrieved by an overarching guideline search, with evidence sifted and mapped to each key question, or whether specific supplementary searches are required for each key question (step 8). Online supplementary table 2 includes what should be considered at each stage of the question formulation process.

Considerations when formulating review questions for QES

In summary, systematic reviewers and guideline developers should decide whether a systematic review will adopt an ‘intervention perspective’ or extend a ‘systems perspective’ beyond the setting to include the wider environment. An intervention perspective can use population-based and intervention-based frameworks perhaps with the addition of context (eg, PICOC) or ‘setting’ (eg, SPICE). Where a ‘systems perspective’ offers insight a nuanced approach, using either the innovative PerSPEcTiF framework or a logic model structure is valuable.

PerSPEcTiF allows systematic reviewers and guideline developers to factor in contextual variation relating to time and/or space. Some variation can be anticipated; literature published prior to a particular date may be irrelevant to the decision in hand. Alternatively, the study contexts informing a particular guideline may emerge as substantively different from the target context and the review and guideline development teams may explicitly exclude such studies.34 The first paper in this series offers guidance on how teams might judge contextual relevance.4

If a QES accompanies a systematic review of quantitative evidence, then teams must consider the extent to which the quantitative and qualitative review share the same question scope. As illustrated in the first paper of this series,1 quantitative and qualitative approaches are typically complementary with questions addressed variously by quantitative or qualitative evidence or by a combination of both.

Finally, we acknowledge that the type of question asked is not only determined by the perspective adopted by the guideline but also, in turn, influences selection of an appropriate method for qualitative, or mixed method synthesis. Guidance on how to select a QES method based, for example, on whether a question is ‘fixed or emerging’ is available35 36 and is considered in other papers in this series.37 38

We acknowledge that further work is required to test the utility of the PerSPEcTiF framework for the widest possible variety of questions addressed by QES. Similarly, there is a need to look at the extent to which this framework elicits additional information, not otherwise identified by population-based and intervention-based frameworks, and, importantly, what the implications of the framework are on the subsequent construction of search strategies.

Conclusion

This paper highlights the need for focused, relevant questions in qualitative evidence syntheses that address a complexity perspective. Use of the PerSPEcTiF framework, where a systems perspective is to be employed, should result in more appropriate questions addressing, for example, feasibility and acceptability. Better formulation of questions will, in turn, lead to more informed decisions on choice of synthesis method and, ultimately, to better evidence on aspects of context that are important to consider when guidelines adopt a complexity perspective.