Taking account of context in systematic reviews and guidelines considering a complexity perspective
===================================================================================================

* Andrew Booth
* Graham Moore
* Kate Flemming
* Ruth Garside
* Nigel Rollins
* Özge Tunçalp
* Jane Noyes

## Abstract

Systematic review teams and guideline development groups face considerable challenges when considering context within the evidence production process. Many complex interventions are context-dependent and are frequently evaluated within considerable contextual variation and change. This paper considers the extent to which current tools used within systematic reviews and guideline development are suitable in meeting these challenges. The paper briefly reviews strengths and weaknesses of existing approaches to specifying context. Illustrative tools are mapped to corresponding stages of the systematic review process. Collectively, systematic review and guideline production reveals a rich diversity of frameworks and tools for handling context. However, current approaches address only specific elements of context, are derived from primary studies which lack information or have not been tested within systematic reviews. A hypothetical example is used to illustrate how context could be integrated throughout the guideline development process. Guideline developers and evidence synthesis organisations should select an appropriate level of contextual detail for their specific guideline that is parsimonious and yet sensitive to health systems contexts and the values, preferences and needs of their target populations.

*   complex interventions
*   context
*   systematic reviews
*   clinical guidelines
*   population health guidelines

### Summary box

*   In systematic reviews and guidelines about complex interventions, considering context is challenging because it is diversely defined and dynamic.

*   Context is typically caricatured as a location in time, in an organisational level or within a geographical space. Reviews and guidelines rarely provide sufficient detail of the context of an intervention in terms of its location in social, cultural, political or economic space.

*   The extent to which reviews and guidelines of complex interventions consider context is limited by inadequate information on context in included primary studies.

*   Collectively, existing tools help systematic reviewers to explore or report on context holistically. However, individually the tools differ in use of terms such as context, setting, environment and location and in what they consider as relevant.

*   Reporting standards and templates, as well as improved reporting of context in primary studies, are needed to better account for context in systematic reviews and guidelines taking a complexity perspective.

## Introduction

Complex interventions, whether clinical, health service, public health or from beyond the health sector, include multiple components that interact along non-linear causal pathways within a complex adaptive system.1 2 Interventions may share a common label and yet contain diverse content and variable components and use diverse modes of delivery, making effects unpredictable. By intervention, we include diverse procedures, practices, programmes and policies operating at different levels of system change. Yet the context or setting within which the intervention is introduced and interacts, ‘a dynamic multidimensional environment’3 constitutes the most significant aspect of this complexity.4 Decision makers need to know when, why, how and in what circumstances complex interventions work well.5 6 If a decision-maker does not understand the interplay between context and an intervention or policy (or more accurately the mechanisms by which the intervention or policy achieves its effect), then they can have only limited confidence in whether they would see the same effect in their own context.7 Such a complexity perspective poses particular challenges in a global context when producing and implementing statements that include ‘recommendations intended to optimise patient care or population health, informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options’.8 Local decision makers need to decide whether a particular guideline is relevant at all, whether it is likely to achieve a better or suboptimal effect and whether the intervention needs to be modified for contextual sensitivity.

Complex interventions interact with and modify the context within which they are introduced. Sometimes complex interventions adapt to the context in which they are implemented while some interventions explicitly aim to change the contexts within which they are implemented.2 Complex interventions may alter a discrete aspect of practice (eg, how doctors interact with patients) or something more radical (eg, smoke-free legislation).8 Key decisions when producing a particular guideline are whether it is helpful and informative to view an intervention as complex.2 9 10 Not every topic justifies the added time and resource use required to explore complexity;2 11 guideline developers may conclude that contextual variation is less critical to intervention effects than intervention complexity. Where complexity is to be considered, guideline developers must decide which characteristics or sources of complexity are to be prioritised. Evidence to Decision (EtD) frameworks offer a potential menu but not all aspects need to be addressed in every review or guideline.10 Having decided whether a review will gravitate towards a complex intervention perspective or towards a complex system perspective, a review team can develop specific review questions to consider complexity and select evidence, synthesis methods and review tools accordingly.12 Qualitative13 and other forms of evidence14 may help in understanding how an intervention interacts with, reshapes and is reshaped by its target context.

This paper reviews a selection of tools for considering context and evaluates their suitability for use in reviews and guidelines that take a complexity perspective. The paper offers guidance on good practice when considering context in systematic reviews and guidelines. The paper addresses the following questions: How is context defined and understood?; Why is context important in reviews and guidelines of complex interventions?; What tools and methods are available to handle context in systematic reviews? and What might represent good practice for considering context in reviews and guidelines?

## How is context defined and understood?

A plethora of definitions exists for ‘context’, each varying in the extent to which they adopt an ‘intervention perspective’ or a ‘system perspective’;2 making it problematic to privilege a single frame of reference.15 UK Medical Research Council guidance defines context as ‘factors external to the intervention which may influence its implementation, or whether its mechanisms of impact act as intended’.16 Further guidance defines context as ‘any feature of the circumstances in which an intervention is conceived, developed, implemented and evaluated’.17 Methodologically, a tension exists between the technical advantages of deciding on a single standardised definition of context and the conceptual richness of diverse and competing definitions. A systematic review should clearly articulate its internal definition of context, justify its selection and apply this definition consistently through the review process.

Context and setting, although used interchangeably, are not synonymous. The former typically carries an overarching sense while the latter is often locally bounded (eg, an acute hospital setting in a low-income or middle-income development context). Research and evaluation, broadly speaking, seek either to ignore, control, document or explore contextual variation. Many intervention and evaluation designs seek to ensure that the context for the intervention group is the same as that in the control group, so that the intervention is the only meaningful difference between groups. Certain contexts are more easily explored by randomised controlled trials (eg, secondary or tertiary care)18 but become less amenable to controlled studies on implementation. Current design and statistical approaches do not adequately deal with contextual variations within source studies and imperfectly represent ‘real world’ conditions within the target context.19 Demarcation between the outer setting (the economic, political and social context within which an organisation resides) and the inner setting (features of structural, political and cultural contexts through which an implementation process will proceed) are ‘not always clear’, given that the interface is ‘dynamic and sometimes precarious’.20 One concept analysis states: ‘Context reflects a set of characteristics and circumstances that consist of active and unique factors, within which the implementation is embedded’.21 Subsequently, the concept analysis unpacks context into eight domains (ie, locational, geographical, epidemiological, sociocultural, socioeconomic, ethical, legal and political). Context typically incorporates the institutional context (setting), the wider environment (eg, epidemiological, genetic, geographical, sociocultural, health system specific, political and others22) or a composite of both.

Authors of primary and secondary research find context challenging to operationalise, research and evaluate.23 Primary studies are criticised for failing to characterise context,17 yet many guidance documents recommend that contextual factors are examined within systematic reviews and syntheses.24 Review teams face particular challenges resulting from (1) the lack of contextual detail in included studies and (2) unsophisticated approaches to handling context within existing review tools.

The Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) reporting standards,25 an evidence-based minimum set of items to help authors improve reporting of systematic reviews and meta-analyses, refer to ‘context’ as the circumstances that explain why a review is currently needed (ie, the need for decision support and the prevalent state of knowledge). PRISMA does not even reference ‘context’ in its common usage as ‘the population targeted by implementation’. The PRISMA-CI extension for Complex Interventions, a supplemental set of items for reporting systematic reviews of complex interventions, partially addresses such limitations,26 27 extending to include the elements of Time and Setting. Nevertheless, this ‘intervention perspective’ proves unable to accommodate a complex systems perspective.2

Adding an additional component to a complex intervention (eg, a practice, behaviour or a way of organising care) almost invariably displaces something else, whether planned or not, thereby challenging conventional evaluation.4 While the context of an intervention is the focus for change, it is important to acknowledge that ‘usual care’ equally operates within a complex adaptive system. Interventions may not transfer to other contexts because usual care in those contexts is already more effective than the intervention or at least more responsive to local needs. Furthermore, studies frequently offer only a sketchy picture of what constitutes ‘usual care’.

## Why is context important in reviews and guidelines of complex interventions?

Key tools and methods used in the systematic review process similarly lack clarity and consensus surrounding the term ‘context’. Context is typically either reported in insufficient detail or omitted altogether from many systematic reviews. Systematic reviews are frequently characterised as seeking to ‘strip away’ context.28 Transferability of review findings, particularly to Low and Middle Income Countries (LMICs), depends on resources, health systems, cultural norms and the physical environment.29 Wider considerations include geographic, seasonal, environmental, cultural, economic, sociopolitical, regulatory and organisational inner (eg, payment policies) and outer (eg, organisational cultures) settings.30 Differences between national health services and insurance-based health systems are similarly important within high income countries.30 Above all, those examining context should seek to extend analysis beyond the caricaturisation of time and place to explore other, less salient, features of contextual variation.

Table 1 characterises three different evaluation perspectives ranging from an intervention perspective, through a more hybrid ‘partial system’ approach, to the system perspective. The table reveals increasing complexity with subsequent rows increasingly being likely to require, and to reward, examination of contextual variation.

View this table:
[Table 1](http://gh.bmj.com/content/4/Suppl_1/e000840/T1)

Table 1 
Perspectives mapped to role of context and evaluation activities

Systematic review and guideline production seeks to ‘globalise the evidence, localise the decisions’.31 This globalise-localise strategy distinguishes three ‘contexts’ of evidence production: (1) the context of the review question (the review context); (2) the contexts of included studies (the source contexts) and (3) the implementation context into which recommendations are introduced (the target context). So, a review focused on low-income and middle-income countries (the review context) may access and include studies conducted in Africa and South America (source contexts) to inform practice in South East Asia (the target context). The context within which a study is conducted may differ considerably from the target context within which study results are later used.32

## What tools and methods are available to handle context in systematic reviews?

Context is currently factored into systematic review processes in two main ways:

1.  Contextual items included within existing review tools (eg, specific tools for describing interventions and for assessing complexity).33 Many review tools already include one or more items for context.

2.  Use of a supplementary framework to explore contextual issues.30 A guideline group can use a context framework when planning reviews (eg, when formulating questions within a review) or while conducting reviews (eg, when representing subgroups within the synthesis).

Table 2 provides examples of known tools and frameworks for taking account of context. Full text searching of Google Scholar for references to ‘context’ within articles reporting systematic review methodology was used to identify candidate tools. This paper seeks to illustrate well-known or recently developed approaches to be used at different stages of the systematic review process and to summarise their potential strengths and weaknesses. While not seeking to be exhaustive, the table can be used to identify composite approaches that explore complementary perspectives of context.

View this table:
[Table 2](http://gh.bmj.com/content/4/Suppl_1/e000840/T2)

Table 2 
Tools and frameworks for handling context, mapped to stages of a systamatic review process

### How to consider context when planning, conducting and implementing reviews

Review authors have numerous opportunities to consider contextual details when planning and conducting the review and, subsequently when making recommendations (box 1).

Box 1 
### Factoring in context within systematic reviews for complex interventions

#### Throughout the review

*   0. Use a review method (eg, realist synthesis or meta-ethnography) that allows exploration of context and/or

#### Planning the review

*   1. Consider contextual details during question formulation and when defining eligibility criteria.

*   2. Describe contextual considerations in the review protocol.

#### Conducting the review

*   3. Extract contextual details as part of data extraction.

*   4. Seek missing contextual details.

*   5. Consider contextual characteristics during statistical and qualitative analyses and explore contextual heterogeneity when appropriate.

#### Making use of review findings

*   6. Report contextual details in supplementary materials.

*   7. Develop support tools to facilitate implementation and adaptation.

*   8. Describe implications for future research.

### Throughout the review

One approach to handling context within systematic reviews and guidelines is to use a review method that explicitly articulates context within its programme logic. Realist synthesis34 may identify contextual variation to be explored through complementary use of quantitative and qualitative data sources.16 Realist synthesis seeks to explain why interventions may or may not work, within which contexts, how and in what circumstances’.35 Specific techniques for juxtaposing quantitative and qualitative evidence are elaborated elsewhere in this series.13 14 Synthesis methods that specifically examine context, as with realist review, or that seek to preserve context when ‘translating’ across studies (as with a meta-ethnography alongside an effects review) offer a holistic approach. However, they struggle to deliver within the constrained time and resource timetable36 that characterises guideline production.

The value of systems thinking4 is explored elsewhere. However, although applications of system thinking in public health are emerging,37 few systematic reviews have encapsulated systems thinking within their review methods. EtD frameworks, such as the GRADE EtD38 framework and the WHO-INTEGRATE framework,10 itemise many contextual factors impacting on decision-making. Nevertheless, exhaustive frameworks may prove resource-intensive when harnessed to a systematic review. Guideline developers and the review team should consider which, if any, elements from this expansive menu will prove informative for their particular guideline/review.

Logic models are graphic descriptions of a system and are designed to identify important elements and relationships within that system.2 In seeking to portray multiple interacting elements, they represent a light touch response to context; the focus is on the intervention or programme itself, together with associated inputs and activities, and context offers a frame for intervention delivery. So a logic model may be useful for initiating discussion of context but will probably require further, more granular, specification.

### Planning the review

#### Consider contextual details during question formulation and when defining eligibility criteria

Methodologists emphasise that a review team should consider context from initiation of the review.11 39 Assumptions about context first surface within the Population, Intervention and Comparison elements of a review question; a review may define its context as ‘low and middle-income settings’, as ‘Africa’ or as a named country. However, a review question may favour geopolitical aspects of context over equally important considerations such as rural-urban differences, ethnicity, infrastructure and policy. Health systems frequently characterise context as primary, secondary, tertiary and community but this demarcation may run counter to a systems perspective.

Question formulation frameworks represent flat, linear representations of context11 that do not accommodate a systems perspective. In contrast, a systems perspective, where macrolevel environmental factors and microlevel factors of setting dynamically interplay, may require a comprehensive question framework.40 Few question formulation frameworks specify context (or associated subconcepts, eg, setting or environment). Nevertheless, question structures such as Population-Intervention-Comparison-Outcome-Context (PICOC),41 Context-Intervention-Mechanisms-Outcomes (CIMO)42 and Setting-Perspective-Interest, phenomenon of-Comparison-Evaluation (SPICE)43 offer alternatives to the PICO structure, which omits context (environment or setting) altogether and may prompt guideline developers or review authors to articulate context concerns. A review team may incorporate question structures that include environment/setting within their inclusion and exclusion criteria and the data extraction form. An innovative, yet untried, question formulation, PerSPE©TiF11 may prompt a review team to identify elements of context not typically articulated within an intervention perspective.

#### Describe how context is to be defined within the review protocol

Other aspects of the review protocol have an implicit role in defining context, for example, inclusion and exclusion criteria, the choice of data sources and search terms as well as language and date restrictions. However, two principal structures for codifying the review protocol, PRISMA-P44 and PROSPERO45 noticeably lack detail of context. PRISMA-P, a reporting standard for review protocols, simply lists setting and time frame alongside the PICO elements. No detail is given on specific details to be included. The template used to register reviews in PROSPERO, an international registry of systematic reviews, requires contextual details ‘if these form part of the review’s eligibility criteria but are not reported elsewhere in the PROSPERO record’. Examples include: ‘studies in hospital accident and emergency departments’ or ‘research in low-income and middle-income countries.’ Detailed reporting in a protocol template can help in agreeing the scope of the review and to surface assumptions of similarity and difference across contexts (eg, combining studies with broadly comparable generalisable study features, eg, low-income and middle-income countries) and splitting (only combining studies with highly comparable populations for example, East Africa).45 However, such templates require the review team to preidentify contextual differences before starting to undertake the review.

### Conducting the review

#### Extract contextual details as part of data extraction

Numerous purpose-specific frameworks exist that offer a potential structure for data extraction forms. Generic frameworks, designed for primary research, are suggested in the Medical Research Council Process Evaluation Guidance.16 Examples include the Cluster randomised controlled trials46 framework, PARIHS47 and RE-AIM.48 However, frameworks designed for use within primary studies may include too much detail to sustain data extraction for a review. Other frameworks remain experimental such as the Context and Implementation of Complex Interventions (CICI) framework, produced for a European Union-funded project as a generic framework for exploring for the context of any health technology.22 Similarly, the WHO-INTEGRATE framework enumerates the ‘Health system and broader context’.10 When conducting a systematic review of a complex intervention, a team must prioritise important contextual elements considered important for data extraction rather than seeking to use frameworks in their entirety.

Implementation frameworks (such as Normalisation Process Theory49 and the Theoretical Domains Framework50) offer an alternative lens on context. A systematic review of facilitators and barriers to team working in primary care used Normalisation Process Theory to identify the importance of contextual variation in funding models and colocation.51 The Theoretical Domains Framework was used in a systematic review on barriers and enablers to acute stroke management in the emergency department revealing important contextual differences in social influences.52 Frameworks must optimise usability and detail; the Diffusion of Innovations framework40 does not appear sufficiently nuanced for data extraction whereas the meta-framework, Consolidated Framework for Implementation Research,20 appears overly detailed. Craig and colleagues have combined and consolidated a comprehensive list of features of context.17

Potential is offered by PROGRESS-Plus, devised by the Cochrane Equity Methods Group to examine any intervention or programme, where a review team identifies concerns of equity.53 The PROGRESS-Plus framework includes contextually sensitive details such as race, religion and ethnicity and may therefore help in isolating important variations that impact on the effects of a complex intervention.54

The Intervention Complexity Assessment Tool for Systematic Reviews (iCAT-SR)33 and the Template for Intervention Description and Replication (TIDieR)55 specify elements of context alongside salient details of the target intervention. The iCAT_SR tool,33 which seeks to locate an intervention on a simple-complex continuum, references context as how ‘the effects of an intervention may be dependent on the societal, political, economic, health system or environmental context in which the intervention is delivered’. Such an approach helps to ‘embed’ context alongside intervention complexity.

The TIDieR55 checklist is used for describing interventions in sufficient detail for replication and can help to extract information on WHERE an intervention was delivered, which is important in considering context. The TIDieR framework encourages a detailed description of location including any necessary infrastructure or relevant features.55 The importance of specific factors depends on the type of health decision and the decision-making context, but the TIDieR framework may be helpful when extracting detailed information about spatial aspects of context.55 A subsequent paper evaluating TIDieR56 contrasts full reporting of interventions (to enhance replicability) with detailed reporting of context (to surface implementation issues). TIDieR, as the first major reporting tool specifically to examine implications when reporting context, signals an opportunity to recalibrate away from seeking to replicate context to the more realistic aspiration of seeking to understand and interpret its effects.

While these tools and frameworks present possible structures for extracting data on context, they tend to artificially separate intervention characteristics and contextual considerations, which is unhelpful when reviews and guidelines specifically need to identify and take account of sources of complexity.

#### Seek missing contextual details

Where the review team considers that important details about context are missing from the included studies, there are three principal options for locating this information.57 The CLUSTER technique58 has been devised for searching systematically for associated (‘sibling’) reports, such as pilot, feasibility studies, doctoral theses, book chapters, process evaluations and qualitative research, that offer supplementary detail beyond that retrieved from a systematic subject search.

Alternatively, a team can conduct supplementary web searches to build up context templates from routine government data and from policy sources for the study context of each ‘index study’. Policy and sociodemographic characteristics appear as national profiles on websites of government agencies or of international organisations such as the World Bank or WHO. Institutional websites also host organisational details, for example, the number of hospital beds or the catchment area for a health service. Finally, the team can harness the collective knowledge and expertise from an expert advisory group, but such approaches pose challenges to the need to be both systematic and explicit.

#### Consider contextual characteristics during statistical and qualitative analyses and explore contextual heterogeneity when appropriate

Where appropriate and feasible, a review team should consider context, as identified a priori in the protocol, when grouping studies, conducting analyses and exploring heterogeneity.59 Higgins and colleagues describe statistical and non-statistical methods available for synthesising quantitative data,19 including subgroup analyses and meta-regression for examining sources of heterogeneity including context. Flemming and colleagues describe approaches for qualitative evidence synthesis, particularly helpful for exploring variation in context and explaining why certain interventions work in some contexts and not others.13 Noyes and colleagues address methods for mixed method synthesis and how such evidence can help a decision-maker to understand how context may affect feasibility and acceptability.14 All these methods possess advantages and disadvantages; essentially, though, the choice of methods centres on the extent to which a review team aims to gain a greater understanding of the study context (by examining potential explanatory factors within the study data) or the extent to which the team seeks to focus on transferability from the study context to the target context (by exploring shared contextual characteristics).

### Making use of the review findings

#### Report contextual details in supplementary materials

Reporting standards for primary studies seek to encourage more complete reporting of context; however, they prescribe diverse levels of suggested reporting detail.60 Craig and colleagues affirm that ‘indicating and describing clearly the contexts in which the research was carried out, how they affect implementation and variation in impacts and the limits to the applicability of the results should make reports of primary studies and evidence syntheses more useful’17 to researchers and decision-makers alike. Shoveller and colleagues report from a snapshot of current journals that context was typically treated as a ‘black box’ or problematised as something to be ‘controlled for’.61 A systematic review of a complex intervention can only report contextual details if they are available in the index primary trial reports or in supplementary publications. CONSORT (2010) suggests reporting ‘other information about setting or location that could have influenced the observed results’62 yet offers limited guidance on how context should be reported.

Currently, PRISMA25 demonstrates superficial engagement with the need to report details of context. The specific PRISMA-CI extension for complex interventions does, however, extend coverage of contextual detail and hint at potential future directions.26 27 Thus, the PICO structure becomes a six-element PICOTS mnemonic to include Time and Setting.27 Furthermore, PRISMA-CI recognises organisational features, geographic location; financial setting and clinical setting and, optionally, ‘rival activities’27. In contrast, the Enhancing Transparency in Reporting the synthesis of Qualitative research (ENTREQ) reporting standard63 does not include any item specifically targeted at context and, indeed only refers to population and country. ENTREQ does, however, advocate the need to specify context when formulating the review question and also alludes to concerns of transferability.

The WHO has recently sought to enhance reporting of programme level detail, recognising that the impact of many social and behavioural programmes is ‘tied to the local context (eg, sociocultural, socioeconomic, geographical, legal, political, health system) and to the processes of implementation, which may not be easily described’.64

Tools for assessing the quality (or certainty) of evidence contained in systematic reviews, that is the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach for quantitative outcomes65 and the Confidence in the Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach for synthesised qualitative findings66 facilitate post hoc consideration of context when moving from evidence to recommendations. GRADE assessment criteria recognise contextual variation (eg, geographic setting, culture, subcultures and resources) and also differences in perspective (patients, providers, public health and society and of individual vs population based evaluation frames).67 Both the GRADE domain of indirectness68 and the GRADE-CERQual component of ‘relevance’ require a reviewer to assess the extent to which study contexts match the target contexts.69

Grading systems are constrained by their reliance on the level of detail in the source primary studies; detail that is often absent. Furthermore, they depend on the ability of the reviewer to identify salient contextual detail a priori and to make this detail the focus for analysis. Finally, summary of findings65 66 are limited in the contextual detail they can accommodate, necessitating difficult choices.

#### Develop support tools to facilitate implementation and adaptation of systematic review findings

Applicability or generalisability checklists, of which more than 25 frameworks exist,70 help when translating systematic review findings into specific contexts. The SUPPORT tools ([https://health-policy-systems.biomedcentral.com/articles/supplements/volume-7-supplement-1](https://health-policy-systems.biomedcentral.com/articles/supplements/volume-7-supplement-1)), written for decision makers and those who support these decision makers, include tools on assessing the applicability of review findings71 and using local contextual evidence.72

Tools to support implementation include case studies, country profiles and vignettes which prompt the user to consider potential variation in context. Individual country case studies can help decision-makers to visualise differences between a source context and a target context or to benchmark across countries.73 However, numbers of case studies are necessarily limited and they may lack the depth required to unearth critical differences. Country profiles require a study author to make an a priori ‘editorial’ judgement on what is ‘relevant’. Vignettes use narrative techniques to characterise complex circumstances and to trigger critical thinking about similarities and differences with local practice. Typically vignettes simplify population characteristics (eg, age or severity of a condition) but they could equally capture contextual characteristics. For example, Gauld *et al,* (2012) use vignettes to characterise qualitatively the primary care systems of 12 different countries, supported by routine quantitative data on key indicators.74

Spreadsheets and calculators require users to add local contextual data to be factored within resultant calculations and projections. So, for example, the National Institute for Health and Clinical Excellence in the UK offers return on investment tools that allow local health providers to configure optimal packages of interventions for physical activity and alcohol or tobacco control for their populations.75 The NICE Guidelines programme routinely produces three types of implementation support tool to accompany each guideline:76

*   a baseline assessment tool,

*   clinical audit tools,

*   a costing report and costing template, or a costing statement.

Such tools use common software, such as Microsoft Excel, to improve uptake.

The Oxford Implementation Index offers a rare example of a tool to help reviewers identify and extract contextual information associated with implementation from studies included in a systematic review.77 Other tools include implementation workbooks and worksheets.78 79 The ADAPTE toolkit80 highlights criteria for assessing whether a recommendation is applicable to a local context.

Considering contextual factors on completion of a review, using applicability checklists and decision-making frameworks, helps to make review findings more useful to guideline developers. Alternatively, strengthening the conduct of the systematic review19 29 can lead a review team to consider a complexity perspective a priori using an EtD-making framework.10

Context is pivotal when localising guideline recommendations. Guidance from the GRADE Working Group directs guideline developers in deciding when to adopt (use a guideline ‘as is’), adapt (introduce local evidence and expertise to rework guidelines) or contextualise (reframe recommendations in light of local resources and constraints) and when to initiate guideline development de novo. 81 82

#### Describe implications for future research

An important outcome of any systematic review, or guideline, is articulating gaps in knowledge or evidence on the topic. However, often reports of reviews and guidelines provide only general observations and miss the opportunity to identify important gaps relating to different contexts. Where important contexts are missing, resulting in ‘absolute gaps’ or ‘synthesis gaps’83 the review team can identify implications for future research. The Evidence Population Intervention Comparison Outcome Timestamp (EPICOT) framework, developed by organisations involved in commissioning or summarising research, is recommended for formulating research recommendations.84 Optional elements in the framework (current state of evidence (E), appropriate study type (s), disease burden and relevance (d) and timeliness (t)) help to ensure more detailed definition of context.84 The WHO Guidelines on Indoor Air Quality85 used this EPICOT reporting structure. However, the EPICOT framework shares limitations of the PICO question framework in providing a static linear framework and it does not take into account other important characteristics of the immediate setting or the wider environment.

## What might represent good practice for considering context in reviews and guidelines?

This paper reaffirms that, where possible, systematic review authors and guideline developers should identify relevant aspects of context from the beginning of the process, when the scope of the review or guideline is determined and the questions identified.86 Sandelowski offers some useful ‘rules of thumb’ on how to anchor text to context in restatements of findings.87 Little consensus exists on how best to identify and select relevant aspects of context that may be important in reviews and guidelines considering a complexity perspective. Other papers2 10 in this series offer practical suggestions on how to achieve this.

A review team should recognise that different types of evidence can contribute to an improved understanding of context. For example, Craig and colleagues illustrate how inclusion of qualitative evidence, in the form of a meta-ethnography can help in exploring context.17 Natural experiments and uncontrolled studies can present opportunities for understanding context and yet are frequently excluded from systematic reviews.17 Systematic reviews can further be designed to incorporate implementation and outcome evidence, rarely reported within a single primary study but usefully juxtaposed within implementation reviews. Ideally, a review team will plan a priori analyses based on anticipated differences in context.88 Nevertheless, if they have not preplanned such analyses, exploratory post hoc analyses, clearly identified as such, may be appropriate in generating further hypotheses.88

From initiation of a review, the review team should look out for studies reporting interventions from highly disparate contexts, whether differences are openly apparent or masked by incomplete reporting.77 Variations in context impact on the review question, the search strategy, the inclusion and exclusion criteria, the data extraction forms and any subgroup analyses and the resultant recommendations. Inevitably, further contextual features emerge throughout the review; a review team should document these, together with implications for the final report. Ongoing interaction with stakeholders is one way to ensure that contextual factors do not surface unexpectedly during the course of a review.

Box 2 illustrates how a joined-up approach to handling context might be built within an existing guideline process. The chosen example of a complex intervention, antenatal care, is delivered in diverse ways across different countries and settings. Therefore, results from individual trials may not generalise to other types of population and setting.89 Box 2 illustrates multiple points at which context might be introduced within a systematic review while illustrating the diversity of approaches to context employed by each tool or framework.

Box 2 
### Factoring in Context—a hypothetical example

#### Planning the review

A guideline development group on antenatal care selects the WHO-INTEGRATE framework10 to identify and discuss the importance of context throughout the development process. The group uses the PerSPE©TiF11 question formulation (From the *Per*spective of a Pregnant Woman in the *S*etting of Rural Communities how does the *P*henomenon of Facility Based Care within an *E*nvironment of Poor Transport Infrastructure and Distantly-Located Facilities (*C*ompare with Traditional Birth Attendants at Home) up to and including the *Ti*me of Delivery in terms of *F*indings related to the Woman's Perceptions and Experiences?) to define the scope of the review. Using the PerSPE©TiF structure, they craft a review protocol together with inclusion and exclusion criteria to define and anticipate salient aspects of context.

#### Conducting the review

The review team takes details from the protocol and selects an appropriate data extraction framework (the Context and Implementation of Complex Interventions (CICI) framework,22 deciding that broader environmental concerns are more critical to this population-based guideline than the specific setting. They access websites to elicit extra contextual data and identify ‘clusters’ of associated studies using the CLUSTER technique.58 Where critical details are missing, the review team contacts authors of included studies requesting additional information, setting a deadline for response. The review team juxtaposes results from a quantitative meta-analysis and a qualitative synthesis using a logic model.95 Important details of context inform plans for statistical and qualitative analyses and the statistical team explore contextual heterogeneity across predefined subgroups.19

#### Making recommendations

The review team provides appendices detailing the context described in each included study. The guideline development group pays particular attention to factors relating to Indirectness and Relevance when formulating their GRADE65 and GRADE-CERQual66 recommendations. The group uses an adaptation framework (ADAPTE94) to provide contextually-sensitive recommendations for major regions of the world, allowing each country to tailor country-specific recommendations, where feasible. The guideline developers provide accompanying spreadsheets and vignettes to facilitate roll-out and the group frames recommendations or using the EPICOT-dts framework.84

## Avoiding interpretation errors in connection with context

Based on the above scenario (box 2), we have identified and labelled three ways in which readers may misinterpret context within systematic review findings; variously termed the ‘universal fallacy’, the ‘focal fallacy’ and the ‘evidence availability fallacy’ (also see online supplementary appendix):

### Supplementary data

[[bmjgh-2018-000840supp001.pdf]](pending:yes)

1.  A reader may think that a review finding that is not qualified by contextual descriptors (eg, time, place, socioeconomic factors, religious beliefs and so on), is a universal finding (ie, applying to all contexts) (the universal fallacy).

2.  A reader may think that a finding only applies to the context that is the focus of the review question (when it is actually more generalisable) (the focal fallacy).

3.  A reader may think that a finding only applies to contexts studied within included studies (when this simply characterises where research has been undertaken) (the evidence availability fallacy).

To avoid these interpretative errors, review authors and guideline developers can better articulate context-sensitivity when framing review findings and subsequent recommendations:

*   **By Exception**—A review team could highlight that exceptions (divergent study contexts) are different from all the other study contexts or from all other contexts encompassed by a review question, for example, ‘This finding seems to apply irrespective of context, except in X, Y, Z where primary care systems continue to be disrupted by conflict and migration’.

*   **By Inclusion**—A review team could explicitly state contexts to which findings apply. This flags the distinction between ‘We have evidence that this only applies to these contexts’ and ‘We only have evidence that this applies to these contexts’.

*   **By Exclusion**—A review team could explicitly state contexts to which findings do not apply.

*   **By Assumption**—A review team could define relevance according to the presence or absence of common mechanisms or contextual factors, for example, ‘This finding is likely to apply in those contexts with a developed method for priority setting for nutrition’.

Supporting recommendations by assumption should be treated with caution, comparable to circumspect use of indirect comparison in quantitative research. Nevertheless, the value of explanatory theory is increasingly recognised and its contribution undoubtedly extends beyond the theories of change or behavioural and social theories that are the focus of this paper.90

## Limitations of this analysis

Our brief selection of high-profile or recently developed frameworks and tools for taking account of context in systematic reviews and guidelines reveals that such tools remain limited in their facility to accommodate a complex systems perspective.2 This selection is necessarily arbitrary given that information retrieval strategies to identify review tools that target context systematically do not currently exist. Highlighted tools and frameworks are illustrative requiring that the example given in box 2 is hypothetical and, indeed aspirational.

## Conclusion

Approaches to context within systematic reviews typically share the following limitations:

*   Tools typically capture context at a single timepoint and are barely able to capture prevailing trends.

*   Many tools derive from use in primary research and either have limited utility within synthesis or have not been fully evaluated.

*   Particular tools focus on a single specific level of context, particularly true of implementation tools that target the direct setting within which the intervention is delivered.

We have also identified a potential tension given that tools that are best equipped for considering context may, by their very nature, not prove practical as a structure or framework for data extraction or analysis. In the absence of empirical evidence on features of context that impact most on intervention success, we observe that tools tend to become increasingly expansive. This, in turn, makes data extraction and analysis more time consuming and resource intensive.

For reviewers and guideline developers, this paper emphasises the practical need to select from this toolkit judiciously. Context should be addressed early in guideline development and then revisited throughout the review process. Frameworks offer a useful structure for ensuring that data on context are extracted for the review. Checklists prompt decision-makers to consider important factors and reporting standards ensure that, where reported, requisite details are present in the report or in supplementary material accompanying each report. Knowledge translation tools such as vignettes, country profiles, calculators and spreadsheets can help in translating between the study contexts and the target context. With such diverse approaches, the current way forward requires composite use of different tools at appropriate stages of the process. Our brief scan of illustrative tools reveals a need for a coherent approach that runs throughout the review process and that underpins the individual tools used at each stage. Approaches to context to date suggests that the components already exist; the outstanding methodological challenge is to agree a coherent approach within which existing tools might be integrated.

Recent years have seen collective recognition of the importance of context when selecting, evaluating and implementing complex interventions. This paper consolidates collective learning in identifying and using tools for handling context and meeting the ever more complex demands of guideline developers and other decision-makers. This overview signals how context might possibly be addressed within systematic reviews and guidelines. For the present, review teams are encouraged to extract contextual details when available, elicit them when feasible and reconstruct them when available data allows. Review teams and guideline producers should seek to navigate a path that is parsimonious in terms of resource and data extraction and yet sensitive to their specific contexts and the values, preferences and needs of their target populations.

## Acknowledgments

We gratefully acknowledge input provided during the drafting process by Anayda Portela and Dr Susan L Norris from the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health and by members of the other Complex Interventions Working Groups and by Dr Helen Smith at the editorial stage.

## Footnotes

*   Handling editor Valery Ridde

*   Contributors AB drafted the article. GM, KF and JN contributed to sections of the final draft article and RG, NR and ÖT provided comments on final drafts prior to completion and submission. All read comments from reviewers and contributed to the amended version. For the final draft, AB made changes and coauthors read and agreed to changes.

*   Funding Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation.

*   Disclaimer OT and NR are staff members of WHO. The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions or policies of WHO.

*   Competing interests None declared.

*   Patient consent Not required.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement No additional data are available.

*   Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

[https://creativecommons.org/licenses/by-nc/3.0/igo/](https://creativecommons.org/licenses/by-nc/3.0/igo/)
This is an open access article distributed under the terms of the Creative Commons Attribution IGO License ([CC BY NC 3.0 IGO](https://creativecommons.org/licenses/by-nc/3.0/igo/)), which permits use, distribution,and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.Disclaimer: The author is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.

## References

1.  1.Petticrew M. When are complex interventions 'complex'? When are simple interventions 'simple'? Eur J Public Health 2011;21:397–8.[doi:10.1093/eurpub/ckr084](http://dx.doi.org/10.1093/eurpub/ckr084)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/eurpub/ckr084&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21771736&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000293025800001&link_type=ISI) 

2.  2.Petticrew M, Knai C, Thomas J, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Health 2019;.
    
    

3.  3.Kelly MP, Noyes J, Kane RL, et al. AHRQ series on complex intervention systematic reviews-paper 2: defining complexity, formulating scope, and questions. J Clin Epidemiol 2017;90:11–18.[doi:10.1016/j.jclinepi.2017.06.012](http://dx.doi.org/10.1016/j.jclinepi.2017.06.012)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

4.  4.Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol 2009;43267–76.[doi:10.1007/s10464-009-9229-9](http://dx.doi.org/10.1007/s10464-009-9229-9)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1007/s10464-009-9229-9&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19390961&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000265685500008&link_type=ISI) 

5.  5.Webster J, Chandramohan D, Hanson K. Methods for evaluating delivery systems for scaling-up malaria control intervention. BMC Health Serv Res 2010;10(Suppl 1):S8.[doi:10.1186/1472-6963-10-S1-S8](http://dx.doi.org/10.1186/1472-6963-10-S1-S8)
    
    

6.  6.Mills A, Gilson L, Hanson K, et al. What do we mean by rigorous health-systems research? Lancet 2008;372:1527–9.[doi:10.1016/S0140-6736(08)61633-5](http://dx.doi.org/10.1016/S0140-6736(08)61633-5)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(08)61633-5&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=18984174&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

7.  7.Pawson R, Tilley N. Realistic evaluation. London: Sage, 1997.
    
    

8.  8.Porter S, O'Halloran P. The use and limitation of realistic evaluation as a tool for evidence-based practice: a critical realist perspective. Nurs Inq 2012;19:18–28.[doi:10.1111/j.1440-1800.2011.00551.x](http://dx.doi.org/10.1111/j.1440-1800.2011.00551.x)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/j.1440-1800.2011.00551.x&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22212367&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

9.  9.Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health 2017;3:132–5.[doi:10.1016/j.ssmph.2016.12.005](http://dx.doi.org/10.1016/j.ssmph.2016.12.005)
    
    

10. 10.Rehfuess EA, Stratil JM, Scheel IB, et al. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating who norms and values and a complexity perspective. BMJ Glob Health 2019;.
    
    

11. 11.Booth A, Noyes J, Flemming K, et al. Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Glob Health 2019;.
    
    

12. 12.Petticrew M, Rehfuess E, Noyes J, et al. Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute. J Clin Epidemiol 2013;66:1230–43.[doi:10.1016/j.jclinepi.2013.06.005](http://dx.doi.org/10.1016/j.jclinepi.2013.06.005)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.06.005&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23953082&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

13. 13.Flemming K, Booth A, Garside R, et al. Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods. BMJ Glob Health 2019;.
    
    

14. 14.Noyes J, Booth A, Moore G, et al. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: Clarifying the purpose, designs and outlining some methods. BMJ Glob Health 2019;.
    
    

15. 15.McKillop A, Shaw J, Sheridan N, et al. Understanding the attributes of implementation frameworks to guide the implementation of a model of community-based integrated health care for older adults with complex chronic conditions: a metanarrative review. Int J Integr Care 2017;17:10.[doi:10.5334/ijic.2516](http://dx.doi.org/10.5334/ijic.2516)
    
    

16. 16.Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: medical research council guidance. BMJ 2015;350:h1258.[doi:10.1136/bmj.h1258](http://dx.doi.org/10.1136/bmj.h1258)
    
    [FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTAvbWFyMTlfNi9oMTI1OCI7czo0OiJhdG9tIjtzOjI5OiIvYm1qZ2gvNC9TdXBwbF8xL2UwMDA4NDAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17. 17.Craig P, Di Ruggiero E, Frohlich KL. Taking account of context in population health intervention research: guidance for producers, users and funders of research. Southampton: NIHR Evaluation, Trials and Studies Coordinating Centre, 2018.
    
    

18. 18.Poland B, Krupa G, McCall D. Settings for health promotion: an analytic framework to guide intervention design and implementation. Health Promot Pract 2009;10:505–16.[doi:10.1177/1524839909341025](http://dx.doi.org/10.1177/1524839909341025)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1177/1524839909341025&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19809004&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

19. 19.Higgins JPT, López-López JA, Becker BJ, et al. Synthesising quantitative evidence in systematic reviews of complex health interventions. BMJ Glob Health 2019;.
    
    

20. 20.Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50.[doi:10.1186/1748-5908-4-50](http://dx.doi.org/10.1186/1748-5908-4-50)
    
    

21. 21.Pfadenhauer LM, Mozygemba K, Gerhardus A, et al. Context and implementation: a concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhwes 2015;109:103–14.[doi:10.1016/j.zefq.2015.01.004](http://dx.doi.org/10.1016/j.zefq.2015.01.004)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

22. 22.Pfadenhauer LM, Gerhardus A, Mozygemba K, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implement Sci 2017;12:21.[doi:10.1186/s13012-017-0552-5](http://dx.doi.org/10.1186/s13012-017-0552-5)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1186/s13012-017-0552-5&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=28202031&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

23. 23.Howarth E, Devers K, Moore G. Contextual issues and qualitative research. NIHR Journals Library 2016.
    
    

24. 24.Mathes T, Antoine SL, Prengel P, et al. Health technology assessment of public health interventions: a synthesis of methodological guidance. Int J Technol Assess Health Care 2017;33:135–46.[doi:10.1017/S0266462317000228](http://dx.doi.org/10.1017/S0266462317000228)
    
    

25. 25.Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535.[doi:10.1136/bmj.b2535](http://dx.doi.org/10.1136/bmj.b2535)
    
    

26. 26.Guise JM, Butler ME, Chang C, et al. AHRQ series on complex intervention systematic reviews-paper 6: PRISMA-CI extension statement and checklist. J Clin Epidemiol 2017;90:43–50.[doi:10.1016/j.jclinepi.2017.06.016](http://dx.doi.org/10.1016/j.jclinepi.2017.06.016)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

27. 27.Guise JM, Butler M, Chang C, et al. AHRQ series on complex intervention systematic review-paper 7: PRISMA-CI elaboration and explanation. J Clin Epidemiol 2017;90:51–8.[doi:10.1016/j.jclinepi.2017.06.017](http://dx.doi.org/10.1016/j.jclinepi.2017.06.017)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

28. 28.Greenhalgh T. Outside the box: Why are Cochrane reviews so boring? Br J Gen Pract 2012;62:371.[doi:10.3399/bjgp12X652418](http://dx.doi.org/10.3399/bjgp12X652418)
    
    [FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNjIvNjAwLzM3MSI7czo0OiJhdG9tIjtzOjI5OiIvYm1qZ2gvNC9TdXBwbF8xL2UwMDA4NDAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

29. 29.Gruen RL, Morris PS, McDonald EL, et al. Making systematic reviews more useful for policy-makers. Bull World Health Organ 2005;83:480.[doi:/S0042-96862005000600020](http://dx.doi.org//S0042-96862005000600020)
    
    

30. 30.Butler M, Epstein RA, Totten A, et al. AHRQ series on complex intervention systematic reviews-paper 3: adapting frameworks to develop protocols. J Clin Epidemiol 2017;90:19–27.[doi:10.1016/j.jclinepi.2017.06.013](http://dx.doi.org/10.1016/j.jclinepi.2017.06.013)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2017.06.013&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

31. 31.Eisenberg JM. Globalize the evidence, localize the decision: evidence-based medicine and international diversity. Health Aff 2002;21:166–8.[doi:10.1377/hlthaff.21.3.166](http://dx.doi.org/10.1377/hlthaff.21.3.166)
    
    [Abstract/FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjg6IjIxLzMvMTY2IjtzOjQ6ImF0b20iO3M6Mjk6Ii9ibWpnaC80L1N1cHBsXzEvZTAwMDg0MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

32. 32.Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet 2005;365:82–93.[doi:10.1016/S0140-6736(04)17670-8](http://dx.doi.org/10.1016/S0140-6736(04)17670-8)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(04)17670-8&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=15639683&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000226213100032&link_type=ISI) 

33. 33.Lewin S, Hendry M, Chandler J, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol 2017;17:76.[doi:10.1186/s12874-017-0349-x](http://dx.doi.org/10.1186/s12874-017-0349-x)
    
    

34. 34.Brennan C, Greenhalgh J, Pawson R. Guidance on guidelines: Understanding the evidence on the uptake of health care guidelines. J Eval Clin Pract 2018;24:105–16.[doi:10.1111/jep.12734](http://dx.doi.org/10.1111/jep.12734)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

35. 35.Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implement Sci 2012;7:33.[doi:10.1186/1748-5908-7-33](http://dx.doi.org/10.1186/1748-5908-7-33)
    
    

36. 36.Booth A, Noyes J, Flemming K, et al. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions. In: Integrate-HTA, 2016.
    
    

37. 37.Carey G, Malbon E, Carey N, et al. Systems science and systems thinking for public health: a systematic review of the field. BMJ Open 2015;5:e009002.[doi:10.1136/bmjopen-2015-009002](http://dx.doi.org/10.1136/bmjopen-2015-009002)
    
    

38. 38.Alonso-Coello P, Schünemann HJ, Moberg J, et al. Grade Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ 2016;353:i2016.[doi:10.1136/bmj.i2016](http://dx.doi.org/10.1136/bmj.i2016)
    
    

39. 39.Rehfuess EA, Akl EA. Current experience with applying the GRADE approach to public health interventions: an empirical study. BMC Public Health 2013;13:9.[doi:10.1186/1471-2458-13-9](http://dx.doi.org/10.1186/1471-2458-13-9)
    
    

40. 40.Rogers EM. The diffusion of innovation. 5th edn. New York, 2003.
    
    

41. 41.Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Blackwell Publishing: Oxford, 2006.
    
    

42. 42.Denyer D, Tranfield D, van Aken JE. Developing design propositions through research synthesis. Organization Studies 2008;29:393–413.[doi:10.1177/0170840607088020](http://dx.doi.org/10.1177/0170840607088020)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1177/0170840607088020&link_type=DOI) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000254720400005&link_type=ISI) 

43. 43.Booth A. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech 2006;24:355–68.[doi:10.1108/07378830610692127](http://dx.doi.org/10.1108/07378830610692127)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1108/07378830610692127&link_type=DOI) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000241513500004&link_type=ISI) 

44. 44.Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015;4:1.[doi:10.1186/2046-4053-4-1](http://dx.doi.org/10.1186/2046-4053-4-1)
    
    

45. 45.Sideri S, Papageorgiou SN, Eliades T. Registration in the international Prospective Register of Systematic Reviews (PROSPERO) of systematic review protocols was associated with increased review quality. J Clin Epidemiol 2018;100:103–10.[doi:10.1016/j.jclinepi.2018.01.003](http://dx.doi.org/10.1016/j.jclinepi.2018.01.003)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

46. 46.Grant A, Treweek S, Dreischulte T, et al. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013;14:15.[doi:10.1186/1745-6215-14-15](http://dx.doi.org/10.1186/1745-6215-14-15)
    
    

47. 47.Helfrich CD, Damschroder LJ, Hagedorn HJ, et al. A critical synthesis of literature on the promoting action on research implementation in health services (PARIHS) framework. Implement Sci 2010;5:82.[doi:10.1186/1748-5908-5-82](http://dx.doi.org/10.1186/1748-5908-5-82)
    
    

48. 48.Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health 2013;103:e38–e46.[doi:10.2105/AJPH.2013.301299](http://dx.doi.org/10.2105/AJPH.2013.301299)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.2105/AJPH.2013.301299&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23597377&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

49. 49.Murray E, Treweek S, Pope C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med 2010;8:63.[doi:10.1186/1741-7015-8-63](http://dx.doi.org/10.1186/1741-7015-8-63)
    
    

50. 50.Mosavianpour M, Sarmast HH, Kissoon N, et al. Theoretical domains framework to assess barriers to change for planning health care quality interventions: a systematic literature review. J Multidiscip Healthc 2016;9:303–10.[doi:10.2147/JMDH.S107796](http://dx.doi.org/10.2147/JMDH.S107796)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

51. 51.O'Reilly P, Lee SH, O'Sullivan M, et al. Assessing the facilitators and barriers of interdisciplinary team working in primary care using normalisation process theory: an integrative review. PLoS One 2017;12:e0177026.[doi:10.1371/journal.pone.0177026](http://dx.doi.org/10.1371/journal.pone.0177026)
    
    

52. 52.Craig LE, McInnes E, Taylor N, et al. Identifying the barriers and enablers for a triage, treatment, and transfer clinical intervention to manage acute stroke patients in the emergency department: a systematic review using the theoretical domains framework (TDF). Implement Sci 2016;11:157.[doi:10.1186/s13012-016-0524-1](http://dx.doi.org/10.1186/s13012-016-0524-1)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

53. 53.O'Neill J, Tabish H, Welch V, et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol 2014;67:56–64.[doi:10.1016/j.jclinepi.2013.08.005](http://dx.doi.org/10.1016/j.jclinepi.2013.08.005)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.08.005&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=24189091&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

54. 54.Attwood S, van Sluijs E, Sutton S. Exploring equity in primary-care-based physical activity interventions using PROGRESS-Plus: a systematic review and evidence synthesis. Int J Behav Nutr Phys Act 2016;13:60.[doi:10.1186/s12966-016-0384-8](http://dx.doi.org/10.1186/s12966-016-0384-8)
    
    

55. 55.Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687.[doi:10.1136/bmj.g1687](http://dx.doi.org/10.1136/bmj.g1687)
    
    

56. 56.Cotterill S, Knowles S, Martindale AM, et al. Getting messier with TIDieR: embracing context and complexity in intervention reporting. BMC Med Res Methodol 2018;18:12.[doi:10.1186/s12874-017-0461-y](http://dx.doi.org/10.1186/s12874-017-0461-y)
    
    

57. 57.Noyes J, Hendry M, Lewin S, et al. Qualitative "trial-sibling" studies and "unrelated" qualitative studies contributed to complex intervention reviews. J Clin Epidemiol 2016;74:133–43.[doi:10.1016/j.jclinepi.2016.01.009](http://dx.doi.org/10.1016/j.jclinepi.2016.01.009)
    
    

58. 58.Booth A, Harris J, Croot E, et al. Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol 2013;13:118.[doi:10.1186/1471-2288-13-118](http://dx.doi.org/10.1186/1471-2288-13-118)
    
    

59. 59.Lavis JN, Oxman AD, Lewin S, et al. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Res Policy Syst 2009;7(Suppl 1):I1.[doi:10.1186/1478-4505-7-S1-I1](http://dx.doi.org/10.1186/1478-4505-7-S1-I1)
    
    

60. 60.Altman DG, Simera I. A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. J R Soc Med 2016;109:67–77.[doi:10.1177/0141076815625599](http://dx.doi.org/10.1177/0141076815625599)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1177/0141076815625599&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=26880653&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

61. 61.Shoveller J, Viehbeck S, Di Ruggiero E, et al. A critical examination of representations of context within research on population health interventions. Critical Public Health 2016;26:487–500.[doi:10.1080/09581596.2015.1117577](http://dx.doi.org/10.1080/09581596.2015.1117577)
    
    

62. 62.Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332.[doi:10.1136/bmj.c332](http://dx.doi.org/10.1136/bmj.c332)
    
    

63. 63.Tong A, Flemming K, McInnes E, et al. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol 2012;12:181.[doi:10.1186/1471-2288-12-181](http://dx.doi.org/10.1186/1471-2288-12-181)
    
    

64. 64.Kågesten AE, Tunçalp Ö, Portela A, et al. Programme Reporting Standards (PRS) for improving the reporting of sexual, reproductive, maternal, newborn, child and adolescent health programmes. BMC Med Res Methodol 2017;17:117.[doi:10.1186/s12874-017-0384-7](http://dx.doi.org/10.1186/s12874-017-0384-7)
    
    

65. 65.Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6.[doi:10.1136/bmj.39489.470347.AD](http://dx.doi.org/10.1136/bmj.39489.470347.AD)
    
    [FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzYvNzY1MC85MjQiO3M6NDoiYXRvbSI7czoyOToiL2JtamdoLzQvU3VwcGxfMS9lMDAwODQwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

66. 66.Lewin S, Glenton C, Munthe-Kaas H, et al. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med 2015;12:e1001895.[doi:10.1371/journal.pmed.1001895](http://dx.doi.org/10.1371/journal.pmed.1001895)
    
    

67. 67.Andrews JC, Schünemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength. J Clin Epidemiol 2013;66:726–35.[doi:10.1016/j.jclinepi.2013.02.003](http://dx.doi.org/10.1016/j.jclinepi.2013.02.003)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.02.003&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23570745&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

68. 68.Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol 2011;64:1303–10.[doi:10.1016/j.jclinepi.2011.04.014](http://dx.doi.org/10.1016/j.jclinepi.2011.04.014)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2011.04.014&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21802903&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

69. 69.Noyes J, Booth A, Lewin S, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 6: how to assess relevance of the data. Implement Sci 2018;13(Suppl 1):4.[doi:10.1186/s13012-017-0693-6](http://dx.doi.org/10.1186/s13012-017-0693-6)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1186/s13012-017-0693-6&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

70. 70.Burchett H, Umoquit M, Dobrow M. How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks. J Health Serv Res Policy 2011;16:238–44.[doi:10.1258/jhsrp.2011.010124](http://dx.doi.org/10.1258/jhsrp.2011.010124)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1258/jhsrp.2011.010124&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21965426&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

71. 71.Lavis JN, Oxman AD, Souza NM, et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 9: assessing the applicability of the findings of a systematic review. Health Res Policy Syst 2009;7(Suppl 1):S9.[doi:10.1186/1478-4505-7-S1-S9](http://dx.doi.org/10.1186/1478-4505-7-S1-S9)
    
    

72. 72.Lewin S, Oxman AD, Lavis JN, et al. SUPPORT tools for evidence-informed policymaking in health 11: Finding and using evidence about local conditions. Health Res Policy Syst 2009;7(Suppl 1):S11.[doi:10.1186/1478-4505-7-S1-S11](http://dx.doi.org/10.1186/1478-4505-7-S1-S11)
    
    

73. 73.Hunter BM, Requejo JH, Pope I, et al. National health policy-makers' views on the clarity and utility of Countdown to 2015 country profiles and reports: findings from two exploratory qualitative studies. Health Res Policy Syst 2014;12:40.[doi:10.1186/1478-4505-12-40](http://dx.doi.org/10.1186/1478-4505-12-40)
    
    

74. 74.Gauld R, Blank R, Burgers J. The world health report 2008 - primary healthcare: how wide is the gap between its agenda and implementation in 12 high-income health systems? Healthc Policy 2012;7:38–58.
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.12927/hcpol.2013.22778&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23372580&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

75. 75.Pokhrel S, Owen L, Coyle K. Supporting decision making. London: Palgrave, 2017.
    
    

76. 76.National Institute for Health and Care Excellence (NICE). Assessing resource impact process manual: technology appraisals and highly specialised technologies. London: National institute for Health and Care Excellence (NICE), 2017.
    
    

77. 77.Montgomery P, Underhill K, Gardner F, et al. The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses. J Clin Epidemiol 2013;66:874–82.[doi:10.1016/j.jclinepi.2013.03.006](http://dx.doi.org/10.1016/j.jclinepi.2013.03.006)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.03.006&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23810026&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

78. 78.Alvarez E, Lavis J. Implementing health systems guidance: a workbook to support the contextualization of recommendations at the national or subnational level. Hamilton, Canada: McMaster Health Forum, 2016.
    
    

79. 79.Alvarez E, Lavis JN, Brouwers M, et al. Developing a workbook to support the contextualisation of global health systems guidance: a case study identifying steps and critical factors for success in this process at WHO. Health Res Policy Syst 2018;16:19.[doi:10.1186/s12961-018-0297-x](http://dx.doi.org/10.1186/s12961-018-0297-x)
    
    

80. 80.Adapte Collaboration. The ADAPTE process: resource toolkit for guideline adaptation Version 2.0, 2009.
    
    

81. 81.Tugwell P, Knottnerus JA. Adolopment - a new term added to the Clinical Epidemiology Lexicon. J Clin Epidemiol 2017;81:2.[doi:10.1016/j.jclinepi.2017.01.002](http://dx.doi.org/10.1016/j.jclinepi.2017.01.002)
    
    

82. 82.Schünemann HJ, Wiercioch W, Brozek J, et al. GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol 2017;81:101–10.[doi:10.1016/j.jclinepi.2016.09.009](http://dx.doi.org/10.1016/j.jclinepi.2016.09.009)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2016.09.009&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=27713072&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

83. 83.Snilstveit B, Vojtkova M, Bhavsar A, et al. Evidence & Gap Maps: a tool for promoting evidence informed policy and strategic research agendas. J Clin Epidemiol 2016;79:120–9.[doi:10.1016/j.jclinepi.2016.05.015](http://dx.doi.org/10.1016/j.jclinepi.2016.05.015)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2016.05.015&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

84. 84.Brown P, Brunnhuber K, Chalkidou K, et al. How to formulate research recommendations. BMJ 2006;333:804–6.[doi:10.1136/bmj.38987.492014.94](http://dx.doi.org/10.1136/bmj.38987.492014.94)
    
    [FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzMvNzU3Mi84MDQiO3M6NDoiYXRvbSI7czoyOToiL2JtamdoLzQvU3VwcGxfMS9lMDAwODQwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

85. 85.World Health Organization. Guidelines approved by the guidelines review committee. In: Who indoor air quality guidelines: household fuel combustion. Geneva: World Health Organization, 2014.
    
    

86. 86.Burford B, Lewin S, Welch V, et al. Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making. J Clin Epidemiol 2013;66:1251–61.[doi:10.1016/j.jclinepi.2013.06.017](http://dx.doi.org/10.1016/j.jclinepi.2013.06.017)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.06.017&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

87. 87.Sandelowski M, Leeman J, Knafl K, et al. Text-in-context: a method for extracting findings in mixed-methods mixed research synthesis studies. J Adv Nurs 2013;69:1428–37.[doi:10.1111/jan.12000](http://dx.doi.org/10.1111/jan.12000)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/jan.12000&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22924808&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

88. 88.Oxman AD, Guyatt GH. A consumer's guide to subgroup analyses. Ann Intern Med 1992;116:78–84.[doi:10.7326/0003-4819-116-1-78](http://dx.doi.org/10.7326/0003-4819-116-1-78)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-116-1-78&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=1530753&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=A1992GX15700012&link_type=ISI) 

89. 89.Dowswell T, Carroli G, Duley L, et al. Alternative versus standard packages of antenatal care for low-risk pregnancy. Cochrane Database Syst Rev 2015;(7):CD000934.[doi:10.1002/14651858.CD000934.pub3](http://dx.doi.org/10.1002/14651858.CD000934.pub3)
    
    

90. 90.Noyes J, Hendry M, Booth A, et al. Current use was established and Cochrane guidance on selection of social theories for systematic reviews of complex interventions was developed. J Clin Epidemiol 2016;75:78–92.[doi:10.1016/j.jclinepi.2015.12.009](http://dx.doi.org/10.1016/j.jclinepi.2015.12.009)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=26772607&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

91. 91.Kirk MA, Kelley C, Yankey N, et al. A systematic review of the use of the consolidated framework for implementation research. Implement Sci 2016;11:72.[doi:10.1186/s13012-016-0437-z](http://dx.doi.org/10.1186/s13012-016-0437-z)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

92. 92.Moore GF, Evans RE, Hawkins J, et al. All interventions are complex, but some are more complex than others: using iCAT_SR to assess complexity. Cochrane Database Syst Rev 2017;7:Ed000122.[doi:10.1002/14651858.ED000122](http://dx.doi.org/10.1002/14651858.ED000122)
    
    

93. 93.Hultcrantz M, Rind D, Akl EA, et al. The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol 2017;87:4–13.[doi:10.1016/j.jclinepi.2017.05.006](http://dx.doi.org/10.1016/j.jclinepi.2017.05.006)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2017.05.006&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=28529184&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

94. 94.Darzi A, Abou-Jaoude EA, Agarwal A, et al. A methodological survey identified eight proposed frameworks for the adaptation of health related guidelines. J Clin Epidemiol 2017;86:3–10.[doi:10.1016/j.jclinepi.2017.01.016](http://dx.doi.org/10.1016/j.jclinepi.2017.01.016)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom) 

95. 95.Rehfuess EA, Booth A, Brereton L, et al. Towards a taxonomy of logic models in systematic reviews and health technology assessments: a priori, staged, and iterative approaches. Res Synth Methods 2018;9:13–24.[doi:10.1002/jrsm.1254](http://dx.doi.org/10.1002/jrsm.1254)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/jrsm.1254&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=28677339&link_type=MED&atom=%2Fbmjgh%2F4%2FSuppl_1%2Fe000840.atom)