Considering Complexity in Systematic Review of Interventions
Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute

https://doi.org/10.1016/j.jclinepi.2013.06.005Get rights and content

Abstract

Objectives

Although there is increasing interest in the evaluation of complex interventions, there is little guidance on how evidence from complex interventions may be reviewed and synthesized, and the relevance of the plethora of evidence synthesis methods to complexity is unclear. This article aims to explore how different meta-analytical approaches can be used to examine aspects of complexity; describe the contribution of various narrative, tabular, and graphical approaches to synthesis; and give an overview of the potential choice of selected qualitative and mixed-method evidence synthesis approaches.

Study Design and Setting

The methodological discussions presented here build on a 2-day workshop held in Montebello, Canada, in January 2012, involving methodological experts from the Campbell and Cochrane Collaborations and from other international review centers (Anderson L, Petticrew M, Chandler J, et al. Introduction: systematic reviews of complex interventions. In press). These systematic review methodologists discussed the broad range of existing methods and considered the relevance of these methods to reviews of complex interventions.

Results

The evidence from primary studies of complex interventions may be qualitative or quantitative. There is a wide range of methodological options for reviewing and presenting this evidence. Specific contributions of statistical approaches include the use of meta-analysis, meta-regression, and Bayesian methods, whereas narrative summary approaches provide valuable precursors or alternatives to these. Qualitative and mixed-method approaches include thematic synthesis, framework synthesis, and realist synthesis. A suitable combination of these approaches allows synthesis of evidence for understanding complex interventions.

Conclusion

Reviewers need to consider which aspects of complex interventions should be a focus of their review and what types of quantitative and/or qualitative studies they will be including, and this will inform their choice of review methods. These may range from standard meta-analysis through to more complex mixed-method synthesis and synthesis approaches that incorporate theory and/or user's perspectives.

Section snippets

Introduction: why is guidance needed on synthesizing evidence on complex interventions?

What is new?

Key findings

  1. There are many potential sources of complexity in the relationship between an intervention and its outcomes. There is also an increasing range of synthesis methods that can help with reviewing complex interventions.

What this adds to what was known?
  1. This article sets out how complexity can be taken into account when integrating evidence from qualitative and quantitative studies and sets out some of the main methodological options for integrating different types of evidence, and how they may shed light on intervention

Synthesizing evidence on complex interventions

The task of synthesizing evidence on complex interventions can be daunting. However it can be made simpler by breaking it down into a series of stages (Fig. 2). First, we might consider the potential sources of complexity in the relationship between an intervention and health or other outcomes; then, we can frame appropriate research questions and hypotheses relating to these sources of complexity. This may be aided by the development of a logic model [9] that can clarify the reviewer's

How might complexity relate to approaches to synthesis?

An earlier article in this series was concerned with how to frame the question for a review of complex interventions [12]. We reiterate here that the choice of empirical approach, study design, and component data to be synthesized and the choice of a meta-analytical, narrative, qualitative, or mixed-method approach depend critically on the research question. Investigation of complexity in a systematic review is not an end in itself but should be strongly driven by the research question(s). In

The contribution of statistical approaches

The traditional strengths of applying meta-analytical synthesis methods in Cochrane and other systematic reviews of the effects of interventions are that they can address questions about: (1) whether an overall effect exists across a larger body of evidence than an individual study; (2) whether effects are consistent across studies; (3) what is the actual magnitude and variation of effects across studies; and (4) whether particular study-level factors are associated with the magnitude of

The contribution of narrative summary approaches

Narrative summary methods can be used in conjunction with meta-analytical and other quantitative approaches to explain and summarize reasons for differences in findings between studies. This systematic organization and presentation of the data can help the reviewer and reader to identify themes across studies and can facilitate the testing of prespecified theory by exploring similarities and differences among studies [22]. This is an important first step in any synthesis, whether or not a

The contribution of qualitative and mixed-method approaches

As shown previously and in Table 2, evidence synthesis conducted in the tradition of systematic reviews of effectiveness can contribute to address a variety of questions of relevance to complex interventions. In many circumstances, however, answering these questions will require a broader approach to enquiry [18]. Many methods to synthesize qualitative studies have been described in the literature, with some of them having been applied to issues of complexity, whereas others have potential

Conclusions and areas for future research

Despite the range of methods already available, there are many areas where new methodological research is needed. Among these are further development of methods for incorporating diverse sources of evidence in single reviews, technical examination of specific analytical methods [26], [27], and further assessment of the role of Bayesian analyses to combine quantitative evidence with prior distributions derived from qualitative evidence.

In many reviews of complex interventions, it makes little

References (68)

  • T. Greenhalgh et al.

    Protocol—realist and meta-narrative evidence synthesis: evolving standards (RAMESES)

    BMC Med Res Methodol

    (2011)
  • C. Pope et al.

    Synthesizing qualitative and quantitative health evidence. A guide to methods

    (2007)
  • Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of...
  • L.M. Anderson et al.

    Using logic models to capture complexity in systematic reviews

    Res Synth Methods

    (2011)
  • S. Shepperd et al.

    Can we systematically review studies that evaluate complex interventions?

    PLoS Med

    (2009)
  • A. Rithalia et al.

    Impact of presumed consent for organ donation on donation rates: a systematic review

    BMJ

    (2009)
  • M. Petticrew

    When are complex interventions “complex”? When are simple interventions “simple”?

    Eur J Public Health

    (2011)
  • M. Joffe et al.

    Causal diagrams in systems epidemiology.

    Emerg Themes Epidemiol

    (2012)
  • S. Galea et al.

    Causal thinking and complex system approaches in epidemiology

    Int J Epidemiol

    (2010)
  • D. Gough et al.

    An introduction to systematic reviews

    (2012)
  • Higgins J, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. In:...
  • E.A. Kristjansson et al.

    School feeding for improving the physical and psychosocial health of disadvantaged elementary school children

    Cochrane Database Syst Rev

    (2007)
  • T. Greenhalgh et al.

    Realist review to understand the efficacy of school feeding programmes

    BMJ

    (2007)
  • M. Petticrew et al.

    Systematic reviews in the social sciences. A practical guide

    (2006)
  • S. Shepherd et al.

    Identifying, documenting and examining heterogeneity in systematic reviews of complex interventions

    J Clin Epidemiol

    (2013)
  • E. Barnett-Page et al.

    Methods for the synthesis of qualitative research: a critical review

    BMC Med Res Methodol

    (2009)
  • K. Hannes et al.

    Synthesizing qualitative research: choosing the right approach

    (2012)
  • N. Ivers et al.

    Audit and feedback: effects on professional practice and healthcare outcomes

    Cochrane Database Syst Rev

    (2012)
  • C. Arditi et al.

    Computer-generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes

    Cochrane Database Syst Rev

    (2012)
  • H. Thomson et al.

    Do urban regeneration programmes improve public health and reduce health inequalities? A synthesis of the evidence from UK policy and practice (1980-2004)

    J Epidemiol Community Health

    (2006)
  • G. Wong et al.

    RAMESES publication standards: meta-narrative reviews

    BMC Med

    (2013)
  • J. Grimshaw et al.

    Complexity and systematic reviews. Report to the U.S. Congress of Technology Assessment

    (1995)
  • WHO

    The solid facts: palliative care

    (2004)
  • D. De Savingny et al.

    Systems thinking for health systems strengthening

  • Cited by (0)

    Financial disclosure: No relevant conflicts relating to this article. M.P. receives funding from the Canadian Institutes of Health Research-funded International Collaboration on Complex Interventions, which supported some of the work reported in this article.

    E.R. acknowledges that her contribution to this paper was partially funded from the European Union Seventh Framework Programme under grant agreement no. 306141. M.P. and E.R. led the development of this paper, with M.P., E.R. and J.N. preparing the first draft of the manuscript. All other authors, listed in alphabetical order, contributed to the revision and completion of the paper. All authors read and approved the final paper.

    View full text