Original Article
Series: Clinical Epidemiology in South Africa. Paper 3: Logic models help make sense of complexity in systematic reviews and health technology assessments

https://doi.org/10.1016/j.jclinepi.2016.06.012Get rights and content

Abstract

Objective

To describe the development and application of logic model templates for systematic reviews and health technology assessments (HTAs) of complex interventions.

Study Design and Setting

This study demonstrates the development of a method to conceptualize complexity and make underlying assumptions transparent. Examples from systematic reviews with specific relevance to Sub-Saharan Africa (SSA) and other low- and middle-income countries (LMICs) illustrate its usefulness.

Results

Two distinct templates are presented: the system-based logic model, describing the system in which the interaction between participants, intervention, and context takes place; and the process-orientated logic model, which displays the processes and causal pathways that lead from the intervention to multiple outcomes.

Conclusion

Logic models can help authors of systematic reviews and HTAs to explicitly address and make sense of complexity, adding value by achieving a better understanding of the interactions between the intervention, its implementation, and its multiple outcomes among a given population and context. They thus have the potential to help build systematic review capacity—in SSA and other LMICs—at an individual level, by equipping authors with a tool that facilitates the review process; and at a system-level, by improving communication between producers and potential users of research evidence.

Introduction

Sub-Saharan Africa (SSA) is affected by an overwhelming burden of diseases and injuries [1] and faces considerable challenges in health service provision. Addressing this burden requires a well-functioning health system and a variety of curative and preventive interventions relevant to the African context, many of which can be considered complex. Policy makers and health care practitioners need to consider the evidence about the benefits and harms of these interventions, if they are to make optimal use of limited resources [2]. Systematic reviews provide the most complete and reliable evidence on intervention effectiveness, while taking stock of existing research and critical gaps [3]. This is crucial to reduce wasting resources on unnecessary research, especially in SSA and other low- and middle-income countries (LMICs) [4], [5]. In these settings, a number of challenges hinder research evidence use, including a paucity of existing systematic reviews relevant to LMICs [2], [3], [6] and limited capacity for research synthesis. In a recent situation analysis, Oliver et al. (2015) identified a lack of overall systematic review capacity in LMICs, including individual, team, institutional, and system capacity. The authors highlight a need to develop methods and build capacity to address complex health system and health policy questions; a need linked to strengthening the relationship between producers and users of evidence [7].

The UK Medical Research Council's guidance on complex interventions [8] resulted in wide use of the term. However, the complexity of the intervention itself is only one of many sources of complexity [9]. In evidence synthesis, complexity can relate to the characteristics of any part of the PICO (population, intervention, comparison, or outcomes) question, and to methodological issues inherent in the included primary studies [10]. Additional complexity can be found in the unique circumstances under which the intervention is delivered and in nonlinear pathways and feedback loops between intervention and outcomes, interactions between direct and indirect effects of the intervention, as well as between different intervention components [11]. Petticrew (2011) explains that complexity does not have to be an inherent characteristic of an intervention, but rather that interventions can have simple and complex explanations, depending on the perspective adopted and the research question asked [11].

A series of six articles published in the Journal of Clinical Epidemiology in 2013, provides the first concerted attempt to address complexity in systematic reviews at each stage of the process from formulating the question [10], to synthesizing evidence [12] and assessing heterogeneity [13] to reviewing the applicability of findings [14]. The series concludes with a research agenda, emphasizing methodological areas needing further development and testing [15].

Logic models have been defined in various ways [16] and can be described, inter alia, as conceptual frameworks, concept maps, or influence diagrams. Anderson et al. (2011) argue that logic models “describe theory of change,” “promote systems thinking,” and contribute both in a conceptual and analytical way [17]. This resonates with our understanding of the use of logic models in systematic reviews and health technology assessments (HTAs). For the purpose of this article, we refer to a logic model as “… a graphic description of a system … designed to identify important elements and relationships within that system” [17], [18]. Logic models can help conceptualize complexity [19] by (1) depicting intervention components and the relationships between them, (2) making underlying theories of change and assumptions about causal pathways between the intervention and multiple outcomes explicit [17], and (3) displaying interactions between the intervention and the system within which it is implemented. Such a graphic representation is particularly helpful as a mechanism for making transparent assumptions among researchers and other stakeholders, and making results more accessible to a potentially broad range of decision makers, including clinicians, public health practitioners, and policy makers. In essence, logic models provide a framework to support the entire systematic review or HTA process and help to interpret the results, as well as to identify areas where further evidence is needed.

Two main approaches to logic modeling can be distinguished: a priori and iterative logic modeling. With an a priori approach, the logic model is developed at the protocol stage to refine the research question, identify sources of heterogeneity and subgroups, design the data extraction form, and plan data synthesis. This type of logic model is finalized before data collection and remains unchanged throughout the systematic review or HTA process [17], [20]. In an iterative approach, the logic model is conceived as a mechanism to incorporate the results of the systematic review or HTA and is subject to repeated changes during the process of data collection [21]. Although both approaches have their advantages and drawbacks (Booth et al., article in preparation), this article focuses mainly on a priori logic modeling.

Examples of logic models in systematic reviews and HTAs of public health and health care interventions exist, but specific guidance on how to develop an appropriate logic model is lacking. Noyes et al. (2013) highlight the need for a taxonomy of logic models, logic model templates, and a better understanding of the impact of the choice of logic model [15].

As part of the EU-funded INTEGRATE-HTA project (www.integrate-hta.eu), we designed two distinct logic model templates and applied these across several Cochrane and non-Cochrane systematic reviews and one HTA addressing different types of complex interventions. This article describes how these templates were developed and examines their applicability and usefulness in making sense of complexity. We have included three completed logic models on questions of particular relevance to SSA, that is, interventions to reduce ambient air pollution, community-level interventions for improving access to food in LMICs, and e-learning interventions to increase evidence-based health care competencies in health care professionals.

Section snippets

Development of logic model templates

We conducted systematic searches in the Cochrane Library, the Campbell Library, and Medline via PubMed (date of last search 10 December, 2013) to identify systematic reviews and HTAs that used logic models. After removal of duplicates and exclusion of irrelevant studies, we identified 18 published systematic reviews that included a logic model and one HTA that referred to the different phases of a logic model but did not include a diagram. Thirteen [22], [23], [24], [25], [26], [27], [28], [29]

Distinct logic model templates

A system-based logic model shown in Fig. 1 (also described as a conceptual framework by some authors) depicts the system in which the interaction between the participants, the intervention, and the context takes place. This perspective is mostly static: although it recognizes that interactions between different elements of the model take place, these are not investigated in detail. The PICO elements form the core elements of the logic model, supplemented with context and implementation

Discussion and conclusion

Systematic reviews that can help provide answers for the vast array of challenges in SSA have become a necessity [2], [62]. Our logic model templates equip review authors with a tool to address complexity in an explicit manner, thereby mainly building capacity at an individual level. However, they also have the potential to enhance the capacity of the system [7] through improved communication between producers and users of evidence. They add value to the review process in terms of achieving a

Acknowledgments

INTEGRATE-HTA-Work Package (WP) 5 working group (in alphabetical order): W. Awa, A. Booth, L. Brereton, J. Chilcott, K.B. Lysdahl, K. Mozygemba, A. Gerhardus, W. Oortwijn, L. Pfadenhauer, P. Refolo, E. Rehfuess, A. Rohwer, D. Sacchini, M. Tummers, G.J. van der Wilt, P. Wahlster.

Authors' contributions: A.R., E.R., and L.P. developed the logic model templates with input from the rest of the WP5 working group. A.R., E.R., L.P., J.B., and L.B. were involved in application of the logic model

References (65)

  • R. Harris et al.

    What factors influence the use of contracts in the context of NHS dental practice? A systematic review of theory and logic model

    Soc Sci Med

    (2014)
  • K.H. Urstad et al.

    Limited evidence for the effectiveness of educational interventions for renal transplant recipients. Results from a systematic review of controlled clinical trials

    Patient Educ Couns

    (2013)
  • R.P. Harris et al.

    Current methods of the U.S. Preventive Services Task Force: a review of the process

    Am J Prev Med

    (2001)
  • P.W. Stone

    Popping the (PICO) question in research and evidence-based practice

    Appl Nurs Res

    (2002)
  • L.M. Pfadenhauer et al.

    Context and implementation: a concept analysis towards conceptual maturity

    Z Evid Fortbild Qual Gesundhwes

    (2015)
  • M. Meremikwu et al.

    Priority setting for systematic review of health care interventions in Nigeria

    Health Policy

    (2011)
  • M. English et al.

    Getting to grips with GRADE-perspective from a low-income setting

    J Clin Epidemiol

    (2011)
  • P. Chinnock et al.

    Is evidence-based medicine relevant to the developing world?

    PLoS Med

    (2005)
  • G.L. Birbeck et al.

    Global health: the importance of evidence-based medicine

    BMC Med

    (2013)
  • M.G. Wilson et al.

    The global stock of research evidence relevant to health systems policymaking

    Health Res Policy Syst

    (2013)
  • S. Oliver et al.

    Capacity for conducting systematic reviews in low- and middle-income countries: a rapid appraisal

    Health Res Policy Syst

    (2015)
  • P. Craig et al.

    Developing and evaluating complex interventions: the new Medical Research Council guidance

    BMJ

    (2008)
  • M. Petticrew

    When are complex interventions ‘complex’? When are simple interventions ‘simple’?

    Eur J Public Health

    (2011)
  • L.P. Wildschut

    Theory-based evaluation, logic modelling and the experience of SA non-governmental organisations

    (2014)
  • L.M. Anderson et al.

    Using logic models to capture complexity in systematic reviews

    Res Synth Methods

    (2011)
  • W.K. Kellog

    Using logic models to bring together planning, evaluation, and action: logic model development guide

    (2004)
  • J.M. Guise et al.

    Systematic reviews of complex multicomponent health care interventions

    (2014)
  • A. Rohwer et al.

    Guidance on the use of logic models in health technology assessments of complex interventions

    (2016)
  • S. Baird et al.

    Relative effectiveness of conditional and unconditional cash transfers for schooling outcomes in developing countries: a systematic review

    Campbell Syst Rev

    (2013)
  • P.R.A. Baker et al.

    Community wide interventions for increasing physical activity

    Cochrane Database Syst Rev

    (2011)
  • C. Chamberlain et al.

    Psychosocial interventions for supporting women to stop smoking in pregnancy

    Cochrane Database Syst Rev

    (2013)
  • E. Coren et al.

    Interventions for promoting reintegration and reducing harmful behaviour and lifestyles in street-connected children and young people

    Cochrane Database Syst Rev

    (2016)
  • Cited by (73)

    • Matching the Outcomes to Treatment Targets of Exercise for Low Back Pain: Does it Make a Difference? Results of Secondary Analyses From Individual Patient Data of Randomised Controlled Trials and Pooling of Results Across Trials in Comparative Meta-analysis

      2023, Archives of Physical Medicine and Rehabilitation
      Citation Excerpt :

      Treatment targets may be described as intermediate variables or surrogate outcomes, as they may sit on the pathway to a patient relevant outcome such as pain or function. However, this may not always be the case, and the treatment targets reported by the authors of these RCTs may not have been based on clear program development theory or logic modeling.36,37 Many of the treatment targets identified by the RCT authors were captured by some of their outcomes, but there were no published intervention development or program evaluation38 papers for any of the included RCTs within which to test the degree that these treatment targets were indeed the focus of their intervention.

    • Treatment targets of exercise for persistent non-specific low back pain: a consensus study

      2021, Physiotherapy (United Kingdom)
      Citation Excerpt :

      These agreed treatment targets may guide the design of trials of exercise interventions for persistent NSLBP by helping to target exercise interventions to achievable, measurable outcomes that match the aim(s) of the intervention. Future trial design may benefit from intervention logic models to map out the role of treatment targets, and select the most appropriate outcome domains and measures for complex interventions, such as exercise, with multiple intervention targets [44,45]. The identified treatment targets may, therefore, help in the identification of potential mediators of exercise that should be measured in future studies and used in pre-specified mediation analysis within RCTs.

    View all citing articles on Scopus

    Funding: The research leading to this publication is part of the project INTEGRATE-HTA and has received funding from the European Union Seventh Framework Programme under grant agreement n° 306141.

    Conflicts of interest: None.

    View full text