Original ArticleSeries: Clinical Epidemiology in South Africa. Paper 3: Logic models help make sense of complexity in systematic reviews and health technology assessments
Introduction
Sub-Saharan Africa (SSA) is affected by an overwhelming burden of diseases and injuries [1] and faces considerable challenges in health service provision. Addressing this burden requires a well-functioning health system and a variety of curative and preventive interventions relevant to the African context, many of which can be considered complex. Policy makers and health care practitioners need to consider the evidence about the benefits and harms of these interventions, if they are to make optimal use of limited resources [2]. Systematic reviews provide the most complete and reliable evidence on intervention effectiveness, while taking stock of existing research and critical gaps [3]. This is crucial to reduce wasting resources on unnecessary research, especially in SSA and other low- and middle-income countries (LMICs) [4], [5]. In these settings, a number of challenges hinder research evidence use, including a paucity of existing systematic reviews relevant to LMICs [2], [3], [6] and limited capacity for research synthesis. In a recent situation analysis, Oliver et al. (2015) identified a lack of overall systematic review capacity in LMICs, including individual, team, institutional, and system capacity. The authors highlight a need to develop methods and build capacity to address complex health system and health policy questions; a need linked to strengthening the relationship between producers and users of evidence [7].
The UK Medical Research Council's guidance on complex interventions [8] resulted in wide use of the term. However, the complexity of the intervention itself is only one of many sources of complexity [9]. In evidence synthesis, complexity can relate to the characteristics of any part of the PICO (population, intervention, comparison, or outcomes) question, and to methodological issues inherent in the included primary studies [10]. Additional complexity can be found in the unique circumstances under which the intervention is delivered and in nonlinear pathways and feedback loops between intervention and outcomes, interactions between direct and indirect effects of the intervention, as well as between different intervention components [11]. Petticrew (2011) explains that complexity does not have to be an inherent characteristic of an intervention, but rather that interventions can have simple and complex explanations, depending on the perspective adopted and the research question asked [11].
A series of six articles published in the Journal of Clinical Epidemiology in 2013, provides the first concerted attempt to address complexity in systematic reviews at each stage of the process from formulating the question [10], to synthesizing evidence [12] and assessing heterogeneity [13] to reviewing the applicability of findings [14]. The series concludes with a research agenda, emphasizing methodological areas needing further development and testing [15].
Logic models have been defined in various ways [16] and can be described, inter alia, as conceptual frameworks, concept maps, or influence diagrams. Anderson et al. (2011) argue that logic models “describe theory of change,” “promote systems thinking,” and contribute both in a conceptual and analytical way [17]. This resonates with our understanding of the use of logic models in systematic reviews and health technology assessments (HTAs). For the purpose of this article, we refer to a logic model as “… a graphic description of a system … designed to identify important elements and relationships within that system” [17], [18]. Logic models can help conceptualize complexity [19] by (1) depicting intervention components and the relationships between them, (2) making underlying theories of change and assumptions about causal pathways between the intervention and multiple outcomes explicit [17], and (3) displaying interactions between the intervention and the system within which it is implemented. Such a graphic representation is particularly helpful as a mechanism for making transparent assumptions among researchers and other stakeholders, and making results more accessible to a potentially broad range of decision makers, including clinicians, public health practitioners, and policy makers. In essence, logic models provide a framework to support the entire systematic review or HTA process and help to interpret the results, as well as to identify areas where further evidence is needed.
Two main approaches to logic modeling can be distinguished: a priori and iterative logic modeling. With an a priori approach, the logic model is developed at the protocol stage to refine the research question, identify sources of heterogeneity and subgroups, design the data extraction form, and plan data synthesis. This type of logic model is finalized before data collection and remains unchanged throughout the systematic review or HTA process [17], [20]. In an iterative approach, the logic model is conceived as a mechanism to incorporate the results of the systematic review or HTA and is subject to repeated changes during the process of data collection [21]. Although both approaches have their advantages and drawbacks (Booth et al., article in preparation), this article focuses mainly on a priori logic modeling.
Examples of logic models in systematic reviews and HTAs of public health and health care interventions exist, but specific guidance on how to develop an appropriate logic model is lacking. Noyes et al. (2013) highlight the need for a taxonomy of logic models, logic model templates, and a better understanding of the impact of the choice of logic model [15].
As part of the EU-funded INTEGRATE-HTA project (www.integrate-hta.eu), we designed two distinct logic model templates and applied these across several Cochrane and non-Cochrane systematic reviews and one HTA addressing different types of complex interventions. This article describes how these templates were developed and examines their applicability and usefulness in making sense of complexity. We have included three completed logic models on questions of particular relevance to SSA, that is, interventions to reduce ambient air pollution, community-level interventions for improving access to food in LMICs, and e-learning interventions to increase evidence-based health care competencies in health care professionals.
Section snippets
Development of logic model templates
We conducted systematic searches in the Cochrane Library, the Campbell Library, and Medline via PubMed (date of last search 10 December, 2013) to identify systematic reviews and HTAs that used logic models. After removal of duplicates and exclusion of irrelevant studies, we identified 18 published systematic reviews that included a logic model and one HTA that referred to the different phases of a logic model but did not include a diagram. Thirteen [22], [23], [24], [25], [26], [27], [28], [29]
Distinct logic model templates
A system-based logic model shown in Fig. 1 (also described as a conceptual framework by some authors) depicts the system in which the interaction between the participants, the intervention, and the context takes place. This perspective is mostly static: although it recognizes that interactions between different elements of the model take place, these are not investigated in detail. The PICO elements form the core elements of the logic model, supplemented with context and implementation
Discussion and conclusion
Systematic reviews that can help provide answers for the vast array of challenges in SSA have become a necessity [2], [62]. Our logic model templates equip review authors with a tool to address complexity in an explicit manner, thereby mainly building capacity at an individual level. However, they also have the potential to enhance the capacity of the system [7] through improved communication between producers and users of evidence. They add value to the review process in terms of achieving a
Acknowledgments
INTEGRATE-HTA-Work Package (WP) 5 working group (in alphabetical order): W. Awa, A. Booth, L. Brereton, J. Chilcott, K.B. Lysdahl, K. Mozygemba, A. Gerhardus, W. Oortwijn, L. Pfadenhauer, P. Refolo, E. Rehfuess, A. Rohwer, D. Sacchini, M. Tummers, G.J. van der Wilt, P. Wahlster.
Authors' contributions: A.R., E.R., and L.P. developed the logic model templates with input from the rest of the WP5 working group. A.R., E.R., L.P., J.B., and L.B. were involved in application of the logic model
References (65)
- et al.
Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010
Lancet
(2012) - et al.
How to increase value and reduce waste when research priorities are set
Lancet
(2014) - et al.
Strengthening research capacity
Lancet
(2002) - et al.
Complex interventions and their implications for systematic reviews: a pragmatic approach
J Clin Epidemiol
(2013) - et al.
Systematic reviews of complex interventions: framing the review question
J Clin Epidemiol
(2013) - et al.
Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute
J Clin Epidemiol
(2013) - et al.
Identifying, documenting, and examining heterogeneity in systematic reviews of complex interventions
J Clin Epidemiol
(2013) - et al.
Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making
J Clin Epidemiol
(2013) - et al.
A research and development agenda for systematic reviews that ask complex questions about complex interventions
J Clin Epidemiol
(2013) - et al.
Investigating complexity in systematic reviews of interventions by using a spectrum of methods
J Clin Epidemiol
(2013)
What factors influence the use of contracts in the context of NHS dental practice? A systematic review of theory and logic model
Soc Sci Med
Limited evidence for the effectiveness of educational interventions for renal transplant recipients. Results from a systematic review of controlled clinical trials
Patient Educ Couns
Current methods of the U.S. Preventive Services Task Force: a review of the process
Am J Prev Med
Popping the (PICO) question in research and evidence-based practice
Appl Nurs Res
Context and implementation: a concept analysis towards conceptual maturity
Z Evid Fortbild Qual Gesundhwes
Priority setting for systematic review of health care interventions in Nigeria
Health Policy
Getting to grips with GRADE-perspective from a low-income setting
J Clin Epidemiol
Is evidence-based medicine relevant to the developing world?
PLoS Med
Global health: the importance of evidence-based medicine
BMC Med
The global stock of research evidence relevant to health systems policymaking
Health Res Policy Syst
Capacity for conducting systematic reviews in low- and middle-income countries: a rapid appraisal
Health Res Policy Syst
Developing and evaluating complex interventions: the new Medical Research Council guidance
BMJ
When are complex interventions ‘complex’? When are simple interventions ‘simple’?
Eur J Public Health
Theory-based evaluation, logic modelling and the experience of SA non-governmental organisations
Using logic models to capture complexity in systematic reviews
Res Synth Methods
Using logic models to bring together planning, evaluation, and action: logic model development guide
Systematic reviews of complex multicomponent health care interventions
Guidance on the use of logic models in health technology assessments of complex interventions
Relative effectiveness of conditional and unconditional cash transfers for schooling outcomes in developing countries: a systematic review
Campbell Syst Rev
Community wide interventions for increasing physical activity
Cochrane Database Syst Rev
Psychosocial interventions for supporting women to stop smoking in pregnancy
Cochrane Database Syst Rev
Interventions for promoting reintegration and reducing harmful behaviour and lifestyles in street-connected children and young people
Cochrane Database Syst Rev
Cited by (73)
Matching the Outcomes to Treatment Targets of Exercise for Low Back Pain: Does it Make a Difference? Results of Secondary Analyses From Individual Patient Data of Randomised Controlled Trials and Pooling of Results Across Trials in Comparative Meta-analysis
2023, Archives of Physical Medicine and RehabilitationCitation Excerpt :Treatment targets may be described as intermediate variables or surrogate outcomes, as they may sit on the pathway to a patient relevant outcome such as pain or function. However, this may not always be the case, and the treatment targets reported by the authors of these RCTs may not have been based on clear program development theory or logic modeling.36,37 Many of the treatment targets identified by the RCT authors were captured by some of their outcomes, but there were no published intervention development or program evaluation38 papers for any of the included RCTs within which to test the degree that these treatment targets were indeed the focus of their intervention.
Treatment targets of exercise for persistent non-specific low back pain: a consensus study
2021, Physiotherapy (United Kingdom)Citation Excerpt :These agreed treatment targets may guide the design of trials of exercise interventions for persistent NSLBP by helping to target exercise interventions to achievable, measurable outcomes that match the aim(s) of the intervention. Future trial design may benefit from intervention logic models to map out the role of treatment targets, and select the most appropriate outcome domains and measures for complex interventions, such as exercise, with multiple intervention targets [44,45]. The identified treatment targets may, therefore, help in the identification of potential mediators of exercise that should be measured in future studies and used in pre-specified mediation analysis within RCTs.
Development of a Health Equity Framework for the US Preventive Services Task Force
2024, JAMA Network OpenImplementation of child-centred outcome measures in routine paediatric healthcare practice: a systematic review
2023, Health and Quality of Life OutcomesWeak Adoption and Performance of Hepatitis B Birth-Dose Vaccination Programs in Africa: Time to Consider Systems Complexity?—A Scoping Review
2023, Tropical Medicine and Infectious Disease
Funding: The research leading to this publication is part of the project INTEGRATE-HTA and has received funding from the European Union Seventh Framework Programme under grant agreement n° 306141.
Conflicts of interest: None.