You are here

  • Home
  • Archive
  • Volume 11, Issue 7
  • Implementation science protocol for a participatory, theory-informed implementation research programme in the context of health system strengthening in sub-Saharan Africa (ASSET-ImplementER)

Article Text

Article menu
XML
Global health
Protocol
Implementation science protocol for a participatory, theory-informed implementation research programme in the context of health system strengthening in sub-Saharan Africa (ASSET-ImplementER)
  1. Nadine Seward1,
  2. Jamie Murdoch2,
  3. Charlotte Hanlon3,4,
  4. Ricardo Araya3,
  5. Wei Gao5,
  6. Richard Harding5,
  7. Crick Lund3,6,
  8. Saba Hinrichs-Krapels7,8,
  9. Rosie Mayston8,9,
  10. Muralikrishnan Kartha10,
  11. Martin Prince8,
  12. Jane Sandall11,
  13. Graham Thornicroft1,
  14. Ruth Verhey12,
  15. Nick Sevdalis1
  1. 1Centre for Implementation Science, Department of Health Services and Population Research, King’s College London, London, UK
  2. 2School of Health Sciences, University of East Anglia, Norwich, UK
  3. 3Institute of Psychiatry, Psychology and Neuroscience, Health Service and Population Research Department, Centre for Global Mental Health, King’s College London, London, UK
  4. 4School of Medicine, Department of Psychiatry, WHO Collaborating Centre for Mental Health Research and Capacity-Building, Addis Ababa University College of Health Sciences, Addis Ababa, Ethiopia
  5. 5Department of Palliative Care and Policy, King’s College London, London, UK
  6. 6Alan J Flisher Centre for Public Mental Health, Department of Psychiatry and Mental Health, University of Cape Town, Rondebosch, Western Cape, South Africa
  7. 7The Policy Institute, King's College London, London, UK
  8. 8King's Global Health Institute, King’s College London, London, UK
  9. 9Global Health and Social Medicine, King’s College London, London, UK
  10. 10King's Health Economics, King’s College London, London, UK
  11. 11Dept of Women and Children’s Health, School of Life Course Sciences, FoLSM, King’s College London, London, UK
  12. 12Research Support Centre, College of Health Sciences, University of Zimbabwe, Harare, Zimbabwe
  1. Correspondence to Dr Nadine Seward; nadine.seward{at}kcl.ac.uk

Abstract

Objectives ASSET (Health System Strengthening in sub-Saharan Africa) is a health system strengthening (HSS) programme involving eight work-packages (ie, a research study that addresses a specific need for HSS) that aims to develop solutions that support high-quality care. Here we present the protocol for the implementation science (IS) theme within ASSET (ASSET-ImplmentER) that aims to understand what HSS interventions work, for whom and how, and how IS methodologies can be adapted to improve the HSS interventions within resource-poor contexts.

Settings Publicly funded health facilities in rural and urban areas in in Ethiopia, South Africa, Sierra Leone, and Zimbabwe.

Participants Research staff including principal investigators, coinvestigators, field staff, PhD students, and research assistants.

Interventions Work-packages use a mixed-methods effectiveness–effectiveness hybrid designs. At the end of the pre-implementation phase, a workshop is held whereby the IS theme, jointly with ASSET work-packages apply IS determinant frameworks to research findings to identify factors that influence the effectiveness of delivering evidence-informed care. Determinants are used to select a set of HSS interventions for further evaluation, where work-packages also theorise selective mechanisms.

In the piloting and rolling implementation phase, work-packages pilot the HSS interventions. An iterative process then begins involving evaluation, reflection and adaptation. Throughout this phase, IS determinant frameworks are applied to monitor and identify barriers/enablers to implementation. Selective mechanisms of action are also investigated. Implementation outcomes are evaluated using qualitative and quantitative methods. The psychometric properties of outcome measures including acceptability, appropriateness and feasibility are also evaluated. In a final workshop, work-packages come together, to reflect and explore the utility of the selected IS methods and provide suggestions for future use.

Structured templates are used to organise and analyse common and heterogeneous patterns across work-packages. Qualitative data are analysed using thematic analysis and quantitative data are analysed using means and proportions.

Conclusions We use a novel combination of IS methods at a programmatic level to facilitate comparisons of determinants and mechanisms that influence the effectiveness of HSS interventions in achieving implementation outcomes across different contexts. The study also contributes conceptual development and clarification at the underdeveloped interface of IS, HSS and global health.

The ASSET-ImplementER theme is considered minimal risk as we only interview researchers involved in the different work-packages. To this effect we have received approval from King’s College London Ethics Committee for research that is considered minimal risk (Reference number: MRA-20/21-21772).

  • public health
  • statistics & research methods
  • tropical medicine
https://creativecommons.org/licenses/by/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

https://doi.org/10.1136/bmjopen-2021-048742

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • ASSET-ImplementER aims to advance understanding of how to use implementation science to design and evaluate health system strengthening interventions across different health systems and contexts.

    • Implementation science determinant and evaluation frameworks were selected to help conceptualise what health systems strengthening interventions work for whom and how.

    • Research staff working within the different work-packages are using determinant frameworks to inform their theorising of how proposed determinants interact to produce mechanisms and the identified outcomes.

    • Work-packages will also validate measures of implementation outcomes in low-income and middle-income countries (LMICs), including Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM) and Feasibility of Intervention Measure.

    • The research will provide recommendations for harnessing implementation science to effectively strengthen health systems in resource-poor contexts in LMICs.

    Background

    The United Nations Third Sustainable Development Goal includes a call for Universal Health Coverage (UHC), particularly within research-poor contexts within low-income and middle-income countries (LMICs).1 Achieving UHC that explicitly addresses the availability and delivery of high-quality evidence-informed care has been identified as an urgent priority for health system strengthening (HSS) in LMICs.2 3

    The scale of unmet need is reflected in estimates that that suggest 5 million people in LMICs who engaged with the health system died due to poor-quality healthcare in 2016, and that 3.6 million deaths occurred due to people not accessing care at all.4 Simply increasing access to care, does not necessarily improve quality of care and improve health outcomes.5 A stark example of this issue is evident with the Janani Suraksha Yojana, a cash incentives programme targeting women who lived below the poverty line in India, to deliver in a health facility.6 7 Despite the increased coverage of facility-based deliveries, there was no corresponding improvement with maternal and newborn outcomes. Improving population health will require not only increased access to health services, but also the provision of high-quality care.8

    There is lack of consensus as to what constitutes health systems strengthening (HSS).9 Initially, the term HSS was a reaction to the limitations of vertical programmes targeting specific diseases, that do not strengthen the broader health system.9 Over time, the definition of HSS changed to one that has system-level effects and not just organisational level effects. It is now recognised that health systems are dynamic and complex sociotechnical structures, composed of multiple interacting components that are constantly adapting to changes in the local context and therefore behave unpredictably.10 HSS to improve quality of care is about ‘permanently making the system function better, and not just about filling gaps or supporting the system to produce better short-term outcomes’.11 HSS involves comprehensive changes to policies and regulations, organisational structures and relationships across the health system building blocks (eg, service delivery, health workforce, health information systems, access to essential medicines, financing and leadership/governance) that motivate changes in behaviour, and/or allow more effective use of resources across multiple care platforms.10 12

    Interventions to strengthen health systems, by their very nature, improve health outcomes by providing strategies that influence several mechanisms, both simultaneously and in isolation, at various time points and at various levels of the health system.10 As such, HSS requires an approach to design and evaluate complex public health interventions, in real-world contexts, that accounts for the multiple interconnecting components and actors.13 A systems-level approach to the design and evaluation of HSS interventions views a complex intervention as a system in itself, interacting with other building blocks of the underlying health system in which the intervention embeds itself, setting off reactions that may well be unexpected or unpredictable.10 Applying this approach to the design and evaluation of HSS interventions requires an evaluation of not only their main effects, but also inputs, outputs, initial, intermediate and eventual outcomes, feedback processes and contexts within the underlying health system.10

    Applying implementation science concepts and methods to HSS

    Implementation research is a rapidly expanding discipline that seeks to understand what, why and how interventions work in real-world settings.14 Implementation research can consider any aspect of implementation, including understanding contextual and behavioural barriers that influence implementation efforts, the process of implementation and the evaluation of implementation efforts for outcomes such as reach, fidelity and sustainability.14 15 By addressing implementation challenges through the participation of actors from multiple disciplines, implementation research can help to apply a systems-level approach and extract lessons that contribute to building stronger and more resilient and responsive health systems.16

    Whereas implementation research seeks to understand what, why and how interventions work in real-world settings,14 implementation science (IS) is the study of methods used to carry out this research.17 Specifically, IS offers theories, models, frameworks and other methodologies to optimise and evaluate the implementation of evidence-informed care.17 The methods offered through IS can inform decision-makers on how best to select and adapt HSS interventions to implement the evidence-informed care within the dynamic environments in which they work.18 For example, IS determinant frameworks describe contextual and behavioural barriers that are known to influence implementation outcomes, and include key factors to consider in the evaluation of the process of implementation.19 HSS interventions are selected to address specific barriers to implement the evidence-informed care.

    ASSET (Health System Strengthening in Sub-Saharan Africa) is a HSS programme that aims to develop solutions that support high-quality care. The IS theme within ASSET (ASSET-ImplmentER) aims to ensure that rigorous and appropriate methodologies are applied across ASSET. Box 1 describes key IS definitions as used within the ASSET-ImplementER theme.

    Box 1

    Implementation science terminology used within the ASSET-ImplementER programme

    Implementation strategies: methods or techniques used to enhance the adoption, implementation and sustainability of a clinical programme or practice. Other terminology includes HSS interventions, quality improvement strategies.

    Implementation outcomes: implementation outcomes assess how well implementation has occurred or provide insights about how this contributes to an individual’s health status or other important health outcomes. Proctor et al articulate the following core set of eight implementation outcomes: acceptability, feasibility, uptake, penetration, cost, fidelity and sustainability.40

    Implementation effectiveness: the impact of implementation efforts on implementation outcomes.

    Evaluation framework: specifies implementation outcomes that can be evaluated to determine implementation effectiveness.50

    Context: any feature of the circumstances in which an intervention is conceived, developed, implemented and evaluated.51 Contextual features are intervention specific but include (1) whole population and compositional features (varying between individuals within a population); (2) features of the physical location or geographical setting of interventions, as well as cultural, social, economic and political aspects; and (3) features affecting implementation (organisation, funding, policy, etc), as well as those directly affecting outcomes.

    Determinant frameworks: identify contextual barriers and/or enablers that are known to impact on the effectiveness of implementation efforts.19

    Theoretical frameworks: identify determinants of behaviours that are known to influence implementation outcomes.38

    Implementation theories/middle-range theories: describe the mechanisms behind how a proposed intervention works. These theories can also be used to identify barriers and/or enablers to change and what needs to change.19

    Programme theory: describes how a specific intervention is expected to lead to its effects and under what conditions.52

    Need for improved methods to design and evaluate HSS interventions in LMICs

    Despite the need to account for the complexity of HSS interventions to implement evidence-informed care, current methods used to design and evaluate such interventions in LMICs typically lack robust methodologies that enable an understanding of what interventions worked, for whom and how.20 As an example, few studies to-date have been supported by a conceptual framework (including a programme theory) explaining how an intervention is theorised to work, or a detailed process evaluation that helps to explain whether the intervention worked as intended, and if so, for whom, and under what circumstances.20–23 There are also limitations with how studies report how the local context influences the effectiveness of HSS interventions on important implementation outcomes such as coverage, acceptability and fidelity. Importantly, much of the evidence base for HSS interventions still comes from high-income countries, with uncertain generalisability of evidence to low-resourced settings.24

    Compounding the methodological shortcoming with HSS in LMICs, there are inconsistencies in the terminology used to describe the science of promoting and supporting the use of evidence in health and healthcare policy25 26 and also its reporting. As an example, IS has been described variously as quality improvement, knowledge translation, knowledge transfer and even complex intervention evaluation.26 Although there are differences between fields such IS and improvement science, they share the objective of improving the quality of healthcare delivery and therefore clinical outcomes.17 27

    Perhaps even more important is the lack of consistency in reporting and describing the implementation strategies to improve the quality of service delivery and therefore clinical outcomes.28 29 This results in the inability to replicate or generalise research studies to different contexts, or to allow for research synthesis such as meta-analysis or systematic reviews.30 31 To help address this issue, the Expert Recommendations for Implementing Change (ERIC) study created a taxonomy of implementation strategies30—to allow researchers to apply a common language when describing how evidenced interventions are being implemented. Taxonomies for quality improvement strategies have also been created that are conceptually similar (in some cases identical) to the ERIC taxonomy.30 32 As an example, both taxonomies include audit and feedback strategies, provider education, patient education, patient reminders and organisational change.30 32 There is also a taxonomy for HSS interventions that includes implementation strategies, the Effective Practice and Organisation of Care (EPOC) taxonomy, that are also similar to the taxonomies for implementation and quality improvement strategies.30 33

    There is, therefore, an urgent need to improve not only the quality of implementation research for HSS in LMICs, but also consistency in reporting the implementation strategies used to deliver evidence-informed care and improve service delivery and patient outcomes. For purposes of the ASSET-ImplementER theme, we will refer to these interventions as HSS interventions. Embedding high-quality IS methodology in HSS research, that includes consistent reporting of the HSS interventions used to deliver evidence-informed care, will help unpack the ‘black box’ of how such interventions work (or fail to reach their expected potential) for certain populations in a given setting.21 The ASSET-ImplementER study aims to start addressing some of these issues.

    The National Institute of Health Research Global Research Unit on HSS in sub-Saharan Africa (ASSET) and the ASSET-implementation research theme (ASSET-ImplementER)

    ASSET is a 4-year HSS research programme with a 10 month no-cost extension (2017–2022) that addresses the imperative of using implementation research with robust IS methods that use consistent reporting of the HSS interventions to deliver high-quality care. ASSET spans three healthcare platforms (primary healthcare for the integrated treatment of chronic conditions in adults, maternal and newborn and surgical care) involving eight work-packages within four different countries within sub-Saharan Africa. A work-package can be defined as a research study that aims to address an identified need for HSS within a care platform. The overall aim of ASSET is to develop, implement and evaluate high-quality HSS interventions that are effective and sustainable. ASSET also aims to ensure care is equitable, people-centred and respectful.34 ASSET is one of the first implementation research programmes for HSS that applies an ‘effectiveness–implementation hybrid’ approach, blending components of study designs that evaluate the effectiveness of interventions on patient outcomes, with study designs that focus on evaluating the effectiveness of HSS interventions on implementation outcomes.35 This methodology allows ASSET to understand both clinical and implementation effectiveness as well as contextual factors influencing implementation outcomes, including the potential sustainability of the different HSS interventions.

    The ASSET programme includes two phases of implementation research including the pre-implementation phase, and piloting and rolling implementation phase. The pre-implementation phase aims to understand requirements for HSS through the evaluation of the structure and function of the health system (such as arrangements for financing, governing and delivering care, and implementation strategies)33 that may limit the potential to deliver evidence-informed and person-centred care. At the end of this phase, each work package use research findings to select a set of HSS interventions.

    The piloting and rolling implementation phase initially pilot the interventions to identify factors that influence the implementation of the proposed interventions. After adjusting the initial programme theory and set of HSS interventions, an adaptive and iterative process is used to test the effectiveness of the set of HSS interventions on both clinical and implementation outcomes. The influence of context on the effectiveness of the HSS interventions delivering evidence-informed care is also assessed.34 To date, ASSET has completed the pre-implementation phase of the programme and is now starting the piloting and rolling implementation phase of research. ASSET-ImplementER, the IS theme within ASSET, embeds robust methodology within and across the different work-packages to help ensure ASSET meet its overall aim and objectives.

    Aim and objectives

    The overall aim of the ASSET-ImplementER theme is to advance our understanding of how to design and evaluate HSS interventions using a systems-level approach informed by IS, across different health systems and contexts. In doing so, we expect to achieve two equally important objectives for HSS in resource-poor contexts in LMICs: (1) advance our understanding of what HSS interventions work, for whom and how; and (2) improve IS methodologies to design and evaluate HSS interventions within LMIC settings.

    The following specific objectives will be addressed within the ASSET-ImplementER theme:

    Pre-implementation phase

    1. Contrast and compare the contextual determinants identified by each work package that influence the effectiveness of evidence-informed care.

    2. Contrast and compare HSS interventions selected by each work package that have the potential to improve the effectiveness of evidence-informed care.

    3. Determine how and why the investigators from the different work-packages selected the HSS interventions and implementation outcomes and compare findings between ASSET work-packages.

    Piloting and rolling implementation phase

    1. Contrast and compare findings between the different work-packages on how context influences the delivery of the selected HSS interventions on standardised implementation outcome measures for acceptability, appropriateness and feasibility.

    2. Evaluate the usefulness and utility of the selected IS methods/frameworks in achieving a systems-level approach in the evaluation of HSS interventions in resource-poor contexts and provide suggestions for future applications of IS to HSS.

    Methods

    Setting

    ASSET is working on three care platforms: (1) integrated primary care; (2) maternal and newborn care; and (3) surgical care, across four sub-Saharan African countries: Ethiopia, Sierra Leone, South Africa and Zimbabwe. Within the three care platforms are eight work-packages (table 1).

    View this table:
    Table 1

    Description of the ASSET work-packages for the different healthcare platforms; implementation research cuts across all of them (ASSET-ImplementER theme)

    Design

    ASSET-ImplementER will be embedded within the timelines for ASSET (2017–2022). The work-packages use mixed-methods throughout ASSET to select HSS interventions in the pre-implementation phase and evaluate the intervention through ‘effectiveness–implementation hybrid’ designs in the piloting and rolling implementation phase. Hybrid designs are essential with implementation research as they blend the components of study designs used to evaluate clinical effectiveness, with those of implementation study designs that focus on the evaluation of the influence of context on the effectiveness of HSS interventions.35

    Throughout ASSET, the ASSET-ImplementER stream uses mixed-methods including workshops, semistructured interviews and documentary analyses, to standardise, record and synthesise findings from the implementation component of the different work-packages. Findings include information from the different frameworks such as context, intervention, selected HSS interventions and implementation outcomes. Figure 1 describes the flow of methods for the ASSET-ImplementER theme.

    Figure 1
    Figure 1

    Flow of methods for the ASSET-ImplementER theme. 1Note: AIM, IAM and FIM have demonstrated promising psychometric properties in high-income settings.41 AIM, Acceptability of Intervention Measure; FIM, Feasibility of Intervention Measure; HSS, health system strengthening; IAM, Intervention Appropriateness Measure; LMICs, low-income and middle-income countries; ToC, Theory-of-Change; WP, work-package.

    To standardise methods and facilitate cross-site comparisons across ASSET, work-packages select contextual and behavioural determinants and implementation outcomes from a defined set IS frameworks. These frameworks are relevant to the programme as a whole, yet at the same time account for the specific characteristics of the different work-packages. Table 2 describes the frameworks that the various work-packages use and how this is relevant to the overall ASSET programme.

    View this table:
    Table 2

    Selected implementation science frameworks and theories used within ASSET-ImplementER

    The Consolidated Framework of Implementation Research (CFIR) is a determinant framework that is used as it provides an overview of a broad range of determinants that influence implementation effectiveness, such as the inner setting (ie, characteristics of the health facility), characteristics of the intervention (eg, complexity and adaptability) and implementation processes (eg, regular feedback about progress and quality of implementation).36 The Context and Implementation of Complex Intervention (CICI) framework is another framework that we use as it offers a detailed approach to identifying determinants from the external context (eg, sociocultural, socioeconomic, political, epidemiological, ethical and legal) that are known to influence implementation effectiveness that are particularly relevant to LMICs.37 It is expected that both of these frameworks will provide a detailed spectrum of determinants that are relevant to the ASSET programme. However, the frameworks are not exhaustive: any determinant identified that is not a part of either frameworks will be documented as such.

    Further to contextual determinants, it is also important to understand in some detail the characteristics of the users who deliver the healthcare. Changing behaviour ingrained in both individuals working within health systems and users of the health system will help to ensure the adoption and longer-term sustainability of the HSS interventions. To address this, we will use the Theoretical Domains Framework (TDF), a determinant framework that brings together evidence-based determinants of behaviour.38 We also use the Behaviour Change Wheel (BCW) that explicitly maps behavioural-change interventions onto determinants of behaviour identified with the TDF.39 Each work-package selects the contextual and behavioural determinants, implementation outcomes and HSS interventions from the different frameworks that are relevant to their aims and objectives. Table 3 describes the objectives of the different IS frameworks across the different phases of ASSET. However, identifying determinants that influence the effectiveness of HSS interventions in delivering evidence-informed practices is not enough to address complexity associated with HSS. We will therefore also explore how context influences the mechanisms introduced by the HSS interventions on implementation outcomes.

    View this table:
    Table 3

    Application of implementation science frameworks across asset HSS interventions and research phases

    Measures and taxonomies

    Implementation outcomes

    To evaluate the effectiveness of the implementation efforts, ASSET uses a combination of implementation outcomes as defined by the framework by Proctor et al.40 The selected implementation outcomes are aligned with the aims and objectives for the different work-packages.

    Many measures used to evaluate implementation outcomes have not been validated, making it difficult to compare the effectiveness of alternative interventions both within and between studies.41 However, three implementation outcomes measures including Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM) and Feasibility of Intervention Measure (FIM) have demonstrated promising psychometric properties in high-income settings.41 Work-packages test the applicability of these measures in low-resourced settings by adapting them to their specific context. Each measure includes four questions measured on a Likert scale (completely disagree, disagree, neither agree nor disagree, agree, completely agree). Questions for each measure will be translated and back translated to ensure linguistic and cross-cultural equivalence. The questions will then be reviewed and cognitive testing with local participants performed prior to the tools being used.

    Taxonomies

    Initially we use the EPOC taxonomy for HSS interventions to label the different interventions selected by work-packages.33 If an appropriate intervention is not included in this taxonomy, work-packages use the ERIC taxonomy for implementation strategies to label the intervention.30 To help alleviate the ambiguity in reporting implementation strategies and HSS interventions, we also report any important differences and similarities between the two taxonomies.

    To provide consistency in labelling of the behavioural-change interventions, we use the taxonomy of behavioural-change interventions that is also aligned with the TDF and the BCW.42 However, there are important limitations to applying the TDF and BCW to help select interventions that improve the quality of people-centred care that also address issues such as illness-related stigma. As an example, these behavioural-change methods were not specifically designed to address the issues surrounding implementation research in low-resource setting such as stigma, that ASSET is focusing on. As such the behavioural-change interventions in this taxonomy may not be appropriate. To address this issue and similar issues with the other taxonomies, where the HSS interventions, quality improvement strategy or behavioural-change intervention is not described in any taxonomy, we clearly describe the intervention and highlight the issue with the selected taxonomy not describing our intervention.

    Reporting frameworks

    To ensure consistent and accurate reporting of implementation studies, work-packages will apply the Standards for Reporting Implementation Studies.43

    Data collection procedures

    Data collection in the pre-implementation phase

    Workshops

    At the end of the pre-implementation phase of ASSET, a 2-day workshop is held involving relevant participants from all work-packages (ie, principal investigators, coinvestigators, field staff and research assistants). The objectives of the workshop are to: (1) standardise the implementation methods being applied across the different work-packages; and (2) collect data from each work-package on contextual and behavioural determinants associated with the delivery of high-quality evidence-informed care and the associated HSS interventions and behavioural-change interventions that were selected to help deliver this care. Findings from this exercise will help to address our first and second objectives of contrasting and comparing contextual and behavioural determinants between the different work-packages and the selected HSS interventions.

    Using data collection tools that identify and describe determinants of the implementation process, participants within the different work-packages apply findings from the formative research to identify and record barriers and/or enablers to the delivery of high-quality care. Participants then use the selected determinants to theorise how potential HSS interventions, described in the EPOC taxonomy, can assist in delivering high-quality evidence-informed care relevant to their work-package. Specifically, work-packages theorise and describe the mechanisms behind how the different determinants interact to produce a consequence (ie, poor quality of care, delayed access to care, loss to follow-up, lack of people-centred care). Similar data collection methods and forms are applied to label and record determinants of behaviours that influence the delivery of evidence-informed care. The identified determinants are then used to select and record behavioural-change interventions. Templates for the workshops can be found in online supplemental appendices 1 and 2.

    Interviews and focus groups

    To understand our third objective of the pre-implementation phase of ASSET (ie, how contextual determinants influence investigators in selecting HSS interventions, implementation outcomes), the workshop is followed-up with semistructured virtual interviews involving investigators from the different work-packages. The purpose of the workshop is to help understand the ‘how and why’, which led them to select the HSS interventions and associated strategies to facilitate their implementation (or virtual focus groups if more feasible for country teams). The same interviews/focus groups are also used to review and finalise the findings from the workshop and to understand if there are any gaps in evidence that will require the review any additional documentation. We anticipate including a minimum of two investigators from each ASSET work-package (16 investigators minimum sample) to reach saturation of the thematic areas that emerge from their responses. A guide for the interviews can be found in online supplemental appendix 3.

    Data collection in the piloting and rolling implementation phase

    Workshops

    Throughout the piloting and rolling implementation phase, workshops are held with each work-package team, involving relevant participants (principal and coinvestigators, field staff, research assistants and PhD students). The purpose of the workshops is to guide investigators on how to design the data collection tools including interview guides, using the IS determinant frameworks, HSS taxonomies and implementation outcomes. In doing so, we hope to ensure findings are standardised and therefore comparable across work-packages.

    At the end of the piloting and rolling implementation phase, a separate workshop will be held with each of different work-package teams to feedback the following information collected using the data collection tools: (1) the effectiveness of HSS interventions on standardised implementation outcomes, (2) the influence of context on the effectiveness of HSS intervention in delivering evidence-informed care and (3) the direct influence of context on the mechanisms introduced by the intervention to produce change. Participants are strongly encouraged to also theorise how the identified determinants interact with one another to produce mechanisms and the identified outcomes. It is through these workshops that we will be able to standardise the labelling of contextual and behavioural determinants to allow cross site comparisons on their influence of the standardised implementation outcomes (AIM, IAM, FIM) between the different work-packages (objective 4).

    At the end of the piloting and rolling implementation phase, another workshop lasting 1 day and involving investigators from all work-packages is used to reflect on the different IS methodologies and suggest improvements for further use. Specifically, we seek investigators’ opinions about the extent to which different frameworks capture the complexity of how the health system impacts on the overarching problem each work-package is trying to address. Again, we anticipate including a minimum of two investigators from each ASSET work-package (16 investigators minimum sample) to reach saturation of the thematic areas that emerge from their responses. All meetings will be audio-recorded with descriptive notes of discussion. Findings from these workshops are used to suggest how frameworks can be adapted to capture this complexity (objective 5).

    Surveys

    In work-packages that had available research capacity (WP1, WP2, WP5, WP6, WP8), staff delivering the interventions (ie, nurses, community health workers) are asked to complete the linguistically adapted tools for the three outcome measures, including AIM, IAM and FIM.41 Approximately 60 staff members from each of the participating work-packages contribute to these surveys. A subset of these participants are then interviewed to discuss the usefulness of the data collection tools. In particular, we review methods to capture the influence of context on both the effectiveness of the HSS interventions in influencing implementation outcomes as well as the influence of context on influencing the mechanisms introduced by the HSS interventions on implementation outcomes. Participants are also questioned on recommendations to improve our ability to capture the influence of context on implementation outcomes (Objective 5).

    Documentary analysis

    Theory-of-Change (ToC) methodology is a participatory approach involving key stakeholders that allows the articulation of the ‘theory’ of how a complex interventional programme will work in reality, describing the necessary interventions to bring about the change, as well as the assumptions inherent to the programme and importantly the context of implementation.44 ASSET work-packages are developing ToCs to support their implementation and evaluation planning. ToCs are effectively programme theories, contextualised within each one of the ASSET work-packages, offering an overview of how the selected HSS interventions are theorised to achieve specific implementation outcomes. ToCs also include information on key assumptions and work-package context. Each work package, including all relevant stakeholders, develops an initial ToC in the pre-implementation phase. This programme theory is then adapted throughout the phases. We examine whether and how the ToCs align with findings from both the pre-implementation and piloting and rolling implementation phases of the different work-packages including determinants identified using the IS frameworks outlined in table 3. We will also use ToC to better articulate how context influences mechanisms introduced by the HSS interventions described by each work-package (Objective 4).

    Data analysis

    Psychometric assessment of outcome measures AIM, IAM and FIM

    The outcome measures AIM, IAM and FIM that have demonstrated promising psychometric properties in high-income countries are tested for similar properties including substantive and discriminant content validity (the extent to which a measure is judged to be reflective of a construct of interest), and interitem consistency (extent to which scale items are scored in a similar manner) by the different work-packages45 to determine their relevance in low-resource settings. Measures are adapted to the local context, translated and back translated.

    To test for substantive validity (extent to which a measure is reflective of the construct of interest) and discriminant content validity (extent to which a measure that is not supposed to be related is actually unrelated), different cadres of workers who are responsible for delivering the intervention in the different work-packages assign 31 items reflecting the three constructs, to each of the three constructs and rate their confidence in the assignments in order test. The Wilcoxon one-sample signed-rank test or t-test (as appropriate) is used to determine whether items measured the intended construct, or whether items measured a combination of constructs. Hochberg’s correction is used to correct for multiple tests.46 Intraclass correlation coefficients using a two-way mixed-effects model to assess the level of agreement in item assignments among all participants, and also within key stakeholder groups, across 31 items and for each construct.47 The same data are also used to assess the factorial validity of the three measures, initially through exploratory factor analyses.41 We assess interitem consistency by computing Cronbach’s alpha for each of the four-item scales. For each measure, we also calculate means and SDs. Higher scores represent more favourable responses. If the measures demonstrate adequate psychometric properties, they are used to facilitate cross-site comparisons across the different work-packages.

    Data collected by ASSET-ImplementER

    For each work-package, a thematic analysis is used to analyse the qualitative data collected in semi-structured interviews and workshops that identify key issues pertinent to using IS frameworks and implementation outcome measures for HSS in low-resourced contexts.

    Analysis and synthesis of data collected by different work-packages

    At the end of each phase of research, data are collected from each of the work-packages (ie, contextual and behavioural determinants, mechanisms, selected HSS interventions and implementation outcomes) and entered into a template created in Excel software (vs 16.49). As an example, each work-package records information on identified barriers/enablers, associated data source, relevant IS framework, EPOC HSS interventions and implementation outcomes. Table 4 demonstrates an example of the Excel template.

    View this table:
    Table 4

    Sample template used to synthesise findings from the workshops and interviews

    We adapt the matrixed multiple case study approach to analyse and synthesise the data we record in the templates.48 This method facilitates comparisons between relevant work-packages, organising, analysing and presenting common and heterogeneous findings across implementation sites. Such an approach also aims to create generalisable knowledge regarding what and how local factors influence implementation. The matrixed multiple case study approach uses a combination of quantitative and qualitative methods that will allow us to identify associations between specific implementation processes and contextual factors on the one hand, and implementation outcomes on the other.

    Initially, data are analysed separately for each work package. Quantitative data are analysed using means and proportions. Given the heterogeneous nature of the care platforms and associated work-packages, context and selected HSS interventions, it will not be useful to compare quantitatively the influence of context on the effectiveness of HSS interventions on implementation outcomes between the work-packages. Instead, we use a qualitative approach that aims to understand why implementation outcomes were similar or different by describing the associated HSS interventions and contextual and behavioural determinants.

    Results from the matrixed multiple case study approach as well as findings from the interviews and the final workshop will be triangulated to identify points of convergence and divergence between different work-packages. Using these methods, we analyse how each work-package defined and operationalised different IS constructs (eg, context, intervention mechanisms, HSS interventions, behaviour change techniques), and how implementation theories and frameworks were used to support specific HSS interventions throughout implementation. The final part of our analysis integrates these elements to offer an overarching understanding of how IS frameworks have been operationalised across all ASSET work-packages and whether the use of these frameworks offered added utility (from a design, implementation or evaluation perspectives).

    Patient public involvement

    Patients and the public were not involved in the designing/writing protocol for this protocol. However, extensive participatory methods that involve both the patients and public will be used by the individual work-packages to design, select and evaluate the HSS interventions for ASSET.

    Discussion

    To our knowledge, ASSET-ImplementER is one of the first global health implementation research programmes that attempts to standardise methodologies to design and evaluate HSS interventions across different healthcare platforms and settings. In doing so, we expect to improve our understanding of what HSS interventions work for whom, by creating a compilation of HSS interventions used by the different work-packages, the associated barriers, and the effectiveness on implementation outcomes.

    ASSET-ImplementER will also attempt to develop novel insights into how we can improve IS methodologies for designing implementation research on HSS in LMICs. Importantly, there is a lack of standardisation in not only the terminology used to describe implementation research and HSS, but also the methods applied to this research. By labelling contextual and behavioural determinants using established frameworks and testing psychometric properties of implementation outcome measures that have demonstrated similar properties in high-income countries, we hope to improve our ability to compare the effectiveness of different HSS interventions. Importantly, many researchers find the plethora of IS frameworks and theories overwhelming and difficult to apply in practice. Through interviews with different stakeholders, we hope to improve methods for HSS in LMICs including our ability to learn how to best apply and simplify frameworks across different health systems.

    ASSET is explicitly addressing scalability through extensive participatory work, especially in the ToC workshops to develop and refine a programme theory. As an example, ToC workshops engage a wide range of stakeholders including policy makers that has a focus on ensuring scalability. Additionally, the different work-packages embed the different HSS interventions within the health systems, using existing healthcare staff, facilities and policies. This approach is in agreement with recommendations for ensuring scalability.49

    There are limitations to ASSET-ImplementER. Although our programme is theory-informed through the use of ToC workshops and IS determinant frameworks, we have not used IS theories or other middle-range theories to guide the design of ASSET as a programme. Indeed, many people criticise determinant frameworks (which we apply heavily within ASSET) as being ‘a’ theoretical. Nevertheless, given many of the work-packages teams were unfamiliar with IS methods at the time of ASSET set-up, we feel this is a pragmatic approach to applying a theory-informed approach to HSS. To mitigate the effects of this static approach to our research, we work with the different work-packages throughout the different phases of research, to theorise and conceptualise how the selected determinants interact with mechanisms introduced by the selected HSS on implementation outcomes.

    The COVID-19 pandemic has impacted on the pre-implementation phase of ASSET that will have consequences in what can be achieved in the longer-term. In some cases, work-packages experienced disruptions as key research activities such as focus group meetings, interviews with patients could not take place in person. Other issues include principal investigators for the different work-packages being seconded away to COVID-19-related activities.

    ASSET-ImplementER applies a system-level approach to both the design and evaluation of HSS interventions for the ASSET programme. Although there are limitations to our approach, we expect to begin advance our understanding of what HSS work for whom, and how. It is also hoped that we will start to address the issues in understanding the complexity surrounding how to effectively strengthen health systems in resource-poor contexts within LMICs.

    Ethics statements

    References

    1. ↵
      1. United Nations
      . Transforming our world: the 2030 agenda for sustainable development, 2015. Available: http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E [Accessed 17 Oct 2019].
    2. ↵
      1. Sachs JD
      . From millennium development goals to sustainable development goals. The Lancet 2012;379:2206–11.doi:10.1016/S0140-6736(12)60685-0
      OpenUrl
    3. ↵
      1. United Nations
      . Transforming our world: the 2030 agenda for sustainable development, 2015. Available: http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E
    4. ↵
      1. Bickler SN WT,
      2. Kassebaum N, et al.
      Debas HT, Donkor P, Gawande A, eds. Essential surgery: disease control priorities, third edition (volume 1. Washington (DC: The International Bank for Reconstruction and Development / The World Bank, 2015. https://www.ncbi.nlm.nih.gov/books/NBK333518/
    5. ↵
      1. Kruk ME,
      2. Gage AD,
      3. Joseph NT
      . Mortality due to low-quality health systems in the universal health coverage era: a systematic analysis of amenable deaths in 137 countries. Lancet 2018;392:2203–12.doi:10.1016/S0140-6736(18)31668-4pmid:30195398
      OpenUrlCrossRefPubMed
    6. ↵
      1. Ng M,
      2. Misra A,
      3. Diwan V, et al
      . An assessment of the impact of the JSY cash transfer program on maternal mortality reduction in Madhya Pradesh, India. Glob Health Action 2014;7:24939. doi:10.3402/gha.v7.24939pmid:http://www.ncbi.nlm.nih.gov/pubmed/25476929
      OpenUrlPubMed
    7. ↵
      1. Powell-Jackson T,
      2. Mazumdar S,
      3. Mills A
      . Financial incentives in health: new evidence from India's Janani Suraksha Yojana. J Health Econ 2015;43:154–69.doi:10.1016/j.jhealeco.2015.07.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/26302940
      OpenUrlCrossRefPubMed
    8. ↵
      1. World Health Organization
      . Everybody business: strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva, Switzerland, 2007.
    9. ↵
      1. Witter S,
      2. Palmer N,
      3. Balabanova D, et al
      . Health system strengthening-Reflections on its meaning, assessment, and our state of knowledge. Int J Health Plann Manage 2019;34:e1980–9.doi:10.1002/hpm.2882pmid:http://www.ncbi.nlm.nih.gov/pubmed/31386232
      OpenUrlPubMed
    10. ↵
      1. De Savigny D,
      2. Adam T
      . Systems thinking for health systems strengthening. alliance for health policy and systems research. WHO, 2009.
    11. ↵
      1. Chee G,
      2. Pielemeier N,
      3. Lion A, et al
      . Why differentiating between health system support and health system strengthening is needed. Int J Health Plann Manage 2013;28:85–94.doi:10.1002/hpm.2122pmid:http://www.ncbi.nlm.nih.gov/pubmed/22777839
      OpenUrlCrossRefPubMed
    12. ↵
      1. World Health Organization
      . Everybody business: strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: WHO, 2007. https://apps.who.int/iris/handle/10665/43918
    13. ↵
      1. Borghi J,
      2. Chalabi Z
      . Square peg in a round hole: re-thinking our approach to evaluating health system strengthening in low-income and middle-income countries. BMJ Glob Health 2017;2:e000406. doi:10.1136/bmjgh-2017-000406pmid:http://www.ncbi.nlm.nih.gov/pubmed/29082021
      OpenUrlFREE Full Text
    14. ↵
      1. Peters DH,
      2. Adam T,
      3. Alonge O
      . Implementation research: what it is and how to do it. BMJ 2013;347.doi:10.1136/bmj.f6753
    15. ↵
      1. Peters DTN,
      2. Adam T
      . Implementation research in health: a practical guide. alliance for health policy and systems research. World Health Organization, 2013.
    16. ↵
      1. Theobald S,
      2. Brandes N,
      3. Gyapong M, et al
      . Implementation research: new imperatives and opportunities in global health. Lancet 2018;392:2214–28.doi:10.1016/S0140-6736(18)32205-0pmid:http://www.ncbi.nlm.nih.gov/pubmed/30314860
      OpenUrlCrossRefPubMed
    17. ↵
      1. Eccles MP,
      2. Mittman BS
      . Welcome to implementation science. Implement Sci 2006;1:1.doi:10.1186/1748-5908-1-1
      OpenUrlCrossRef
    18. ↵
      1. Tabak RG,
      2. Khoong EC,
      3. Chambers DA, et al
      . Bridging research and practice: models for dissemination and implementation research. Am J Prev Med 2012;43:337–50.doi:10.1016/j.amepre.2012.05.024pmid:http://www.ncbi.nlm.nih.gov/pubmed/22898128
      OpenUrlCrossRefPubMed
    19. ↵
      1. Nilsen P
      . Making sense of implementation theories, models and frameworks. Implement Sci 2015;10:53.doi:10.1186/s13012-015-0242-0pmid:http://www.ncbi.nlm.nih.gov/pubmed/25895742
      OpenUrlCrossRefPubMed
    20. ↵
      1. Adam T,
      2. Hsu J,
      3. de Savigny D, et al
      . Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy Plan 2012;27 Suppl 4:iv9–19.doi:10.1093/heapol/czs086pmid:http://www.ncbi.nlm.nih.gov/pubmed/23014156
      OpenUrlPubMed
    21. ↵
      1. Alonge O,
      2. Rodriguez DC,
      3. Brandes N, et al
      . How is implementation research applied to advance health in low-income and middle-income countries? BMJ Glob Health 2019;4:e001257. doi:10.1136/bmjgh-2018-001257pmid:http://www.ncbi.nlm.nih.gov/pubmed/30997169
      OpenUrlAbstract/FREE Full Text
    22. ↵
      1. Ridde V
      . Need for more and better implementation science in global health. BMJ Glob Health 2016;1:e000115. doi:10.1136/bmjgh-2016-000115pmid:http://www.ncbi.nlm.nih.gov/pubmed/28588947
      OpenUrlFREE Full Text
    23. ↵
      1. Ridde V,
      2. Pérez D,
      3. Robert E
      . Using implementation science theories and frameworks in global health. BMJ Glob Health 2020;5:e002269. doi:10.1136/bmjgh-2019-002269pmid:http://www.ncbi.nlm.nih.gov/pubmed/32377405
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. Pantoja T,
      2. Opiyo N,
      3. Lewin S, et al
      . Implementation strategies for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev 2017;9:CD011086–CD.doi:10.1002/14651858.CD011086.pub2pmid:http://www.ncbi.nlm.nih.gov/pubmed/28895659
      OpenUrlCrossRefPubMed
    25. ↵
      1. Colquhoun H,
      2. Leeman J,
      3. Michie S, et al
      . Towards a common terminology: a simplified framework of interventions to promote and integrate evidence into health practices, systems, and policies. Implement Sci 2014;9:781.doi:10.1186/1748-5908-9-51pmid:http://www.ncbi.nlm.nih.gov/pubmed/24885553
      OpenUrlPubMed
    26. ↵
      1. McKibbon KA,
      2. Lokker C,
      3. Wilczynski NL, et al
      . A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a tower of Babel? Implement Sci 2010;5:16.doi:10.1186/1748-5908-5-16pmid:http://www.ncbi.nlm.nih.gov/pubmed/21080976
      OpenUrlCrossRefPubMed
    27. ↵
      1. Shojania KG,
      2. Grimshaw JM
      . Evidence-Based quality improvement: the state of the science. Health Aff 2005;24:138–50.doi:10.1377/hlthaff.24.1.138pmid:http://www.ncbi.nlm.nih.gov/pubmed/15647225
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Cotterill S,
      2. Knowles S,
      3. Martindale A-M, et al
      . Getting messier with TIDieR: embracing context and complexity in intervention reporting. BMC Med Res Methodol 2018;18:12. doi:10.1186/s12874-017-0461-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/29347910
      OpenUrlPubMed
    29. ↵
      1. Hoffmann TC,
      2. Glasziou PP,
      3. Boutron I, et al
      . Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687. doi:10.1136/bmj.g1687pmid:http://www.ncbi.nlm.nih.gov/pubmed/24609605
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Powell BJ,
      2. Waltz TJ,
      3. Chinman MJ,
      4. Thomas W,
      5. Matthew C, et al
      . A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci 2015;10:21.doi:10.1186/s13012-015-0209-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/25889199
      OpenUrlCrossRefPubMed
    31. ↵
      1. Booth A,
      2. Moore G,
      3. Flemming K, et al
      . Taking account of context in systematic reviews and guidelines considering a complexity perspective. BMJ Glob Health 2019;4:e000840. doi:10.1136/bmjgh-2018-000840pmid:http://www.ncbi.nlm.nih.gov/pubmed/30775011
      OpenUrlAbstract/FREE Full Text
    32. ↵
      1. Shojania KG MK,
      2. Wachter RM
      . Closing the quality gap: a critical analysis of quality improvement strategies (Vol. 1: series overview and methodology). Rockville (MD): agency for healthcare research and quality (US), 2004Aug. (Technical Reviews, No. 9.1.) Available from2004. Available: https://www.ncbi.nlm.nih.gov/books/NBK43908/
    33. ↵
      1. Effective Practice and Organisation of Care (EPOC).
      . The EpoC taxonomy of health systems interventions. EPOC resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services, 2016.
    34. ↵
      1. ASSET Health System StrEngThening in Sub Saharan Africa
      . The web page for the NIHR funded asset programme, 2020
    35. ↵
      1. Curran GM,
      2. Bauer M,
      3. Mittman B, et al
      . Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012;50:217–26.doi:10.1097/MLR.0b013e3182408812pmid:http://www.ncbi.nlm.nih.gov/pubmed/22310560
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Damschroder LJ,
      2. Aron DC,
      3. Keith RE, et al
      . Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50.doi:10.1186/1748-5908-4-50pmid:http://www.ncbi.nlm.nih.gov/pubmed/19664226
      OpenUrlCrossRefPubMed
    37. ↵
      1. Pfadenhauer LM,
      2. Gerhardus A,
      3. Mozygemba K, et al
      . Making sense of complexity in context and implementation: the context and implementation of complex interventions (CICI) framework. Implement Sci 2017;12:21.doi:10.1186/s13012-017-0552-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/28202031
      OpenUrlCrossRefPubMed
    38. ↵
      1. Atkins L,
      2. Francis J,
      3. Islam R, et al
      . A guide to using the theoretical domains framework of behaviour change to investigate implementation problems. Implement Sci 2017;12:77.doi:10.1186/s13012-017-0605-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/28637486
      OpenUrlCrossRefPubMed
    39. ↵
      1. Michie S,
      2. van Stralen MM,
      3. West R
      . The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011;6:42.doi:10.1186/1748-5908-6-42pmid:http://www.ncbi.nlm.nih.gov/pubmed/21513547
      OpenUrlCrossRefPubMed
    40. ↵
      1. Proctor E,
      2. Silmere H,
      3. Raghavan R, et al
      . Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38:65–76.doi:10.1007/s10488-010-0319-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/20957426
      OpenUrlCrossRefPubMed
    41. ↵
      1. Weiner BJ,
      2. Lewis CC,
      3. Stanick C, et al
      . Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017;12:108.doi:10.1186/s13012-017-0635-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/28851459
      OpenUrlCrossRefPubMed
    42. ↵
      1. Michie S,
      2. Richardson M,
      3. Johnston M, et al
      . The behavior change technique taxonomy (V1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med 2013;46:81–95.doi:10.1007/s12160-013-9486-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/23512568
      OpenUrlCrossRefPubMed
    43. ↵
      1. Pinnock H,
      2. Epiphaniou E,
      3. Sheikh A, et al
      . Developing standards for reporting implementation studies of complex interventions (STARI): a systematic review and e-Delphi. Implement Sci 2015;10:42.doi:10.1186/s13012-015-0235-zpmid:http://www.ncbi.nlm.nih.gov/pubmed/25888928
      OpenUrlCrossRefPubMed
    44. ↵
      1. Vogel I
      . Review of the use of theory of change in international development. UK: Department for International Development, 2012.
    45. ↵
      1. Noar SM,
      2. Zimmerman RS
      . Health behavior theory and cumulative knowledge regarding health behaviors: are we moving in the right direction? Health Educ Res 2005;20:275–90.doi:10.1093/her/cyg113pmid:http://www.ncbi.nlm.nih.gov/pubmed/15632099
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      1. Yosef H
      . A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988;75:800–2.
      OpenUrlCrossRefWeb of Science
    47. ↵
      1. Dixon D,
      2. Pollard B,
      3. Johnston M
      . What does the chronic pain grade questionnaire measure? Pain 2007;130:249–53.doi:10.1016/j.pain.2006.12.004pmid:http://www.ncbi.nlm.nih.gov/pubmed/17257751
      OpenUrlCrossRefPubMedWeb of Science
    48. ↵
      1. Kim B,
      2. Sullivan JL,
      3. Ritchie MJ, et al
      . Comparing variations in implementation processes and influences across multiple sites: what works, for whom, and how? Psychiatry Res 2020;283:112520. doi:10.1016/j.psychres.2019.112520pmid:http://www.ncbi.nlm.nih.gov/pubmed/31627960
      OpenUrlPubMed
    49. ↵
      1. Barker PM,
      2. Reid A,
      3. Schall MW
      . A framework for scaling up health interventions: lessons from large-scale improvement initiatives in Africa. Implement Sci 2016;11:12.doi:10.1186/s13012-016-0374-xpmid:http://www.ncbi.nlm.nih.gov/pubmed/26821910
      OpenUrlCrossRefPubMed
    50. ↵
      1. Glasgow RE,
      2. McKay HG,
      3. Piette JD
      . The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns 2001;44:119–27.doi:10.1016/S0738-3991(00)00186-5pmid:11479052
      OpenUrlCrossRefPubMedWeb of Science
    51. ↵
      1. Craig P,
      2. Di Ruggiero E,
      3. Frohlich KL
      . Chapter 3, Taking account of context in the population health intervention research process. Canadian Institutes of Health Research (CIHR) - National Institute for Health Research (NIHR). Southampton (UK): NIHR Journals Libary, 2018.
    52. ↵
      1. Rogers PJ
      . Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 2008;14:29–48.doi:10.1177/1356389007084674
      OpenUrlCrossRef
      1. Grol R,
      2. Grimshaw J
      . From best evidence to best practice: effective implementation of change in patients' care. Lancet 2003;362:1225–30.doi:10.1016/S0140-6736(03)14546-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/14568747
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    Footnotes

    • Twitter @nadineseward, @RHardingCSI, @sabziehin

    • Contributors NadineS wrote the first and subsequent drafts of the paper. NadineS and NickS conceptualised the idea for the paper. NadineS, NickS, JM and CH offered insight and edited all drafts of the paper. All other authors (RA, WG, RH, CL, SH-K, RM, MK, MP, JS, AS, GT, RV) edited and offered input to various drafts of the paper.

    • Funding The study is funded by the National Institute of Health Research (NIHR) Global Health Research Unit on Health System Strengthening in Sub-Saharan Africa, King’s College London (GHRU 16/136/54) using UK aid from the UK Government to support global health research. GT, WG and NS’ research is also supported by the National Institute for Health Research (NIHR) Applied Research Collaboration (ARC) South London at King’s College Hospital NHS Foundation Trust. GT and NS are members of King’s Improvement Science, which offers co-funding to the NIHR ARC South London and is funded by King’s Health Partners (Guy’s and St Thomas’ NHS Foundation Trust, King’s College Hospital NHS Foundation Trust, King’s College London and South London and Maudsley NHS Foundation Trust), Guy’s and St Thomas’ Charity and the Maudsley Charity. NS’ research is further supported by the ASPIRES research programme (Antibiotic use across Surgical Pathways - Investigating, Redesigning and Evaluating Systems), funded by the Economic and Social Research Council. GT also receives support from the National Institute of Mental Health of the National Institutes of Health under award number R01MH100470 (Cobalt study) and by the UK Medical Research Council in relation the Emilia (MR/S001255/1) and Indigo Partnership (MR/R023697/1) awards. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, the ESRC, the MRC, the charities or the Department of Health and Social Care. CH receives support from AMARI as part of the DELTAS Africa Initiative (DEL-15-01).

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.