You are here

Article Text

Article menu
Original research
Data-driven decision-making for district health management: a cluster-randomised study in 24 districts of Ethiopia
  1. Bilal Iqbal Avan1,
  2. Mehret Dubale2,
  3. Girum Taye3,
  4. Tanya Marchant4,
  5. Lars Åke Persson2,
  6. Joanna Schellenberg4
  1. 1Population Health, London School of Hygiene & Tropical Medicine, London, UK
  2. 2London School of Hygiene & Tropical Medicine, London, UK
  3. 3Health System and Reproductive Health Research Directorate, Ethiopian Public Health Institute, Addis Ababa, Ethiopia
  4. 4Disease Control, London School of Hygiene & Tropical Medicine, London, UK
  1. Correspondence to Dr Bilal Iqbal Avan; bilal.avan{at}lshtm.ac.uk

Abstract

Background Use of local data for health system planning and decision-making in maternal, newborn and child health services is limited in low-income and middle-income countries, despite decentralisation and advances in data gathering. An improved culture of data-sharing and collaborative planning is needed. The Data-Informed Platform for Health is a system-strengthening strategy which promotes structured decision-making by district health officials using local data. Here, we describe implementation including process evaluation at district level in Ethiopia, and evaluation through a cluster-randomised trial.

Methods We supported district health teams in 4-month cycles of data-driven decision-making by: (a) defining problems using a health system framework; (b) reviewing data; (c) considering possible solutions; (d) value-based prioritising; and (e) a consultative process to develop, commit to and follow up on action plans. 12 districts were randomly selected from 24 in the North Shewa zone of Ethiopia between October 2020 and June 2022. The remaining districts formed the trial’s comparison arm. Outcomes included health information system performance and governance of data-driven decision-making. Analysis was conducted using difference-in-differences.

Results 58 4-month cycles were implemented, four or five in each district. Each focused on a health service delivery challenge at district level. Administrators’ practice of, and competence in, data-driven decision-making showed a net increase of 77% (95% CI: 40%, 114%) in the regularity of monthly reviews of service performance, and 48% (95% CI: 9%, 87%) in data-based feedback to health facilities. Statistically significant improvement was also found in administrators’ use of information to appraise services. Qualitative findings also suggested that district health staff reported enhanced data use and collaborative decision-making.

Conclusions This study generated robust evidence that 20 months’ implementation of the Data-Informed Platform for Health strengthened health management through better data use and appraisal practices, systemised problem analysis to follow up on action points and improved stakeholder engagement.

Trial registration number NCT05310682.

  • Health systems
  • Cluster randomized trial
  • Maternal health
  • Child health
  • Health systems evaluation

Data availability statement

Data are available upon reasonable request. The data are available from the principal investigator of the IDEAS project and co-authors for this manuscript.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjgh-2023-014140

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Decentralised health systems depend on local data and effective decision-making at the district level.

    • Although the District Health Information System 2 initiative has transformed data management, evidence is sparse on how to facilitate data-driven decision-making at the district level.

    WHAT THIS STUDY ADDS

    • We developed the Data-Informed Platform for Health, a health system-strengthening intervention, which promotes structured decision-making by health administrators and managers using local data sources, in 12 districts of Ethiopia.

    • Using a randomised study design, we found that the Data-Informed Platform for Health strengthened existing decision-making fora at the district level: it improved data management and appraisal practices, systemised problem analysis to follow up on action points and created a culture of positive engagement among stakeholders.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

    • The Data-Informed Platform for Health could be adapted for other contexts and levels in the health system, enabling managers and administrators to improve service delivery by making the best use of data.

    Introduction

    Health systems in most low-income and middle-income countries (LMICs) have become devolved and decentralised, shifting decision-making from national to district level. Thus, ‘decision space’ at the district level—the lowest administrative unit for primary healthcare—has expanded.1 2 District health systems comprise networks of primary healthcare facilities, including hospitals, and are managed by district health management teams (DHMTs). It is the responsibility of autonomous DHMTs to improve healthcare through responsive, local decision-making.

    Rapid advancement in health information systems has accompanied the decentralisation. Governments have committed to strengthening health data management—through enhanced information technology (IT) services and focal development initiatives such as District Health Information System 2 (DHIS2)—rendering local data more available.3 Nevertheless, despite better data gathering and expanded decision space, the use of local data in decision-making for maternal, newborn and child health services remains limited in LMICs.4

    Research, long-established, to strengthen health systems has addressed performance improvement,5 supported human resources and financial services, and promoted evidence-based practices.6 Facility-level service delivery processes have been a priority, with scant attention paid to health system management—the critical back end to effective service delivery. Evidence on health management interventions to identify and address practice-need gaps is limited, yet building managers’ capacity while embedding technology-based solutions can enhance data use and improve DHMTs’ decision-making.

    Our earlier research in LMICs found a limited range of well-developed, contextually embedded processes and tools facilitating data use for comprehensive decision-making by district administrators and health managers.7 Among current efforts are cycles of data review and planning through externally facilitated workshops, and focused approaches such as human resource management.8 9

    Data-driven decision-making appraises all sources of evidence to understand health service challenges and develop solutions and follow-up. District-level Health Management Information System data are underused but possess great promise in relation to the six critical health system components: service delivery, medical supplies, workforce, governance, information and finance.10 DHIS2 is a transformative electronic data management system for collecting, validating, storing, analysing and communicating data collected routinely in health facilities. However, there are gaps: for example, databases beyond service delivery and functions for using DHIS data for problem-solving. Decentralisation means that local health administrators need more capacity to analyse and use data for decision-making. Yet the culture of data-sharing and collaborative planning among health stakeholders needs to be enhanced.11 12

    To address capacity and needs, we developed the Data-Informed Platform for Health (DIPH)—a health management systems intervention, which promotes structured decision-making by district health administrators and managers using local data. The DIPH strategy promotes quality decision-making by: (a) defining themes using a health policy and system framework; (b) reviewing data to identify problems; (c) considering alternative options; (d) value-based prioritising; and (e) using a consultative process to develop, commit to and follow up on action plans. The DIPH package included job aids and guidelines, providing tools and knowledge for structured decision-making using available data. The strategy was conceptualised, developed and tested over 10 years: the theory of change is shown in online supplemental annex Ia, which summarises the problem, solution and logic model. A prototype phase, piloted in three health districts in West Bengal, India, reported multisectoral data-sharing and data-based decision-making by district health management.13

    Supplemental material

    Committing to data use for decision-making rendered Ethiopia a pioneer among sub-Saharan African countries. The government’s ‘Information Revolution’ initiative was part of the Ethiopian ‘Health Sector Transformation Plan’ (HSTP; 2015–2020) to nurture a culture of information use and strengthen health system governance.14 This remained a priority in the HSTP (2021–2025),15 resulting in the Woreda Transformation initiative (districts in Ethiopia are ‘Woredas’) to create high-performing district health offices.16 In this context, two co-creation workshops with the Ethiopian Ministry of Health (MoH) and leading local public health institutions highlighted challenges faced by district management, especially in data-driven decision-making. The participants were oriented about the nature of the proposed research solution, and they endorsed the DIPH approach. A collaborative strategy for implementing and evaluating DIPH was finalised, and an advisory group was set up, to support the research within the public health system.

    This paper describes the implementation of the DIPH interventions at district level in Ethiopia, and its evaluation using a cluster-randomised controlled trial (RCT) to estimate its effect on health information system performance and data-driven decision-making.

    Methods

    Our research was embedded in the Ethiopian district health management system.17 A cluster-RCT with pre-comparison and post-comparison design was used. Study districts were ‘clusters’. This section details the study context, DIPH implementation and the evaluation of outcomes.

    Study context

    DIPH was implemented in all 24 districts of North Shewa zone in Amhara region, Ethiopia (online supplemental annex Ib). This zone was selected in co-creation workshops with the Ethiopian MoH because of its manageable distance from Addis Ababa and because it was felt to reflect the average functioning of the country’s health system. Districts were categorised into 12 pairs based on (a) performance level based on health outcomes, (b) distance from the zonal capital (Debrebirhan), and (c) the presence of Transform PHCU (Primary Healthcare Unit, a non-governmental organisation working locally on data). In each pair, one district was randomly chosen for the intervention group, the other being for the comparison group (online supplemental annex Ic).

    The co-creation workshops clarified that the pre-existing Performance Monitoring Team (PMT) meetings were the main platform for monthly performance reviews, priority-setting and decision-making by DHMT staff. However, PMT meetings were held irregularly, attendees lacked the capacity to conduct them and procedural details were unclear. DIPH supported PMTs by: (a) strengthening the forum for regularity and engagement of government and non-governmental stakeholders (health and non-health) to identify challenges, find solutions, and assess resource allocation and responsibilities, with the aim of consensus-building and collective decision-making; and (b) promoting critical review and regular use of diverse local data (DHIS and other) to understand health system progress.

    Implementation (October 2020−June 2022)

    Four 4-month cycles of DIPH were conducted in the 12 DIPH-intervention districts over 16 months. The first two cycles were to embed the DIPH strategy within the system by actively addressing any teething problems with the research implementation team, and the subsequent two cycles were fully integrated into the health system. Effectiveness estimation by the research evaluation team followed this. An additional cycle (20 February−20 June 2022) assessed the sustainability of DIPH without the research implementation team supporting on the ground, and information from this fifth cycle will be reported elsewhere.

    Before the first cycle, DHMT staff of intervention districts had 3 days of intensive training supported by the DIPH handbook. The research team (BIA, MD and GT) trained five staff members in each district: the District Health Office head, maternal and reproductive health officer, child and nutrition officer, monitoring and evaluation (M&E) officer, and health information technician (HIT) or their delegates.

    A field team was recruited and trained, comprising a coordinator, four support supervisors and a data manager. Each support supervisor was responsible for three districts, which they visited monthly to provide technical support, including inducting, orienting and handholding district stakeholders during the implementation of initial DIPH cycles, participating in monthly PMT meetings and monitoring implementation data.

    Monitoring

    Data were collected by DIPH support supervisors for each 4-month cycle. Synthesised findings were presented periodically to district staff to improve the next cycle.

    Process evaluation

    Qualitative semistructured key informant interviews were conducted in March 2022 with members of DHMTs from five DIPH and five comparison districts. The districts were selected based on their previous year’s performance. In each DHMT, the following personnel were interviewed: head of the Health Office, M&E officer, reproductive and maternal health officer, and HIT. 43 interviews were conducted from 2 March to 28 March 2022. Digital recordings of the interviews in Amharic were transcribed verbatim and translated into English. The transcripts were imported into NVivo and coded under themes related to practices of data use and stakeholder collaboration, and the nature and mechanisms of change of those practices. Key themes included dimensions of data use and stakeholder collaboration, differences in practices between DIPH and comparison districts, mechanisms for changes introduced by DIPH, and the effect of contextual factors on the nature and mechanisms of those changes.

    Outcome assessment

    Outcomes were assessed through baseline (September 2020) and endline (March 2022) surveys in all 24 districts. Each survey included a review of District Health Office documents and interviews with the DHMT. Before-and-after comparison of health system outcomes (primary) in intervention and comparison districts assessed changes resulting from DIPH in terms of (1) data management practices, especially health information system performance at the district level, including (a) essential infrastructure for data management, (b) data diversity, (c) reporting timelines, (d) data quality assessment mechanisms, and (e) data use; and (2) practices of data-driven decision-making in relation to district-level governance, including (a) evidence-based decision-making, (b) participatory decision-making, (c) understanding data value, (d) health system support for data use or data-driven decision-making, and (e) accountability. The key secondary outcome was the perception of data use culture, a composite index of 33 items encompassing six aspects of decision-making culture: evidence-based decision-making, emphasis on data quality, use of information, problem-solving, responsibility and motivation to use data. For the cluster-randomised trial overall, with 60 participants per arm, equally distributed across 12 clusters, the study had 80% power to detect a 22% increase in the intervention arm compared with the control arm, assuming a 50% proportion of primary outcomes in the control arm and an intraclass correlation of 0.03, with a 95% confidence level.18 Survey respondents included all five eligible staff per District Health Office: head of the District Health Office, the HIT, M&E officer, maternal and reproductive health officer, and child health and nutrition officer. The survey instruments were adapted from the Demographic and Health Survey and Performance of Routine Information System Management (PRISM) tools.19 20

    Summary indices were developed, with conceptual and statistical underpinning based on standard PRISM tools. Indices were calculated for each of the 24 districts at baseline and endline, ranging from 0% to 100%. PMT practices were assessed at two levels: (a) respondent perception of regularity and types of decision-making in PMTs, and (b) content analysis of PMT records to assess the types of issues discussed. We used a difference-in-differences (DiD) approach to evaluate effectiveness by comparing outcomes in districts that were exposed to the intervention with those that were not. In district-level health system research, there is always a potential risk of competing initiatives or programmes existing in the study area, which might introduce bias if a conventional post-intervention comparison is made in an RCT. DiD is a useful analytical technique in such a context to estimate a causal effect with a valid counterfactual. This is because in DiD analysis, the comparability assumption between the two study arms is less strict: in the absence of intervention, differences between study arms will be the same over time. DiD requires assessing change pre-intervention to post-intervention, and comparing this between the two groups. This eliminates biases resulting from trends in the outcome due to causes other than the intervention. All analyses were adjusted for clustering at the district level.

    Results

    This section is organised into four sections: (a) establishing comparability of the two study arms, (b) data management practices, (c) practices of data-driven decision-making and (d) perceptions about the culture of data use.

    58 4-month DIPH cycles were completed across 12 intervention districts between October 2020 and June 2022 (figure 1). In one district, security concerns prevented the initiation of two cycles. The diverse themes included antenatal care, birth attendant skills, community hygiene and various aspects of immunisation (table 1). Over 50% of action points were completed in nearly all (54 of 58, 93%) cycles. The completion rate of the action was at least 80% in most cycles (37 of 58, 63%).

    View this table:
    Table 1

    Total implemented cycles of DIPH in Ethiopia

    Figure 1
    Figure 1

    Flow chart of cluster-randomised controlled trial of the DIPH intervention. DIPH, Data-Informed Platform for Health.

    Comparability of study arms

    Sociodemographic characteristics of respondents were summarised by study arm over the study period (table 2). There were 60 respondents, on average, in each of the two arms, five from each district. The average age of respondents was 33 years, the mean professional experience in the health department was 11 years and the mean duration of service in the same district was 6 years. Over time, respondents were relatively more academically qualified, and positive gender changes were observed in both study arms.

    View this table:
    Table 2

    Characteristics of study participants and districts by study arm

    The configuration of health facilities in the DIPH and comparison arms was similar. The availability of resources related to data management—including local data storage devices, continuous grid electricity supply, mobile network and access to the national data server—improved comparably in both study arms between baseline and endline, except for wi-fi internet access which improved by 27% in the intervention districts and only 7% in comparison districts.

    Data management practices

    Three critical aspects of data management were explored and verified by direct observation: (a) resources (availability of trained staff and MoH guidelines), (b) completeness and record-keeping of key reports, and (c) practices of data quality assessment and data analysis (online supplemental annex IIb).

    Availability of written guidelines or manuals for data review and data quality was considerably improved in DIPH districts: for example, the availability of DHIS2-specific data management guidelines rose from 17% (baseline) to 87% (endline) in intervention districts, as against 17–50% in comparison districts, resulting in 37 percentage points more in the intervention than in comparison arms (95% CI: 4%, 70%) (figure 2A).

    Figure 2
    Figure 2

    Practices of data management and perception of data use culture. (A) Changes in the availability of DHIS2 guidelines over time. (B) Changes in producing maternal infographics based on DHIS2 guideline over time (verified). (C) Changes in overall decision-making culture among district health management teams’ record over time (summary measure). (D) Changes in perception about commitment to data use overall among district health management teams’ record over time. DHIS2, District Health Information System 2; DIPH, Data-Informed Platform for Health.

    Data processing and analysis practices improved in both arms, but changes were greater in the intervention districts. The practice of producing up-to-date maternal infographics from DHIS2 was observed in less than half of the districts at baseline. There was a 36 percentage point greater improvement in intervention districts than in comparison districts (figure 2B).

    In the process evaluation, it was reported that the digital interconnectedness of information systems including interoperability introduced by DIPH—especially optimal linkages to DHIS2—resulted in the functional advantage of district management teams taking an integrated view of service delivery challenges and decision-making processes.

    Practices of data-driven decision-making

    In the intervention districts, holding regular PMT meetings in the last 3 months increased from 12% to 77% between baseline and endline, with a fall from 30% to 18% in comparison districts, resulting in DiD of 77 percentage points (95% CI: 40%, 114%) (figure 3A). Keeping of minutes in the latest PMT meeting increased from 77% at baseline to 98% at endline in intervention districts, yet fell from 85% to 75% in the comparison districts (figure 3B).

    Figure 3
    Figure 3

    Practices of data-driven decision-making. (A) Changes in the regularity of PMT meetings over time. (B) Changes in the availability of PMT meeting record over time: last meeting minutes recorded (observed). (C) Changes in diversity of decision-making in PMT meeting record over time: decisions made on the health facilities’ performance (reported)—summary measure. (D) Changes in feedback on data quality for PHCU in last 3 months (verified). DIPH, Data-Informed Platform for Health; PHCU, Primary Healthcare Unit; PMT, Performance Monitoring Team.

    In the process evaluation, all intervention districts reported an increased regularity of PMT meetings. The adoption of DIPH procedures for data use and planning, use of the PMT logbook and strengthened ownership of the process also enhanced commitment to the meetings.

    DIPH helps us to easily access information and deliver quality data by introducing the software. It is helping us to timely conduct review meetings by better coordination among district management members. (Head of DIPH District Health Office)

    In the last recorded PMT meeting minutes, the types of decision-making were noted and a summary measure was developed to assess the diversity of decision-making: themes included coverage of specific services, human resource management and emerging issues (online supplemental annex IIa–c). DiD analysis showed a 42 percentage point greater increase (95% CI: 6%, 77%) in the intervention districts over time, suggesting diverse decision-making with regard to assessing performance and problem-solving (figure 3C). At the end of the study period, providing feedback on data quality to primary healthcare facilities (PHCU) based on PMT proceedings was almost universal in intervention districts as against only 48% in comparison districts, with a resulting DiD of 48 percentage point positive change (95% CI: 9%, 87%) in the intervention group (figure 3D).

    Perceptions of data use culture

    The decision-making culture improved over time in intervention districts from 70% to 80%, with little change in the comparison districts: the DiD estimate suggested a net 12 percentage point improvement as a result of the intervention (95% CI: 8%, 16%) (figure 2C). Decision-making culture reflects the values and beliefs of district health administrators regarding collection, analysis and use of information to improve performance. Among its domains, the largest gains were in the perception of commitment to data use, with a net 16% (95% CI: 8%, 23%) improvement over 18 months (figure 2D). However, no meaningful effect was observed in acknowledging and rewarding good performance within district management teams (for further details, see online supplemental annex IId).

    In the process evaluation, respondents reported changes in their abilities in data use and collaborative decision-making culture as a result of DIPH activities:

    DIPH training empowered us to closely monitor down to the lowest level of the healthcare system, and in a fixed time frame. Responsibilities are shared among all stakeholders. We continually assess whether activities have met their target and set action points for those that didn’t. Now, everyone does their job with a sense of accountability which was lacking before. (M&E officer, DIPH District Health Office)

    Data use and collaborative procedures introduced by DIPH have been adopted in other district-level review meetings and routine tasks. It has especially influenced managers, who consult data before making decisions. Analysis comes before making a decision, and I believe this is one of the results of the DIPH training. (IT officer, DIPH District Health Office)

    Discussion

    We promoted structured decision-making by district health administrators and managers through their use of local data by embedding a new approach in their routine work. The DIPH intervention improved competencies in reviewing data, identifying problems by health system blocks, prioritising issues and implementing practical solutions through integrated, collaborative decision-making. The most remarkable results were observed in monthly PMT meetings. Each meeting, without exception, discussed the quality of data reported by the health facilities, resulting in structured feedback to the facilities on their performance, and formulated theme-specific action points which were followed through to completion. In addition, statistically significant effects were observed in the beliefs and opinions of district health administrators and managers: for example, their commitment to improving local data quality and promoting evidence-based decision-making.

    The key DIPH mechanisms of change (online supplemental annex Ia) in data use and stakeholder collaboration were observable during implementation. These included training, supervision and a tools package, leading to improvements in data quality and use in PMT meetings—both in the participation and contributions of stakeholders, and in collaborative decision-making. Data quality improved through group reviews, regular screening for errors in DHIS2 and reporting. Data use improved by strengthening analysis skills, digital use and data-sharing, and through the PMT meetings themselves. Stakeholders’ contributions were enhanced through formalised roles and task assignments, joint planning, follow-up, the digital platform and enhanced data analysis, as well as planning and implementation skills. Collaborative decision-making improved in terms of consensual decisions and task-sharing. The DIPH intervention worked in synergy with DHIS2 in the effective use of data for performance review and problem-solving at the local level: to the best of our knowledge, no similar tool or strategy exists in the DHIS2 platform.

    Our study took place in a context of health system decentralisation in Ethiopia. Institutional and national governance contexts affect the embedding and sustainability of any health system management intervention. In the Ethiopian devolved health system, the national MoH still issues policy directives. Nonetheless, districts have sizeable space for evidence-based decision-making to interpret and operationalise national directives. The relevance of approaches such as DIPH is commensurate with the degree of autonomy available to district health teams. The role of district management is crucial in translating national and regional policies into action while addressing local health needs, including resource management, staff supervision, coordination among projects and services, and responsiveness to community needs and performance targets. Each aspect requires evidence-based decision-making and commitment to follow-up.21 Among the obstacles to reaching population health targets in the primary healthcare system are limited core competencies and a lack of system-level opportunities to enable quality decision-making by district managers.22

    DHMTs work within broader sociocultural, organisational, structural and relational norms.23 In addition to structural changes—that is, decentralisation and devolution—behavioural hierarchies and power dynamics set the tone for interactions, collaborations and decision-making boundaries among stakeholders. DHMTs possess a unique understanding of their local health context: with improved compilation and interpretation of local data, and tools for collaborative decision-making, DHMTs can devise the most practical and feasible solutions for their local healthcare system. Collaboration among district stakeholders for joint decision-making was a key theme of the DIPH intervention. Diverse members of the district cabinet—including the district administrator, deputy administrator and sectoral heads—were contacted for participation.

    DIPH is fundamentally an application of ‘systems thinking’ principles in that data-driven decision-making is explicitly and critically linked with subcomponents of the health system. District managers reviewed available routine data with consideration to organisational and population contexts, and found feasible solutions within their means, bringing efficiency to their district’s primary healthcare performance.24 25 Unlike linear thinking, which assumes a single cause-and-effect model, systems thinking breaks down the complexity of a problem into manageable, interconnected components, incorporating multiperspective, participatory and iterative approaches while defining solutions.26 Despite its promise, there is negligible evidence of systems thinking being translated into health system processes and district-level decision-making.27–29

    While use of technology can enhance the efficiency of DIPH implementation, it is not a prerequisite for its success. As exemplified by our initial paper-based prototype testing in West Bengal, DIPH can be effectively implemented using traditional manual methods.13

    A proactive, consolidated management tier is central to building an effective and efficient health system.25 Most health management initiatives in LMICs, especially in sub-Saharan Africa, focus on core competencies and skills, while lacking evidence on systems operations and contextual dimensions necessary to embed and sustain their interventions.30

    There have been other recent attempts, contemporary with DIPH, to strengthen district health management, especially in African settings including Ghana Essential Health Interventions Programme, Tanzania Essential Health Interventions Project and African Health Initiative.31–34 Only a few have undergone rigorous evaluation leading to peer-reviewed publications. Notable among these are PERFORM (2011−2015)35 and PERFORM2 scale (2017–2021),31 carried out in Ghana, Malawi and Uganda to strengthen the capacity of district health managers—in particular, to develop better human resource management and health system strategies to improve performance. The broader PERFORM strategy was based on an action research cycle delivered in workshop-style formats involving four steps: plan (analyse the primary causes of health service delivery challenges); act (prioritise problems and solutions, design human resource management strategies linked to health systems); observe (follow up on implementation for 3 months); and reflect (on processes and effects). The evaluation of PERFORM was primarily qualitative, reporting a strengthened workforce management and highlighting the potential for workforce performance improvement. Similarly, a workshop-based approach was explored in Nigeria, where district management teams’ capacity to appraise the quality of local data was enhanced through periodic workshops and intense facilitation.7 This approach highlighted that cycles of self-assessment of data quality, review and feedback, enabled through regular district-level workshops along with technical support, can enhance the completeness, accuracy and internal consistency of routine health data.

    While sharing with these initiatives a focus on district health management and the introduction of structured processes, the DIPH strategy is distinct in that it was designed to be embedded in existing decision-making fora, strengthening those fora to make optimal use of all available key data sources to appraise challenges and implement solutions.

    Limitations

    Our outcomes were limited to practices and performances in data-driven decision-making for health system planning, problem-solving and follow-up on an action plan. Though population-level health outcomes remain the ultimate aim, the translation of health management improvement into population-level effects is a long-term process and beyond the scope and timelines of our research. The field implementation team conducted the baseline survey before DIPH intervention training and random allocation of districts to the study arms. An independent data collection team carried out the endline survey assessment, and this team were unaware of the nature of the intervention and the study arm of each district. Thus, although the study was not blinded, and the respondents themselves were inevitably aware of the intervention, we took steps to minimise potential bias. Furthermore, to mitigate any influence of change in the data collection team, our analyses were based on DiD. Although the DIPH intervention was multicomponent, it is not possible to assess the quantitative contribution of individual components nor how they worked together to result in the observed effect. Nevertheless, we have described above possible mechanisms of change through the process evaluation. Among contextual factors, the COVID-19 pandemic and changing law and order situation in some study districts slowed down DIPH implementation. However, of the 60 cycles planned, 58 were completed. Stakeholder participation in PMTs increased over time in both study arms, which may have been due partially to government reforms aiming to promote data use for better district planning and management.

    The cost of implementation—that is, financial resources needed across the design, initiation and maintenance stages of DIPH—would have been helpful information for health system stakeholders, especially policymakers, when embedding the strategy.36 It is important to note that the intervention is not resource-intensive, as it is about streamlining the decision-making process of the district health administration within an existing routine platform and with existing resources. The key costing element for scaling up would be the training of district administration staff and of supportive supervision, which would be 0.3–0.25 full-time equivalent for each district, particularly during the first months of implementation. The technological aspect would also involve a limited one-off cost. Our lack of costing estimation was a missed opportunity. However, we plan to incorporate a cost estimation in our sustainability research.

    Sustainability and scale-up by practitioners and policymakers are of critical importance to health system interventions. In related research, our group has explored broader determinants and frameworks of scalability underpinning the current study.37 38 Specifically, additional qualitative research was carried out to assess the possible mechanisms of sustainability and scale-up of DIPH after the assessment reported here. Analysis is in progress and will be shared elsewhere.

    Conclusions

    From this robust evaluation of a strategy to enable data-driven decision-making for solving local primary healthcare challenges, we conclude that the intervention improved data management and appraisal practices; systemised problem analysis to follow up on action points; and improved the culture of stakeholder engagement. Further research potential lies in adapting this strategy for various tiers of health systems, with diverse thematic focus.

    Data availability statement

    Data are available upon reasonable request. The data are available from the principal investigator of the IDEAS project and co-authors for this manuscript.

    Ethics statements

    Patient consent for publication

    Ethics approval

    This study involves human participants and was approved by the ethical review boards of the Ethiopian Public Health Institute and the London School of Hygiene & Tropical Medicine (reference numbers: EPHI 6-13/194 and LSHTM 17578). Participants gave informed consent to participate in the study before taking part.

    Acknowledgments

    We appreciate the support of the DIPH field staff, Amhara Regional Health Bureau head, Amhara Public Health Institute head, Health Department deputy head and District Health Office heads of North Shewa zone; the Tattva Foundation India (IT partner), and members of the advisory board including the MoH Policy, Planning, Monitoring and Evaluation director; and the EPHI Health System and Reproductive Research Directorate director and the EPHI deputy director general of Research and Knowledge Translation.

    References

    1. ↵
      1. Roman TE,
      2. Cleary S,
      3. McIntyre D
      . Exploring the functioning of decision space: a review of the available health systems literature. Int J Health Policy Manag 2017;6:365–76. doi:10.15171/ijhpm.2017.26
      OpenUrl
    2. ↵
      1. Bossert TJ,
      2. Mitchell AD,
      3. Janjua MA
      . Improving health system performance in a decentralised health system: capacity building in Pakistan. Health Syst Reform 2015;1:276–84. doi:10.1080/23288604.2015.1056330
      OpenUrlCrossRefPubMed
    3. ↵
      1. Bergum BI,
      2. Kusumasindra F,
      3. Øren M, et al
      . Deliverable 2 analyzing DHIS2 as an information infrastructure, submission date. University of Oslo; 2015. 22.
    4. ↵
      1. Panda B,
      2. Thakur HP
      . Decentralisation and health system performance - a focused review of dimensions, difficulties, and derivatives in India. BMC Health Serv Res 2016;16:561. doi:10.1186/s12913-016-1784-9
    5. ↵
      1. Freeman T
      . Using performance indicators to improve health care quality in the public sector: a literature review. Health Serv Manage Res 2002;15:126–37. doi:10.1258/0951484021912897
      OpenUrlCrossRefPubMed
    6. ↵
      1. Shortell SM,
      2. Rundall TG,
      3. Hsu J
      . Improving patient care by linking evidence-based medicine and evidence-based management. JAMA 2007;298:673–6. doi:10.1001/jama.298.6.673
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Wickremasinghe D,
      2. Hashmi IE,
      3. Schellenberg J, et al
      . District decision-making for health in low-income settings: a systematic literature review. Health Policy Plan 2016;31 Suppl 2(Suppl 2):ii12–24. doi:10.1093/heapol/czv124
      OpenUrlPubMed
    8. ↵
      1. Bhattacharya AA,
      2. Allen E,
      3. Umar N, et al
      . Improving the quality of routine maternal and newborn data captured in primary health facilities in Gombe state, northeastern Nigeria: a before-and-after study. BMJ Open 2020;10:e038174. doi:10.1136/bmjopen-2020-038174
    9. ↵
      1. Bulthuis S,
      2. Kok M,
      3. Onvlee O, et al
      . Assessing the scalability of a health management-strengthening intervention at the district level: a qualitative study in Ghana, Malawi and Uganda. Health Res Policy Sys 2022;20. doi:10.1186/s12961-022-00887-2
    10. ↵
      1. Bhattacharyya S,
      2. Berhanu D,
      3. Taddesse N, et al
      . District decision-making for health in low-income settings: a case study of the potential of public and private sector data in India and Ethiopia. Health Policy Plan 2016;31 Suppl 2:ii25–34. doi:10.1093/heapol/czw017
      OpenUrlPubMed
    11. ↵
      1. Dehnavieh R,
      2. Haghdoost A,
      3. Khosravi A, et al
      . The district health information system (DHIS2): a literature review and meta-synthesis of its strengths and operational challenges based on the experiences of 11 countries. Health Inf Manag 2019;48:62–75. doi:10.1177/1833358318777713
      OpenUrlPubMed
    12. ↵
      1. Mekebo M,
      2. Gobena T,
      3. Hawulte B, et al
      . Level of implementation of district health information system 2 at public health facilities in Eastern Ethiopia. Digit Health 2022;8:20552076221131151. doi:10.1177/20552076221131151
    13. ↵
      1. IDEAS-LSHTM
      . Data informed platform for health. Structured decision-making using local data. Prototype phase, West Bengal, India. External evaluation report. London, UK London School of Hygiene & Tropical Medicine; 2018.
    14. ↵
      1. Ethiopia Ministry of Health (MOH)
      . Health sector transformation plan [MOH]. 2015. Available: https://www.globalfinancingfacility.org/sites/gff_new/files/Ethiopia-health-system-transformation-plan.pdf [Accessed 11 May 2022].
    15. ↵
      1. Ethiopia Ministry of Health (MOH)
      . Health sector transformation Plan-II [MOH]. 2021. Available: https://fp2030.org/sites/default/files/HSTP-II.pdf [Accessed 22 Jun 2023].
    16. ↵
      1. Ethiopia Ministry of Health (MOH)
      . Woreda transformation implementation manual [MOH]. 2017. Available: http://repository.iifphc.org/bitstream/handle/123456789/1480/Woreda-Transform-Summary.pdf?sequence=1&isAllowed=y [Accessed 22 Jun 2023].
    17. ↵
      1. Theobald S,
      2. Brandes N,
      3. Gyapong M, et al
      . Implementation research: new Imperatives and opportunities in global health. Lancet 2018;392:2214–28. doi:10.1016/S0140-6736(18)32205-0
      OpenUrlCrossRefPubMed
    18. ↵
      1. Hayes RJ,
      2. Bennett S
      . Simple sample size calculation for cluster-randomized trials. Int J Epidemiol 1999;28:319–26. doi:10.1093/ije/28.2.319
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. The DHS Program
      . Demographic and health survey (DHS). Available: https://dhsprogram.com [Accessed 20 Jun 2023].
    20. ↵
      1. Hotchkiss DR,
      2. Aqil A,
      3. Lippeveld T, et al
      . Evaluation of the performance of routine information system management (PRISM) framework: evidence from Uganda. BMC Health Serv Res 2010;10:188. doi:10.1186/1472-6963-10-188
    21. ↵
      1. World Health Organization
      . Who are health managers: case studies from three African countries [World Health Organization]. 2009. Available: https://apps.who.int/iris/handle/10665/44119
    22. ↵
      1. World Health Organization
      . Managing the health millennium development goals: the challenge of management strengthening: lessons from three countries [World Health Organization]. 2007. Available: https://apps.who.int/iris/handle/10665/70010
    23. ↵
      1. Arts B,
      2. Tatenhove JV
      . Policy and power: a conceptual framework between the ‘Old’And ‘New’Policy idioms. Policy Sci 2004;37:339–56. doi:10.1007/s11077-005-0156-9
      OpenUrlCrossRefWeb of Science
    24. ↵
      1. Adam T
      . Advancing the application of systems thinking in health. Health Res Policy Syst 2014;12:50. doi:10.1186/1478-4505-12-50
    25. ↵
      1. Kwamie A,
      2. van Dijk H,
      3. Agyepong IA
      . Advancing the application of systems thinking in health: realist evaluation of the leadership development programme for district manager decision-making in Ghana. Health Res Policy Syst 2014;12:29. doi:10.1186/1478-4505-12-29
      OpenUrlPubMed
    26. ↵
      1. Thelen J,
      2. Sant Fruchtman C,
      3. Bilal M, et al
      . Development of the systems thinking for health actions framework: a literature review and a case study. BMJ Glob Health 2023;8:e010191. doi:10.1136/bmjgh-2022-010191
    27. ↵
      1. Wilkinson J,
      2. Goff M,
      3. Rusoja E, et al
      . The application of systems thinking concepts, methods, and tools to global health practices: an analysis of case studies. J Eval Clin Pract 2018;24:607–18. doi:10.1111/jep.12842
      OpenUrl
    28. ↵
      1. Chughtai S,
      2. Blanchet K
      . Systems thinking in public health: a bibliographic contribution to a meta-narrative review. Health Policy Plan 2017;32:585–94. doi:10.1093/heapol/czw159
      OpenUrlPubMed
    29. ↵
      1. Mutale W,
      2. Balabanova D,
      3. Chintu N, et al
      . Application of system thinking concepts in health system strengthening in low-income settings: a proposed conceptual framework for the evaluation of a complex health system intervention: the case of the BHOMA intervention in Zambia. J Eval Clin Pract 2016;22:112–21. doi:10.1111/jep.12160
      OpenUrl
    30. ↵
      1. Curry L,
      2. Taylor L,
      3. Chen PG-C, et al
      . Experiences of leadership in health care in sub-Saharan Africa. Hum Resour Health 2012;10:33. doi:10.1186/1478-4491-10-33
    31. ↵
      1. Awoonor-Williams JK,
      2. Bawah AA,
      3. Nyonator FK, et al
      . The Ghana essential health interventions program: a plausibility trial of the impact of health systems strengthening on maternal & child survival. BMC Health Serv Res 2013;13 Suppl 2:S3. doi:10.1186/1472-6963-13-S2-S3
    32. ↵
      1. De Savigny D,
      2. Kasale H,
      3. Mbuya C, et al
      . Fixing health systems, 2nd edition. Canada International Development Research Centre; 2008. Available: https://idrc-crdi.ca/en/book/fixing-health-systems-2nd-edition
    33. ↵
      1. De Savigny D,
      2. Kasale H,
      3. Mbuya C, et al
      . Tanzania essential health interventions project: TEHIP “interventions”; an overview. 2002. Available: http://hdl.handle.net/10625/40912 [Accessed 02 Dec 2023].
    34. ↵
      1. Hirschhorn LR,
      2. Adedokun L,
      3. Ghaffar A
      . Implementing, improving and learning: cross-country lessons learned from the African health initiative. BMC Health Serv Res 2017;17:773. doi:10.1186/s12913-017-2655-8
    35. ↵
      1. Martineau T,
      2. Raven J,
      3. Aikins M, et al
      . Strengthening health district management competencies in Ghana, Tanzania and Uganda: lessons from research to improve health workforce performance. BMJ Glob Health 2018;3:e000619. doi:10.1136/bmjgh-2017-000619
    36. ↵
      1. Sohn H,
      2. Tucker A,
      3. Ferguson O, et al
      . Costing the implementation of public health interventions in resource-limited settings: a conceptual framework. Implement Sci 2020;15:86. doi:10.1186/s13012-020-01047-2
    37. ↵
      1. Spicer N,
      2. Berhanu D,
      3. Bhattacharya D, et al
      . 'The stars seem aligned': a qualitative study to understand the effects of context on scale-up of maternal and newborn health innovations in Ethiopia. Global Health 2016;12:75. doi:10.1186/s12992-016-0218-0
    38. ↵
      1. Spicer N,
      2. Bhattacharya D,
      3. Dimka R, et al
      . 'Scaling-up is a craft not a science': catalysing scale-up of health innovations in Ethiopia. Soc Sci Med 2014;121:30–8. doi:10.1016/j.socscimed.2014.09.046
      OpenUrlCrossRefPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Handling editor Seema Biswas

    • Contributors BIA and JS conceived the research project. MD and GT collected data. BIA analysed data with contributions from JS and GT. BIA and JS drafted the manuscript with the involvement of all authors. TM and LAP provided critical revisions. BIA is the guarantor. All authors approved the final version for the manuscript submission.

    • Funding This study was supported by IDEAS—Informed Decisions for Actions to improve maternal and newborn health (http://ideas.lshtm.ac.uk), funded through a grant from the Bill & Melinda Gates Foundation to the London School of Hygiene & Tropical Medicine (Gates Global Health grant number: INV-007644).

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.