Discussion
As part of the PCE, we developed conceptual frameworks and health system models for HIV, TB and malaria and applied them to evaluate delivery of a complex package of interventions in three countries. Although the models do come with some important limitations, we believe this study shows that an HSM approach is a valuable addition to other implementation science methods to gain greater insight into the functioning of health service delivery.
Statistical and mathematical models to evaluate global health interventions are rarely done in a way that emphasises multiple, interdependent interventions, indirect effects and relationships along causal pathways. Modelling approaches are being employed to measure important health indicators with ever more precision and validity,33–35 but despite advances, they usually focus on the nuances of disease transmission processes or on forming highly accurate predictions, with the near-universal objective of helping policy-makers prioritise and allocate resources.36 In other words, most statistical models in global health are used to aid decision science, but may be equally valuable to aid implementation science, as this study has aimed to do.
Through the PCE, teams are already disseminating these results for use in informing national programme strategies, grant implementation processes, and topics for further investigation. For example, the suggestion from some of the findings in DRC are that these new estimates could have practical utility for budgeting during the next funding request (estimates of cost per unit delivered), as well as setting realistic targets for grant implementation (estimates of increases in coverage per additional output). In Guatemala, the potential bottlenecks identified have been considered as an evidence-based starting point for further qualitative evaluation. In Senegal, diverse subnational patterns are being disseminated as a way of drawing increased attention to equity considerations for areas that may need further prioritisation by the national programme. In each country, model results were also used in triangulation with other evaluation results to support the findings of other methods or identify areas needing further study. In these ways, the PCE has used an HSM approach to gain pragmatic insights about how to improve implementation in grants and programmes.
This work extends earlier efforts in the application of HSM in multiple ways. First, by collaboratively developing conceptual frameworks and models with in-country teams, these models are locally-relevant, tailored to the specific interventions occurring in a country, and incorporate the most robust data sources available in each setting. As others have advocated, purposively inclusive, theory-based practices should be further emphasised in global health systems research.14 37 38 Second, by carrying out these models as part of a prospective evaluation, we have produced timely results that can be readily used by stakeholders, demonstrating that HSM approaches—especially post-modelling analysis—can potentially aid programme implementation directly. Finally, this work adds to the broader literature of assessing the contribution of development assistance towards health outcomes. As Ataya et al13 describe measuring the effect of health investment by tracking trends in mortality, morbidity and coverage indicators overlooks the complex causal pathways by which investments become changes in outcomes. This work demonstrates an approach toward detailing those causal pathways.
Another strength of this work is the use of routine national programme data, an underused data source in LMICs.39 National programme data and administrative data, sometimes referred to as health management information system (HMIS) data, are collected in increasing volumes in most LMICs, but are rarely analysed in academic literature across a whole health system or disease area. There are important caveats to these data, as low completeness can render them poorly representative of population-level health outcomes. However, through over a decade of financial investment in HMIS and sustained work by in-country monitoring and evaluation teams, HMIS data are increasing in completeness and coverage.40 41 We expect these data systems to continue to improve, making it imperative to advance analytical methods that use them for purposes such as implementation science.
There are also important limitations to these analyses. Already mentioned, data quality and availability among HMIS data can vary. Although we have taken steps to systematically correct for issues such as missing data and outliers, systematic misclassification (either overcounting or undercounting) may persist and contribute to bias in the model results. Further, some data included in the model were found to be highly variable, at times resulting in wide uncertainty intervals described above. For this reason, the direction and magnitude of the coefficients represented in the model should be interpreted in the context of other data, such as from stakeholder interviews. Finally, the availability of data limited our ability to comprehensively reflect the conceptual framework, especially with indicators that pertain to non-financial inputs and activities that lack traceable commodities.
Apart from data quality limitations, a number of model limitations are important to discuss as well. First, the models presented here are all static models. No time-varying or otherwise-dynamic coefficients were built into the models, although dynamic HSM have already been identified as the standard.15 Second, the models presented are necessarily a simplification of the complete conceptual framework. Numerous variables and elements were included in the conceptual model but not in the statistical model, leaving important gaps between adjacent indicators and some pathways only superficially represented. Similarly, we elected to aggregate some variables, especially financial variables to construct the model, which loses some detail in terms of expenditure by intervention. Third, the lagged relationships between inputs and activities, as well as certain outputs and treatment success rates were uniformly applied. In reality, the delay between expenditure and activity is likely to both fluctuate over time and by intervention, but this model assumed them to be constant. Finally, many factors from outside the health system are not reflected in this model, and thus confounding may be of concern. While many linkages, such as the linkage between shipment of a commodity and facility output of it, may be assumed to be unaffected by changes in community indicators, others, such as the linkage between coverage of services and changes in burden of disease, are inarguably confounded. This limited our ability to reliably measure those aspects of the results chain, hence the models that conclude with outcomes rather than impact. Each of these limitations is important, but they are also the result of an applied, use-focused approach; we developed these models expressly for the purpose of evaluating implementation in a way that it can be used in a timely fashion.
Future directions of this work may seek to mitigate some of these limitations. For example, more detailed examination of specific aspects of the results chain, such as supply chains, already exists and could be incorporated into these models.42 43 Some authors have advocated an approach that focuses on the least-certain pathways for secondary analysis.14 Other HSM work has implemented both systems dynamics models and disease transmission models in tandem to reach the end of the results chain.18 25
Nevertheless, we believe HSM approaches have important utility in complex evaluations and implementation science. Among global policy-makers, the mechanisms of local health service delivery are often perceived as micro-planning concerns outside the control of donors and the international community. But to improve population health it is imperative to both identify effective interventions and to deliver them efficiently and effectively, and both global and local stakeholders benefit from better delivery. This is especially true in the context of stagnating development assistance for health as has been the case for several years.6 To do so, a continued focus on the complexity of health systems and the delivery of services through them is necessary, and the approach demonstrated here may be an important new approach for doing just that in global health evaluations.
