Discussion
Using the PRISM framework, we examined components of interventions designed to increase the quality of data generated through RHISs at peripheral levels of the health system. We screened 5294 references and identified 17 interventions in 56 studies designed to increase DQ and that typically measured improvements of three outputs: accuracy, completeness and timeliness-- although definitions of those outputs differed between studies (online supplemental file S1(2.4)). Most studies were conducted in Africa, and in particular Kenya (20%). This may reflect a concentration of research and NGO (Non-Governmental Organisation) communities located in Africa and Kenya working in the field of DQ.
Most studies (93%) reported improvements in DQ, regardless of the types of interventions or context. This suggests that any investment in improving data accuracy, completeness and timeliness may result in DQ improvements, though not necessarily to a satisfactory level of quality. We attempted to identify more effective interventions by setting a DQ threshold of 80% for any of the DQ outputs measured. Intervention components that improved all three DQ outputs from below to above the 80% threshold were training (either normal or enhanced) and eHMIS. Examining DQ outputs separately, completeness was improved most through meetings, engagement of stakeholders and eHMIS. Accuracy was improved through training, equipment purchases and maintenance, data checks and timeliness through mHealth, enhanced training and eHMIS.
Overall, eHMIS was the intervention that was most frequently associated with increased DQ above the 80% threshold. DHIS2 was the most used eHMIS system. Training was the second most used intervention that seemed to effectively improved DQ in terms of accuracy and completeness (figures 5 and 6). The subject of the training varied and included data aggregation,27 new data entry form28 and utilisation of new digital systems.29–45 The effect of enhanced training with longer training period did not seem to exceed the effect of short-term training in improving accuracy and completeness (figure 5 and 6), this is consistent with findings by Rowe et al.46 The use of mHealth solutions alone was only effective in improving timeliness, using mHealth interventions in reducing the time lag between collection and usage points had an impact on improving timeliness. Conversely, it did not show an effect in preventing input errors at the point of collection and thus was not enough to ensure data accuracy.37 47 48 For this reason, authors of the studies included in this review recommended that in addition to an automated system at the point of collection, DQ checking and supervision in multiple levels of the system is crucial to ensure better accuracy and completeness.36 49–56
Overall, technical interventions alone were not shown to be ‘silver-bullets’, but required careful consideration of context.9 57 58 For example, eHMIS implemented as part of the intervention in most studies, consisted of health system-wide components that addressed multiple processes from data collection through to feedback. Indeed, while we have described single components most frequently associated with effective DQ improvements, our findings also suggest that more comprehensive approaches in the design of DQ interventions, that is, applying more intervention components and addressing all the technical, organisational, and behavioural aspects of RHIS, were likely lead to greater improvements in DQ than implementing a single component interventions (figures 5 and 6). Organisational/behavioural factors seem particularly important given that all bar one of the top five most effective interventions that improved data completeness included interventions that included these such as meetings, engagement of stakeholder, task-shifting, enhanced supervision. This finding of the need for a more holistic approach is not particularly novel. With the advent of the ‘microcomputer’ in the 90s, Sandiford et al had already identified that technical approaches alone would not improve DQ.9
Different DQ outputs are interlinked (ie, accuracy without completeness cannot exist), but this review showed that the mechanisms of improving each output may differ. Pilot, pre–postevaluations or controlled trials provide important insights into elements that are likely to impact on primary outcomes, however, beyond the study period these outcomes must be regularly monitored when implemented at scale.59 Nearly half of the studies (43%) were undertaken over periods of less than 12 months. One might expect optimal results during intense investigate periods. The frequency of studies of a duration >3 years was relatively low (18%), therefore limiting our ability to examine sustainability or real-life implementation constraints.
Fewer qualitative (13%, table 1) and mixed-method studies (34%, table 1) were identified during the review of the literature. These studies often reported improved DQ based on ‘perceptions’ of the users who were involved in the interventions. Further studies including qualitative methods are necessary to examine appropriateness of interventions or perceived usefulness of different data component applications in various contexts in order to unpack why interventions may not work or could be further optimised.
Our findings show a remarkable diversity in both the methodology to test interventions and the measurement of DQ. Most studies included in the review were pre–postintervention comparisons (table 1), only five studies were formal RCTs. The latter are considered to provide the highest quality evidence. The design of quality improvement interventions is a critical area for future studies focusing on RHIS.60
In terms of measuring DQ outputs, our review highlights the need to design studies of RHIS interventions with a clear set of measurable outputs, which are comparable beyond a trial or pilot phase. While we defined ‘above-threshold’ studies as those that showed improvement and achieved ≥80% DQ postintervention for any of the three DQ outputs, a more comprehensive approach would have been to apply thresholds set out in the DQ review61 in which multiple thresholds are suggested according to different levels of the health system and core indicators for health data used universally (ie, Antenatal Carefirst visit, third-dose DTP (diphtheria. tetanus. pertussis)-containing vaccine). Going forward, studies investigating the effect of interventions on DQ should aim to align evaluations with thresholds and targets set out in recent years.
Limitations across studies
Most studies provided limited details on their interventions and this could have missed components. We relied on authors to report key intervention activities, which could introduce an inherent bias. Also, some components might appear more effective, due to the number of studies implementing that component. For example, components that were included in less studies, such as ‘task-shifting’, which was only in five studies, appeared more effective in improving completeness as defined as the difference between studies that showed improvement vs those that did not. Comparatively, training was included in many studies and while still ranked as a top five component appears ‘less effective’ than task shifting. It is still important to note that within our analysis the top three to five components showed a clear difference.
While cost-effectiveness was not an outcome measure for this study, we note that only three of the studies34 62 63 considered costs such as cost–benefit analysis of staff time or comparison of cost between different digital interventions, but none assessed costs per quantity of data improvement and this is something often ignored. This omission was certainly a limitation across studies and prevented the authors’ ability to assess intervention components based on opportunity costs and budget allowances.
Limitations of this review
There are several limitations to this review related to the number of studies that we identified and were able to include into the study and the variability in measurement of three DQ outputs. First, we did not conduct a meta-analysis in this review due to varying outcome measures for each of the DQ outputs (ie, accuracy, completeness, timeliness, relevance) which could not be consolidated (online supplemental file S1(2.4)), and were not clinical data. Furthermore, only five RCTs were identified, making it unviable to consolidate quantitative measures to conduct a meta-analysis.
Throughout the studies, the number of times an intervention component was studied varied as did the number of DQ outputs that studies measured. This limited the analyses that we could undertake and the strength of the conclusions we could draw from the review. In examining DQ outputs in an aggregated way, we risk being unable to disentangle which interventions contributed to which DQ output measures. However, disaggregating studies by DQ output measured means reviewing small numbers of studies. We tried to strengthen our analyses by both aggregating and then disaggregating DQ outputs measured to assess whether there were key intervention components emerging as key to improving overall DQ. Additionally, due to length considerations we have presumed that the interventions, distinguished across 17 different categories, were implemented with similar intensity across contextually similar settings, which is not reflective of reality.
More broadly, our study did not address data use—which is likely to act as an intervention in and of itself because the use of data leads to feedback and several studies, including this review, have shown that feedback does improve DQ.
Finally, the interventions we identified do not address some fundamental factors identified as challenging to good DQ previously17 such as ‘technical infrastructure, issues such as unreliable electric power and erratic Internet connectivity and clinicians’ limited computer skills… good communication and networking actions among all stakeholders of HMIS, and information culture at different levels of district health information systems’.64 Given studies included in this review tended to take place over relatively short periods of time, these more fundamental issues may not have been identified as barriers to uptake of the intervention more widely.