Article Text
PDF
Abstract
This manuscript describes the process and impact of strengthening the WHO Regional Office for Africa (WHO AFRO)’s COVID-19 vaccination information system. This system plays a critical role in tracking vaccination coverage, guiding resource allocation and supporting vaccination campaign roll-out for countries in the African region. Recognising existing data management issues, including complex reporting prone to human error, compromised data quality and underutilisation of collected data, WHO AFRO introduced significant system improvements during the COVID-19 pandemic. These improvements include shifting from an Excel-based to an online Azure-based data collection system, automating data processing and validation, and expansion of collected data. These changes have led to improvements in data quality and quantity including a decrease in data non-validity, missingness, and record duplication, and expansion of data collection forms to include a greater number of data fields, offering a more comprehensive understanding of vaccination efforts. Finally, the creation of accessible information products—including an interactive public dashboard, a weekly data pack and a public monthly bulletin—has improved data use and reach to relevant partners. These resources provide crucial insights into the region’s vaccination progress at national and subnational levels, thereby enabling data-driven decision-making to improve programme performance. Overall, the strengthening of the WHO AFRO COVID-19 vaccination information system can serve as a model for similar efforts in other WHO regions and contexts. The impact of system strengthening on data quality demonstrated here underscores the vital role of robust data collection, capacity building and management systems in achieving high-quality data on vaccine distribution and coverage. Continued investment in information systems is essential for effective and equitable public health efforts.
- COVID-19
- Epidemiology
- Health systems
- Immunisation
Data availability statement
Data may be obtained from a third party and are not publicly available. Data are not publicly available.
This is an open access article distributed under the terms of the Creative Commons Attribution IGO License (CC BY NC 3.0 IGO), which permits use, distribution,and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.
Statistics from Altmetric.com
Summary box
In accordance with International Health Regulations, the 47 African member states of the WHO Regional Office for Africa (WHO AFRO) are mandated to report events that threaten public health security to WHO AFRO. As part of the response to the COVID-19 pandemic, member states report weekly national-level COVID-19 vaccination data and WHO AFRO collects, collates and publishes those data. Recognising gaps in their information system, WHO AFRO initiated an effort to improve data collection, management and analysis for COVID-19 vaccination data.
This study documents the process of improving WHO AFRO’s COVID-19 vaccine information system and demonstrates that there were measurable improvements in data quality after system enhancements.
This work can serve as a model for other regions and contexts and highlights the importance of investment in information systems to achieve high-quality vaccination data and enhance evidence-based public health decision-making.
Introduction
The COVID-19 pandemic resulted in high levels of mortality and morbidity around the world1 requiring urgent global coordination to roll out COVID-19 vaccines after they were first authorised for use in late 2020.1 The WHO set ambitious global coverage targets in 2021 and 2022, aiming to vaccinate 70% of the total population and 100% of high priority groups including healthcare workers and older adults.2 While these unprecedented efforts have led to significant uptake of COVID-19 vaccine around the world, vaccine coverage in the African region had the slowest progress of any global region.3 As of June 2023, Africa has achieved 30% full coverage, defined here as completion of a primary COVID-19 vaccine series, compared with 64% globally.3 This subtarget coverage is due to many factors, including but not limited to poor vaccine delivery infrastructure, uneven vaccine supply and low vaccine demand4; inadequate COVID-19 information systems and reporting have been cited as playing a crucial role in the efficacy of vaccine roll-out programmes.5
The WHO Regional Office for Africa (WHO AFRO), which oversees WHO operations for 47 member states in Africa, plays a critical role in overseeing the collection, compilation and reporting of data and information on outbreaks and other public health emergencies of international concern and shares that information with WHO Headquarters, member states and partners.6 At the onset of the COVID-19 pandemic, WHO AFRO coordinated the compilation, management and dissemination of weekly national-level COVID-19 vaccination data for all 47 African member states. Data collection was initiated in March 2021.7
These data are critical for decision-making because they aid in understanding individual member state’s progress towards the coverage targets across the entire region,8 9 and they provide essential information to guide the roll-out of vaccination campaigns such as allocation of financial resources, vaccine supply distribution, infrastructure support for vaccine roll-out and human resource management.10 WHO AFRO must ensure that the data they share and publish are high quality and contain enough information to enable informed decision-making.6 Recognising these needs, in January 2022, WHO AFRO launched an initiative to scale and speed up COVID-19 vaccine roll-out regionally, and as part of this effort a WHO AFRO Monitoring and Evaluation team conducted a rapid analysis of WHO AFRO’s COVID-19 vaccine information system including review of reporting mechanisms, data collected and available information products. Based on those results, in February 2022, WHO AFRO launched an effort to overhaul their COVID-19 vaccination information system with the objective of increasing data usage and data quality. Here, we describe the steps taken by WHO AFRO to strengthen their COVID-19 vaccine information system and increase data usage. We measure the success of those efforts by comparing data quality before and after.
Approach
This manuscript first describes WHO AFRO’s initial COVID-19 vaccine information system including the gaps and challenges it presented, and then describes WHO AFRO’s efforts to strengthen its COVID-19 vaccination information system through improvements in data collection, management and validation.
We then identify all new information products developed as part of WHO AFRO’s efforts to improve usage of COVID-19 vaccination data. We describe the audience and summarise the content of each information product, as well as the platform in which each was developed, its mode of distribution and how frequently it is updated or published.
Finally, we measure the impact these efforts had on increasing data quality by comparing data quality measures before and after system strengthening was implemented. To measure the impact of data system strengthening on data quality, we compared 4725 records total including 2753 records before the system was overhauled (baseline) and 1992 records from after. All data are aggregate weekly national-level data submitted by member states that represent total cumulative COVID-19 vaccination statistics rather than individual-level vaccination data. Data compared for both time points were final endpoint data sets after all processing, validation and cleaning had been conducted. Data quality was measured by calculating missingness, data non-validity (entries outside the acceptable value range) and record duplication. Missingness and non-validity were measured for each of five core variables that were included in the data collection forms at both time points. Core variables included (1) total doses administered (total vaccine doses administered including first, second and any recommended booster doses), (2) total doses received (total vaccine doses received or purchased by the member state), (3) total doses received via COVAX (total vaccine doses received via the COVAX mechanism,11 (4) number of people fully vaccinated (total number of people in the member state who have completed a primary series as recommended per vaccine manufacturer) and (5) number of people vaccinated with at least one dose (total number of people in the member state who have completed a minimum of a single dose of a vaccine series). We report the change from before to after system improvements in missingness and non-validity for each core variable as well as overall and report the change in duplicated entries.
Initial COVID-19 vaccine information system
Initially, WHO AFRO’s COVID-19 vaccination data in the region was managed through an Excel-based system. Data were uploaded weekly by member state focal persons into a shared Excel file accessible and jointly used by all member states, manually reviewed by WHO AFRO for non-valid entries, duplication and missing entries, and shared with WHO headquarters and other partners as needed. Data submitted to the Excel sheet were aggregate weekly totals on COVID-19 vaccination statistics at the national level per member state rather than individual level vaccination data (one row of entered data per country per week). This approach introduced multiple challenges, including susceptibility to human error (eg, missed non-valid entries or faulty corrections), time-intensive workload for WHO AFRO staff overseeing data management, and limited production and distribution of Africa-specific information products and thus utilisation of collected data.
System strengthening process
Improvements to the data collection system and capacity building for end-users
In February 2022, an online Azure (Microsoft, Redmond, Washington, USA) data collection system was developed to replace the Excel-based system for data collection of the COVID-19 Vaccination dataset. Azure was selected to ensure data security and based on concurrence with WHO AFRO’s existing Microsoft-based infrastructure to maximise system streamlining and leverage existing personnel capacity. All data entered in the portal are automatically backed up in a cloud database for disaster recovery, and the system automatically maintains data history. The portal is login and password protected, and system users are given unique access roles with varied permissions based on user requirements (eg, countries, subregional users, partners, system administrators, data system team, data extractors). Data entry forms are developed with data validation checks and data type constraints for accuracy and reliability such as selection from drop-down menus for categorical variables and range limitations on entered dates and numbers. Data submitted through the portal are also aggregate weekly totals on COVID-19 vaccination statistics at the national level by member state rather than individual-level vaccination data. Member state focal points can view their entered data in real time, and the portal offers a range of user controls including creation, reading, updating and deleting records. The portal allows users to sort, filter and export data in multiple file formats including csv, pdf and xlsx.
The designated focal points from member states are required to report data by 8:00 hours Central Africa Time (CAT) every Monday for the preceding week. Prior to system roll-out, at least one member state focal person from each of the 47 WHO AFRO member states participated in a virtual 1-day training demonstrating all portal functionalities and walking users through data entry and management. Additionally, users were provided with contact information for troubleshooting in the event of any system challenges. Users are additionally invited to provide feedback to the WHO AFRO team on the portal, and iterative improvements are made based on incoming feedback.
Increasing data collection
On roll-out of the data collection portal, the data collection forms were updated to increase the number of data collection fields for weekly submission to the COVID-19 vaccination data set from five fields to 32. Additional data fields added to the new data collection form included: vaccine reception by source and manufacturor, vaccines administered by manufacturor, priority group, and sex, vaccine coverage for the total population, target population, and per priority group, total vaccines expired, reported AEFI’s, and total, target, and priority group population references for each member state.
A second data collection form was created in March 2022 to gather operational information on COVID-19 vaccine roll-out. This form includes data on vaccination strategies and policies, priority groups, available vaccination sites by subnational region, vaccine supply and administration nationally and by subnational region, mass vaccination campaigns planned and implemented, partners, funding and available human resources. This form was created with the goal of providing information to explain progress on vaccine coverage and identify targeted areas for better support. No operational data were collected in the original COVID-19 vaccine information system, and between March and June 2022, operational data were originally collected on a form developed in Excel due to existing resources at the time of creation. Starting on 24 June 2022, data collection was transitioned to the online data collection portal. As part of the transition to the online data collection portal, member state focal points from each member state were trained virtually on system use. This form began in the portal as a separate form and on 28 March 2023 the form was integrated into the COVID-19 vaccination data collection form.
Automation of data processing and validation
On implementation of the online data collection portal, data processing was updated from a manual system to automatically merge and validate weekly data. After 8:00 hours CAT each week, WHO AFRO data staff extract the update data from the Azure platform and execute an R script (a text file containing a set of commands executed within the R software) that collates and merges new data submitted by each member state with all data from all previous weeks. An R script is then executed on the merged data to perform data quality checks including checks for data validity (values within acceptable ranges set for each field) and data missingness (missing fields for countries entering data and countries that did not report any data for a week’s submission). If any data quality issues are identified, the relevant focal person is notified by phone or email to rectify issues identified. In cases when a member state fails to enter data for a given week and the focal person cannot be reached, WHO AFRO data staff will use official public data published by the corresponding member state’s Ministry of Health to fill in missing records for that week. The final merged and cleaned data set for each week is stored in a centralised SharePoint folder owned by WHO AFRO, and this is used as the single source of truth for all information products and data sharing. The full data flow is shown in figure 1.
A summary of the updated data flow for COVID-19 Vaccination Data for the WHO Regional Office for Africa (WHO AFRO). Data are entered by member state focal points weekly into the Data Collection Tool, where it is stored in a cloud-based database. Data are extracted, automatically merged, validated, and stored in a secure WHO AFRO SharePoint where they are used to generate information products and share data.
New information products created and published
Three new information products were developed and are routinely published as part of WHO AFRO’s effort to improve data use, summarised here in table 1 .
A summary of new information products developed as part of the effort by WHO Regional Office for Africa (WHO AFRO) to strengthen data usage
A full description of each information product is as follows:
A dashboard on COVID-19 vaccination (figure 2): This interactive, publicly available dashboard was developed in PowerBI. It is updated weekly and summarises and visualises data on vaccine supply, administration, coverage and operational information for 47 AFRO member states. This dashboard also includes the total first dose and complete primary series vaccine administration data on seven additional countries on the African continent that are not part of the AFRO region. The dashboard is interactive and can be viewed for the region as a whole or per member state. Data on vaccine supply and administration are shown overall as well as disaggregated by vaccine manufacturer, vaccine source, priority group and sex. Since the original publication, the dashboard has been expanded to include data for select countries on vaccine demand, vaccine expiry, mass vaccine campaigns and vaccination policies. Link: https://afrocovid19vaccination.azurewebsites.net/vaccination-dashboard/
A weekly COVID-19 vaccination data pack (figure 3): The data pack is a slide set developed to visualise performance results for key COVID-19 vaccination indicators using the same graphs and figures each week in a simplified, digestible format. This data pack also includes full data on the 47 AFRO member states and more limited data on the 7 additional countries on the African continent that are not part of the AFRO region. The slide set is automatically produced each week using updated data and the resulting PowerPoint file is distributed via an email listserv to key partners.
A COVID-19 vaccination bulletin (figure 4): This bulletin was published monthly starting February 2022 and ending December 2022. All published issues are publicly available for download on the WHO AFRO website. This bulletin summarises progress in COVID-19 vaccination uptake across 47 AFRO regional member states in a narrative format. Each monthly issue provides updates on a standardised set of indicators and summarises key interpretations and trends. The bulletins include a narrative update on vaccination progress for the region overall and highlight progress in one or more countries in each issue. Link: https://www.afro.who.int/health-topics/coronavirus-covid-19/vaccines/monthly-bulletin
Screenshot of the home page from the dashboard on COVID-19 Vaccination developed as part of the WHO Regional Office for Africa effort to improve data usage. Link: https://afrocovid19vaccination.azurewebsites.net/vaccination-dashboard/.
Screenshot of the title slide and an example data slide from an issue of the weekly COVID-19 Vaccination Data Pack developed as part of the WHO Regional Office for Africa effort to improve data usage.
Screenshot of the cover page of an issue of the COVID-19 vaccination bulletin developed as part of the WHO Regional Office for Africa effort to improve data usage. Link: https://www.afro.who.int/health-topics/coronavirus-covid-19/vaccines/monthly-bulletin.
Impact of system strengthening on data quality
Changes in data quality
Changes in data missingness and validity for core variables are shown in figure 5. After implementing the COVID-19 vaccination data collection portal in February 2022, missing data decreased for all five core vaccine administration variables. Missingness decreased from 2.2% (n=6/2753) to 1.3% (n=26/1992) for total doses administered, 1.9% (n=53) to 1.1% (n=22) for total doses received, 11.8% (n=324) to 2.3% (n=46) for COVAX doses received, 17.8% (n=489) to 1.3% (n=26) for fully vaccinated, and 5.6% (n=155) to 1.3% (n=26) for vaccinated first dose (figure 5A). Core variable non-validity decreased for three out of five core variables, stayed the same for one variable and rose slightly for one core variable. Non-validity decreased from 0.9% (n=24) to 0.4% (n=7) for total doses received, decreased from 0.5% (n=13) to 0.3% (n=6) for total doses administered, increased from 0.1% (n=3) to 0.2% (n=4) for fully vaccinated, stayed the same at 0.6% for vaccinated first dose (n=17 pre and n=13 post), and decreased from 1.1% (n=29) to 0.45% (n=9) for COVAX doses received. (figure 5B). Duplication dropped from 4.0% (118) to 0.0% (n=0) of entries of health information system strengthening measures (February 2022).
Change in (A) missingness and (B) non-validity for each five core variables for WHO Regional Office for Africa vaccine administration data between 1 June 2021 and 30 October 2022, before and after implementation.
Conclusions
This manuscript describes the process of strengthening WHO AFRO’s COVID-19 vaccine data collection, management and analytical system and provides evidence that these system improvements resulted in changes in the quality of data. The transition to an online data collection system, the automation of data processing and validation, the expansion of the data that are collected, and the development of new, accessible information products have all led to measurable improvements in data quality, quantity and data usage.
The implementation of a new data collection portal, along with automation of data processing and validation, resulted in reduced data missingness for all variables, reduced data non-validity for all but two variables and eliminated duplication. While non-validity increased for one core variable, the number of invalid records only increased by one and the percent of records invalid both before and after is less than one percent; therefore, we believe that while there may be some human error in the system, the level of variable non-validity is well within acceptable levels. However, review of the data entry, processing, and validation system should be reviewed as a precaution to ensure data validity is maximised across all variables.
The increase in data fields after expansion of the data collection form facilitated the capture of more comprehensive and varied data, leading to greater insights and a richer understanding of the vaccination landscape in the African region. The new system yielded data that was not only higher in quality but also much more detailed and comprehensive, allowing WHO AFRO to track operational progress more effectively and identify gaps more accurately.
The dashboards, data packs and bulletins produced for the African Region to monitor progress on COVID-19 vaccination have enabled better tracking of vaccination progress and dissemination of more information, which support the ability to make data-driven decisions.3 8–10 In the context of other dashboards available on COVID-19 vaccination,9 12–14 efforts by WHO AFRO contribute to the larger system of information products informing vaccine roll-out.
While this study provides insight on the process of information system improvements and the resulting impact on the data, it also has several limitations. First, the accuracy of the data is not assessed here as this depends on member states collecting and reporting accurate data, and this cannot be easily addressed at the WHO AFRO level and may depend on having a second data source for comparison. Additionally, our analysis was unable to measure the qualitative impacts of system improvements such as changes in end-user experience or labour time required from WHO AFRO staff, particularly as the number of variables required was increased. While not captured here, impacts on operational efficiency and user satisfaction are also important objectives. Finally, our study lacks a comprehensive measure of the audience for the information products generated. Future processes developed to capture those statistics would provide a better understanding of the reach of those products.
Overall, the information system strengthening described here can serve as a model for monitoring vaccination roll-out and improving data quality and use at a regional level, including other WHO regions and those working in similar contexts. As has been documented previously, ensuring strong health information systems is critical for public health programmes to enable accurate decision-making15–17 and improving underlying health information systems can strengthen data quality and usage.18 This work highlights the value of robust data collection and management systems. It also emphasises the importance of data monitoring in achieving global coverage targets, contributing to global datasets and guiding resource allocation including vaccination campaign implementation. Continued investments in such information system strengthening efforts are essential for public health interventions as well as to ensure effective and equitable vaccine distribution and coverage. Future efforts should consider the lessons learnt from such system improvements to enhance data collection, management, utilisation and information products that are critical to decision-makers. As continued health information system strengthening efforts are made both within WHO AFRO and in other organisations, studies like this that document the impact of those efforts will help provide quantifiable justification for investment in public health data.
Data availability statement
Data may be obtained from a third party and are not publicly available. Data are not publicly available.
Ethics statements
Patient consent for publication
Ethics approval
This activity was reviewed by CDC and was conducted consistent with applicable federal law and CDC policy (45 C.F.R. part 46.102(l)(2), 21 C.F.R. part 56; 42 U.S.C. 145 Sect. 241(d); 5 U.S.C. Sect. 552a; 44 U.S.C. Sect. 3501 et seq).
Acknowledgments
The authors are thankful for the contributions of the WHO African Regional Office for their support of this work.
Footnotes
Handling editor Seye Abimbola
Contributors TS led the data analysis and manuscript writing. All authors contributed to manuscript review and finalisation. BI, FM and EL oversaw the conception of the project, the execution of the project and the development of the manuscript. TS, AB, AOA, DRO, RS, KV, LK, NS, BF and MKW executed the project.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Disclaimer The author is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.
The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the US Centers for Disease Control and Prevention.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.