Article Text
PDF
Abstract
Nigeria began administering COVID-19 vaccines on 5 March 2021 and is working towards the WHO’s African regional goal to fully vaccinate 70% of their eligible population by December 2022. Nigeria’s COVID-19 vaccination information system includes a surveillance system for COVID-19 adverse events following immunisation (AEFI), but as of April 2021, AEFI data were being collected and managed by multiple groups and lacked routine analysis and use for action. To fill this gap in COVID-19 vaccine safety monitoring, between April 2021 and June 2022, the US Centers for Disease Control and Prevention, in collaboration with other implementing partners led by the Institute of Human Virology Nigeria, supported the Government of Nigeria to triangulate existing COVID-19 AEFI data. This paper describes the process of implementing published draft guidelines for data triangulation for COVID-19 AEFI data in Nigeria. Here, we focus on the process of implementing data triangulation rather than analysing the results and impacts of triangulation. Work began by mapping the flow of COVID-19 AEFI data, engaging stakeholders and building a data management system to intake and store all shared data. These datasets were used to create an online dashboard with key indicators selected based on existing WHO guidelines and national guidance. The dashboard went through an iterative review before dissemination to stakeholders. This case study highlights a successful example of implementing data triangulation for rapid use of AEFI data for decision-making and emphasises the importance of stakeholder engagement and strong data governance structures to make data triangulation successful.
- COVID-19
- Vaccines
- Health systems
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary box
Strong vaccine safety surveillance systems that produce timely, high-quality data are critical for effective rollout of COVID-19 vaccines, but adverse events following immunisation (AEFI) COVID-19 vaccine data in many low-and-middle-income countries are collected non-systematically across space and time, by many non-coordinated groups, and sometimes using outdated tools. In 2020, the WHO, UNICEF and the US Centers for Disease Control collaborated to develop draft guidance on triangulation for improved decision-making in immunisation programmes, with support from Gavi, the Vaccine Alliance, as a method for immunisation programmes to analyse multiple data sources.
This case study describing data triangulation in Nigeria for COVID-19 vaccine safety data documents a country’s experience implementing data triangulation, including the challenges and successes, and shows how data triangulation was used to rapidly analyse existing COVID-19 vaccine AEFI data sources.
Lessons learnt here can be applied to country preparedness, stakeholder engagement, data mapping and analyses, and human resource capacity building for strategic information system building.
Introduction
The COVID-19 pandemic has had a major impact on Nigeria, with more than 266 000 documented confirmed cases and 3100 deaths as of October 2022.1 Nigeria began administering COVID-19 vaccines on 5 March 2021 and over the first year administered 18.4 million doses to their 200-million-person population.2 Nigeria is now working towards the WHO African Regional goal to fully vaccinate (provide two shots of a two-dose vaccine, or one shot of a one-dose vaccine) 70% of their eligible population.3 As of October 2022, they have reached 21% coverage of their total population.2
Adverse events following immunisation (AEFI) passive surveillance to assess COVID-19 vaccine safety has been identified as a critical component of COVID-19 vaccine information systems.4 Tracking AEFIs for novel vaccines is important to detect safety signals, as these novel vaccines are administered with WHO Emergency Use Listing.4 Careful surveillance of vaccine safety can increase public trust in vaccinations, improving uptake.4
Because existing routine immunisation tools in Nigeria were determined insufficient for COVID-19 vaccination data tracking, the country rapidly built a separate COVID-19 vaccine information system with responsibility for maintaining components of the information system allocated across a variety of agencies. As a part of the COVID-19 vaccine information system, COVID-19 AEFI passive surveillance data were being collected and managed by multiple groups, using a mix of paper, Microsoft Excel and digital systems, with inconsistent data sharing practices and lack of routine analysis. Briefly, Nigeria’s vaccination system structure is decentralised across 36 states and 774 local government areas (LGAs) and the federal government provides partial funding, vaccines, immunisation guidelines and technical support to lower administrative levels.5
To fill gaps in AEFI surveillance, In April 2021, the US Centers for Disease Control and Prevention (US-CDC) proposed implementing data triangulation of existing COVID-19 AEFI passive surveillance data in Nigeria. Data triangulation is a method that intakes and compares multiple data sources to address questions for programme planning and decision-making.6 It has been used in multiple settings7–9 to better understand overall trends, to cross-check data across systems and to improve data sharing. This article describes the process—including implementation approach, outcomes, challenges, achievements and lessons learnt—of triangulating Nigeria’s COVID-19 AEFI data for improved decision-making regarding COVID-19 vaccine safety. We focus here on key steps to implement data triangulation rather than analysing its results and impacts, which will be the focus of a follow-up paper.
Approach to implementation
Partnerships
In 2021, the Institute of Human Virology Nigeria (IHVN), with funding and technical support from the US-CDC, implemented data triangulation of COVID-19 AEFI data on a live, web-based dashboard. IHVN provided leadership on the implementation of data triangulation, centralised AEFI data management and the dashboard creation and management process. IHVN created a subcontract with the University of Maryland, Baltimore (UMB) for the backend development of the dashboard. All activities were conducted in consultation with the National Primary Health Care Development Agency (NPHCDA) with further technical support by the African Field Epidemiology Network (AFENET). Other technical partners included the National Agency for Food and Drug Administration and Control (NAFDAC), the WHO Nigeria country office, the Health Information System Programme of Nigeria, a global organisation that supports District Health Information 2 (DHIS2) implementation and Sydani, a Bill and Melinda Gates Foundation funded partner.
Data triangulation
Methods for data triangulation implementation followed established steps in the WHO, UNICEF and US-CDC publication, ‘triangulation for improved decision-making in immunisation programmes’.6 The approach taken is described below.
System characterisation
All COVID-19 AEFI data collection sources were identified via engagement of relevant agencies and stakeholders. Based on desk review of existing and shared documentation, in-person site visits and interviews with stakeholders, the flow of COVID-19 AEFI data through all existing systems, from data collection to reporting, was described and mapped. Key issues, system gaps and potential areas for improved data access and sharing were documented.
Data management
Data sharing and management processes were established for identified data sources that agreed to participate in triangulation. Data sharing agreements and Memoranda of Understanding (MOUs) for data use were developed between relevant stakeholders and IHVN/UMB as necessary. Data shared with IHVN and UMB were stored in a central database. Data security measures for the central database were developed and implemented. Data dictionaries were created by IHVN and UMB for shared datasets. All processes were compiled into a standard operating procedures (SOPs) document for reference.
Visualisation and communication of results and use
Based on available data, IHVN adapted key indicators and visualisations for AEFI surveillance from established indicators in WHO and Nigeria’s national AEFI surveillance guidelines.4 10 These were visualised onto a web-based dashboard written in C# (Microsoft Corporation, 2013). The dashboard underwent external review by national level stakeholders in February 2022, and a second review was conducted in July 2022. The beta version of the dashboard was deployed for use by the national COVID-19 vaccination Thematic Working Group in July 2022 with wider dissemination to healthcare workers and subnational pharmacovigilance and disease surveillance and notification officers (DSNOs) planned for October–December 2022.
Capacity building
As part of the data triangulation process, the Growing E-health Expertise Knowledge and Skill (GEEKS) programme11 was leveraged to develop informatics workforce capacity for agencies involved with COVID-19 AEFI surveillance. GEEKS is an applied apprenticeship programme supported by US-CDC and AFENET, where fellows across agencies work as a team on a competency-based, project-based learning approach to develop informatics capacity.11 A team in the GEEKS Nigeria programme was formed with representatives from across the agencies supporting COVID-19 vaccine AEFI data triangulation with the implementation of data triangulation as their project.
Outcomes
Data triangulation
System characterisation
Overall oversight and leadership of implementation of COVID-19 AEFI surveillance, including direction of the data collection system, training of staff and supportive supervision, was divided among implementing partner organisations, IHVN, AFENET and Sydani with the NPHCDA providing government guidance. COVID-19 AEFI data were mapped to four sources described below.
A line list data system collected at the health facility level and managed by DSNOs, LGA-level administrators who oversee health surveillance data in their district. This system is paper based and Excel based; healthcare workers at facilities complete a client-level paper AEFI reporting form using the standard WHO AEFI reporting variables for each suspected case, detailing patient demographics, vaccination information and symptoms. These forms are collected in person by LGA staff and are given to the DSNO to transcribe weekly into a Microsoft Excel spreadsheet, then emailed to the state level. LGA data are combined by state, and then shared with the national level. While all states share data in person, the DSNOs for the five states with COVID-19 vaccine AEFI activities supported by IHVN (Lagos, Rivers, Federal Capital Territory, Oyo and Kano) also enter these data weekly into REDCap V.12.2.6 (Research Electronic Data Capture), an electronic data capture software.
A call-in data system collected by healthcare workers and managed by the NPHCDA in collaboration with WHO Nigeria. This system is the data source for Nigeria’s official reported COVID-19 AEFI statistics and is mandated country-wide. This system is paper based and Excel based; healthcare workers at all healthcare facilities who administer COVID-19 vaccines complete a health facility aggregate paper form tallying daily AEFI reports characterised by sex, vaccine brand, dose number received and priority group (healthcare workers and older adults). These tallies are reported in person or by phone daily to the ward focal person and submitted to the LGA level where they are aggregated into a Microsoft Excel spreadsheet and emailed and aggregated at the state and national levels.
A DHIS2 platform with data collected by healthcare workers and managed by the Federal Ministry of Health. DHIS2 is used country-wide in Nigeria as a client-level digital system to collect COVID-19 vaccination data. An AEFI module was planned as part of the DHIS2 platform. However, development and rollout were stalled, and at the time of triangulation initiation, no AEFI data were being collected through this system. Over the course of this project, development restarted and rollout of the DHIS2 COVID-19 AEFI system began in April 2022 in health facilities in two pilot states, Akwa Ibom and Ekiti, with nationwide rollout ongoing during April–December 2022.
A Med Safety phone app system with data collected by individuals receiving vaccine and managed by NAFDAC. Med Safety is a globally used app that has been applied in Nigeria for individuals to enter their own data if they experience an AEFI after COVID-19 vaccination. This app was developed for reporting the harmful effects of a variety of health products but is used in Nigeria just for COVID-19 vaccine AEFI reporting. There was some education about Med Safety to patients in select healthcare facilities, but not in any consistent or systematic way. All data collected via Med Safety is reported to WHO’s VigiBase, WHO’s reporting platform within the Uppsala Monitoring Center. At the national level, reports are managed by NAFDAC but are not routinely used for decision-making at the time of writing.
These sources are summarised in table 1.
Summary of adverse events following immunisation (AEFI) data sources mapped in Nigeria as a first step of data triangulation
Data management
After characterising all COVID-19 vaccine AEFI data sources in Nigeria, the next step was to set up data sharing and management systems for those available for the dashboard. Initially, call-in data were available nationally and line list data were available for the five states where AEFI surveillance is supported by IHVN (Lagos, Rivers, Federal Capital Territory, Oyo and Kano). Over the timeline of the project, the DHIS2 AEFI module was completed, and data are currently available in the two states where the module is rolled out (Akwa Ibom and Ekiti). Med Safety data are not available, with no immediate plans to make them available based on continued inability to reach a data sharing agreement. The flow of data into the dashboard is visualised in figure 1.
Data flow for COVID-19 adverse events following immunisation (AEFI) data from AEFI event to the COVID-19 AEFI data triangulation dashboard. At time of publication, line list data, call-in data and District Health Information 2 (DHIS2) data are incorporated into the dashboard, while Med Safety app data are not.
Data sharing agreements were implemented to incorporate national call-in data and line list data in the five states where IHVN provides AEFI surveillance support (Lagos, Rivers, Federal Capital Territory, Oyo and Kano). LGA-level call-in data were made accessible to IHVN and UMB in real time through a secure shared Dropbox, and facility-level line list data for the five states with agreements were made accessible weekly to IHVN through REDCap. Although access to data from DHIS2 was delayed due to the need to finalise customisation, training and rollout of the module, in April 2022, the DHIS2 AEFI module was piloted in two states and configured into the data triangulation dashboard. Because DHIS2 is a national system, no MOU was needed. Data are pulled directly from the DHIS2 database in real time into the dashboard for the two states (Akwa Ibom and Ekiti) where the module is currently rolled out.
NAFDAC approved an MOU with IHVN to initiate sharing of the Med Safety call-in data. However, Med Safety data are not currently included in the triangulation process because agreements are still in development as of October 2022.
In collaboration with WHO Nigeria and NPHCDA, IHVN and UMB developed a data dictionary to define all variables in all available databases. All processes for data sharing, pulling data into the central database, data cleaning and data storage were documented in an SOP developed and maintained by IHVN. Data security measures, including encryption of data transfers and uploaded files from web browsers to the dashboard, using transport layer security and encryption of the operating system drive and forced redirection of all HTTP requests to HTTPS protocol, were implemented to ensure privacy and protection of all incoming data.
Visualisation and communication of results and current use
Key indicators and visualisations were drafted by IHVN and developed into a dashboard an interactive web-based dashboard by UMB, including counts of serious and non-serious AEFIs and rate of serious and non-serious AEFI per 100 000 vaccinations, both overall and disaggregated by week, state, LGA, vaccine brand, outcome, sex, age group and dose number, and distribution of symptoms of reported AEFIs. (figure 2). The dashboard underwent iterative review beginning with internal stakeholders from UMB, IHVN and CDC and expanding to a day-long dashboard review workshop in February 2022 with external stakeholders including NPHCDA, NAFDAC and WHO to assess choice of indicators, visualisations, layout, formatting and accessibility. A second external stakeholder review workshop was conducted in July 2022. These reviews were used to inform updates to data sources, indicators and visualisations on the dashboard. The dashboard is deployed to the same stakeholders who have participated in the review meetings via a password-protected login. While the dashboard is housed and maintained by UMB, it is currently used by the NPHCDA AEFI national technical working group to guide policy and decision-making.
Screenshots from the COVID-19 adverse events following immunisation (AEFI) interactive dashboard. (A) The opening page of the dashboard, which begins on The Report Summary page showing key summary statistics for COVID-19 AEFI surveillance. Summary data can be toggled between data sources using the Data Sources button. More in-depth statistics for each data source can be viewed using the page selector on the left-hand bar. (B) Geographical representation of AEFI reports per 100 000 persons vaccinated from the call-in data page, shown as a map. (C) Ten most common AEFI reactions disaggregated by sex, calculated from the line list data. (D) Proportion of AEFIs disaggregated by age group and sex, calculated from the line list data.
Capacity building
The GEEKS team for COVID-19 AEFI data triangulation was formed in January 2022 with six mentees selected from NAFDAC and the NPHCDA Department of Planning Research and Statistics and four mentors from IHVN, UMB and AFENET. The team received an orientation and introductory training covering topics of foundational informatics, problem-solving, critical thinking skills and evaluation. This training also included introduction to the DHIS2 AEFI module and Med Safety data sources. In February 2022, the team received in-person trainings conducted by US-CDC on data triangulation using the draft WHO, UNICEF and CDC global data triangulation guidance. The capacity of the team is continuously developed through 1:1 on-the-job training and mentorship sessions. Mentors and mentees meet bi-weekly, either virtually or in-person. All work by the GEEKS team supports the data triangulation effort.
Summary of key lessons learned
During April 2021–June 2022, partners used data triangulation to aggregate COVID-19 AEFI data from multiple streams of existing COVID-19 AEFI data in Nigeria. This culminated in a live, web-based dashboard accessible to key stakeholders. This work has rapidly expanded analyses of COVID-19 AEFI data in Nigeria for all states, which is currently being used by the AEFI national technical working group at NPHCDA as a warning system for emerging issues with COVID-19 vaccine safety.12
Successes
Data triangulation of COVID-19 AEFI data led to several critical successes. First, characterising the COVID-19 vaccine AEFI surveillance system revealed and led to filling documentation gaps. For example, data dictionaries and SOPs for data intake are recognised as best practices to for high-quality data13 and were developed as part of this project. The characterisation process also highlighted the need for integrating COVID-19 AEFI surveillance with the COVID-19 vaccination system in DHIS2;12 subsequently, NPHCDA completed development and initiated scale-up of the COVID-19 AEFI DHIS2 module. Importantly, the process of data triangulation prompted GEEKS, which promotes intergroup collaboration and capacity building. Lastly, this work expanded analysis of AEFI data. The dashboard is currently Nigeria’s only tool analysing all available COVID-19 vaccine safety data and is used by the NPHCDA AEFI national technical working group for decision-making.
Challenges/limitations
Despite the successes, several significant barriers were encountered. Availability of data sources was initially more limited than expected; this project planned to incorporate call-in data, Med Safety data and DHIS2 data, but at the time the project began, only call-in data from all states and line list data from five states (Lagos, Rivers, FCT, Oyo and Kano) were available for inclusion. This highlights the value of the initial system characterisation step taken to establish willingness of key stakeholder groups to engage in the data triangulation process, particularly in sharing data, and emphasises the value of strong governance structures for data systems.14 Second, it was challenging to implement an analysis process for COVID-19 AEFI data while vaccine rollout was already underway. Stakeholder resources were often stretched, and available time to work on AEFI surveillance was competing with other immediate COVID-19 vaccine rollout-related issues; an established analytic plan for AEFI data prior to vaccine rollout would have been useful. Finally, implementation of COVID-19 AEFI surveillance was supported by partner organisations with varying levels of funding, human resources, and technical capacity, resulting in challenges coordinating and agreeing on standardised procedures. When responsibilities for information systems are divided among groups, extra care is necessary to ensure resource distribution commensurate with each group’s responsibilities and needs.
Conclusion
Lessons learnt here can be used by other low-and-middle-income countries encountering issues with COVID-19 vaccine safety data and information systems. First, data triangulation requires cooperation across agencies who own the data, and the success was contingent on their buy-in. Continuous and interactive stakeholder engagements, including development of data sharing agreements, should be established early in the process of triangulation. Second, challenges analysing COVID-19 AEFI data post-vaccine rollout may have been avoided if information system issues were addressed earlier on. Implementation of health information systems cannot be an afterthought to public health action.15 Next, there were concurrent information and surveillance system developments that resulted from the data triangulation process, including the rollout of the DHIS2 AEFI module. Other groups implementing similar approaches should consider additional deliverables from triangulation, such as improved data collection systems or improved standardisation of system documentation. Lastly, while the limited number of data sources and buy-in from stakeholders was an initial challenge, we were able to start the data triangulation process by leveraging existing data sources, human resources and infrastructure to establish a working dashboard that acted as proof of concept. This then helped justify to a greater number of stakeholders the value of providing additional data, which then led to an expanded tool. Demonstrating value of data triangulation to stakeholders can be a powerful way to increase buy-in, data sharing and participation.
In conclusion, data triangulation and COVID-19 AEFI dashboard development in Nigeria has rapidly established expanded analysis of COVID-19 AEFI data for decision-making.12 This in promotes vaccine safety by monitoring adverse events post vaccination, and may also improve uptake as concerns regarding AEFIs can drive vaccine hesitancy.16 17 Data triangulation can be a powerful tool for other countries to quickly bring together existing data to support decision-making and programme performance.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
Ethics statements
Patient consent for publication
Ethics approval
This activity was reviewed by CDC and was conducted consistent with applicable federal law and CDC policy (45 C.F.R. part 46.102(l)(2), 21 C.F.R. part 56; 42 U.S.C. 145 Sect. 241(d); 5 U.S.C. Sect. 552a; 44 U.S.C. Sect. 3501 et seq). This project was additionally reviewed and approved by the Nigeria Ministry of Health AEFI Technical Working Group.
Acknowledgments
The authors are thankful for the contributions of the Nigerian National adverse events following immunisation technical working group and the Nigeria US Centers for Disease Control and Prevention Country Office Team for their support of this work.
Footnotes
Handling editor Seye Abimbola
Contributors TS led the analysis and interpretation of data for the work, the drafting of the work and the final approval of the published version. OJA contributed to the conception and design of the work and the drafting and revision of the content. HI, EL and OB contributed to the conception and design of the work and the revision of content. TS, AH, WI, OR, KO, AA, MG, EI, AIA, AR and BO contributed to the design of the work and the acquisition of data for the work and revised it critically for content. JG revised the work for content.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Map disclaimer The inclusion of any map (including the depiction of any boundaries therein), or of any geographic or locational reference, does not imply the expression of any opinion whatsoever on the part of BMJ concerning the legal status of any country, territory, jurisdiction or area or of its authorities. Any such expression remains solely that of the relevant source and is not endorsed by BMJ. Maps are provided without any warranty of any kind, either express or implied.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.