Use of routinely collected electronic healthcare data for postlicensure vaccine safety signal detection: a systematic review
============================================================================================================================

* Yonatan Moges Mesfin
* Allen Cheng
* Jock Lawrie
* Jim Buttery

## Abstract

**Background** Concerns regarding adverse events following vaccination (AEFIs) are a key challenge for public confidence in vaccination. Robust postlicensure vaccine safety monitoring remains critical to detect adverse events, including those not identified in prelicensure studies, and to ensure public safety and public confidence in vaccination. We summarise the literature examined AEFI signal detection using electronic healthcare data, regarding data sources, methodological approach and statistical analysis techniques used.

**Methods** We performed a systematic review using the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines. Five databases (PubMed/Medline, EMBASE, CINAHL, the Cochrane Library and Web of Science) were searched for studies on AEFIs monitoring published up to 25 September 2017. Studies were appraised for methodological quality, and results were synthesised narratively.

**Result** We included 47 articles describing AEFI signal detection using electronic healthcare data. All studies involved linked diagnostic healthcare data, from the emergency department, inpatient and outpatient setting and immunisation records. Statistical analysis methodologies used included non-sequential analysis in 33 studies, group sequential analysis in two studies and 12 studies used continuous sequential analysis. Partially elapsed risk window and data accrual lags were the most cited barriers to monitor AEFIs in near real-time.

**Conclusion** Routinely collected electronic healthcare data are increasingly used to detect AEFI signals in near real-time. Further research is required to check the utility of non-coded complaints and encounters, such as telephone medical helpline calls, to enhance AEFI signal detection.

**Trial registration number** CRD42017072741

*   electronic healthcare records
*   post-licensure safety surveillance
*   adverse event following immunization
*   systematic review

### Key questions

#### What is already known?

*   Adverse event(s) following immunisation (AEFI) signal detection has primarily relied on passive surveillance reporting.

#### What are the new findings?

*   AEFIs signal monitoring using population-based electronic health records (EHRs) is increasing, but has been primarily limited to diagnostic data from hospital settings.

*   Continuous sequential (rapid cycle) analysis method allows AEFIs signal monitoring in near real-time.

*   Data delays (data accrual lags) are the key challenges to perform near real-time AEFI monitoring using EHRs.

#### What do the new findings imply?

*   A complementary and efficient AEFI signal monitoring system is feasible using EHRs.

*   Further research is required to evaluate the utility of syndromic data/proxy measures to enhance the timeliness of monitoring AEFIs.

## Introduction

Vaccination is one of the most effective public health interventions. Current immunisation programmes provide protection against up to 26 diseases and prevent an estimated 2–3 million deaths every year.1 2 It is estimated that 1.5 million more deaths could be saved through further increasing vaccination coverage of existing vaccines.3 However, this remarkable success has been challenged due to vaccine safety concerns and increasing vaccine hesitancy, largely due to fear of adverse event following immunisation (AEFIs). Notably, following the sharp reduction of incidence of vaccine-preventable diseases the public attention to AEFI has increased. This can result in loss of confidence in vaccination, a resultant drop in vaccine coverage and eventually lead to a re-emergence of controlled disease (figure 1).4 Hence, timely detection of potentially causally related adverse events (AEs) and more rapidly refute spurious claims regarding AEs using real-world data is critical to maintain the community and providers confidence in vaccine programmes. Nevertheless, recent analysis of global AEFI reporting found that more than 36% of WHO member countries do not have a functional postlicensure safety monitoring system for vaccines.5

![Figure 1](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001065/F1.medium.gif)

[Figure 1](http://gh.bmj.com/content/4/4/e001065/F1)

Figure 1 
Potential stage in the evolution of an immunisation programme, vaccine safety. Diagram adapted from Chen *et al*. The Vaccine Adverse Effect Reporting System (VAERS). *Vaccine* 1994:12(6):542–50.

Postlicensure AEFIs monitoring is often classified into three stages: signal detection, signal refinement and signal confirmation. A vaccine safety signal is defined as *‘*reported information on a possible causal relationship between an adverse event and a vaccine, the relationship being unknown or incompletely documented previously’.6 Generally, AEFI signal detection has been undertaken using passive surveillance or active surveillance system. Passive surveillance systems, the prevailing AEFI monitoring system, monitor reports of AEs that are spontaneously submitted by healthcare providers, vaccinated individuals/their caregivers or others. Its wide population coverage allows for detection of new and unanticipated AEs but has limitations of under-reporting and imprecise risk estimates due to lack of appropriate denominator data.7 According to the 2015 Global Vaccine Safety Initiative meeting report, low passive AEFI reporting rates are a significant barrier to detect vaccine safety signal timely.8 In contrast, active surveillance of AEFI involves proactively seeking information from healthcare providers, vaccinated individuals/their caregivers, or related datasets using well-designed study protocols. These surveillance systems provide more detail, less biased information and appropriate denominators. However, active surveillance systems are resource intensive and takes substantial time to achieve the required sample size to study rare AEs. Hence, their use in many settings are largely limited to investigate signals detected from the passive surveillance systems, literature review or possible prelicensure trial safety questions.7 9 10

Encouragingly, in recent years, new studies have shown that routinely collected electronic health records (EHRs) can be used as an alternative data source to monitor for AEFI signals in near real-time.11 12 For example, in the USA, newly marketed vaccines are monitored for potential AEFIs weekly using the Vaccine Safety Datalink (VSD) collaboration between the US Centre for Disease Control and eight healthcare organisations. In the VSD, patient encounters and diagnoses made in an emergency department, outpatient clinic and hospital are linked with previous vaccine via patient-specific study identification numbers. Though the regular use of VSD is to investigate known AEFI signals identified from passive surveillance, published studies also show that VSD and other EHR detection systems are suitable for rapid detection of AEFIs signals.13–15

Considering the increasing availability of EHRs and the necessity of further improving the capacity of vaccine safety monitoring, particularly in low-income and middle-income countries, EHRs can offer an alternative data source to establish complementary active AEFI surveillance systems. By systematically summarising these literature, we intend to provide valuable information for countries considering establishing AEFI signal detection system based on EHRs. Therefore, we aimed to: (1) describe the features of postlicensure vaccine safety studies employing EHRs primarily for safety signal detection and (2) catalogue the nature of data sources, methodological approaches and analysis techniques applied

## Methods

### Search strategy

A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines,16 as provided in online supplementary file 1. The protocol was registered at the international prospective register of systematic reviews (registration number CRD42017072741). We searched OVID Medline (1946 to September week 3 2017), OVID Embase (1974 to 2017 September 10), the Cochrane Library, Scopus and Web of Science. Comprehensive search terms for all databases were developed in consultation with a medical librarian to identify all potentially relevant studies. A combination of keywords and Medical Subject Headings (MeSH) were used in each database with appropriate adjustment. Final searches were performed on 25 September 2017. An example of the search strategy used in Ovid MEDLINE is shown in online supplementary file 1. In addition, bibliographies of relevant studies, conference papers/proceedings and grey literature databases, such as who.int and greylit.org, were searched to identify further important and unpublished studies.

### Supplementary data

[[bmjgh-2018-001065supp001.pdf]](pending:yes)

### Studies selection criteria and screening

We included studies primarily focussing on AEFI signal detection using EHRs. Studies were included regardless of vaccine type, population group studied, study setting and methodology used. However, studies based on passive pharmacovigilance data or administrative (claim) data; studies conducted solely to test or verify the previously identified signals and feasibility studies or studies conducted to evaluate methodologies were excluded from the review. We also excluded non-English records and conference abstracts.

Search results were downloaded and managed in EndNote X8. Articles were screened in three stages (titles alone, abstracts and then full-text review) based on the PRISMA flow of information (figure 2). At the initial stage, titles and abstracts were screened to remove duplicate records and studies clearly outside the scope of the review. Then, two reviewers conducted a full-text review to assess the eligibility based on the inclusions criteria. Study screening stages and the reasons for articles exclusion during full-text review are described in figure 2.

![Figure 2](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001065/F2.medium.gif)

[Figure 2](http://gh.bmj.com/content/4/4/e001065/F2)

Figure 2 
Flow diagram shows stages of study selection and screening. Articles may have been excluded for more than one reasons.

### Quality assessment and data extraction

We used a checklist adapted from the Food and Drug Authority (Best Practices for Conducting and Reporting Pharmaco-epidemiologic Safety Studies Using EHR).17 Many of the critical appraisal tools extensively used to appraise observational studies, such as Ottawa-Newcastle tool and strengthening the reporting of observational studies in epidemiology (STROBE), are not suitable for evaluating pharmaco-epidemiological studies and public health surveillance as they are reasonably different from the standard epidemiological studies. The lead author (YMM) assessed risk of bias of all the included studies, and the second independent reviewer (TK) evaluated 25% of the studies randomly for verification. As there was no substantial risk of bias identified, we considered all appraised studies for the final review. The methodological quality and risk of bias assessment criteria were:

*   Well defined research questions.

*   Sample representativeness.

*   Clear inclusion and exclusion criteria.

*   Appropriateness of study design and comparison groups.

*   Follow-up (risk interval) long enough for the events to occur.

*   Appropriateness of data integration method, when relevant.

*   Adjustment of confounders.

*   Employed appropriate statistical analyses method.

*   Used objective criteria to measure outcomes.

The lead author consistently extracted the required data using pretested data abstraction template. The following information were extracted across the included studies:

*   Study author.

*   Publication year.

*   Study setting and period.

*   Data source(s) and nature of the data (diagnostic vs prediagnostic).

*   Study design(s) employed.

*   Studied population.

*   Vaccine(s) and AE(s) studied.

*   Statistical analysis approaches and signal detection method used.

*   Frequency of assessment.

*   Method(s) of controlling confounders reported and challenges reported.

*   Main findings (signal (s) identified or not).

### Data analysis

Key features of the studies are described quantitatively. Results from the selected studies are synthesised in a narrative analysis. The structure of the detailed review includes: vaccines monitored; AEs studied; study design(s) used; data analysis approach and signal detection method employed.

### Patient and public involvement statement

No patient data were consided in this study.

## Result

### Studies identified and characteristics

After removal of duplicate articles, we screened the titles and abstracts of 606 articles and excluded articles clearly out of the scope of this review. Then, we screened the remaining 235 full-text articles according to the exclusion criteria (figure 2). Studies could be excluded for more than one reason. Forty-seven articles, conducted between 2002 and 2017, were included in the final synthesis.18–64 No studies were excluded based on quality or bias.

Almost all studies included in this review were conducted in the USA (n=45).18–25 27–33 35–65 Two additional studies were conducted in the UK26 and Taiwan.34 A considerable number of studies (n=13, 28%) assessed the safety of vaccines administered to high-risk groups (pregnant women or elderly subjects). Fourteen (30%) studies assessed the AEFIs in near real-time (table 1).

View this table:
[Table 1](http://gh.bmj.com/content/4/4/e001065/T1)

Table 1 
Summary characteristics of selected studies

### Vaccines studied

Multiple types of vaccines, including live, inactivated, monovalent and combined, were monitored after licensure for potential AEFI. Seasonal influenza vaccines (trivalent inactivated influenza vaccines (TIIV), live attenuated influenza vaccines, monovalent influenza vaccines and live attenuated monovalent influenza vaccines) were most frequently studied (n=17), followed by combined diphtheria-tetanus toxoid-acellular pertussis (Tdap) vaccines (n=5)(figure 3).

![Figure 3](http://gh.bmj.com/https://gh.bmj.com/content/bmjgh/4/4/e001065/F3.medium.gif)

[Figure 3](http://gh.bmj.com/content/4/4/e001065/F3)

Figure 3 
Type of vaccines studied by the selected studies.

### AEFIs studied and data source

Most of the reviewed studies (n=35) studied preidentified AEs using a fixed postvaccination risk interval. AEs were selected based on the safety concerns from passive surveillance reports and prelicensure clinical trials. Frequently studied AEs were Guillain-Barré syndrome, febrile convulsions, seizures, anaphylaxis, meningitis/encephalitis and local reactions. Potential maternal and infant outcome (AEFIs), such as pre-eclampsia/eclampsia, maternal death, small for gestational age, preterm birth, stillbirth and neonatal death were also evaluated. Studied AEFIs were mainly identified using International Classification of Diseases (ICD) Clinical Modification codes as well as relevant ICD-9 or ICD-10 codes from electronic records (outpatient, inpatient and emergency department settings). In some studies, patients’ charts/medical records were manually reviewed to verify the AEs.

In this review, 14 statistically elevated vaccine-AE pairs (signals) were detected, and 6 were confirmed. These were measles, mumps, rubella and varicella vaccine and seizure/febrile convulsion,38 43 2010–2011 TIIV and febrile seizure,57 monovalent rotavirus vaccine and intussusception,61 2014–2015 TIIV and febrile seizures48 and Tdap vaccine and chorioamnionitis.41

### Study designs employed

Self-controlled design was the most frequently used study design (n=22),18–21 25 27 28 30–34 36 38 39 44 46–48 53 57–59 62 63 followed by cohort design with historical comparison (also called observed vs expected analysis) (n=20).18 22–26 29 34 38 39 43 45 47–49 57 60 61 63 64 Self-controlled design can be self-controlled risk interval (SCRI) or self-controlled case series (SCCS). Cohort design with concurrent/parallel comparison group,19 20 29 40–42 50–52 mostly to examine vaccines administered to pregnant women, and case-crossover study designs were also employed.28 32 Of note, 18 studies (38.3%) employed more than one study design; of these, SCRI and current versus historical designs were often used together.25 34 38 39 47 48 57 63

### Statistical analysis and signal detection method

Two broad data analytic approaches, non-sequential analysis and sequential analysis, were employed to identify elevated risk of AEs associated with a given vaccine. In studies that employ a non-sequential analysis approach (n=33), statistical tests are performed after all the data are collected/accumulated. Detailed description of these studies and their analytic approaches are provided in online supplementary file 2. The sequential analysis approach allows repeated examination of data to check for AEFI increased occurrence. This was implemented in two different ways in the included studies: (i) as group sequential analysis (n=2), which involved a periodic statistical test and limited number of statistical tests over time and (ii) as continuous sequential analyses (n=12), also called ‘rapid cycle analysis’, which involved a weekly statistical test until the end of the study period (table 2).

### Supplementary data

[[bmjgh-2018-001065supp002.pdf]](pending:yes)

View this table:
[Table 2](http://gh.bmj.com/content/4/4/e001065/T2)

Table 2 
Included studies implemented near real-time vaccine safety monitoring methods (sequential analysis)

The choice of specific statistical tests was guided by the data analysis approach used. Standard analytic tests, such as logistic and Cox regression, were used to examine the data at the end of the study period (end-of-study analysis). A sequential hypothesis test statistic, the sequential probability ratio test (SPRT), was used to examine data for an elevated risk of AEFI continually over time. In particular, maximised sequential probability ratio test (MaxSPRT) was the most frequently applied sequential hypothesis test statistic.22 24 29 34 39 43 47 48 57 61 62 64 It has different versions: Poisson MaxSPRT, Binomial MaxSPRT and Conditional MaxSPRT (table 2). Further, supplementary analyses were performed to verify the detected signals and instances of elevated risks. These included temporal scan statistics, to evaluate clustering of events after vaccination, and case-centred regression and logistic regression.29 39 43 47–49 60 61 64

### Confounder adjustment and potential challenges

Many different potential confounders were measured including age, gender, chronic conditions, site, seasonality, trend, concomitant vaccines and delay in the arrival of patient data. Generally, studies adjusted confounding variables in three ways: using data restriction, matching and stratification (alone or in combination). Strategies chosen were often design-based and included the following: (i) using a matched control design to adjust baseline confounders and seasonal trends; (ii) using self-controlled design, which automatically addresses time-invariant confounders and (iii) adjusting the expected rate calculated from historical data. Interestingly, during analysis, MaxSPRT inherently allows controlling bias due to repeated tests. In this review, the most cited challenges, particularly in the case of continuous sequential analysis, were uncertainty in estimating background rates, outcome misclassification, partially elapsed risk window and late-arriving data (data accrual lags).

## Discussion

Routinely collected EHRs are increasingly used for the detection of AEFIs signal besides for testing hypothesis based on known signals. Evidence from this review suggests that electronic healthcare data have a significant potential to establish a near real-time AEFI surveillance systems. All the included studies used coded diagnostic medical data to get information about the studied AEs. Further, non-pharmacovigilance studies have also suggested that alternative non-coded medical information, such as telephone triage data and ambulance data, have potential for near real-time syndromic surveillance and rapidly detection of outbreak signal.66 67

A near real-time surveillance systems involves continuous checking (rapid cycle analysis (RCA)) of the EHRs for an elevated occurrence of AEs as the new data are added over the study period. It was first used to evaluate the safety of meningococcal conjugate vaccine using electronic healthcare data from the VSD in the USA,14 though Davis *et**al* established its feasibility by replicating the previously recognised rotavirus-intussusception signal.68 Since then, we identified 12 studies that examined AEFI signal using RCA method.14 22 24 29 39 43 47 48 57 61 62 64 The RCA method has been also used based on an alternative data sources other than EHRs. For example, in the UK, H1N1 vaccine was monitored using passive surveillance data,69 and in Australia seasonal influenza vaccines have been monitored since 2015, based on data collected directly from consumers using SMS-messaging and email (AusVaxSafety).70

The near real-time AEFI surveillance systems use sequential analysis approach, primarily MaxSPRT, to continuously evaluate data for signals while adjusting bias due to multiple testing. MaxSPRT is an improved type of the classical SPRT, which uses a two-sided alternative hypothesis and a predefined relative risk (RR) value usually other than 1. MaxSPRT uses one-sided composite alternative hypothesis by defining the RR usually as >1 to declare statistically significant risk.71 The key advantage of MaxSPRT over the classical SPRT is that it helps to minimise the risk of late detection of AEs due to an incorrect choice of RR and make it suitable for data monitoring more frequently.14 Indications, advantages and weakness of both classical and MaxSPRT, including the three variants of MaxSPRT, are provided in table 3.24 47

View this table:
[Table 3](http://gh.bmj.com/content/4/4/e001065/T3)

Table 3 
Sequential statistical approaches for postlicensure vaccine safety surveillance (description, indication and challenges)

As vaccines are often recommended for all persons in a given age group, traditional epidemiological cohort and case-control designs are usually not suitable to study vaccines AEs after licensure. The main reasons include an inadequate number of comparison groups (unvaccinated individuals), concern regarding comparability of the vaccinated to unvaccinated groups (selection bias), insufficient power and timeliness.72 Rather, self-controlled design (SCRI and SCCS) and cohort design, with a historical comparison, are the preferred design choice in postlicensure vaccine safety studies (table 4). In self-controlled design, comparisons are made with individuals in two different periods, vaccination risk period and control period. The incidence of AEFI is compared between prespecified postvaccination risk period and control period (unexposed period).73 Studies showed that including a prevaccination control period is essential to facilitate timely data analysis for vaccines administered in a short period, mostly in case of seasonal influenza vaccine. However, if there are clinical confounders that are a contraindication for vaccination (eg, allergic reaction) or indications for vaccination (eg, seizure disorder), a prevaccination control period is not recommended.39 47 48 57 74 75

View this table:
[Table 4](http://gh.bmj.com/content/4/4/e001065/T4)

Table 4 
Commonly used study designs in postlicensure vaccine safety monitoring (study population, comparison group, indication, strength and weakness)

A cohort study design with a historical comparison is used frequently for detecting AEFI signals. This design compares the observed incidence of AEFI in the risk period after vaccination of the studied vaccine(s) against the expected incidence of AEFI projected based on the historical data.22 It helps to improve the timeliness of detecting the AEFI signal because only data for the risk window is collected rather than waiting for data for the comparison window.48 However, studies showed that accurate baseline risk estimation is a very challenging task, and it may introduce bias if the historical population are considerably different from the studied population. Nevertheless, this problem can be minimised through simultaneous use of the self-controlled design as they have complementary strengths (table 4).14 48

The essential requirement to conduct a near real-time AEFI surveillance based on EHRs is the availability of timely data. Both data accrual lag and partially elapsed risk window, the risk windows might not be fully elapsed for some AEs at the time of each analysis, can deter performing RCA.74 76 Data accrual lag in EHRs can occur due to several reasons and the level of delay may vary depending on the outcomes studied. A study from UK showed that up to 30 days or more are required to completely record AEFI diagnoses at general practice level.77 Two studies were included in this review,39 48 and methodological evaluation studies suggested that various design-based measures can be taken for adjusting partially elapsed risk window and data accrual lags. These include: (i) calculating the expected counts of AEFIs comparable to the elapsed risk window length; (ii) restricting comparison periods proportional to the elapsed risk period or (iii) AEFIs occurring in later weeks in the risk window can be ignored if the matching weeks in the control period have not elapsed.48 71 78–80

## Conclusion

The utility of routinely collected EHRs for AEFI monitoring globally has been demonstrated, with most published experience drawn from US literature. In addition, the advancement of statistical analysis techniques and RCA provide a significant potential to detect AEFI signal in near real-time.

To date, AEFI monitoring based on EHRs use is limited to diagnostic medical information. Potential incorporation of other electronic health information, including non-coded complaints and encounters, offers further opportunities to improve AEFI real-time surveillance systems to help maintain safe immunisation programmes and maximise confidence in those programmes.

## Acknowledgments

The Australian Government through the International Postgraduate Research Scholarship funds YMM for his PhD study tuition fee. The authors are grateful to Ms Lorena Romero, a Senior Medical Librarian at the Ian Potter Library, Alfred Hospital, Melbourne for providing a feedback on the pilot search strategy for the review. We thank Mr. Kelmu T Kibret for his help in evaluating the methodological quality of studies considered in this review

## Footnotes

*   Handling editor Seye Abimbola

*   Contributors YMM conceived the original research idea with guidance from JB. All authors contributed to the design of the study. YMM searched and screened the studies, extracted the data and wrote the initial drafts of the paper. JB, AC and JL revised the article critically. All authors contributed to and approved the final manuscript.

*   Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests None declared.

*   Patient consent for publication Not required.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data availability statement The authors are happy to provide further data up on request.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  1.Duclos P, Okwo-Bele J-M, Gacic-Dobo M, et al. Global immunization: status, progress, challenges and future. BMC Int Health Hum Rights2009;9 Suppl 1.[doi:10.1186/1472-698X-9-S1-S2](http://dx.doi.org/10.1186/1472-698X-9-S1-S2)
    
    

2.  2.WHO global health days:World Immunization Week, 2017. Available: [http://wwwwhoint/campaigns/immunization-week/2017/en/](http://wwwwhoint/campaigns/immunization-week/2017/en/) [Accessed 19 Jun 2017].
    
    

3.  3.Immunization WHO. Vaccines and biologicals: vaccines and diseases. Available: [http://wwwwhoint/immunization/diseases/en/](http://wwwwhoint/immunization/diseases/en/) [Accessed 19 Jun 2017].
    
    

4.  4.Poland GA, Jacobson RM. The re-emergence of measles in developed countries: time to develop the next-generation Measles Vaccines?Vaccine2012;30:103–4.[doi:10.1016/j.vaccine.2011.11.085](http://dx.doi.org/10.1016/j.vaccine.2011.11.085)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2011.11.085&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22196079&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000299971800001&link_type=ISI) 

5.  5.Lei J, Balakrishnan MR, Gidudu JF, et al. Use of a new global indicator for vaccine safety surveillance and trends in adverse events following immunization reporting 2000–2015. Vaccine2018;36:1577–82.[doi:10.1016/j.vaccine.2018.02.012](http://dx.doi.org/10.1016/j.vaccine.2018.02.012)
    
    

6.  6.Organization WH. Immunization safety surveillance: guidelines for immunization programme managers on surveillance of adverse events following immunization. who regional office for the western Pacific: Manila, 2016.
    
    

7.  7.Griffin MR, Braun MM, Bart KJ. What should an ideal vaccine postlicensure safety system be?Am J Public Health2009;99 Suppl 2:S345–S350.[doi:10.2105/AJPH.2008.143081](http://dx.doi.org/10.2105/AJPH.2008.143081)
    
    

8.  8.O S. Achievements and challenges in global vaccine safety. 25. Uppsala, 2016.
    
    

9.  9.Waldman EA, Luhm KR, Monteiro SAMG, et al. Surveillance of adverse effects following vaccination and safety of immunization programs. Rev Saude Publica2011;45:173–84.[doi:10.1590/s0034-89102011000100020](http://dx.doi.org/10.1590/s0034-89102011000100020)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21181055&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

10. 10.Danova J, Kocourkova A, Celko AM. Active surveillance study of adverse events following immunisation of children in the Czech Republic. BMC Public Health2017;17.[doi:10.1186/s12889-017-4083-4](http://dx.doi.org/10.1186/s12889-017-4083-4)
    
    

11. 11.Coloma PM, Trifirò G, Patadia V, et al. Postmarketing safety surveillance. Drug Saf2013;36:183–97.[doi:10.1007/s40264-013-0018-x](http://dx.doi.org/10.1007/s40264-013-0018-x)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1007/s40264-013-0018-x&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23377696&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

12. 12.Leite A, Andrews NJ, Thomas SL. Near real-time vaccine safety surveillance using electronic health records-a systematic review of the application of statistical methods. Pharmacoepidemiol Drug Saf2016;25:225–37.[doi:10.1002/pds.3966](http://dx.doi.org/10.1002/pds.3966)
    
    

13. 13.Stehr-Green P, Radke S, Kieft C, et al. The risk of simple febrile seizures after immunisation with a new group B meningococcal vaccine, New Zealand. Vaccine2008;26:739–42.[doi:10.1016/j.vaccine.2007.12.001](http://dx.doi.org/10.1016/j.vaccine.2007.12.001)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=18187240&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

14. 14.Lieu TA, Kulldorff M, Davis RL, et al. Real-time vaccine safety surveillance for the early detection of adverse events. Medical Care2007;45(Suppl 2):S89–S95.[doi:10.1097/MLR.0b013e3180616c0a](http://dx.doi.org/10.1097/MLR.0b013e3180616c0a)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/MLR.0b013e3180616c0a&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=17909389&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000250333800016&link_type=ISI) 

15. 15.Bryan P, Seabroke S. No increased risk of febrile convulsions after seasonal influenza immunisation in UK. The Lancet2011;377.[doi:10.1016/S0140-6736(11)60352-8](http://dx.doi.org/10.1016/S0140-6736(11)60352-8)
    
    

16. 16.Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med2009;6:e1000097.[doi:10.1371/journal.pmed.1000097](http://dx.doi.org/10.1371/journal.pmed.1000097)
    
    

17. 17.Safety P. Best practices for conducting and reporting Pharmacoepidemiologic safety studies using electronic healthcare dataCenter for Biologics Evaluation and Research; 2013.
    
    

18. 18.Baxter R, Hansen J, Timbol J, et al. Post-licensure safety surveillance study of routine use of tetanus toxoid, reduced diphtheria toxoid and 5-component acellular pertussis vaccine. Hum Vaccin Immunother2016;12:2742–8.[doi:10.1080/21645515.2016.1201622](http://dx.doi.org/10.1080/21645515.2016.1201622)
    
    

19. 19.Baxter R, Toback SL, Sifakis F, et al. A postmarketing evaluation of the safety of Ann Arbor strain live attenuated influenza vaccine in adults 18–49 years of age. Vaccine2012;30:3053–60.[doi:10.1016/j.vaccine.2012.02.080](http://dx.doi.org/10.1016/j.vaccine.2012.02.080)
    
    

20. 20.Baxter R, Toback SL, Sifakis F, et al. A postmarketing evaluation of the safety of Ann Arbor strain live attenuated influenza vaccine in children 5 through 17 years of age. Vaccine2012;30:2989–98.[doi:10.1016/j.vaccine.2012.02.039](http://dx.doi.org/10.1016/j.vaccine.2012.02.039)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2012.02.039&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22386746&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000303429100012&link_type=ISI) 

21. 21.Baxter R, Tran TN, Hansen J, et al. Safety of Zostavax™—A cohort study in a managed care organization. Vaccine2012;30:6636–41.[doi:10.1016/j.vaccine.2012.08.070](http://dx.doi.org/10.1016/j.vaccine.2012.08.070)
    
    

22. 22.Belongia EA, Irving SA, Shui IM, et al. Real-time surveillance to assess risk of intussusception and other adverse events after pentavalent, bovine-derived rotavirus vaccine. Pediatr Infect Dis J2010;29:1–5.[doi:10.1097/INF.0b013e3181af8605](http://dx.doi.org/10.1097/INF.0b013e3181af8605)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/INF.0b013e3181af8605&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19907356&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000273184100001&link_type=ISI) 

23. 23.Chao C, Klein NP, Velicer CM, et al. Surveillance of autoimmune conditions following routine use of quadrivalent human papillomavirus vaccine. J Intern Med2012;271:193–203.[doi:10.1111/j.1365-2796.2011.02467.x](http://dx.doi.org/10.1111/j.1365-2796.2011.02467.x)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2796.2011.02467.x&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21973261&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

24. 24.Daley MF, Yih WK, Glanz JM, et al. Safety of diphtheria, tetanus, acellular pertussis and inactivated poliovirus (DTaP–IPV) vaccine. Vaccine2014;32:3019–24.[doi:10.1016/j.vaccine.2014.03.063](http://dx.doi.org/10.1016/j.vaccine.2014.03.063)
    
    

25. 25.Davis RL, Black S, Shinefield H, et al. Post-marketing evaluation of the short term safety of COMVAX®. Vaccine2004;22:536–43.[doi:10.1016/j.vaccine.2003.06.001](http://dx.doi.org/10.1016/j.vaccine.2003.06.001)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=14670337&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

26. 26.Donegan K, King B, Bryan P. Safety of pertussis vaccination in pregnant women in UK: observational study. BMJ2014;349. no pagination)(g4219)..[doi:10.1136/bmj.g4219](http://dx.doi.org/10.1136/bmj.g4219)
    
    

27. 27.Duffy J, Lewis M, Harrington T, et al. Live attenuated influenza vaccine use and safety in children and adults with asthma. Ann Allergy Asthma Immunol2017;118:439–44.[doi:10.1016/j.anai.2017.01.030](http://dx.doi.org/10.1016/j.anai.2017.01.030)
    
    

28. 28.France EK, Glanz JM, Xu S, et al. Safety of the trivalent inactivated influenza vaccine among children: a population-based study. Arch Pediatr Adolesc Med2004;158:1031–6.[doi:10.1001/archpedi.158.11.1031](http://dx.doi.org/10.1001/archpedi.158.11.1031)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/archpedi.158.11.1031&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=15520339&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000224837700002&link_type=ISI) 

29. 29.Gee J, Naleway A, Shui I, et al. Monitoring the safety of quadrivalent human papillomavirus vaccine: findings from the vaccine safety Datalink. Vaccine2011;29:8279–84.[doi:10.1016/j.vaccine.2011.08.106](http://dx.doi.org/10.1016/j.vaccine.2011.08.106)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2011.08.106&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21907257&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000296988500016&link_type=ISI) 

30. 30.Glanz JM, et al. Safety of trivalent inactivated influenza vaccine in children aged 24 to 59 months in the vaccine safety Datalink. Arch Pediatr Adolesc Med2011;165:749–55.[doi:10.1001/archpediatrics.2011.112](http://dx.doi.org/10.1001/archpediatrics.2011.112)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/archpediatrics.2011.112&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21810637&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000293342900010&link_type=ISI) 

31. 31.Greene SK, Rett M, Weintraub ES, et al. Risk of confirmed Guillain-Barre syndrome following receipt of monovalent inactivated influenza A (H1N1) and seasonal influenza vaccines in the vaccine safety Datalink project, 2009-2010. Am J Epidemiol2012;175:1100–9.[doi:10.1093/aje/kws195](http://dx.doi.org/10.1093/aje/kws195)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/aje/kws195&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22582210&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000305082300002&link_type=ISI) 

32. 32.Hambidge SJ, Glanz JM, France EK, et al. Safety of trivalent inactivated influenza vaccine in children 6 to 23 months old. JAMA2006;296:1990–7.[doi:10.1001/jama.296.16.1990](http://dx.doi.org/10.1001/jama.296.16.1990)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/jama.296.16.1990&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=17062862&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000241495300021&link_type=ISI) 

33. 33.Hansen J, Timbol J, Lewis N, et al. Safety of DTaP-IPV/Hib vaccine administered routinely to infants and toddlers. Vaccine2016;34:4172–9.[doi:10.1016/j.vaccine.2016.06.062](http://dx.doi.org/10.1016/j.vaccine.2016.06.062)
    
    

34. 34.Huang W-T, Chen W-W, Yang H-W, et al. Design of a robust infrastructure to monitor the safety of the pandemic A(H1N1) 2009 vaccination program in Taiwan. Vaccine2010;28:7161–6.[doi:10.1016/j.vaccine.2010.08.069](http://dx.doi.org/10.1016/j.vaccine.2010.08.069)
    
    

35. 35.Jackson LA, Carste BA, Malais D, et al. Retrospective population-based assessment of medically attended injection site reactions, seizures, allergic responses and febrile episodes after acellular pertussis vaccine combined with diphtheria and tetanus toxoids. Pediatr Infect Dis J2002;21:781–6.[doi:10.1097/00006454-200208000-00016](http://dx.doi.org/10.1097/00006454-200208000-00016)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/00006454-200208000-00016&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=12192169&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000177543300012&link_type=ISI) 

36. 36.Jackson LA, Nelson JC, Whitney CG, et al. Assessment of the safety of a third dose of pneumococcal polysaccharide vaccine in the vaccine safety Datalink population. Vaccine2006;24:151–6.[doi:10.1016/j.vaccine.2005.07.066](http://dx.doi.org/10.1016/j.vaccine.2005.07.066)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=16122845&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

37. 37.Jackson LA, Yu O, Nelson J, et al. Risk of medically attended local reactions following diphtheria toxoid containing vaccines in adolescents and young adults: a vaccine safety Datalink study. Vaccine2009;27:4912–6.[doi:10.1016/j.vaccine.2009.06.038](http://dx.doi.org/10.1016/j.vaccine.2009.06.038)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2009.06.038&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19567245&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

38. 38.Jacobsen SJ, Ackerson BK, Sy LS, et al. Observational safety study of febrile convulsion following first dose MMRV vaccination in a managed care setting. Vaccine2009;27:4656–61.[doi:10.1016/j.vaccine.2009.05.056](http://dx.doi.org/10.1016/j.vaccine.2009.05.056)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2009.05.056&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19520201&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000268351200015&link_type=ISI) 

39. 39.Kawai AT, Li L, Kulldorff M, et al. Absence of associations between influenza vaccines and increased risks of seizures, Guillain-Barré syndrome, encephalitis, or anaphylaxis in the 2012-2013 season. Pharmacoepidemiol Drug Saf2014;23:548–53.[doi:10.1002/pds.3575](http://dx.doi.org/10.1002/pds.3575)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/pds.3575&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=24497128&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

40. 40.Kharbanda EO, Vazquez-Benitez G, Lipkind H, et al. Inactivated influenza vaccine during pregnancy and risks for adverse obstetric events. Obstet Gynecol2013;122:659–67.[doi:10.1097/AOG.0b013e3182a1118a](http://dx.doi.org/10.1097/AOG.0b013e3182a1118a)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/AOG.0b013e3182a1118a&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23921876&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

41. 41.Kharbanda EO, Vazquez-Benitez G, Lipkind HS, et al. Evaluation of the association of maternal pertussis vaccination with obstetric events and birth outcomes. JAMA2014;312:1897–904.[doi:10.1001/jama.2014.14825](http://dx.doi.org/10.1001/jama.2014.14825)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/jama.2014.14825&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=25387187&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000344532600019&link_type=ISI) 

42. 42.Kharbanda EO, Vazquez-Benitez G, Lipkind HS, et al. Maternal Tdap vaccination: coverage and acute safety outcomes in the vaccine safety Datalink, 2007–2013. Vaccine2016;34:968–73.[doi:10.1016/j.vaccine.2015.12.046](http://dx.doi.org/10.1016/j.vaccine.2015.12.046)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2015.12.046&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=26765288&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

43. 43.Klein NP, Fireman B, Yih WK, et al. Measles-mumps-rubella-varicella combination vaccine and the risk of febrile seizures. Pediatrics2010;126:e1–8.[doi:10.1542/peds.2010-0665](http://dx.doi.org/10.1542/peds.2010-0665)
    
    [Abstract/FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6ODoiMTI2LzEvZTEiO3M6NDoiYXRvbSI7czoyMzoiL2JtamdoLzQvNC9lMDAxMDY1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

44. 44.Klein NP, Hansen J, Chao C, et al. Safety of quadrivalent human papillomavirus vaccine administered routinely to females. Arch Pediatr Adolesc Med2012;166:1140–8.[doi:10.1001/archpediatrics.2012.1451](http://dx.doi.org/10.1001/archpediatrics.2012.1451)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/archpediatrics.2012.1451&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23027469&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000311805300011&link_type=ISI) 

45. 45.Klein NP, Lewis E, Baxter R, et al. Measles-containing vaccines and febrile seizures in children age 4 to 6 years. Pediatrics2012;129:809–14.[doi:10.1542/peds.2011-3198](http://dx.doi.org/10.1542/peds.2011-3198)
    
    [Abstract/FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6OToiMTI5LzUvODA5IjtzOjQ6ImF0b20iO3M6MjM6Ii9ibWpnaC80LzQvZTAwMTA2NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

46. 46.Klein NP, Lewis E, Fireman B, et al. Safety of measles-containing vaccines in 1-year-old children. Pediatrics2015;135:e321–9.[doi:10.1542/peds.2014-1822](http://dx.doi.org/10.1542/peds.2014-1822)
    
    [Abstract/FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjEzNS8yL2UzMjEiO3M6NDoiYXRvbSI7czoyMzoiL2JtamdoLzQvNC9lMDAxMDY1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

47. 47.Lee GM, Greene SK, Weintraub ES, et al. H1N1 and seasonal influenza vaccine safety in the vaccine safety Datalink project. Am J Prev Med2011;41:121–8.[doi:10.1016/j.amepre.2011.04.004](http://dx.doi.org/10.1016/j.amepre.2011.04.004)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.amepre.2011.04.004&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21767718&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

48. 48.Li R, Stewart B, McNeil MM, et al. Post licensure surveillance of influenza vaccines in the vaccine safety Datalink in the 2013-2014 and 2014-2015 seasons. Pharmacoepidemiol Drug Saf2016;25:928–34.[doi:10.1002/pds.3996](http://dx.doi.org/10.1002/pds.3996)
    
    

49. 49.Nelson JC, Yu O, Dominguez-Islas CP, et al. Adapting group sequential methods to observational postlicensure vaccine safety surveillance: results of a pentavalent combination DTaP-IPV-Hib vaccine safety study. Am J Epidemiol2013;177:131–41.[doi:10.1093/aje/kws317](http://dx.doi.org/10.1093/aje/kws317)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/aje/kws317&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23292957&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000313528700005&link_type=ISI) 

50. 50.Nordin JD, Kharbanda EO, Benitez GV, et al. Maternal safety of trivalent inactivated influenza vaccine in pregnant women. Obstet Gynecol2013;121:519–25.[doi:10.1097/AOG.0b013e3182831b83](http://dx.doi.org/10.1097/AOG.0b013e3182831b83)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/AOG.0b013e3182831b83&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23635613&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000316607600005&link_type=ISI) 

51. 51.Nordin JD, Kharbanda EO, Vazquez Benitez G, et al. Maternal influenza vaccine and risks for preterm or small for gestational age birth. J Pediatr2014;164:1051–7.[doi:10.1016/j.jpeds.2014.01.037](http://dx.doi.org/10.1016/j.jpeds.2014.01.037)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jpeds.2014.01.037&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=24582484&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000335516000023&link_type=ISI) 

52. 52.Nordin JD, Kharbanda EO, Vazquez-Benitez G, et al. Monovalent H1N1 influenza vaccine safety in pregnant women, risks for acute adverse events. Vaccine2014;32:4985–92.[doi:10.1016/j.vaccine.2014.07.017](http://dx.doi.org/10.1016/j.vaccine.2014.07.017)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=25045808&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

53. 53.Nordin JD, Parker ED, Vazquez-Benitez G, et al. Safety of the yellow Fever vaccine: a retrospective study. J Travel Med2013;20:368–73.[doi:10.1111/jtm.12070](http://dx.doi.org/10.1111/jtm.12070)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/jtm.12070&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=24118538&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

54. 54.Sukumaran L, McCarthy NL, Kharbanda EO, et al. Association of Tdap vaccination with acute events and adverse birth outcomes among pregnant women with prior Tetanus-Containing immunizations. JAMA2015;314:1581–7.[doi:10.1001/jama.2015.12790](http://dx.doi.org/10.1001/jama.2015.12790)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/jama.2015.12790&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=26501534&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

55. 55.Sukumaran L, McCarthy NL, Kharbanda EO, et al. Safety of tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis and influenza vaccinations in pregnancy. Obstet Gynecol2015;126:1069–74.[doi:10.1097/AOG.0000000000001066](http://dx.doi.org/10.1097/AOG.0000000000001066)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/AOG.0000000000001066&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=26444109&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

56. 56.Tartof SY, Sy LS, Ackerson BK, et al. Safety of quadrivalent meningococcal conjugate vaccine in children 2-10 years. Pediatr Infect Dis J2017;36:1087–92.[doi:10.1097/INF.0000000000001696](http://dx.doi.org/10.1097/INF.0000000000001696)
    
    

57. 57.Tse A, Tseng HF, Greene SK, et al. Signal identification and evaluation for risk of febrile seizures in children following trivalent inactivated influenza vaccine in the vaccine safety Datalink project, 2010–2011. Vaccine2012;30:2024–31.[doi:10.1016/j.vaccine.2012.01.027](http://dx.doi.org/10.1016/j.vaccine.2012.01.027)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2012.01.027&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22361304&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000302662800016&link_type=ISI) 

58. 58.Tseng HF, Liu A, Sy L, et al. Safety of zoster vaccine in adults from a large managed-care cohort: a vaccine safety Datalink study. J Intern Med2012;271:510–20.[doi:10.1111/j.1365-2796.2011.02474.x](http://dx.doi.org/10.1111/j.1365-2796.2011.02474.x)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2796.2011.02474.x&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22026504&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

59. 59.Tseng H-F, Sy LS, Ackerson BK, et al. Safety of quadrivalent meningococcal conjugate vaccine in 11- to 21-year-olds. Pediatrics2017;139.[doi:10.1542/peds.2016-2084](http://dx.doi.org/10.1542/peds.2016-2084)
    
    

60. 60.Tseng HF, Sy LS, Liu I-LA, et al. Postlicensure surveillance for pre-specified adverse events following the 13-valent pneumococcal conjugate vaccine in children. Vaccine2013;31:2578–83.[doi:10.1016/j.vaccine.2013.03.040](http://dx.doi.org/10.1016/j.vaccine.2013.03.040)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2013.03.040&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23579258&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000320495000007&link_type=ISI) 

61. 61.Weintraub ES, Baggs J, Duffy J, et al. Risk of intussusception after monovalent rotavirus vaccination. N Engl J Med2014;370:513–9.[doi:10.1056/NEJMoa1311738](http://dx.doi.org/10.1056/NEJMoa1311738)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1056/NEJMoa1311738&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=24422678&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000331073500007&link_type=ISI) 

62. 62.Wise ME, Viray M, Sejvar JJ, et al. Guillain-Barre syndrome during the 2009-2010 H1N1 influenza vaccination campaign: population-based surveillance among 45 million Americans. Am J Epidemiol2012;175:1110–9.[doi:10.1093/aje/kws196](http://dx.doi.org/10.1093/aje/kws196)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/aje/kws196&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22582209&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000305082300003&link_type=ISI) 

63. 63.Yih WK, Lee GM, Lieu TA, et al. Surveillance for adverse events following receipt of pandemic 2009 H1N1 vaccine in the Post-Licensure rapid immunization safety monitoring (PriSM) system, 2009-2010. Am J Epidemiol2012;175:1120–8.[doi:10.1093/aje/kws197](http://dx.doi.org/10.1093/aje/kws197)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/aje/kws197&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22582207&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000305082300004&link_type=ISI) 

64. 64.Yih WK, Nordin JD, Kulldorff M, et al. An assessment of the safety of adolescent and adult tetanus–diphtheria–acellular pertussis (Tdap) vaccine, using active surveillance for adverse events in the vaccine safety Datalink. Vaccine2009;27:4257–62.[doi:10.1016/j.vaccine.2009.05.036](http://dx.doi.org/10.1016/j.vaccine.2009.05.036)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2009.05.036&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19486957&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

65. 65.Burwen DR, Sandhu SK, MaCurdy TE, et al. Surveillance for Guillain-Barré syndrome after influenza vaccination among the Medicare population, 2009-2010. Am J Public Health2012;102:1921–7.[doi:10.2105/AJPH.2011.300510](http://dx.doi.org/10.2105/AJPH.2011.300510)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.2105/AJPH.2011.300510&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22970693&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000309435200020&link_type=ISI) 

66. 66.Todkill D, Loveridge P, Elliot AJ, et al. Utility of ambulance data for real-time syndromic surveillance: a pilot in the West Midlands region, United Kingdom. Prehosp Disaster Med2017;32:667–72.[doi:10.1017/S1049023X17006690](http://dx.doi.org/10.1017/S1049023X17006690)
    
    

67. 67.Caudle JM, van Dijk A, Rolland E, et al. Telehealth Ontario detection of gastrointestinal illness outbreaks. Can J Public Health2009;100:253–7.
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19722336&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

68. 68.Davis RL, Kolczak M, Lewis E, et al. Active surveillance of vaccine safety: a system to detect early signs of adverse events. Epidemiology2005;16:336–41.
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1097/01.ede.0000155506.05636.a4&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=15824549&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

69. 69.Seabroke S, Bryan P, Wise L. 444. The use of observed vs expected analyses in real-time vaccine safety surveillance in the UK during the H1N1 pandemic. Pharmacoepidemiology and Drug Safety2010;19.
    
    

70. 70.Pillsbury A, Cashman P, Leeb A, et al. Real-time safety surveillance of seasonal influenza vaccines in children, Australia, 2015. Euro Surveill2015;20.[doi:10.2807/1560-7917.ES.2015.20.43.30050](http://dx.doi.org/10.2807/1560-7917.ES.2015.20.43.30050)
    
    

71. 71.Kulldorff M, Davis RL, Kolczak† M, et al. A maximized sequential probability ratio test for drug and vaccine safety surveillance. Seq Anal2011;30:58–78.[doi:10.1080/07474946.2011.539924](http://dx.doi.org/10.1080/07474946.2011.539924)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1080/07474946.2011.539924&link_type=DOI) 

72. 72.Li R, Stewart B, Weintraub E. Evaluating efficiency and statistical power of self-controlled case series and self-controlled risk interval designs in vaccine safety. J Biopharm Stat2016;26:686–93.[doi:10.1080/10543406.2015.1052819](http://dx.doi.org/10.1080/10543406.2015.1052819)
    
    

73. 73.Farrington CP. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics1995;51:228–35.[doi:10.2307/2533328](http://dx.doi.org/10.2307/2533328)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.2307/2533328&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=7766778&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=A1995QX73300020&link_type=ISI) 

74. 74.Greene SK, Kulldorff M, Yin R, et al. Near real-time vaccine safety surveillance with partially accrued data. Pharmacoepidemiol Drug Saf2011;20:583–90.[doi:10.1002/pds.2133](http://dx.doi.org/10.1002/pds.2133)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/pds.2133&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=21538670&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

75. 75.Greene SK, Kulldorff M, Lewis EM, et al. Near real-time surveillance for influenza vaccine safety: proof-of-concept in the vaccine safety Datalink project. Am J Epidemiol2010;171:177–88.[doi:10.1093/aje/kwp345](http://dx.doi.org/10.1093/aje/kwp345)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/aje/kwp345&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19965887&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000273224700005&link_type=ISI) 

76. 76.Silva IR, Kulldorff M. Continuous versus group sequential analysis for post-market drug and vaccine safety surveillance. Biometrics2015;71:851–8.[doi:10.1111/biom.12324](http://dx.doi.org/10.1111/biom.12324)
    
    

77. 77.Leite A, Andrews NJ, Thomas SL. Assessing recording delays in general practice records to inform near real-time vaccine safety surveillance using the clinical Practice Research Datalink (CPRD). Pharmacoepidemiol Drug Saf2017;26:437–45.[doi:10.1002/pds.4173](http://dx.doi.org/10.1002/pds.4173)
    
    

78. 78.Li L, Kulldorff M. A conditional maximized sequential probability ratio test for pharmacovigilance. Stat Med2010;29:284–95.[doi:10.1002/sim.3780](http://dx.doi.org/10.1002/sim.3780)
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19941282&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000273666600011&link_type=ISI) 

79. 79.Nelson JC, Cook AJ, Yu O, et al. Methods for observational post-licensure medical product safety surveillance. Stat Methods Med Res2015;24:177–93.[doi:10.1177/0962280211413452](http://dx.doi.org/10.1177/0962280211413452)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1177/0962280211413452&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22138688&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom) 

80. 80.Nelson JC, Cook AJ, Yu O, et al. Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data. Pharmacoepidemiol Drug Saf2012;21:62–71.[doi:10.1002/pds.2324](http://dx.doi.org/10.1002/pds.2324)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/pds.2324&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22262594&link_type=MED&atom=%2Fbmjgh%2F4%2F4%2Fe001065.atom)