Methodological frontiers in vaccine safety: qualifying available evidence for rare events, use of distributed data networks to monitor vaccine safety issues, and monitoring the safety of pregnancy interventions
==================================================================================================================================================================================================================

* Caitlin Dodd
* Nick Andrews
* Helen Petousis-Harris
* Miriam Sturkenboom
* Saad B Omer
* Steven Black

## Abstract

While vaccines are rigorously tested for safety and efficacy in clinical trials, these trials do not include enough subjects to detect rare adverse events, and they generally exclude special populations such as pregnant women. It is therefore necessary to conduct postmarketing vaccine safety assessments using observational data sources. The study of rare events has been enabled in through large linked databases and distributed data networks, in combination with development of case-centred methods. Distributed data networks necessitate common protocols, definitions, data models and analytics and the processes of developing and employing these tools are rapidly evolving. Assessment of vaccine safety in pregnancy is complicated by physiological changes, the challenges of mother-child linkage and the need for long-term infant follow-up. Potential sources of bias including differential access to and utilisation of antenatal care, immortal time bias, seasonal timing of pregnancy and unmeasured determinants of pregnancy outcomes have yet to be fully explored. Available tools for assessment of evidence generated in postmarketing studies may downgrade evidence from observational data and prioritise evidence from randomised controlled trials. However, real-world evidence based on real-world data is increasingly being used for safety assessments, and new tools for evaluating real-world evidence have been developed. The future of vaccine safety surveillance, particularly for rare events and in special populations, comprises the use of big data in single countries as well as in collaborative networks. This move towards the use of real-world data requires continued development of methodologies to generate and assess real world evidence.

*   vaccines
*   epidemiology
*   review

### Summary box

*   Assessment of vaccine safety has been enabled through availability of observational data sources and development of new methods, but gaps still exist for the study of rare events and assessment in special populations.

*   Collaborative studies in distributed data networks increase power but require the use of common protocols, data models, analytics and definitions as well as methods for data privacy preservation.

*   Inclusion of low-income and middle-income country data sources and efficient reuse of tools and expertise generated in collaborative studies requires sustainable funding for capacity building and readiness.

*   As more vaccines are recommended in pregnancy, increased attention to the quality and completeness of vaccine safety data surrounding pregnancy is necessary, as is additional research into sources of bias relevant to the study of exposure during pregnancy.

*   While randomised controlled trials represent a gold standard, the inability of trials to answer all relevant questions must be accepted and criteria for assessing quality of real-world evidence incorporated into decision making on the safety of vaccines.

## Introduction

The awareness that medicines can cause harm has been known since ancient times. However, it was the thalidomide incident of the late 1950s and early 1960s that stimulated the monumental shift to proactively regulate drugs internationally. While new mandates around drug registration were implemented, there were also new regulations for postmarketing surveillance. Developments in ways to code adverse events emerged through the 60s and the 70s and continue today, and drug adverse event reporting systems have been established. In 1978, a feasibility study to evaluate standardising spontaneous reports funded by the US Food and Drug Administration (FDA) evolved to what is now known as the International Pharmacovigilance Monitoring Centre based in Uppsala, Sweden. However, while modern pharmacovigilance incorporates spontaneous reporting data and experience from over 128 countries exists to detect signals, the data are not suitable to estimate risk.1

At the time of their licensure, vaccines have been assessed for efficacy, quality and safety in randomised controlled trials (RCTs). While randomised studies are traditionally viewed as the gold standard for vaccine outcomes, they are usually limited by their sample size when it comes to detecting very rare events or long-term outcomes.2 Additionally, during public health emergencies such as the 2009 H1N1 influenza pandemic, recent Ebola epidemics and the 2020 SARS-CoV-2 pandemic, vaccines may by necessity undergo expedited clinical trials and licensing. This is where postmarketing surveillance, based on observational studies, is required.

Pharmacoepidemiology is the study of the use and effects of drugs and other medicinal products in populations. It uses epidemiological methods to study effects in large numbers of people and can contribute important information about the effectiveness and safety of a vaccine that is not available from a prelicensure clinical evaluation. The importance of observational studies in complementing clinical trials of drugs is a well-established cornerstone of pharmacoepidemiology. Observational studies in vaccine safety include a range of methodologies that essentially compare the occurrence of outcomes of interest among people exposed and unexposed to a vaccine. These include the cohort, case-referent (ie, case-control) studies, self-controlled studies, analyses of secular trends (ecological studies), case series and case reports, in descending order of levels of evidence.3 Vaccine safety studies rely on the first three approaches to assess risk, while the others serve to generate hypotheses. In recent years, sequential analyses have also been used to prospectively monitor suspected vaccine safety concerns.4

Over the past three decades, the ability to conduct observational studies has evolved to a point that extremely large, heterogeneous populations can be used to assess vaccine safety outcomes. Traditional methods have used cohort and case-control designs but more recently newer designs such as the self-controlled case series have found favour in overcoming potential confounder and selection bias (table 1).5

View this table:
[Table 1](http://gh.bmj.com/content/6/Suppl_2/e003540/T1)

Table 1 
Postmarketing evidence generation in vaccine safety76

While monitoring the safety of vaccines to the highest standard achievable using the tools available is fundamentally ‘right’ and scientifically important, there is also a need to address perceptions about vaccine safety. In part due to the well-coordinated international antivaccine movement, vaccine hesitancy has risen to become one of the top 10 threats to global public health.6 Achieving and maintaining public confidence in vaccines needs rapid and credible responses to vaccine safety concerns, real or perceived. It is concerns about vaccine safety that contribute the most to vaccine hesitancy and being able to offer reassurance and effective communications supported by high-quality data is essential.7

Another reason for conducting large observational postlicensure vaccine safety studies is to examine potential associations with rare events, differences between vaccine brands and formulations and possible risks in special subpopulations. Rare events such as Guillain-Barre syndrome (GBS) pose challenges with respect to study power. For example, to detect a twofold increased relative risk with a background incidence of 1/100 000 and assuming a 1:1 ratio of vaccinated to non-vaccinated subjects, a study population of over 4.7 million is required.3 Such risks cannot be assessed in clinical trials, nor among smaller populations. Only through large administrative data collections for populations in the millions can a vaccine-associated risk for rare events be assessed. In addition, the globalisation of vaccine manufacturing increases the need to be able to compare brands, as illustrated by measles vaccines and infant mortality,8 mumps vaccines and aseptic meningitis9 and influenza vaccines in association with febrile convulsions10 or narcolepsy.11 12 Finally, there is also the need to assess the safety of vaccines in subpopulations such as pregnant women and their infants,13 people with certain health conditions such as immune suppression14 and among different ethnic groups15 and indigenous peoples.16 These subpopulations are often excluded from RCTs. To these ends, linkable data for large and diverse populations is required.

In recent years, there have been major advances in the tools and methodologies required to undertake the types of large, robust observational studies needed to address these diverse challenges. In 1990, the US Centers for Disease Control and Prevention in collaboration with large managed care organisations created the Vaccine Safety Datalink (VSD).2 This project linked medical events and demographic information with vaccine exposure information. Since then the system has evolved to include more populations, changes in the way data are collected, the development of near real-time data files and the ability to monitor new vaccines or changes in vaccine use in near real-time.17 Since the early 1990s, the number of publications internationally referring to multiple observational databases has exploded thanks to increased existence of electronic databases, the appreciation of the importance of data-linkage and information technology for analysing multiple databases.18 Recent collaborations such as Vaccine Adverse Events Monitoring and Communication (VAESCO), Accelerated Development of Vaccine Benefit-risk Collaboration in Europe (ADVANCE/VAC4EU), Systematic Observational Method for Narcolepsy and Influenza Immunisation Assessment (SOMNIA) and the WHO Global Vaccine Safety-Multi Country Collaboration (GVS-MCC) have successfully incorporated data from multiple countries to produce estimates of vaccine safety and effectiveness.12 19–22 The purpose of this paper is to appraise current methodologies in postmarketing vaccine safety in terms of how these can be used to assess rare events and special populations, and how evidence derived from these studies can be evaluated.

## Studying rare events

### Self-controlled case-only designs and other methods

Over the past 20 years, there have been an increasing number of epidemiological studies assessing vaccine safety for rare events which cannot be easily assessed in trials. This has been made possible through the availability of large linked databases and innovative tools for analysis. While traditional case-control and cohort studies continue to be used, novel designs such as case-only, case-coverage and case-centred methods have been developed and evaluated for use with these databases.23–27 Improved understanding of the benefits and limitations of the methods and data sources in terms of potential bias and confounding helps determine which are most suitable for addressing a specific adverse event and vaccine.

Case-only designs, which only require an unbiased set of cases, are particularly useful for assessing rare events due to their efficiency in only needing cases and their elimination of time-invariant measured and un-measured confounding factors. Such factors may be multifactorial and difficult to measure and could include frailty, socioeconomic status and levels of healthcare usage. The most commonly used case-only method is self-controlled case-series (SCCS) and its derivatives (eg, self-controlled risk-interval).23 25 SCCS requires definition of a follow-up period in which cases and their vaccine history are obtained, along with risk intervals postvaccination and baseline intervals for comparison. Another method, developed prior to SCCS and applied in the GVS-MCC study of measles and mumps containing vaccines, is the case crossover method in which vaccine exposure in risk windows just prior to the event is compared with earlier control windows within cases.21 24 Case-coverage and referent designs use data on cases and their vaccination status and compare this with vaccination uptake in a referent population or cases which may be from a different data source.26 27 These novel tools have proved highly useful in recent decades for assessing rare events and with more availability of linked databases should continue to be used and developed.

### Detection, power, specificity, bias and delayed onset

To study an adverse event, it is first necessary to identify it as a possible adverse reaction. Events likely to be missed and therefore requiring methods for identification are those that are rare, new, unexpected and with delayed onset after vaccination. Even when there is a signal, or other reason to study a rare event, it is often the case that this cannot be done rapidly due to the rarity of the event. For new vaccines, this may mean that the full safety profile will remain unknown for a long time and this limitation should be documented and solutions developed. Large-linked databases and collaborative studies will enhance power, but many countries, including many low-income and middle-income countries (LMIC), do not have such databases. This means that, for vaccines used only in these countries, enhanced safety assessment with epidemiological studies is limited. Case-only methods offer a good solution due to their efficiency but still require the ability to identify an unbiased set of cases with linked vaccination records. While some adverse events are well diagnosed and coded in records, others may have non-specific symptoms with unclear onset meaning coded events have a low positive predictive value. Validation, while desirable, may not be possible or expensive. It is therefore important to understand methodologically the implications of assessing events where specificity may be lacking. Further work on bias analysis would be beneficial in this area as well as the merits of validation on subsets of cases. Finally dealing with confounding, particularly in studies where case-only methods are not appropriate remains a challenge, particularly with events with onset many months or years after vaccination.

### Collaborative studies

Although most epidemiological studies of rare events to date have been done within countries, this requires the availability of data from large study populations with high vaccine exposure or over long periods of time. Collaborative studies in which data or results are combined between countries or regions within a country offer the opportunity to assess rare and very rare adverse events where numbers within one site are insufficient to address a question. They may also enable assessment within subpopulations such as pregnant women or using data from several smaller populations such as those obtained through sentinel hospitals. Collaborative studies of vaccine safety are complex due to the variety of potential signals, available data sources and possible study designs. A solution has been successfully employed within the VSD (see ‘Distributed data networks’ section), where a common methodology and analysis is possible when using a network of healthcare organisations.28

There have also been several international collaborative studies in the past decade. This was first done in Europe through the VAESCO consortium for Pandemic influenza and GBS (7 countries) and then internationally through the Global H1N1 GBS consortium (10 countries), the SOMNIA pandemic vaccine narcolepsy study (6 countries) and the proof-of-concept WHO Global vaccine Safety Multi Country Collaboration in LMICs assessing measles-containing vaccines association with thrombocytopenic purpura and aseptic meningitis (16 countries).12 19 29–31 There have also been bipartite collaborations such as Denmark and Sweden for human papillomavirus (HPV) vaccine and autoimmune conditions.32 33 Most recently, the ADVANCE project created a system for monitoring vaccine benefits and risks and is now continuing as the VAC4EU.20 34–39 Most of these studies have involved data sharing to a hub and while some have involved a single methodology, others have allowed different study designs. Key issues to have arisen are the cost and timeliness of such studies and in LMIC having sentinel hospitals that code discharges and link to vaccination data.

## Distributed data networks

### Common data models, protocols, analytics, definitions

Currently, distributed data networks can be described along two axes: first, whether they employ an ad hoc or dedicated network and second, the extent to which they use common standards. Along the ad hoc to dedicated network axis, collaborative studies can be implemented either through a group of data sources assembled to address a specific study question or through a sustained database network via which studies can be deployed rapidly. Many examples of distributed analyses conducted within ad hoc networks exist (see ‘Collaborative studies’ section) while the only examples of dedicated networks specific to the study of vaccines are the VSD2 and the ADVANCE/VAC4EU in Europe. Other dedicated networks of observational data sources include the FDA Sentinel System and the Observational Health Data Sciences and Informatics community, of which only Sentinel has been employed in the assessment of vaccine safety.40–42

These data networks also vary in the extent to which they employ common standards such as common protocols, definitions, data models and analytics. Common study protocols are written and agreed on by the network and implemented either centrally by distributing analysis scripts which can be run locally against data in a common data model (CDM) or implemented locally by each database. Common definitions used to harmonise data from heterogeneous sources include those for adverse events following immunisation (AEFI) developed by the Brighton Collaboration which require medical record review, or harmonisation of codes from various coding systems both for events and vaccine exposures in retrospective database analyses.43–45 CDMs harmonise the structure and content of the participating data sources to enable the use of common analytics developed centrally to run against all data sources. Finally, common analytics include data management, data quality assessment and analysis scripts which can be deployed against data in a CDM and, in a dedicated database network, can be modularised for reuse.

### Models for analysis

Several models exist to analyse data within a distributed data network, the choice of which is dependent on both the privacy restrictions at each contributing data site and the extent to which the network has used common standards. The models vary in how much individual-level information is shared, how analysis results are combined and the autonomy of each site.46 At the extreme of local autonomy and privacy preservation, a common protocol is implemented locally against data in its original format and estimates from all sites are meta-analysed. An alternative which reduces local autonomy is one in which data are converted to a CDM and analysis is conducted locally using a programme written centrally, after which results are meta-analysed. This model can be adapted to allow for pooling of individual-level data by deploying centrally written scripts which generate aggregated or de-identified data sets which can be pooled centrally and analysed. At the extreme of centralisation and data sharing, all individual-level data are pooled and centrally analysed.

### Data privacy and implications for one-stage versus two-stage meta-analysis

Typically, studies are conducted in individual sites and synthesis of evidence is conducted through systematic reviews or meta-analysis. In distributed data networks, depending on the model for analysis chosen, data may be retained locally at each site or shared at the aggregate or individual level. This choice depends primarily on privacy concerns and the desire for local control of the data. The approach of conducting analysis locally using only local data has the drawbacks that some data sources fall out of the analysis due to paucity of cases and resulting non-convergence, and that naturally occurring heterogeneity in vaccine formulations and schedules cannot be exploited. In the SOMNIA study, a hybrid approach was used, allowing for pooling of individual-level data from sites for which sharing was possible followed by meta-analytical pooling with results from sites for which sharing of individual-level data was not possible.12

Approaches to facilitate one-stage pooling have been discussed and include sharing of aggregate data sets or sharing of individual-level data sets. Currently, a lot of research is being conducted on approaches to privacy protecting distributed analysis methods. This includes development of DataSHIELD, which exploits a common underlying database structure to produce coefficients and estimates from data processed in parallel identical to those from data pooled at the individual level.47 More recently, simulation as well as empirical studies have shown that privacy-protecting methods, such as sharing of aggregated data sets, perform similarly to analysis in individual-level pooled data.48–50

### Inclusion of LMIC and sustainability

To date, distributed data networks for assessment of vaccine safety have struggled to include data sources in LMIC due to unavailability of electronic databases, inability to link vaccination data to outcomes and lack of funding for capacity building in these areas. While studies such as the GVS-MCC have successfully been conducted using data from LMIC, further development of capacity in countries where new vaccines will be developed and deployed requires sustained funding and infrastructure. Additionally, sustained funding is required to enable efficient reuse, dissemination and development of tools and expertise developed in funded projects with a limited time frame such as VAESCO, ADVANCE and SOMNIA.

## Studying exposure in pregnancy

Currently, vaccines are increasingly indicated for use in pregnancy, and additional vaccines are undergoing pregnancy trials. Vaccine safety assessment in pregnancy presents unique challenges as pregnancy is a physiologically dynamic state and safety considerations involve both the mother and the fetus. Levels of sex hormones estradiol and progesterone increase during the course of pregnancy and changes in sex hormones are associated with changes in the immune system. For example, as the pregnancy progresses, there is an increase in type 2 helper T-cell (Th2) responses and attenuation of type 1 helper T-cell (Th1); hence a Th1-to-Th2 shift in pregnancy.51 However, the increases in sex hormones and associated immunological changes are not perfectly linear. Moreover, these changes occur relatively quickly. Therefore, exposure assessment in pregnancy should account for the gestational age at vaccination. Often the gestational age at vaccination is grouped by trimester for statistical analyses. However, given the pace of change in pregnancy, trimester-based classification of gestational age might be too crude. If there is sufficient power, week of gestation is a better measure while assessing exposure time in pregnancy.

Considerations of toxic and teratogenic effects (ie, permanent anatomical, functional or developmental disruptions) on the embryo or fetus are dependent on the stage of development. Prior to implantation, adverse events can be a result of injury to a large proportion cells resulting in spontaneous abortion. If there is injury to a small number of cells, then often there is survival without abnormalities. In the embryonic period, that is, 2–9 weeks of embryonic gestational age, there is substantial organogenesis. Therefore, for exposures in this period, outcomes of concern include malformations and altered function. After 9 weeks of gestation through term, there is a period of fetal growth, differentiation and maturation; hence, outcomes of concern include preterm and small for gestational age birth, fetal death, minor malformations and altered function.

Exposure in pregnancy also warrants infant follow-up. While the duration of follow-up depends on the vaccine and the outcome of interest, there is increasing consensus on following the infant for at least 1 year after birth—particularly in maternal vaccine trials. The postpartum period also represents an opportunity to study the effects of antibodies in breast milk due to vaccine exposure in pregnancy or post partum on infant outcomes.

Because conduct of trials in pregnant women has implications for the health and safety of both mother and infant, their conduct is limited, increasing the need for observational studies.52 53 However, only a limited number of observational studies of vaccine exposure in pregnancy have been conducted and include assessments of influenza, tetanus, meningococcal, measles, mumps, oral polio, pertussis and yellow fever vaccines among others.54 55 This limitation to the number of studies conducted in large observational databases may be related both to the difficulty of conducting mother-child linkage and the range of adverse pregnancy and neonatal outcomes, which should be assessed in association with vaccine exposure. Additionally, observational studies of vaccination in pregnancy are prone to sources of biases including differential access to and utilisation of antenatal care, immortal time bias related especially to late-term exposures, seasonal timing of pregnancy and covariates related to adverse pregnancy outcomes such as nutrition, smoking and alcohol use which are typically not captured in observational data sources.56 Adverse pregnancy and birth outcomes must be assessed objectively using standardised definitions and, as is the case for all AEFI, availability of background rates is vital. Towards these ends, the GAIA Consortium has developed and tested case definitions for a set of maternal and neonatal AEFI.57 The US Sentinel system has developed and included a dedicated mother-child linkage table in its CDM.58 59

## Options for evaluating existing evidence

When public health policy decisions are made regarding recommendations for vaccine usage, a potential safety issue or regarding the cost-effectiveness of a vaccine, decision makers often use criteria to evaluate the quality of the evidence that is available. The most common assessment tool is the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) criteria.60 GRADE classifies studies into four levels of evidence—very low, low, moderate and high—where evidence from RCTs is rated as high quality and evidence from observational data is rated as low quality.

Despite widespread acceptance by recommending bodies including the WHO Strategic Advisory Group of Experts (WHO SAGE) committee, the assumption that observational data are routinely of low quality and randomised trials are always the best evidence source for recommendations is itself based on low-quality evidence and is increasingly being challenged.61 62 Frieden pointed out that while RCT data have good internal validity and assess efficacy in perfect situations, they can have very low external validity, which is now generally accepted.62 63 RCTs have inclusion and exclusion criteria that select certain populations and follow-up that does not reflect reality. In fact, the FDA, European Medicines Agency (EMA) and other regulators and health technology organisations are now embracing real-world data and real-world evidence (RWE) as the state of the art for safety assessments, because of the limited size and follow-up for RCTs. The 21st century Act places additional focus on the use of RWE for regulatory decision making.62 The EMA has developed the OPTIMAL framework and is increasingly using RWE for decision making.64

While treatment assignment is not randomised in observational studies (leaving potential for confounding) and data is not always source verified (leaving space for bias), they still can offer a robust ‘real-world’ picture of the effectiveness and safety of a vaccine in the target populations that it is being used, including people who might not have been eligible to participate in a clinical trial. Bias in observational studies is a problem to be addressed, as illustrated by conflicting results from studies of the purported association between hepatitis B vaccine and multiple sclerosis, some of which were subject to information and selection biases.65 However, study designs and methods have been developed to deal with bias and confounding such as the self-controlled case series, and the use and matching on propensity scores emulates a clinical trial. Assessment of the correctness of the outcome can be validated and exposure data can be bench marked.66 The availability of electronic health records in many countries has provided the opportunity to assess vaccine coverage, safety, effectiveness and impact on large populations in a retrospective fashion. The VSD in the USA (8 million source population) and the ADVANCE/VAC4EU infrastructure in Europe (40 million source population) are able to rapidly assess the safety, coverage, effectiveness and benefit-risk of vaccines postmarketing and the same infrastructures can be used to evaluate vaccine impact.28 67 68

To illustrate the poor predictive value of some RCT data, when results of a model based on real-world data for effectiveness for pneumococcal conjugate vaccine were compared with the prelicensure randomised trial for PCV7, it was evident that the prelicensure RCT may have underestimated vaccine impact for pneumonia and otitis media and thus the data from the RCT found the vaccine cost US$50 000 per quality-adjusted life year when in fact it was likely to be cost saving.69 This disparity was likely due to the inability to assess herd effects in the clinical trial setting. In a study assessing potentially reduced efficacy of herpes zoster vaccine when administered concomitantly with 23-valent pneumococcal vaccine, no increased risk of herpes zoster was detected in subjects co-administered both vaccines. This evidence contradicts findings of reduced efficacy from clinical trials based on measured antibody levels as a measure of efficacy.70 Similarly, for rotavirus vaccines, while prelicensure RCT trials with >100 000 individuals found no evidence of an increased risk of intussusception following two different RCTs, large studies using RWE postlicensure found such a risk.71

We should point out that our goal here is not to set up a ‘we versus them’ conflict with GRADE but rather to point out that in the case of vaccine safety, while GRADE has its strengths, observational studies do as well. Hence, blind adherence to the GRADE criteria in evaluating vaccine safety is inappropriate. Rather, we would argue that both randomised trial data and observational data should each be considered in light of their strengths and weaknesses.

### Real-world evidence

The use of real-world data to generate RWE presents opportunities unavailable through randomised trials. For example, observational data in a diverse distributed network present opportunities to study questions such as the impact of differing childhood vaccination schedules including co-administration.38 72 Observational data sources also allow for application of case definitions and case-finding algorithms with differing balances of sensitivity and specificity as well as customisation on a per-data source basis.36 New sources of real-world data such as patient-generated data including that from apps present new opportunities as well as new regulatory challenges for use and interpretation of this type of evidence. Finally, the volume of real-world data allows for as-yet unexplored applications of machine learning and data mining beyond the data mining currently applied to data generated through passive reporting.

## Conclusion

Current methodologies for postmarketing vaccine safety are sufficient, in general, to assess rare events and safety concerns in general populations. There needs to be further effort in identifying and applying these methods to special populations such as pregnant women, persons with comorbidities and diverse ethnicities. Moreover, there is a need to look at combination schedules rather than separate vaccines.

While the study of events too rare to assess within a single population is made possible through collaborative studies at larger scale, these studies also require tools for harmonisation: common protocols, data models, definitions and analytics. The future of vaccine safety surveillance comprises the use of big data in single countries as well as in collaborative networks and requires that networks have sustainable funding rather than being assembled ad hoc. Progress towards this goal is evident in the VAC4EU consortium, which aims to develop a sustained vaccine-benefit risk network for Europe, and the Global Vaccine Data Network currently in development, which aims to maintain a global network of vaccine experts and data owners capable of quickly deploying vaccine safety studies in populations beyond North America and Europe.73

While each of these approaches will lead to increases in the availability of RWE on vaccine safety, each requires access to large volumes of harmonised data, which in turn requires a sustained data network, alongside global capacity building in this area.

Due to barriers to participation in trials for pregnant and lactating women, observational studies are of particular importance in the assessment of vaccine safety in pregnancy. This requires sustainable networks employing CDMs that prioritise mother-child linkage and recording of variables which allow for accurate ascertainment of the timing of exposure during pregnancy. Additionally, there is much room for methodological development in addressing bias and confounding in studies of vaccine exposure during pregnancy.

Our ability to use RWE is growing exponentially. Methods to deal with confounding and bias have matured and guidance documents on the transparency of reporting of such studies and new tools such as Risk Of Bias In Non-Randomised Studies-of Interventions have been developed to evaluate the potential for bias in observational studies.74 75

It may be time to re-evaluate the GRADE criteria themselves and provide more balance in our evaluation of available evidence. It would seem more reasonable to argue, as Frieden has done, that there is no one perfect gold standard for evidence. Rather, RCTs and observational studies each have their own strengths and weaknesses and can complement one another as we make decisions regarding the vaccines that we use.

## Data availability statement

There are no data in this work.

## Ethics statements

### Patient consent for publication

Not required.

## Acknowledgments

The authors would like to thank Patrick Zuber and Christine Guillard for coordination and support in the drafting of this manuscript.

## Footnotes

*   Handling editor Seye Abimbola

*   Contributors All authors contributed to conceptualisation, drafting and revision of the manuscript.

*   Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests HPH and SB are co-directors of the Global Vaccine Data Network.

*   Provenance and peer review Not commissioned; externally peer reviewed.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  1.  Andrews EB, 
    2.  Moore N
    
    Jones JK, Kingery E. History of Pharmacovigilance. In: Andrews EB, Moore N, eds. Mann’s Pharmacovigilance, 2014: 11–24.
    
    

2.  Chen RT, Glasser JW, Rhodes PH, et al. Vaccine safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The vaccine safety Datalink team. Pediatrics 1997;99:765–73.[doi:10.1542/peds.99.6.765](http://dx.doi.org/10.1542/peds.99.6.765)pmid:http://www.ncbi.nlm.nih.gov/pubmed/9164767
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1542/peds.99.6.765&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=9164767&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=A1997XC58800014&link_type=ISI) 

3.  Strom BL, Kimmel SE, Hennessy S. Textbook of pharmacoepidemiology. Wiley Online Library, 2013.
    
    

4.  Yih WK, Kulldorff M, Fireman BH, et al. Active surveillance for adverse events: the experience of the vaccine safety Datalink project. Pediatrics 2011;127 Suppl 1:S54–64.[doi:10.1542/peds.2010-1722I](http://dx.doi.org/10.1542/peds.2010-1722I)pmid:http://www.ncbi.nlm.nih.gov/pubmed/21502252
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

5.  Lao KSJ, Chui CSL, Man KKC, et al. Medication safety research by observational study design. Int J Clin Pharm 2016;51:676–84.[doi:10.1007/s11096-016-0285-6](http://dx.doi.org/10.1007/s11096-016-0285-6)
    
    

6.  Available: [https://www.who.int/news-room/feature-stories/ten-threats-to-global-health-in-2019](https://www.who.int/news-room/feature-stories/ten-threats-to-global-health-in-2019)
    
    

7.  Lane S, MacDonald NE, Marti M, et al. Vaccine hesitancy around the globe: analysis of three years of WHO/UNICEF joint reporting form data-2015–2017 2018;36:3861–7.
    
    

8.  Holt EA, Moulton LH, Siberry GK, et al. Differential mortality by measles vaccine titer and sex. J Infect Dis 1993;168:1087–96.[doi:10.1093/infdis/168.5.1087](http://dx.doi.org/10.1093/infdis/168.5.1087)pmid:http://www.ncbi.nlm.nih.gov/pubmed/8228340
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/infdis/168.5.1087&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=8228340&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=A1993MC89000001&link_type=ISI) 

9.  Bonnet M-C, Dutta A, Weinberger C, et al. Mumps vaccine virus strains and aseptic meningitis. Vaccine 2006;24:7037–45.[doi:10.1016/j.vaccine.2006.06.049](http://dx.doi.org/10.1016/j.vaccine.2006.06.049)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2006.06.049&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=16884835&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000242674300006&link_type=ISI) 

10. Petousis-Harris H, Poole T, Turner N, et al. Febrile events including convulsions following the administration of four brands of 2010 and 2011 inactivated seasonal influenza vaccine in NZ infants and children: the importance of routine active safety surveillance. Vaccine 2012;30:4945–52.[doi:10.1016/j.vaccine.2012.05.052](http://dx.doi.org/10.1016/j.vaccine.2012.05.052)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2012.05.052&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=22664224&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

11. Sarkanen TO, Alakuijala APE, Dauvilliers YA, et al. Incidence of narcolepsy after H1N1 influenza and vaccinations: systematic review and meta-analysis. Sleep Med Rev 2018;38:177–86.[doi:10.1016/j.smrv.2017.06.006](http://dx.doi.org/10.1016/j.smrv.2017.06.006)
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.smrv.2017.06.006&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

12. Weibel D, Sturkenboom M, Black S, et al. Narcolepsy and adjuvanted pandemic influenza A (H1N1) 2009 vaccines - Multi-country assessment. Vaccine 2018;36:6202–11.[doi:10.1016/j.vaccine.2018.08.008](http://dx.doi.org/10.1016/j.vaccine.2018.08.008)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30122647
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

13. Edwards KM. Ensuring vaccine safety in pregnant women. N Engl J Med 2017;376:1280–2.[doi:10.1056/NEJMe1701337](http://dx.doi.org/10.1056/NEJMe1701337)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28355513
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

14. Hechter RC, Qian L, Tartof SY, et al. Vaccine safety in HIV-infected adults within the vaccine safety Datalink project. Vaccine 2019;37:3296–302.[doi:10.1016/j.vaccine.2019.04.080](http://dx.doi.org/10.1016/j.vaccine.2019.04.080)
    
    

15. Huang J, Du J, Duan R, et al. Characterization of the differential adverse event rates by race/ethnicity groups for HPV vaccine by integrating data from different sources. Front Pharmacol 2018;9.[doi:10.3389/fphar.2018.00539](http://dx.doi.org/10.3389/fphar.2018.00539)
    
    

16. Petousis-Harris H, Jackson C, Stewart J, et al. Factors associated with reported pain on injection and reactogenicity to an OMV meningococcal B vaccine in children and adolescents. Hum Vaccin Immunother 2015;11:1872–7.[doi:10.1080/21645515.2015.1016670](http://dx.doi.org/10.1080/21645515.2015.1016670)
    
    

17. Baggs J, Gee J, Lewis E, et al. The vaccine safety Datalink: a model for monitoring immunization safety. Pediatrics 2011;127 Suppl 1:S45–53.[doi:10.1542/peds.2010-1722H](http://dx.doi.org/10.1542/peds.2010-1722H)pmid:http://www.ncbi.nlm.nih.gov/pubmed/21502240
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

18. Bate A, Chuang‐Stein C, Roddam A, et al. Lessons from meta‐analyses of randomized clinical trials for analysis of distributed networks of observational databases 2019;18:65–77.
    
    

19. Dieleman J, Romio S, Johansen K, et al. Guillain-Barre syndrome and adjuvanted pandemic influenza A (H1N1) 2009 vaccine: multinational case-control study in Europe. BMJ 2011;343:d3908. [doi:10.1136/bmj.d3908](http://dx.doi.org/10.1136/bmj.d3908)pmid:http://www.ncbi.nlm.nih.gov/pubmed/21750072
    
    [Abstract/FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDMvanVsMTJfMi9kMzkwOCI7czo0OiJhdG9tIjtzOjI5OiIvYm1qZ2gvNi9TdXBwbF8yL2UwMDM1NDAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

20. Weibel D, Dodd C, Mahaux O, et al. Advance system testing: can safety studies be conducted using electronic healthcare data? an example using pertussis vaccination. Vaccine 2019. doi:[doi:10.1016/j.vaccine.2019.06.040](http://dx.doi.org/10.1016/j.vaccine.2019.06.040). [Epub ahead of print: 31 Oct 2019].pmid:http://www.ncbi.nlm.nih.gov/pubmed/31677946
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

21. Perez-Vilar S, Weibel D, Sturkenboom M, et al. Enhancing global vaccine pharmacovigilance: proof-of-concept study on aseptic meningitis and immune thrombocytopenic purpura following measles-mumps containing vaccination. Vaccine 2018;36:347–54.[doi:10.1016/j.vaccine.2017.05.012](http://dx.doi.org/10.1016/j.vaccine.2017.05.012)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28558983
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

22. Sturkenboom M, Bahri P, Chiucchiuini A, et al. Why we need more collaboration in Europe to enhance post-marketing surveillance of vaccines. Vaccine 2019. doi:[doi:10.1016/j.vaccine.2019.07.081](http://dx.doi.org/10.1016/j.vaccine.2019.07.081). [Epub ahead of print: 31 Oct 2019].pmid:http://www.ncbi.nlm.nih.gov/pubmed/31677952
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

23. Farrington CP. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics 1995;51:228–35.[doi:10.2307/2533328](http://dx.doi.org/10.2307/2533328)pmid:http://www.ncbi.nlm.nih.gov/pubmed/7766778
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.2307/2533328&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=7766778&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=A1995QX73300020&link_type=ISI) 

24. Maclure M. The case-crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol 1991;133:144–53.[doi:10.1093/oxfordjournals.aje.a115853](http://dx.doi.org/10.1093/oxfordjournals.aje.a115853)pmid:http://www.ncbi.nlm.nih.gov/pubmed/1985444
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/oxfordjournals.aje.a115853&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=1985444&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=A1991ET75500006&link_type=ISI) 

25. Li L, Kulldorff M, Russek-Cohen E, et al. Quantifying the impact of time-varying baseline risk adjustment in the self-controlled risk interval design. Pharmacoepidemiol Drug Saf 2015;24:1304–12.[doi:10.1002/pds.3885](http://dx.doi.org/10.1002/pds.3885)pmid:http://www.ncbi.nlm.nih.gov/pubmed/26464236
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

26. Stowe J, Andrews N, Taylor B, et al. No evidence of an increase of bacterial and viral infections following measles, mumps and rubella vaccine. Vaccine 2009;27:1422–5.[doi:10.1016/j.vaccine.2008.12.038](http://dx.doi.org/10.1016/j.vaccine.2008.12.038)pmid:http://www.ncbi.nlm.nih.gov/pubmed/19146903
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2008.12.038&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=19146903&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

27. Greene SK, Rett MD, Vellozzi C, et al. Guillain-Barré syndrome, influenza vaccination, and antecedent respiratory and gastrointestinal infections: a Case-Centered analysis in the vaccine safety Datalink, 2009-2011. PLoS One 2013;8:e67185. [doi:10.1371/journal.pone.0067185](http://dx.doi.org/10.1371/journal.pone.0067185)pmid:http://www.ncbi.nlm.nih.gov/pubmed/23840621
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1371/journal.pone.0067185&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23840621&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

28. McNeil MM, Gee J, Weintraub ES, et al. The vaccine safety Datalink: successes and challenges monitoring vaccine safety. Vaccine 2014;32:5390–8.[doi:10.1016/j.vaccine.2014.07.073](http://dx.doi.org/10.1016/j.vaccine.2014.07.073)pmid:http://www.ncbi.nlm.nih.gov/pubmed/25108215
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2014.07.073&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=25108215&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

29. Romio S, Weibel D, Dieleman JP, et al. Guillain-Barré syndrome and adjuvanted pandemic influenza A (H1N1) 2009 vaccines: a multinational self-controlled case series in Europe. PLoS One 2014;9:e82222. [doi:10.1371/journal.pone.0082222](http://dx.doi.org/10.1371/journal.pone.0082222)pmid:http://www.ncbi.nlm.nih.gov/pubmed/24404128
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

30. Dodd CN, Romio SA, Black S, et al. International collaboration to assess the risk of Guillain Barré syndrome following influenza A (H1N1) 2009 monovalent vaccines. Vaccine 2013;31:4448–58.[doi:10.1016/j.vaccine.2013.06.032](http://dx.doi.org/10.1016/j.vaccine.2013.06.032)pmid:http://www.ncbi.nlm.nih.gov/pubmed/23770307
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2013.06.032&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=23770307&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000324510500027&link_type=ISI) 

31. Guillard-Maure C, Elango V, Black S, et al. Operational lessons learned in conducting a multi-country collaboration for vaccine safety signal verification and hypothesis testing: the global vaccine safety multi country collaboration initiative. Vaccine 2018;36:355–62.[doi:10.1016/j.vaccine.2017.07.085](http://dx.doi.org/10.1016/j.vaccine.2017.07.085)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28780118
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

32. Hviid A, Svanström H, Scheller NM, et al. Human papillomavirus vaccination of adult women and risk of autoimmune and neurological diseases. J Intern Med 2018;283:154–65.[doi:10.1111/joim.12694](http://dx.doi.org/10.1111/joim.12694)pmid:http://www.ncbi.nlm.nih.gov/pubmed/29044769
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/joim.12694&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=29044769&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

33. Andrews N, Stowe J, Miller E, et al. A collaborative approach to investigating the risk of thrombocytopenic purpura after measles-mumps-rubella vaccination in England and Denmark. Vaccine 2012;30:3042–6.[doi:10.1016/j.vaccine.2011.06.009](http://dx.doi.org/10.1016/j.vaccine.2011.06.009)pmid:http://www.ncbi.nlm.nih.gov/pubmed/21699947
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

34. Tin MT, et al. Advance system testing: vaccine benefit studies using multi-country electronic health data–The example of pertussis vaccination. Vaccine 2019.
    
    

35. Bollaerts K, de Smedt T, McGee C, et al. Advance: towards near real-time monitoring of vaccination coverage, benefits and risks using European electronic health record databases. Vaccine 2019. doi:[doi:10.1016/j.vaccine.2019.08.012](http://dx.doi.org/10.1016/j.vaccine.2019.08.012). [Epub ahead of print: 31 Oct 2019].pmid:http://www.ncbi.nlm.nih.gov/pubmed/31677951
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

36. Bollaerts K, Ledent E, de Smedt T, et al. Advance system testing: benefit-risk analysis of a marketed vaccine using multi-criteria decision analysis and individual-level state transition modelling. Vaccine 2019. doi:[doi:10.1016/j.vaccine.2019.09.034](http://dx.doi.org/10.1016/j.vaccine.2019.09.034). [Epub ahead of print: 31 Oct 2019].pmid:http://www.ncbi.nlm.nih.gov/pubmed/31677947
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

37. Gini R, Dodd CN, Bollaerts K, et al. Quantifying outcome misclassification in multi-database studies: the case study of pertussis in the advance project. Vaccine 2019.[doi:10.1016/j.vaccine.2019.07.045](http://dx.doi.org/10.1016/j.vaccine.2019.07.045)
    
    

38. Emborg H-D, Kahlert J, Braeye T, et al. Advance system testing: can coverage of pertussis vaccination be estimated in European countries using electronic healthcare databases: an example. Vaccine 2019. doi:[doi:10.1016/j.vaccine.2019.07.039](http://dx.doi.org/10.1016/j.vaccine.2019.07.039). [Epub ahead of print: 31 Oct 2019].pmid:http://www.ncbi.nlm.nih.gov/pubmed/31677953
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

39. Braeye T, et al. Estimation of vaccination coverage from electronic healthcare records; methods performance evaluation–A contribution of the ADVANCE-project. PLoS One 2019;9.
    
    

40. Ball R, et al. "The FDA’s sentinel initiative—a comprehensive approach to medical product surveillance.". Clin Pharmacol Ther 2016;3:265–8.
    
    

41. Reisinger SJ, Ryan PB, O'Hara DJ, et al. Development and evaluation of a common data model enabling active drug safety surveillance using disparate healthcare databases. J Am Med Inform Assoc 2010;17:652–62.[doi:10.1136/jamia.2009.002477](http://dx.doi.org/10.1136/jamia.2009.002477)pmid:http://www.ncbi.nlm.nih.gov/pubmed/20962127
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1136/jamia.2009.002477&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=20962127&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

42. Yih WK, Kulldorff M, Sandhu SK, et al. Prospective influenza vaccine safety surveillance using fresh data in the sentinel system. Pharmacoepidemiol Drug Saf 2016;25:481–92.[doi:10.1002/pds.3908](http://dx.doi.org/10.1002/pds.3908)pmid:http://www.ncbi.nlm.nih.gov/pubmed/26572776
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

43. Kohl KS, Bonhoeffer J, Chen R, et al. The Brighton collaboration: enhancing comparability of vaccine safety data. Pharmacoepidemiol Drug Saf 2003;12:335–40.[doi:10.1002/pds.851](http://dx.doi.org/10.1002/pds.851)pmid:http://www.ncbi.nlm.nih.gov/pubmed/12812014
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/pds.851&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=12812014&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000183287100011&link_type=ISI) 

44. Becker BFH, Avillach P, Romio S, et al. CodeMapper: semiautomatic coding of case definitions. A contribution from the advance project. Pharmacoepidemiol Drug Saf 2017;26:998–1005.[doi:10.1002/pds.4245](http://dx.doi.org/10.1002/pds.4245)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28657162
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

45. Becker B. Vaccine semantics: automatic methods for recognizing, representing, and Reasoning about vaccine-related information 2019.
    
    

46. Trifirò G, Coloma PM, Rijnbeek PR, et al. Combining multiple healthcare databases for postmarketing drug and vaccine safety surveillance: why and how? J Intern Med 2014;275:551–61.[doi:10.1111/joim.12159](http://dx.doi.org/10.1111/joim.12159)pmid:http://www.ncbi.nlm.nih.gov/pubmed/24635221
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1111/joim.12159&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=24635221&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

47. Wolfson M, Wallace SE, Masca N, et al. DataSHIELD: resolving a conflict in contemporary bioscience--performing a pooled analysis of individual-level data without sharing the data. Int J Epidemiol 2010;39:1372–82.[doi:10.1093/ije/dyq111](http://dx.doi.org/10.1093/ije/dyq111)pmid:http://www.ncbi.nlm.nih.gov/pubmed/20630989
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/ije/dyq111&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=20630989&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000283932000034&link_type=ISI) 

48. Yoshida K, et al. Comparison of privacy‐protecting analytic and data‐sharing methods: a simulation study. Pharmacoepidemiol Drug Saf 2018;9:1034–41.
    
    

49. Li X, et al. Validity of Privacy-Protecting analytical methods that use only Aggregate-Level information to conduct Multivariable-Adjusted analysis in distributed data networks. Am J Epidemiol 2018;4:709–23.
    
    

50. Shu D, Yoshida K, Fireman BH, et al. Inverse probability weighted COX model in multi-site studies without sharing individual-level data. Stat Methods Med Res 2020;29:1668–81.[doi:10.1177/0962280219869742](http://dx.doi.org/10.1177/0962280219869742)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31448681
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

51. Omer SB. Maternal immunization. N Engl J Med 2017;376:1256–67.[doi:10.1056/NEJMra1509044](http://dx.doi.org/10.1056/NEJMra1509044)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28355514
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1056/NEJMra1509044&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

52. Zuber PLF, Moran AC, Chou D, et al. Mapping the landscape of global programmes to evaluate health interventions in pregnancy: the need for harmonised approaches, standards and tools. BMJ Glob Health 2018;3:e001053. [doi:10.1136/bmjgh-2018-001053](http://dx.doi.org/10.1136/bmjgh-2018-001053)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30364289
    
    [Abstract/FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiYm1qZ2giO3M6NToicmVzaWQiO3M6MTE6IjMvNS9lMDAxMDUzIjtzOjQ6ImF0b20iO3M6Mjk6Ii9ibWpnaC82L1N1cHBsXzIvZTAwMzU0MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

53. Kochhar S, Edwards KM, Ropero Alvarez AM, et al. Introduction of new vaccines for immunization in pregnancy - Programmatic, regulatory, safety and ethical considerations. Vaccine 2019;37:3267–77.[doi:10.1016/j.vaccine.2019.04.075](http://dx.doi.org/10.1016/j.vaccine.2019.04.075)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31072733
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

54. Keller-Stanislawski B, Englund JA, Kang G, et al. Safety of immunization during pregnancy: a review of the evidence of selected inactivated and live attenuated vaccines. Vaccine 2014;32:7057–64.[doi:10.1016/j.vaccine.2014.09.052](http://dx.doi.org/10.1016/j.vaccine.2014.09.052)pmid:http://www.ncbi.nlm.nih.gov/pubmed/25285883
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2014.09.052&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=25285883&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

55. Griffin JB, Yu L, Watson D, et al. Pertussis immunisation in pregnancy safety (PIPs) study: a retrospective cohort study of safety outcomes in pregnant women vaccinated with Tdap vaccine. Vaccine 2018;36:5173–9.[doi:10.1016/j.vaccine.2018.07.011](http://dx.doi.org/10.1016/j.vaccine.2018.07.011)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30031662
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.vaccine.2018.07.011&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

56. Vazquez-Benitez G, Kharbanda EO, Naleway AL, et al. Risk of preterm or small-for-gestational-age birth after influenza vaccination during pregnancy: caveats when conducting retrospective observational studies. Am J Epidemiol 2016;184:176–86.[doi:10.1093/aje/kww043](http://dx.doi.org/10.1093/aje/kww043)pmid:http://www.ncbi.nlm.nih.gov/pubmed/27449414
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1093/aje/kww043&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=27449414&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

57. Bonhoeffer J, et al. Global alignment of immunization safety assessment in pregnancy–The Gaia project. Vaccine 2016;49:5993–7.
    
    

58. Kawai AT, et al. Developing a mother-infant cohort in Sentinel’s PRISM Program as a resource to monitor the safety of vaccine use during pregnancy. Pharmacoepidemiology and Drug Safety;26.
    
    

59. Sentinel common data model. Available: [https://www.sentinelinitiative.org/sentinel/data/distributed-database-common-data-model](https://www.sentinelinitiative.org/sentinel/data/distributed-database-common-data-model)
    
    

60. Guyatt GH, Oxman AD, Vist GE, et al. Grade: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6.[doi:10.1136/bmj.39489.470347.AD](http://dx.doi.org/10.1136/bmj.39489.470347.AD)pmid:http://www.ncbi.nlm.nih.gov/pubmed/18436948
    
    [FREE Full Text](http://gh.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzYvNzY1MC85MjQiO3M6NDoiYXRvbSI7czoyOToiL2JtamdoLzYvU3VwcGxfMi9lMDAzNTQwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

61. WHO. Guidance for the development ofevidence-based vaccinationrelated recommendations. Available: [https://www.who.int/immunization/sage/Guidelines\_development\_recommendations.pdf](https://www.who.int/immunization/sage/Guidelines_development_recommendations.pdf)
    
    

62. Real-World data (RWD) and real-world evidence (RWE) are playing an increasing role in health care decisions. Available: [https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence](https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence)
    
    

63. Frieden TR. Evidence for health decision making—beyond randomized, controlled trials. New England Journal of Medicine 2017;5:465–75.
    
    

64. Cave A, Kurz X, Arlett P. Real-World data for regulatory decision making: challenges and possible solutions for Europe. Clin Pharmacol Ther 2019;106:36–9.[doi:10.1002/cpt.1426](http://dx.doi.org/10.1002/cpt.1426)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30970161
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/cpt.1426&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

65. Hernán MA, Jick SS. Hepatitis B vaccination and multiple sclerosis: the jury is still out. Pharmacoepidemiol Drug Saf 2006;15:653–5.[doi:10.1002/pds.1159](http://dx.doi.org/10.1002/pds.1159)pmid:http://www.ncbi.nlm.nih.gov/pubmed/16245367
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/pds.1159&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=16245367&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000240665000007&link_type=ISI) 

66. Khosla S, White R, Medina J, et al. Real world evidence (RWE) - a disruptive innovation or the quiet evolution of medical evidence generation? F1000Res 2018;7:111.[doi:10.12688/f1000research.13585.2](http://dx.doi.org/10.12688/f1000research.13585.2)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30026923
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

67. DeStefano F. Vaccine safety Datalink Research Group. The vaccine safety Datalink project. Pharmacoepidemiol Drug Saf 2001;10:403–6.
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1002/pds.613&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=11802585&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000172704600010&link_type=ISI) 

68. VAccine monitoring Collaboration for Europe. Available: [https://vac4eu.org](https://vac4eu.org)
    
    

69. Lieu TA, Ray GT, Black SB, et al. Projected cost-effectiveness of pneumococcal conjugate vaccination of healthy infants and young children. JAMA 2000;283:1460–8.[doi:10.1001/jama.283.11.1460](http://dx.doi.org/10.1001/jama.283.11.1460)pmid:http://www.ncbi.nlm.nih.gov/pubmed/10732936
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1001/jama.283.11.1460&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=10732936&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 
    
    [Web of Science](http://gh.bmj.com/lookup/external-ref?access_num=000085696700031&link_type=ISI) 

70. Bruxvoort K, Sy LS, Luo Y, et al. Real-World evidence for regulatory decisions: concomitant administration of zoster vaccine live and pneumococcal polysaccharide vaccine. Am J Epidemiol 2018;187:1856–62.[doi:10.1093/aje/kwy076](http://dx.doi.org/10.1093/aje/kwy076)pmid:http://www.ncbi.nlm.nih.gov/pubmed/29659663
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

71. Black S. The costs and effectiveness of large phase III pre-licensure vaccine clinical trials. Expert Rev Vaccines 2015;14:1543–8.[doi:10.1586/14760584.2015.1091733](http://dx.doi.org/10.1586/14760584.2015.1091733)pmid:http://www.ncbi.nlm.nih.gov/pubmed/26414015
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

72. Glanz JM, Newcomer SR, Jackson ML, et al. White paper on studying the safety of the childhood immunization schedule in the vaccine safety Datalink. Vaccine 2016;34 Suppl 1:A1–29.[doi:10.1016/j.vaccine.2015.10.082](http://dx.doi.org/10.1016/j.vaccine.2015.10.082)pmid:http://www.ncbi.nlm.nih.gov/pubmed/26830300
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

73. D7.7 blueprint of a framework to rapidlyprovide scientific evidence on post marketingvaccination benefits and risks for informeddecisions. Available: [https://vac4eu.org/wp-content/uploads/2019/02/D77Blueprint.pdf](https://vac4eu.org/wp-content/uploads/2019/02/D77Blueprint.pdf)
    
    

74. Viswanathan M, Patnode CD, Berkman ND, et al. Recommendations for assessing the risk of bias in systematic reviews of health-care interventions. J Clin Epidemiol 2018;97:26–34.[doi:10.1016/j.jclinepi.2017.12.004](http://dx.doi.org/10.1016/j.jclinepi.2017.12.004)pmid:http://www.ncbi.nlm.nih.gov/pubmed/29248724
    
    [CrossRef](http://gh.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2017.12.004&link_type=DOI) 
    
    [PubMed](http://gh.bmj.com/lookup/external-ref?access_num=29248724&link_type=MED&atom=%2Fbmjgh%2F6%2FSuppl_2%2Fe003540.atom) 

75. Wang SV, Schneeweiss S, Berger ML, et al. Joint ISPE-ISPOR special Task force on real world evidence in health care decision making. reporting to improve reproducibility and facilitate validity assessment for healthcare database studies v1.0. Pharmacoepidemiol Drug Saf 2017.
    
    

76. Available: [http://www.advance-vaccines.eu/app/archivos/publicacion/67/ADVANCE\_D4%202\_appraisal%20safety%20methods\_final\_PU.pdf](http://www.advance-vaccines.eu/app/archivos/publicacion/67/ADVANCE\_D4%202_appraisal%20safety%20methods_final_PU.pdf)