Introduction Vaccines for SARS-CoV-2 have been accessible to the public since December 2020. However, only 58.3% of Americans are fully vaccinated as of 5 November 2021. Numerous studies have supported YouTube as a source of both reliable and misleading information during the COVID-19 pandemic. Misinformation regarding the safety and efficacy of COVID-19 vaccines has negatively impacted vaccination intent. To date, the literature lacks a systematic evaluation of YouTube’s content on COVID-19 vaccination using validated scoring tools. The objective of this study was to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 vaccination.
Methods A search on YouTube was performed on 21 July 2021, using keywords ‘COVID-19 vaccine’ on a cleared-cache web browser. Search results were sorted by ‘views’, and the top 150 most-viewed videos were collected and analysed. Duplicate, non-English, non-audiovisual, exceeding 1-hour duration, or videos unrelated to COVID-19 vaccine were excluded. The primary outcome was usability and reliability of videos, analysed using the modified DISCERN (mDISCERN) score, the modified Journal of the American Medical Association (mJAMA) score and the COVID-19 Vaccine Score (CVS).
Results Approximately 11% of YouTube’s most viewed videos on COVID-19 vaccines, accounting for 18 million views, contradicted information from the WHO or the Centers for Disease Control and Prevention. Videos containing non-factual information had significantly lower mDISCERN (p<0.001), mJAMA (p<0.01) and CVS (p<0.001) scores compared with videos with factual information. Videos from government sources had higher mJAMA and CVS scores, but averaged three times the ratio of dislikes to likes, while videos containing non-factual information averaged 14 times more likes than dislikes.
Conclusion As the COVID-19 pandemic evolves, widespread adoption of vaccination is essential in reducing morbidity, mortality, and returning to some semblance of normalcy. Providing high-quality and engaging health information from reputable sources is essential in addressing vaccine hesitancy.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
What is already known?
Throughout the COVID-19 pandemic, videos on social media questioning the efficacy and safety of COVID-19 vaccines have become more numerous and widespread. It has previously been reported that 27.5% of the most viewed YouTube videos on COVID-19 contained misinformation.
Information from reputable sources, such as the WHO or the Centers for Disease Control and Prevention, is under-represented on social media.
What are the new findings?
Approximately 11% of YouTube’s most viewed videos on COVID-19 vaccines, accounting for over 18 million views, contradicted the reference standard from the WHO, or the Centers for Disease Control and Prevention.
Information from reputable sources may be perceived as less favourable by the public, when proxied by the ratio of likes to dislikes. On average, YouTube videos from government sources had triple the ratio of dislikes to likes, while entertainment videos had 14 times the ratio of likes to dislikes.
What do the new findings imply?
Despite efforts by YouTube in combatting health misinformation, a substantial portion of highly viewed content contains misinformation regarding COVID-19 vaccines continues to be hosted by the second-largest social media platform.
Public health officials and clinicians should recognise the impact of YouTube and other social media platforms as sources of health information and misinformation for patients and leverage these platforms to disseminate high-quality content to combat misinformation given poor vaccination uptake.
COVID-19, caused by the novel SARS-CoV-2, was first noted in December 2019 and has evolved into a global pandemic with over 246 million cases worldwide and nearly 5 million deaths to date.1 Vaccines against SARS-CoV-2 became accessible to the public since December 2020, after being granted emergency use authorisation status by the Federal Drug Administration.2 Despite this, as of 5 November 2021, only 58.3% of Americans are fully vaccinated.3 This is concerning as public trust in vaccination programmes is paramount in reducing morbidity and mortality, limiting the spread of highly transmissible SARS-CoV-2 variants, and putting an end to this pandemic.4
In the digital age, more individuals are relying on the internet and social media as sources of healthcare-related information.5–8 In particular, YouTube is one of the most predominant social media platforms with billions of views daily and is the second most used search engine after Google.9 This readily accessible platform offers multiple genres of content, including entertainment, news, consumers, and education, and allows its users to interact with each other in real time.
Numerous studies have supported YouTube as a source of both reliable and misleading information during the COVID-19 pandemic.10–12 A previous study demonstrated that 27.5% of the most viewed YouTube videos on COVID-19 contained misinformation.11 As vaccine misinformation has been shown to negatively impact vaccination intent,12 propagation of misinformation regarding the safety and efficacy of COVID-19 vaccines on large platforms, such as YouTube, has potential devastating consequences on public health.
To date, the literature lacks a systematic evaluation of YouTube as a source of information on COVID-19 vaccination using validated scoring tools. The objective of this study is to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 vaccination.
A YouTube search was performed on 21 July 2021, using keywords ‘COVID-19 vaccine’ on a cleared-cache web browser. Search results were sorted by ‘views’, and 150 videos with the greatest number of views were collected and analysed.
Study selection and data extraction
Three reviewers (HO-YL, EP and OB-L) reviewed and analysed the 150 videos. Videos that were duplicate, non-English, non-audio, non-visual or unrelated to COVID-19 vaccine were excluded. Discrepancies were resolved by discussion for consensus.13 Video descriptive characteristics including title, hyperlink, number of views, number of likes and dislikes, upload date, duration and publishing category, were collected.
The reliability and quality of video content were assessed using the modified DISCERN (mDISCERN) score (online supplemental table S1) and the modified Journal of the American Medical Association (mJAMA) criteria (online supplemental table S2).14 15 A novel five-point COVID-19 Vaccine Score (CVS), developed based on previously published work, was used to evaluate the usefulness of video content for the average viewer (online supplemental table S3).16
Patient and public involvement
No patients or public were involved in any way in this study.
Following data extraction, two independent reviewers (EP and OB-L) categorised all videos into videos containing factual or non-factual information, and into their video publishing category, in duplicate, with a third reviewer (HO-YL) consultation for consensus. Video sources were categorised into healthcare professionals, network news, education, internet news, consumers, entertainment, government sources and newspapers according to previously published work (online supplemental table S4).17 Videos were considered non-factual if they contained one or more non-factual statements, defined as contradictory to guidelines published by public health agencies (ie, WHO or Centers for Disease Control and Prevention) at the time of video publication.18 19 The CVS score was calculated as follows: if a CVS criterion was discussed, it was given a score of 1 if factually correct, a score of 0.5 if ambiguous and 0 if non-factual (online supplemental table S3). The total score was then divided by the number of items discussed (up to a maximum of 5) and expressed as a proportion. mDISCERN and mJAMA scores were calculated with one point given for each criterion correctly addressed (online supplemental tables S1 and S2).
Descriptive statistics including two tailed t-tests and Mann-Whitney U tests compared factual videos to non-factual videos, with a significance level of p<0.01 to offset multiple hypothesis testing. Percentages, means and ratios compared videos from different sources. All analyses were performed using Microsoft Excel and Mathworks Matlab software (V.R2019b).
Of 150 videos screened, 122 videos (81%) were included. A total of 28 (19%) videos were excluded: 14 for not discussing any CVS criteria, 13 for non-English and one for non-audio (online supplemental figure S1).
Source of videos
Videos were classified into eight categories (figure 1). The total viewership at time of data collection was 169 446 382 views. 24.1% of views were from videos produced by healthcare professionals, 20.9% from network news, 16.9% from education, 10.2% from internet news, 8.4% from consumers, 8.1% from entertainment, 7.3% from government sources and 4.2% from newspapers.
Government videos had the highest average CVS score (0.98) and number of CVS items discussed per minute of video (1.89 items/min). Internet news videos had the lowest CVS score (0.83), and consumer videos had the lowest CVS items discussed per minute (0.20 items/min). Internet news also had the lowest mDISCERN and mJAMA scores, whereas government videos had the highest mJAMA scores and educational videos had the highest mDISCERN scores (table 1).
Government videos had the shortest average time since publication of 65 days, compared with 310 days for newspaper videos. Entertainment videos on average had 14.4 times more likes than dislikes, whereas government videos averaged more dislikes than likes, with 3.1 dislikes to every like.
Videos with factual versus non-factual information
Thirteen videos (10.7%) contained non-factual information and accounted for 11.0% of total viewership, while 109 videos (89.3%) contained only factual information. Of the 13 non-factual videos, 4 were from network news, three from healthcare professionals, two each from education and internet news, and one each from newspaper and consumer sources. Two of the 13 videos (15.4%) containing non-factual information did not contain a YouTube information icon prompting the viewer to seek additional information on COVID-19 vaccination.
When comparing the two groups, there were no significant differences in views per video (p=0.94), likes per video (p=0.40), dislikes per video (p=0.36), duration (p=0.98) and mean days since publication (p=0.56) (table 2). Videos with non-factual information on average had a greater median ratio of likes to dislikes, compared with videos with factual information (13.89 vs 2.33, p<0.01).
Videos with non-factual information were found to have lower CVS (p<0.001), mDISCERN (p<0.001) and mJAMA (p<0.01) scores, compared with videos with factual information. Classification and examples of non-factual statements are shown in table 3.
This study highlights that approximately 11% of YouTube’s most viewed videos on COVID-19 vaccines contradicted the reference standard,18 19 reaching over 18 million viewers. Our findings were significantly lower than other studies reporting 27.5% and 45% of videos containing non-factual information.11 16 This discrepancy is likely due to increased awareness of the impact of social media on disseminating misinformation that has resulted in YouTube’s implementation of a COVID-19 medical misinformation policy in May 2020.20 Furthermore, government sources are increasingly leveraging YouTube as a valuable opportunity to disseminate vaccine information, with an over twofold increase in the percentage of government videos in this study compared with a study published earlier in the pandemic on COVID-19 misinformation.11 Despite producing the most reliable content, videos from government sources had over three times the ratio of dislikes to likes. In comparison, entertainment videos had a ratio of over 14 times the number of likes to dislikes. Similarly, the median ratio of likes to dislikes for videos containing non-factual information was almost six times higher than that of videos containing factual information. This suggests the presence of an active community of users who are sceptical of the COVID-19 vaccine and government sources. It is important to note that this may also represent a selection bias, as a previous study demonstrated that participants who relied on private sources for COVID-19 information, such as network news and social media, were less knowledgeable about COVID-19 and less likely to adhere to social distancing guidelines.21 Additionally, another study demonstrated that YouTube users were less likely to receive the COVID-19 vaccine compared with non-users.22
Developing timely strategies to monitor and optimise the quality of information on YouTube during the COVID-19 pandemic is crucial for two reasons. First, it may be difficult for the YouTube algorithm to discern the accuracy and reliability of information conveyed. For example, the infamous COVID-19 film, Plandemic, which contained significant misinformation and conspiracy theories, was only removed from YouTube after public outcry, but not before it had already garnered millions of views at the time of removal, with the potential for serious consequences.23 Second, the YouTube algorithm allegedly makes recommendations for videos with a higher number of views and a greater proportion of likes.24 25 Moreover, the algorithm also allegedly adjusts video recommendations based on the user’s history of videos watched.25 Thus, it is plausible that individuals who encounter an initial video containing misinformation, may continue to receive recommendations for other videos containing similar misinformation. To address these concerns, in 2020, YouTube implemented an automated flagging system to demonetise and remove videos containing COVID-19 misinformation.26 However, this system’s reliance on automation has resulted in enforcement errors, such as mistakenly removing videos which sought to refute COVID-19 misinformation.27 Potential improvements could include increasing viewership of reputable government sources, as they have been shown to be the most reliable, by suggesting these videos to users. Furthermore, given that over 85% of those aged 13–17 years old use YouTube,28 algorithms suggesting youth-friendly government videos on COVID-19 vaccination to this audience may help increase vaccine compliance in the younger population, which has been reported to be much lower than among adults.29
As the COVID-19 pandemic evolves, widespread vaccination is essential in reducing morbidity, mortality and returning to some semblance of normalcy. Providing high-quality health information from reputable sources that is engaging to a variety of viewers will be essential in addressing vaccine hesitancy and improving the public’s understanding and adherence to public health measures. Policy-makers should consider implementing mandates surrounding medical misinformation on social media platforms. Public health officials and clinicians should recognise the impact of YouTube and other similar social media sites as sources of health information for patients from all demographic backgrounds and leverage these platforms to disseminate high-quality content to combat misinformation.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
Patient consent for publication
Ethical approval was not required for this study as it does not involve human or animal subjects.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
HO-YL, EP, OB-L and MGT are joint first authors.
Handling editor Seye Abimbola
HO-YL, EP, OB-L and MGT contributed equally.
Contributors Conception and design: HO-YL. Data collection: HO-YL, EP and OB-L. Analysis and interpretation: all authors. Manuscript writing: HO-YL, EP, OB-L and MGT. Critical revision of the article: all authors. Final approval of the article: all authors. Obtained funding: not applicable. Overall responsibility and gurantor: MGK.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.