Article Text
PDF
Abstract
Substandard and falsified (SF) medicines are a global issue contributing to antimicrobial resistance and causing economic and humanitarian harm. To direct law enforcement efficiently, halt the spread of SF medicines and antimicrobial resistance, academics, NGOs and government organisations use medicine quality sampling studies to estimate the prevalence of the problem. A systematic review of medicine quality studies was conducted to estimate how the methodological quality of these studies and SF prevalence has changed between 2013 and 2018. We also aimed to critique medicine sampling study methodologies, and the systematic review process which generates prevalence estimates. Based on 33 studies, the overall estimated median (Q1–Q3) prevalence of SF medicines appears to have remained high at 25% (7.7%–34%) compared with 28.5% in 2013. Furthermore, the methodological quality of prevalence studies has improved over the last 25 years. Definitive conclusions regarding the prevalence of SF medicines cannot be drawn due to the variability in sample sizes, consistency of design methods, and a lack of information concerning contextual factors affecting medicine quality studies. We contend that studies which present cumulative average prevalence figures are useful in a broad sense but could be improved to create more reliable estimates. We propose that medicine quality studies record the context of the study environment to allow systematic reviewers to compare like with like. Although, the academic rigour of medicine quality studies is improving, medicine sampling study limitations still exist. These limitations inhibit the accurate estimation of SF medicine prevalence which is needed to support detailed policy changes.
- systematic review
- health policy
- public health
- health systems evaluation
- health services research
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key questions
What is already known?
There are many medicine sampling studies conducted according to the MEDQUARG guidelines which generate SF medicine prevelence data for individual regions. These are often combined in systematic reviews and meta-analysis to generate overall continental or global SF prevelenace rates.
What are the new findings?
The prevelence of SF medicines appears to remain high at 25% (7.7%-34%). The quality of medicine sampling studies is improving according to the MEDQUARG guidelines. However, many medicine quality studies fail to record the contextual factors which may affect medicine quality results in a given region.
What do the new findings imply?
SF medicines are still a problem. Although the quality of measuring medicine quality is improving on an individual study basis, these studies would benefit from recording the specific context from which the samples have been collected. This woud add depth to the studies themselves but also allow systematic reviewers to amalgamate study results from similar contexts to create more accurate and granular prevelence data.
Introduction
Substandard medicines fail to meet their quality standards or specifications, or both. Falsified medicines are medical products that deliberately or fraudulently misrepresent their identity, composition or source.1 Substandard and falsified (SF) medicines are a ‘Wicked Problem’2–4 and a global issue with substantial estimated economic and public health impact.5 6 They affect countries across the full spectrum of financial levels, from low-income countries (LICs) to high-income countries (HICs).6–14 SF medicines exist in many different therapeutic classes, are sold with the explicit intent to deceive the end user of their origin, authenticity and efficacy9 15–21 and can have wide ranging effects including failure of treatment, toxicity and antimicrobial resistance.6 22 Arguably, the most morally reprehensible crime is the willing sale of falsified lifesaving treatments, such as antimalarial products, in areas where the disease is endemic, a practice that continues at great cost to human life.23 Systematic reviews and metanalysis prevalence studies provide some evidence and understanding of the extent of international pharmaceutical falsification. Such studies aim to provide insight as to where efforts should be directed24 and aim to supplement and correct the WHO estimates of SF medicine prevalence.25 Systematic reviews and metanalysis prevalence studies have been described as providing the only feasible means of estimating the burden of poor-quality medicines. This is especially true in LICs and lower-middle income countries (LMICs) where the cost of systematic national testing makes this practice unrealistic. However, the quality of these systematic reviews and metanalysis are only as good as the data that comprises them. Therefore, it is important that medicine quality studies are conducted rigorously, in a reproducible and effective manner.
In 2013, Almuzaini et al produced a systematic analysis of prevalence studies.26 This review included studies from June 1974 to January 2013. The primary aim of our study was to repeat the Almuzaini et al study methodology using contemporary data to understand how medicine sampling studies have changed in terms of methodological quality and the prevalence of reported SF medicines. Our second aim was to critique medicine quality studies and the use of cumulative average SF prevalence study data based on the results of our systematic review and similar systematic reviews and metanalysis, with the goal of suggesting ways to improve the current approach to medicine quality sampling studies, and therefore the systematic reviews and metanalysis which generate international SF medicine prevalence estimates.
Methods
We first conducted a systematic review of medicine quality studies according to the methodology of Almuzaini et al between January 2013 and December 2018 and combined these statistics as seen in other systematic reviews of medicine quality.5 19 26 A strength of the review conducted by Almuzaini et al is the use of a checklist by which to evaluate the methodology of the prevalence studies. These criteria are adopted from the Medicine Quality Assessment Reporting Guidelines checklist (MEDQUARG) of items to be addressed in reports of surveys of medicine quality.27 This review adhered to these same stringent guidelines, collected and amalgamated data and provided a regional representation of the areas in which prevalence studies had been conducted. This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.28
Two literature searches were carried out using the following scholarly databases: Embase, Medline and PubMed. Date ranges were set to 1 January 2013 to 31 December 2018. The first literature search repeats the search conducted in Almuzaini et al. Our search terms were identical in order to enable a valid comparison between the two studies. This search strategy included the MeSH terms associated with published prevalence studies as well as therapeutic areas often implicated with counterfeit, falsified and substandard medicines. These included: ‘counterfeit’, ‘fake’, ‘substandard’, ‘falsified’ which were subsequently combined with ‘drug’, ‘medicine’, ‘pharmaceutical’, ‘antimicrobial’, ‘antimalarial’ or ‘antibiotic’. Eligibility criteria for this review were any studies that evaluated the prevalence of substandard or counterfeit medicines within a defined area. Studies were excluded if they discussed analytical methods for the identification of counterfeit drugs or if these studies took the form of a review, opinion paper, letter or comment. These criteria exclude other systematic analysis and any analysis of studies on collated online data sets. The full search strategy and results can be found in figure 1, online supplementary table 1, figure 1, and table 2.
Supplemental material
Supplemental material
Flow diagram of search and review process for literature search 1.
The second literature search was conducted with the goal of updating the original search criteria of Almuzaini et al to include ‘unlicensed and unregistered’ drugs described in the recent Essential Medicines report.29 These search terms included ‘counterfeit’, ‘fake’, ‘substandard’, ‘falsified’, ‘unlicensed’ and ‘unregistered’ which were subsequently combined with ‘drug’, ‘medicine’, ‘pharmaceutical’, ‘antimicrobial’, ‘antimalarial’ or ‘antibiotic’. At the full-text screening stage, all articles that examined the prevalence of substandard/falsified drugs were removed—with the exception of the study by Khurelbat et al which provided data on the prevalence of both substandard and unregistered medicines.30 The eligibility criteria for this review were any studies that evaluated the prevalence of unlicensed or unregistered medicine use within a defined area. Studies which discussed analytical methods or took the form of review, opinion paper or letters were excluded. The full search strategy and results can be found in figure 2, online supplementary table 2, figure 2 and table 4.
Supplemental material
Flow diagram of search and review process for literature search 2.
Data collection process and data items
All abstracts were screened and evaluated against the inclusion and exclusion criteria. Full articles were then retrieved and the following data were extracted for literature search 1: place of the study, type of drugs sampled, sample size, percentage of substandard or falsified medicines, dosage forms included, chemical analysis performed, origin of the drugs and stated issues of SF medicines.
For literature search 2 some of the data captured in search 1 was not relevant, or was not included in those studies, therefore the following data were extracted: place of the study, type of drugs sampled, sample size, percentage of unlicensed/unregistered medicines, dosage forms included and origin of the drugs.
Studies were grouped geographically according to the World Bank classification of income level as follows: LIC, LMIC, upper middle-income country (UMIC) and high-income country (HIC). These classifications are up to date as of December 2018. Any study that contained information on more than one country was expressed as part of the mixed group. It is important to note that the aim of literature search 2 is to evaluate unlicensed and unregistered medicines only—not off-label use of licensed medicines. Therefore, all studies that examined only off-label use were also excluded at the full-text screening stage. Patient and public involvement was not required in this study.
Quality evaluation assessment
The MEDQUARG checklist of items to be addressed in reports of surveys of medicine quality were adopted to provide 12 criteria against which to assess the quality of each included study’s methodology. This was conducted in order to minimise bias from studies which employed a poor collection methodology. The MEDQUARG criteria used in this review is identical to that used in Almuzaini et al, therefore the methodological strength of the studies included in this review can be directly compared with that of the 2013 review. The cut-off point for a study to be considered methodologically sound and have its data included in this systematic review is a score of between 6 and 12 according to the MEDQUARG checklist criteria.26 27 Quality assessment was not carried out on the unlicensed / unregistered set of studies. This is because these studies were prospective, retrospective or observational analyses of existing patient records, as opposed to field studies of medicines collected from a range of outlets. Quality assessment was carried out by the first author, and papers were assessed against the same criteria detailed in Almuzaini et al.
Results
Literature search one: Studies on the prevalence of substandard and falsified medicines
A total of 34 studies concerning the prevalence of SF medicines were identified between 2013 and 2018. The number of articles screened and assessed are shown in figure 1. Thirty three studies passed quality assessment and were included in the subsequent analysis (figure 3). Studies included in literature search 1 were designed to select and test drug samples from a well-defined target geographical region. These samples were obtained from a range of outlets, which were divided into three distinct categories: public (ie, health facilities, hospitals and other official centres), private (licensed pharmacists) and informal (street vendors and market stalls). The median number (Q1–Q3) of medicine samples collected across the published studies examining SF medicine was 155 (60.25–675)(online supplementary table 3).
Quality assessment criteria for methodology of studies, prior to poor quality study removal.
32 out of 33 studies used random sampling in this review (97%), compared with 4/15 studies in the 2013 review (27%). Random sampling in this case refers to studies which had investigators collect samples from outlets that were randomly chosen from a complete or registered list or outlets in a defined area. The study performed by Osei-Safo et al in Accra and Lagos was the only study that didn't mention any form of randomised sampling.31
Information concerning the person collecting the samples was provided by 32/33 studies in this review (97%), with the study performed by the ACT Consortium in Tanzania being the only exception.32 This is an increase compared with Almuzaini et al. in which 12/15 studies provided the same information (80%).
Median (Q1–Q3) prevalence of SF medicines was similar in the LIC at 26% (11%–45.3%) and LMIC at 30% (9.3%–43.7%) studies. This is also the case in the review conducted by Almuzaini et al. This study encountered three studies from the UMIC which demonstrated a median (Q1–Q3) of 19% (0.5%–55.4%) group, whereas there were no UMIC country studies in the Almuzaini study. There was a decrease in the mixed group for this review with a median (Q1–Q3) of 13% (3.95%–28.5%) compared with the 2013 review (29%).
Averaging all data included in figure 4, it is possible to give an estimate of the median (Q1–Q3) prevalence of SF medicines across all studies included in this review. This review identifies that 25% (7.7%–34%) of medicines tested in these 33 studies were substandard or falsified compared with 28.5% in the 2013 study.
Prevalence of SF medicines according to country income classifications.LIC, low-income country; LMIC, lower middle-income country; SF, substandard and falsified; UMIC, upper middle-income country.
Drugs were assessed according to official pharmacopoeia guidelines. The most common issues with samples are displayed in figure 5. Inadequate amount of active pharmaceutical ingredient is the most common stated problem in both reviews. Impurity was the least common issue in both reviews. No active ingredient—a falsified medicine—was more common in the 2013 review (47%) compared with this review (18%). Dissolution failure was more common in this review (39% of studies) compared with the 2013 review (33% of studies).
Frequency of six different reported issues concerning the quality of tested medicines.
Changes in methodological quality of publications on the prevalence of SF medicines over time
Figure 6 plots all medicine quality studies from 1992 to 2018 with their MEDQUARG quality scores according to publication year. Linear regression shows a moderate positive correlation indicating that recent papers are more likely to score higher on the quality assessment.
The methodological quality of published prevalence studies over time. Linear regression analysis shows a positive correlation between year of publication and number of quality assessment criteria met (adjusted R2=0.38)(p<0.01).
Literature search two: Studies on the prevalence of unlicensed and unregistered medicines
A total of 47 studies on the prevalence of unlicensed/unregistered medicines were identified (figure 7). Data extracted from these studies varied compared with literature search one: packaging analysis was not performed, therefore labelled origin was not applicable; chemical testing did not take place and this was replaced by the type of study conducted on the data extraction regime; the percentage of unlicensed/unregistered medicines is given as either a percentage of prescriptions or as a percentage of the number of patients prescribed an unlicensed drug depending on what form the study presented these statistics in.
The range of the prevalence of unlicensed and unregistered medicines based on the World Bank classification of countries (by income level). HIC, high-income country; LIC, low-income country; LMIC, lower middle-income country.
Studies included in literature search 2 were designed to examine the prescription of unlicensed drugs by healthcare professionals within a well-defined region. In this study, the prescribing centres took the form of primary care hospitals, teaching hospitals, neonatal intensive care units, as well as a range of other public and private outlets.
The median (Q1–Q3) prevalence of unlicensed and unregistered medicines across all studies included in this review, was 11% (5%–26%). The median (Q1–Q3) number of unlicensed or unregistered medicines sampled across the published studies was 1054 (444–2273.5). Two studies did not state the number of unlicensed or unregistered medicines (online supplementary table 4). A full list of studies included in this review can be found in the attached appendices.
Discussion
Finding one: quality assessment of prevalence studies
An overwhelming majority of papers included in this review were successful in passing the MEDQUARG quality assessment. Furthermore, a number of papers stated explicitly that their study aimed to conform to the MEDQUARG guidelines published in 2009 by Newton et al.27 This is in contrast to the 2013 systematic review where 29/44 studies failed the same quality assessment criteria assessment.27 The introduction of these guidelines appears to have had a positive impact on the quality of prevalence study methodology and if this upward trend continues, the accuracy in estimating the extent of the problem of SF medicines in specific regions may improve. Empirically, this methodological improvement is likely to aid researchers and Non-Governmental Organisations (NGOs) to more accurately estimate the prevalence of SF medicines in a given region.
Finding two: prevalence of substandard and falsified (SF) medicines and context
Countries that fall into the LIC and LMIC income categories still have a problem with SF medicines penetrating their supply chains. It has been previously established that SF medicines pose an even greater threat in LICs and LMICs due to the greater need for life-saving treatments for infectious diseases in these geographical areas.33 34 The median (Q1–Q3) prevalence in LICs has remained high in our review, and overall prevalence has also remained high at 25% (7.7%–34%), with little change since the study of Almuzaini in 2013. Comparatively, a systematic review and metanalysis study was published by Ozawa et al in 2018 included studies which had collected 50 samples or greater; mean prevalence data averages were weighted for the MEDQUARG criteria, which found that 13.6% of medicines were of poor quality, with regional prevalence of 18.7% in Africa and 13.7% in Asia. A similar study by the WHO estimated that 10.5% of medicines in LIC and middle-income countries are substandard or falsified. These three systematic reviews have resulted in different estimates.
A limitation of these studies is that they use different inclusion and exclusion criteria and cite a single prevalence rate based on data across many years and failed to capture a recent snapshot or the dynamic nature of the SF medicine issue. An especially positive aspect of the WHO report is that it understands the limitations of its method but describes it as the only available measure, considering the cost and burden of systematic national testing. Furthermore, the results of these systematic reviews are only as reliable as the original study data to which they refer. The reliability issues affecting the amalgamation of these studies concern study methodology, rigour, sample sizes and the intervals at which these studies are performed. There are also several confounding contextual factors which could affect the outcome of amalgamating these sampling studies. These factors include the time of year, sociocultural effects, the geographical location of the study or awareness of the study and communication between social networks.5 There is a sense that systematic reviews and metanalysis are not combining like with like, a point which is supported by Cuzzolin et al35 and Mackey36 who mentioned a recurring issue with systematic reviews of SF medicines due to different authors, different sampling methods and sizes, as well as different settings within the same country. It is difficult to equate studies to representative estimates of prevalence within an entire region.
Case study perspective
The individual small scale medicine quality studies described in this systematic review and other systematic reviews like Ozawa et al 2018, could be described as case studies; which can be defined as ‘An intensive study about a person, a group of people or a unit, which is aimed to generalize over several units’.37 Case studies often include a detailed summary of the case study context. Medicine quality studies are useful as case studies; however, they often fail to appreciate the surrounding context from which samples are collected. These case studies are useful for descriptive purposes, but we advocate that future medicine quality studies should not only strive to improve their methodologies, but also record the specific context of the ecosystem or environment from which they draw their conclusions. This might help to improve the ‘incomplete picture’ that is being drawn from the systematic review and metanalysis of medicine quality studies.36
Context
Johns discusses organisational context and the importance of considering findings in light of the study context.38 We propose that applying the omnibus and discrete context framework (figure 8) to the medicine quality study research area could help to build a richer picture of medicine sampling study results and help to manage some of their limitations.
An illustration of Johns Omnibus and discrete context framework.
We propose a standard set of questions based on Johns (2006) framework (figure 8) to capture this contextual data, and to support its implementation in the medicine quality sampling research field. These questions could be used in medicine quality sampling studies to contextualise research findings.
Omnibus context
WHO
Who was involved in the research? Who has funded the research? Who has provided the sample? Who has collected the sample? Who has tested the sample? It might also be useful to record who was aware that the research study was being performed?
WHERE
Where has the study been performed nationally, regionally and locally and what significant contextual factors (social political or environmental) were affecting the region at the time?
WHEN
When was the data for each part of the study collected? What was the time of year, was there a recent significant event which may affect study results? (social, political or environmental) What were the drivers for this study?
WHY
Why was this research conducted i.e. What were the driving forces behind this study?
Discrete context
Task
Were the three levels of individuals involved in the study autonomous that is, the sample provider, the sample collector, the sample tester? Did these individuals have any conflicts of interest?
Were there any uncertainties associated with the processes of collection, provision or testing of the sample?
Were the three levels of individuals accountable for their actions? Was the provider accountable to a reliable authority, was the collector accountable to the research team, was the tester accountable to the research team?
Were there adequate resources in place to conduct the study? If not, how did this affect the three levels of individuals and their ability to undertake the work?
Social
What were the dominant social groups in the sampling context which could have affected the study outcomes? These groups could include political groups, religious groups, criminal groups.
What was the density of these groups within the study region?
How were these groups structured?
And what influence did these social groups have on the area? And how was the group influence affected by the Where and When dimensions of the omnibus context?
Physical
Were there any physical factors affecting this study? How far did collected samples have to travel to be tested? Did this distance affect the product, under what conditions was the sample stored and transported?
We propose that recording the medicine sampling study context, at the time of data collection, can help researchers to appreciate the factors which may be affecting SF medicine prevalence in a given region at a given time. For example, COVID-19 is likely to increase the prevalence of certain SF medicines in affected regions in late 2019 and throughout 2020.39 Therefore, we would advise that medicine sampling studies being conducted during this period should appreciate the sudden and major context change that is the COVID-19 pandemic and record this factor in their research study as part of a wider contextual analysis.
The proposed questions could be answered by interviewing local community networks in the study environment and/or by conducting desk-based research and should be considered at the study planning stage. The capturing of contextual study data would not only add depth to these studies, but it would also give systematic reviewers the opportunity to adapt inclusion and exclusion criteria to include studies which are both methodologically and contextually similar. In gaining a better understanding of these contextual factors, researchers conducting systematic reviews and metanalysis will be better informed of whether research study outcomes can be combined to create cumulative data. If studies begin to be conducted using similar methodologies while recording the study context, the systematic reviews could state more specific limitations of their reviews and metanalysis.
Medicine quality study scale
Another way to improve the reliability of these individual sampling studies themselves is to ensure that these studies, which are often ‘case study like’ in size, are performed on a much larger scale. Of course, this is easier said than done for complexity and financial reasons. However, considering the recent global call to action on SF medicines published by the Lancet Global health, which includes over 150 authors from academia, policy, private companies and advocacy groups, perhaps now is the time to pool resources to conduct fewer but more robust and wider scale sampling studies.40 Digital methods of systematic, mass authentication could be one way to generate more reliable, medicine quality sampling data at scale.3 41–44 This may take the form of serialisation and authentication of barcodes, mobile phone authentication or the use of low-cost pill scanners by consumers to identify falsified medicines.3 43 45–48
Finding three: Prevalence of unlicensed and unregistered medicines
Since the Almuzaini et al 2013 publication did not assess unlicensed and unregistered medicines, there is no basis for comparison within that paper. However, a 2006 systematic review examined the use of unlicensed, unregistered and off-label medicines: the review sampled 41 studies in 6 HICs, reporting an average (mean) prevalence of 12% unlicensed medicines.49 This figure is similar to the results obtained in HICs in this study (8%, median and 18% mean). However, these estimates are based on limited data and further research is required to estimate the prevalence of unlicensed and unregistered medicines. Several unlicensed/unregistered prevalence studies did not report the exact reasoning for a drug not being approved but used regardless. A suggestion is given in the study performed by Ramadaniati et al—‘drug reformulation (so-called extemporaneously prepared drugs) was a common reason for unlicensed use. Extemporaneously prepared drugs occur due to the fact that the availability of a specific drug or dosage or formulation suitable for paediatric patients was very limited’.50 Man et al found in their study that previous positive experiences of the prescribing physician could explain the use of unapproved medicines in hospital environments. They also offer the explanation that the unavailability of paediatric oral liquid formulations could explain the use of unapproved extemporaneously prepared drugs in their place.51 Unregistered and unlicensed medicines is an under researched area and considering the increase in global medicine shortages, the use of unlicensed and unregistered medicine is open to growth. Therefore, further research concerning the prevalence of unlicensed and unregistered medicines, and the most common contexts in which these medicines are used may be beneficial contributions to this research domain.
Limitations
This review is limited by the methodological strength of the medicine sampling studies which it includes. The International Pharmaceutical Abstracts were searched in accordance with the Almuzaini method. As it yielded no results from 2013 to 2018, it was removed from the methodology. The systematic review in this study would have benefited from an independent check from a second person. However, as the study was unfunded, this was not possible.
Conclusion
Substandard, falsified, unregistered and unlicensed medicines are still an issue, especially in LIC and middle-income countries. Having gained a better understanding of medicine quality studies and the process of systematic review, metanalysis and extent literature, we conclude that the quality of medicine quality studies has improved according to the MEDQUARG guidelines. However, the amalgamation of prevalence data has many limitations. We propose that as well as recording the methodological quality of medicine quality studies, these studies could benefit from recording the context of the ecosystem or environment from which the samples are collected, which may help systematic reviewers better appreciate the differences in medicine quality studies, consider the contextual difference in their inclusion and exclusion criteria and begin to amalgamate studies with methodological and contextual similarities. In order to push this field forward and to tackle the ‘Wicked Problem’ of substandard and falsified medicine, we urge governments, academics (medical and social scientists), industry, healthcare and NGO stakeholders to record the contexts of their medicine quality studies.
References
Footnotes
Handling editor Seye Abimbola
Twitter @bernard_naughton
Collaborators Oksana Pyzik Senior Teaching Fellow UCL School of Pharmacy and Founder of UCL Fight the Fakes.
Michael Munday, Professor of Pharmaceutical Biochemistry at UCL School of Pharmacy.
Contributors BDN proposed the research project idea and design. DMcM gathered the data and analysed the data. The data analysis was checked by BDN. BDN and DMcM wrote the manuscript. BDN made revisions. BDN and DMcM reviewed the paper before submission and resubmission.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Disclaimer The opinions expressed are those of the authors and do not necessarily represent the opinion of their employers.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information. Please see supplementary file for study data.