Review Article
Systematic review finds overlapping reviews were not mentioned in every other overview

https://doi.org/10.1016/j.jclinepi.2013.11.007Get rights and content

Abstract

Objectives

The objective of this study was to determine if the authors mention overlapping reviews in overviews (reviews of reviews). In addition, we aimed to calculate the actual overlap in published overviews using newly introduced, validated measures.

Study Design and Settings

We systematically searched for overviews from 2009 to 2011. Reviews included in the overviews were obtained. Tables (review × primary publication) were generated for each overview. The first occurrence of a primary publication is defined as the index publication. We calculated the “corrected covered area” (CCA) as a measure of overlap by dividing the frequency of repeated occurrences of the index publication in other reviews by the product of index publications and reviews, reduced by the number of index publications. Subgroup analyses were performed to investigate further differences in the overviews.

Results

Only 32 of 60 overviews mentioned overlaps. The median CCA was 4.0. Validation of the CCA and other overlap measures was in accordance with our predefined hypotheses. The degree of overlap tended to be higher in health technology assessment reports than in journal publications and was higher with increasing numbers of publications.

Conclusions

Overlaps must be reported in well-conducted overviews, and this can comprehensively be accomplished using the CCA method.

Introduction

What is new?

Key findings

  1. Almost half of the overviews (53.3%) mentioned overlaps, whereas the remaining overviews did not.

What this adds to what was known?
  1. This is the first systematic analysis of overlaps in reviews of reviews.

  2. Development and validation of a measure (corrected covered area [CCA]) to calculate the actual degree of overlap in overviews.

What is the implication and what should change now?
  1. Insufficient reporting of the quality of systematic reviews complicates the production of overviews, in particular with respect to overlaps.

  2. Overlaps must be reported in well-conducted overviews, and this can comprehensively be done using the CCA method.

Overviews (reviews of reviews), as a new type of evidence synthesis, have recently gained more interest, such that the number of published overviews is steadily increasing [1]. It is possible that overviews are becoming more prevalent because overviews have potential advantages over systematic reviews (SRs). For example, overviews enable data obtained from different interventions or conditions to be compared, which provides decision makers with a broader summary of the current information available. This is a limitation of SRs, which may be overcome by using overviews [2]. Furthermore, overviews can compare the findings of several reviews and determine the reasons for conflicting results. By identifying the reasons for discordance, users are able to base their decisions on the most current, reliable, and suitable data for their situation [3], [4].

It has been stated that many of the methodological standards for SRs can also be applied to overviews [5]. However, little guidance is available for authors on how to conduct methodologically sound overviews. Interestingly, a descriptive analysis concluded that overviews often have limited rigor [4].

Decisions in health care should be based on all of the available evidence to draw reliable conclusions and to support policy making. Therefore, we most often rely on the SRs [6]. When conducting an overview, one might argue that a decision should be based on an enormous body of evidence. Even if this holds to be true, it may be difficult and challenging to survey all of the available evidence that is gained from primary studies mainly because they are often included in more than one review. Additionally, a meta-analysis of meta-analyses may also be difficult to conduct because many of the primary studies will usually be included in more than one meta-analysis. Therefore, pooling the results of all of the reviews would give disproportionate statistical power to multiple primary studies [2]. An informal analysis that sums the results of the reviews could also introduce significant overlap and result in many primary studies being included more than once, which would lead to biased results. This problem should be addressed by developing standard methods for authors of overviews to follow [7].

However, to the best of our knowledge, the degree of overlap in overviews has not been examined systematically. Thus, the amount of overlapping data in overviews remains unknown. First, we aimed to determine whether authors mentioned overlaps in their overviews, and if so, we examined how the authors dealt with these overlaps. Second, we examined the actual overlap in published overviews and suggested potential measures for handling these overlaps.

Section snippets

Literature search

In November 2011, we performed a systematic search for overviews of reviews. An update was performed in May 2012. We searched the databases of Medline, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Physiotherapy Evidence Database (PEDro), and all of the databases of the Cochrane library. Because no subsets, subject headings, or search filters for overviews were available thus far, we searched the databases using text words. We adapted this search strategy based on

Search results

We identified 1,993 potentially relevant studies. After checking the titles and abstracts, we retrieved 273 full-text articles for further consideration. Of these, 200 were excluded for various reasons, namely 44 did not explicitly search systematically for reviews and named at least one database, 100 did not critically assess the included literature, 6 did not use the reviews in evidence synthesis, 4 had a methodological focus, 10 were neither written in English nor in German, 3 were

Discussion

There is a substantial overlap of reviews being included in overviews. However, only half of all authors of overviews acknowledge this problem. The manner in which overlaps are dealt with in the literature does not depend on the degree of overlap. The CCA measure showed a statistically significant difference in the degree of overlap, but the absolute difference was small. The degree of overlap was higher in HTA reports than in journal publications. This is perhaps because the purpose of HTA

Conclusion

Authors often disregard overlaps rather than address the issue in their work, which may lead to false conclusions. Thus, all producers of overviews should analyze the overlaps and report their analysis. Reporting should be done even if the amount of overlap is small and unlikely to have an impact on the conclusion. Otherwise consumers will not know whether there is no meaningful overlap or if the authors simply did not account of it. Consequently, overlaps should be reported by default, and

Acknowledgments

The authors are thankful to all the authors who submitted the requested files or answered our clarification questions.

References (18)

  • D. Pieper et al.

    Overviews of reviews often have limited rigor: a systematic review

    J Clin Epidemiol

    (2012)
  • M. Velasco Garrido et al.

    Developing health technology assessment to address health care system needs

    Health Policy

    (2010)
  • A. Leizorovicz et al.

    Meta-analysis and multiple publication of clinical trial reports

    Lancet

    (1992)
  • L. Hartling et al.

    A descriptive analysis of overviews of reviews published between 2000 and 2011

    PLoS One

    (2012)
  • V. Smith et al.

    Methodology in conducting a systematic review of systematic reviews of healthcare interventions

    BMC Med Res Methodol

    (2011)
  • H. Cooper et al.

    The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship

    Am Psychol

    (2012)
  • J. Higgins et al.

    Cochrane handbook for systematic reviews of interventions

    (2011)
  • D.J. Cook et al.

    Systematic reviews: synthesis of best evidence for clinical decisions

    Ann Intern Med

    (1997)
  • D. Thomson et al.

    The evolution of a new publication type: steps and challenges of producing overviews of reviews

    Res Synth Methods

    (2010)
There are more references available in the full text version of this article.

Cited by (0)

The authors of this article received no financial support for this research and/or authorship of this article.

All of the authors declare that they have no conflicts of interest to disclose.

View full text