Original Article
Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry

https://doi.org/10.1016/j.jclinepi.2007.11.010Get rights and content

Abstract

Objectives

To present the contour-enhanced funnel plot as an aid to differentiating asymmetry due to publication bias from that due to other factors.

Study Design and Setting

An enhancement to the usual funnel plot is proposed that allows the statistical significance of study estimates to be considered. Contour lines indicating conventional milestones in levels of statistical significance (e.g., <0.01, <0.05, <0.1) are added to funnel plots.

Results

This contour overlay aids the interpretation of the funnel plot. For example, if studies appear to be missing in areas of statistical nonsignificance, then this adds credence to the possibility that the asymmetry is due to publication bias. Conversely, if the supposed missing studies are in areas of higher statistical significance, this would suggest the cause of the asymmetry may be more likely to be due to factors other than publication bias, such as variable study quality.

Conclusions

We believe this enhancement to funnel plots (i) is simple to implement, (ii) is widely applicable, (iii) greatly improves interpretability, and (iv) should be used routinely.

Introduction

Publication bias describes the tendency for studies reporting uninteresting or unfavorable results to be less likely to be published [1]. Meta-analysis of published papers is likely to be affected by publication bias [2]. Although the precise mechanisms of publication bias are unknown, evidence suggests that statistical significance of the main outcome in a study is the most important factor; nonsignificant studies are less likely to be published [3], [4], [5]. Absence of statistical significance has also been identified as a factor in a related bias, outcome reporting bias: where multiple outcomes are investigated in a study, but only the outcomes with interesting or statistically significant results are reported in the paper [6], [7], [8].

The funnel plot is the simplest of all techniques to help assess possible publication bias. The effect estimate from each study in the meta-analysis is plotted against some measure of precision from that study [1]. Estimates of effect from smaller studies are more variable than those from the larger studies and so scatter more widely at the base of the plot creating, in the absence of bias, a symmetrical funnel shape. If smaller, nonstatistically significant studies tend to remain unpublished then an asymmetrical shape may be observed [9].

However, publication bias is not the only possible cause of asymmetry observed in a funnel plot [10]. Any factor which is associated with both study effect and study size could confound the true association and cause an asymmetric funnel. For example, if there is an indication of poorer study design in smaller studies [10] and this poor study design leads to systematic exaggeration of effect [11], this could manifest itself as asymmetry on a funnel plot because studies near the top of the plot (small bias) will have smaller effect sizes, on average, than those at the bottom (large bias). We introduce a graphical aid for the interpretation of funnel plots to help differentiate asymmetry caused by statistical significance related publication bias from that caused by other factors.

Section snippets

Funnel plots and contour-enhanced funnel plots

Contour-enhanced funnel plots display areas of statistical significance on a funnel plot. If it is assumed that the treatment effect in each study is normally distributed, then the significance of any effect size can be calculated from the effect size and the standard error. Because effect size and standard error (or some functions of it) are the two axes of a funnel plot, the (two-sided) statistical significance of any point on a funnel plot can be calculated. Thus, contours representing

Discussion

Tools to aid interpretation of funnel plots are clearly needed [22]. A recent study suggests that correct identification of the presence or absence of publication bias using funnel plots is poor [23] with only 52.5% of funnel plots correctly assessed for publication bias in simulated meta-analysis data sets. We believe contour-enhanced funnel plots greatly help interpretation of funnel plot asymmetry.

A number of points need consideration regarding the contour-enhanced funnel plots presented

Acknowledgements

JP was funded through a UK Department of Health Evidence Synthesis Award while undertaking this work. The funding source had no role in any aspect of this study.

References (36)

  • P.R. Williamson et al.

    Outcome selection bias in meta-analysis

    Stat Methods Med Res

    (2005)
  • A.W. Chan et al.

    Empirical evidence for selective reporting of outcomes in randomized trials—Comparison of Protocols to published articles

    JAMA

    (2004)
  • R.J. Light et al.

    Summing up: the science of reviewing research

    (1984)
  • M. Egger et al.

    Bias in meta-analysis detected by a simple, graphical test

    BMJ

    (1997)
  • K.F. Schulz et al.

    Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials

    JAMA

    (1995)
  • Cochrane Database of Systematic Reviews. In: The Cochrane Collaboration. Cochrane Library, 1998; Issue 3,...
  • A.J. Sutton et al.

    Empirical assessment of effect of publication bias on meta-analyses

    BMJ

    (2000)
  • S. Duval et al.

    Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis

    Biometrics

    (2000)
  • Cited by (1252)

    • Preventing postpartum hemorrhage: A network meta-analysis on routes of administration of uterotonics

      2024, European Journal of Obstetrics and Gynecology and Reproductive Biology
    View all citing articles on Scopus
    View full text