Original ArticleContour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry
Introduction
Publication bias describes the tendency for studies reporting uninteresting or unfavorable results to be less likely to be published [1]. Meta-analysis of published papers is likely to be affected by publication bias [2]. Although the precise mechanisms of publication bias are unknown, evidence suggests that statistical significance of the main outcome in a study is the most important factor; nonsignificant studies are less likely to be published [3], [4], [5]. Absence of statistical significance has also been identified as a factor in a related bias, outcome reporting bias: where multiple outcomes are investigated in a study, but only the outcomes with interesting or statistically significant results are reported in the paper [6], [7], [8].
The funnel plot is the simplest of all techniques to help assess possible publication bias. The effect estimate from each study in the meta-analysis is plotted against some measure of precision from that study [1]. Estimates of effect from smaller studies are more variable than those from the larger studies and so scatter more widely at the base of the plot creating, in the absence of bias, a symmetrical funnel shape. If smaller, nonstatistically significant studies tend to remain unpublished then an asymmetrical shape may be observed [9].
However, publication bias is not the only possible cause of asymmetry observed in a funnel plot [10]. Any factor which is associated with both study effect and study size could confound the true association and cause an asymmetric funnel. For example, if there is an indication of poorer study design in smaller studies [10] and this poor study design leads to systematic exaggeration of effect [11], this could manifest itself as asymmetry on a funnel plot because studies near the top of the plot (small bias) will have smaller effect sizes, on average, than those at the bottom (large bias). We introduce a graphical aid for the interpretation of funnel plots to help differentiate asymmetry caused by statistical significance related publication bias from that caused by other factors.
Section snippets
Funnel plots and contour-enhanced funnel plots
Contour-enhanced funnel plots display areas of statistical significance on a funnel plot. If it is assumed that the treatment effect in each study is normally distributed, then the significance of any effect size can be calculated from the effect size and the standard error. Because effect size and standard error (or some functions of it) are the two axes of a funnel plot, the (two-sided) statistical significance of any point on a funnel plot can be calculated. Thus, contours representing
Discussion
Tools to aid interpretation of funnel plots are clearly needed [22]. A recent study suggests that correct identification of the presence or absence of publication bias using funnel plots is poor [23] with only 52.5% of funnel plots correctly assessed for publication bias in simulated meta-analysis data sets. We believe contour-enhanced funnel plots greatly help interpretation of funnel plot asymmetry.
A number of points need consideration regarding the contour-enhanced funnel plots presented
Acknowledgements
JP was funded through a UK Department of Health Evidence Synthesis Award while undertaking this work. The funding source had no role in any aspect of this study.
References (36)
- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias
J Clin Epidemiol
(2005) - et al.
Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis
J Clin Epidemiol
(2001) - et al.
Misleading funnel plots for detection of bias in meta-analysis
J Clin Epidemiol
(2000) - et al.
Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature
J Clin Epidemiol
(2000) - et al.
Methods for meta-analysis in medical research
(2000) - et al.
Publication bias in meta-analysis
(2005) How important is publication bias? A synthesis of available data
AIDS Educ Prev
(1997)Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials
JAMA
(1998)- et al.
Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
BMJ
(2005)
Outcome selection bias in meta-analysis
Stat Methods Med Res
Empirical evidence for selective reporting of outcomes in randomized trials—Comparison of Protocols to published articles
JAMA
Summing up: the science of reviewing research
Bias in meta-analysis detected by a simple, graphical test
BMJ
Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials
JAMA
Empirical assessment of effect of publication bias on meta-analyses
BMJ
Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis
Biometrics
Cited by (1252)
Aftermath on COVID-19 technological and socioeconomic changes: A meta-analytic review
2024, Technological Forecasting and Social ChangePreventing postpartum hemorrhage: A network meta-analysis on routes of administration of uterotonics
2024, European Journal of Obstetrics and Gynecology and Reproductive BiologyNarrative Review to Meta-Analysis: A Cookbook Approach to Evidence Synthesis in Surgical Research
2024, Clinics in Podiatric Medicine and Surgery