Publication bias

Educational level: this is a tertiary (university) resource.

Completion status: this resource is ~25% complete.

Publication bias arises from the tendency for researchers, editors, and companies to handle the reporting of experimental results that are positive (i.e., they show a significant finding) differently from results that are negative (i.e. supported the null hypothesis) or inconclusive.

This page provides an undergraduate-level introduction to the problem of publication bias.

Two counter-acting biases

Low Power: under-estimation of real effects
Publication Bias or File-drawer effect: over-estimation of real effects

What is publication bias?

Publication of results depends on their nature and direction.
Studies that show significant effects are more likely to be published.
Type I publication errors are underestimated to the extent that they are: “frightening, even calling into question the scientific basis for much published literature.” (Greenwald, 1975, p. 15)

Funnel plots

A scatterplot of treatment effect against study size.
Precision in estimating the true treatment effect increases as N increases.
Small studies scatter more widely at the bottom of the graph.
As studies become less precise, results should be more variable, scattered to both sides of the more precise larger studies … unless there is publication bias.
In the absence of bias the plot should resemble a symmetrical inverted funnel.
If there is publication bias this will cause meta-analysis to overestimate effects.
The more pronounced the funnel plot asymmetry, the more likely it is that the amount of bias will be substantial.

File drawer effects

Tendency for non-sig. results to be ‘filed away’ (hidden) and not published.
of null studies which would have to ‘filed away’ in order for a body of significant published effects to be considered doubtful.

Countering the bias

Journal Articles in Support of the Null Hypothesis

External links

Publication bias (delicious.com/jtneill bookmarks)