Publication bias arises from the tendency for researchers, editors, and companies to handle the reporting of experimental results that are positive (i.e., they show a significant finding) differently from results that are negative (i.e. supported the null hypothesis) or inconclusive.
This page provides an undergraduate-level introduction to the problem of publication bias.
Two counter-acting biases
- Low Power: under-estimation of real effects
- Publication Bias or File-drawer effect: over-estimation of real effects
What is publication bias?
- Publication of results depends on their nature and direction.
- Studies that show significant effects are more likely to be published.
- Type I publication errors are underestimated to the extent that they are: “frightening, even calling into question the scientific basis for much published literature.” (Greenwald, 1975, p. 15)
- A scatterplot of treatment effect against study size.
- Precision in estimating the true treatment effect increases as N increases.
- Small studies scatter more widely at the bottom of the graph.
- As studies become less precise, results should be more variable, scattered to both sides of the more precise larger studies … unless there is publication bias.
- In the absence of bias the plot should resemble a symmetrical inverted funnel.
- If there is publication bias this will cause meta-analysis to overestimate effects.
- The more pronounced the funnel plot asymmetry, the more likely it is that the amount of bias will be substantial.
File drawer effects
- Tendency for non-sig. results to be ‘filed away’ (hidden) and not published.
- of null studies which would have to ‘filed away’ in order for a body of significant published effects to be considered doubtful.
Countering the bias
- Publication bias (delicious.com/jtneill bookmarks)