Publication bias

Not to be confused with Reporting bias or Media bias.

In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings in favor of positive results.[1] The study of publication bias is an important topic in metascience.

Despite similar quality of execution and design,[2] papers with statistically significant results are three times more likely to be published than those with null results.[3] This unduly motivates researchers to manipulate their practices to ensure statistically significant results, such as by data dredging.[4]

Many factors contribute to publication bias.[5][6] For instance, once a scientific finding is well established, it may become newsworthy to publish reliable papers that fail to reject the null hypothesis.[7] Most commonly, investigators simply decline to submit results, leading to non-response bias. Investigators may also assume they made a mistake, find that the null result fails to support a known finding, lose interest in the topic, or anticipate that others will be uninterested in the null results.[2] The nature of these issues and the resulting problems form the five diseases that threaten science: "significosis, an inordinate focus on statistically significant results; neophilia, an excessive appreciation for novelty; theorrhea, a mania for new theory; arigorium, a deficiency of rigor in theoretical and empirical work; and finally, disjunctivitis, a proclivity to produce many redundant, trivial, and incoherent works."[8]

Attempts to find unpublished studies often prove difficult or are unsatisfactory.[5] In an effort to combat this problem, some journals require studies submitted for publication pre-register (before data collection and analysis) with organizations like the Center for Open Science.

Other proposed strategies to detect and control for publication bias[5] include p-curve analysis[9] and disfavoring small and non-randomized studies due to high susceptibility to error and bias.[2]

  1. ^ Song, F.; Parekh, S.; Hooper, L.; Loke, Y. K.; Ryder, J.; Sutton, A. J.; Hing, C.; Kwok, C. S.; Pang, C.; Harvey, I. (2010). "Dissemination and publication of research findings: An updated review of related biases". Health Technology Assessment. 14 (8): iii, iix–xi, iix–193. doi:10.3310/hta14080. PMID 20181324.
  2. ^ a b c Easterbrook, P. J.; Berlin, J. A.; Gopalan, R.; Matthews, D. R. (1991). "Publication bias in clinical research". Lancet. 337 (8746): 867–872. doi:10.1016/0140-6736(91)90201-Y. PMID 1672966. S2CID 36570135.
  3. ^ Dickersin, K.; Chan, S.; Chalmers, T. C.; et al. (1987). "Publication bias and clinical trials". Controlled Clinical Trials. 8 (4): 343–353. doi:10.1016/0197-2456(87)90155-3. PMID 3442991.
  4. ^ Pearce, J; Derrick, B (2019). "Preliminary testing: The devil of statistics?". Reinvention: An International Journal of Undergraduate Research. 12 (2). doi:10.31273/reinvention.v12i2.339.
  5. ^ a b c H. Rothstein, A. J. Sutton and M. Borenstein. (2005). Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley. Chichester, England; Hoboken, NJ.
  6. ^ Chopra, Felix; Haaland, Ingar; Roth, Christopher; Stegmann, Andreas (2023). "The Null Result Penalty". The Economic Journal. 134 (657): 193–219. doi:10.1093/ej/uead060. ISSN 0013-0133.
  7. ^ Luijendijk, HJ; Koolman, X (May 2012). "The incentive to publish negative studies: how beta-blockers and depression got stuck in the publication cycle". J Clin Epidemiol. 65 (5): 488–92. doi:10.1016/j.jclinepi.2011.06.022. PMID 22342262.
  8. ^ Antonakis, John (February 2017). "On doing better science: From thrill of discovery to policy implications" (PDF). The Leadership Quarterly. 28 (1): 5–21. doi:10.1016/j.leaqua.2017.01.006.
  9. ^ Simonsohn, Uri; Nelson, Leif D.; Simmons, Joseph P. (2014). "P-curve: A key to the file-drawer". Journal of Experimental Psychology: General. 143 (2): 534–547. doi:10.1037/a0033242. PMID 23855496. S2CID 8505270.