The quest for an optimal alpha

PLoS One. 2019 Jan 2;14(1):e0208631. doi: 10.1371/journal.pone.0208631. eCollection 2019.

Abstract

Researchers who analyze data within the framework of null hypothesis significance testing must choose a critical "alpha" level, α, to use as a cutoff for deciding whether a given set of data demonstrates the presence of a particular effect. In most fields, α = 0.05 has traditionally been used as the standard cutoff. Many researchers have recently argued for a change to a more stringent evidence cutoff such as α = 0.01, 0.005, or 0.001, noting that this change would tend to reduce the rate of false positives, which are of growing concern in many research areas. Other researchers oppose this proposed change, however, because it would correspondingly tend to increase the rate of false negatives. We show how a simple statistical model can be used to explore the quantitative tradeoff between reducing false positives and increasing false negatives. In particular, the model shows how the optimal α level depends on numerous characteristics of the research area, and it reveals that although α = 0.05 would indeed be approximately the optimal value in some realistic situations, the optimal α could actually be substantially larger or smaller in other situations. The importance of the model lies in making it clear what characteristics of the research area have to be specified to make a principled argument for using one α level rather than another, and the model thereby provides a blueprint for researchers seeking to justify a particular α level.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • False Negative Reactions
  • False Positive Reactions
  • Models, Statistical*
  • Research Design
  • Sample Size

Grants and funding

This work was supported by the Deutsche Forschungsgemeinschaft (Statistical Modeling in Psychology, GRK 2277). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.