A note on Type S/M errors in hypothesis testing

Br J Math Stat Psychol. 2019 Feb;72(1):1-17. doi: 10.1111/bmsp.12132. Epub 2018 Mar 23.


Motivated by the recent replication and reproducibility crisis, Gelman and Carlin (2014, Perspect. Psychol. Sci., 9, 641) advocated focusing on controlling for Type S/M errors, instead of the classic Type I/II errors, when conducting hypothesis testing. In this paper, we aim to fill several theoretical gaps in the methodology proposed by Gelman and Carlin (2014, Perspect. Psychol. Sci., 9, 641). In particular, we derive the closed-form expression for the expected Type M error, and study the mathematical properties of the probability of Type S error as well as the expected Type M error, such as monotonicity. We demonstrate the advantages of our results through numerical and empirical examples.

Keywords: design calculation; monotonicity; p-value; power calculation; replication; reproducibility; statistical significance.

Publication types

  • Comment

MeSH terms

  • Probability
  • Reproducibility of Results
  • Research Design