A Comment on Replication, P-Values and Evidence

Stat Med. 1992 May;11(7):875-9. doi: 10.1002/sim.4780110705.

Abstract

It is conventionally thought that a small p-value confers high credibility on the observed alternative hypothesis, and that a repetition of the same experiment will have a high probability of resulting again in statistical significance. It is shown that if the observed difference is the true one, the probability of repeating a statistically significant result, the 'replication probability', is substantially lower than expected. The reason for this is a mistake that generates other seeming paradoxes: the interpretation of the post-trial p-value in the same way as the pre-trial alpha error. The replication probability can be used as a frequentist counterpart of Bayesian and likelihood methods to show that p-values overstate the evidence against the null hypothesis.

MeSH terms

  • Bias*
  • Humans
  • Probability*
  • Reproducibility of Results*