Frequentist versus Bayesian approaches to multiple testing

Abstract

Multiple tests arise frequently in epidemiologic research. However, the issue of multiplicity adjustment is surrounded by confusion and controversy, and there is no uniform agreement on whether or when adjustment is warranted. In this paper we compare frequentist and Bayesian frameworks for multiple testing. We argue that the frequentist framework leads to logical difficulties, and is unable to distinguish between relevant and irrelevant multiplicity adjustments. We further argue that these logical difficulties resolve within the Bayesian framework, and that the Bayesian framework makes a clear and coherent distinction between relevant and irrelevant adjustments. We use Directed Acyclic Graphs to illustrate the differences between the two frameworks, and to motivate our arguments.

Keywords: Bonferroni correction; Data fishing; Multiple comparisons; Multiple tests; Posterior distribution; p-value.

Fig. 1

DAG for the Bygren et al. [6] study

Fig. 2

DAG for tests on dependent data sets

Fig. 3

DAG for data fishing selection mechanism

Fig. 4

$cor(X_f,X_m)$ as a function of $({\rho }_H,{\rho }_Y)$

Fig. 5

${\beta }_m$ as a function of $({\rho }_H,{\rho }_Y)$

Fig. 6

Frequentist estimates and 95% confidence intervals

Fig. 7

Bayesian estimates and 95% credible intervals, for independent elements of $\beta$

Fig. 8

Bayesian estimates and 95% credible intervals, for strongly dependent elements of $\beta$, with $\text{cor}({\beta }_j,{\beta }_k)=0.95$

Fig. 9

Sample variance of the estimates $({\widehat{\beta }}_1,\dots ,{\widehat{\beta }}_J)$ as a function of $\text{cor}({\beta }_j,{\beta }_k)$