Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

Abstract

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

Keywords: Benjamini–Hochberg procedure; Factorial ANOVA; False discovery rate; Familywise error rate; Multiple comparison problem; Multiway ANOVA; Preregistration; Sequential Bonferroni; Type I error.

Fig. 1

A visual representation of the sequential Bonferroni method for controlling familywise error rate. All p values are sorted in ascending order and are assigned a rank number from 1 (smallest) to k (largest). Next, one starts by evaluating the first (smallest) p value (p ⁽¹⁾) against the adjusted α (α _adj), which is—for the first p value—equal to α divided by k. If the p value is smaller than α _adj, then the first hypotheses H ⁽¹⁾ is rejected, and one proceeds to the second p value. If the p value is not smaller than α _adj, then one immediately accepts all null hypotheses and stops testing

Fig. 2

A visual representation of the Benjamini–Hochberg procedure for controlling false discovery rate. All m p values are sorted in ascending order and assigned a rank number from 1 (smallest) to k (largest). Next, one starts by evaluating the last (largest) p value (p ^(k)) against the adjusted α (α _adj), which is—for the last p value—equal to k divided by m times α. If the p value is smaller than α _adj, then all null hypotheses are rejected and testing stops. If the p value is not smaller than α _adj, then one proceeds to the next p value