How to get statistically significant effects in any ERP experiment (and why you shouldn't)

Abstract

ERP experiments generate massive datasets, often containing thousands of values for each participant, even after averaging. The richness of these datasets can be very useful in testing sophisticated hypotheses, but this richness also creates many opportunities to obtain effects that are statistically significant but do not reflect true differences among groups or conditions (bogus effects). The purpose of this paper is to demonstrate how common and seemingly innocuous methods for quantifying and analyzing ERP effects can lead to very high rates of significant but bogus effects, with the likelihood of obtaining at least one such bogus effect exceeding 50% in many experiments. We focus on two specific problems: using the grand-averaged data to select the time windows and electrode sites for quantifying component amplitudes and latencies, and using one or more multifactor statistical analyses. Reanalyses of prior data and simulations of typical experimental designs are used to show how these problems can greatly increase the likelihood of significant but bogus results. Several strategies are described for avoiding these problems and for increasing the likelihood that significant effects actually reflect true differences among groups or conditions.

Keywords: Analysis/statistical methods; ERPs; Other.

Figure 1

Experimental paradigm from the study of Luck et al. (2009). Letters and digits were presented at fixation, with a stimulus duration of 200 ms and a stimulus onset asynchrony of 1500±150 ms. One of these two stimulus categories was rare (20%) and the other was frequent (80%). Participants were instructed to make a left-hand button-press for one category and a right-hand button-press for the other category. Both the rare category and the category-hand response mapping were counterbalanced across trial blocks. The same letter or digit was occasionally presented twice in succession in the frequent category.

Figure 2

Grand average waveforms for the frequent repetitions and frequent non-repetitions. Repetitions yielded a larger P2 wave over posterior scalp sites (the P2 effect) and a larger P1 wave over the right hemisphere (the P1 effect).

Figure 3

Familywise Type I error rate as a function of the number of statistical comparisons in a set of related tests (A) and as a function of the number of factors in a given ANOVA (B).