Background: In recent years, analyses of event related potentials/fields have moved from the selection of a few components and peaks to a mass-univariate approach in which the whole data space is analyzed. Such extensive testing increases the number of false positives and correction for multiple comparisons is needed.
Method: Here we review all cluster-based correction for multiple comparison methods (cluster-height, cluster-size, cluster-mass, and threshold free cluster enhancement - TFCE), in conjunction with two computational approaches (permutation and bootstrap).
Results: Data driven Monte-Carlo simulations comparing two conditions within subjects (two sample Student's t-test) showed that, on average, all cluster-based methods using permutation or bootstrap alike control well the family-wise error rate (FWER), with a few caveats.
Conclusions: (i) A minimum of 800 iterations are necessary to obtain stable results; (ii) below 50 trials, bootstrap methods are too conservative; (iii) for low critical family-wise error rates (e.g. p=1%), permutations can be too liberal; (iv) TFCE controls best the type 1 error rate with an attenuated extent parameter (i.e. power<1).
Keywords: Cluster-based statistics; ERP; Family-wise error rate; Monte-Carlo simulations; Multiple comparison correction; Threshold free cluster enhancement.
Crown Copyright © 2014. Published by Elsevier B.V. All rights reserved.