Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 Jul 30;250:85-93.
doi: 10.1016/j.jneumeth.2014.08.003. Epub 2014 Aug 13.

Cluster-based Computational Methods for Mass Univariate Analyses of Event-Related Brain Potentials/Fields: A Simulation Study

Affiliations
Free PMC article

Cluster-based Computational Methods for Mass Univariate Analyses of Event-Related Brain Potentials/Fields: A Simulation Study

C R Pernet et al. J Neurosci Methods. .
Free PMC article

Abstract

Background: In recent years, analyses of event related potentials/fields have moved from the selection of a few components and peaks to a mass-univariate approach in which the whole data space is analyzed. Such extensive testing increases the number of false positives and correction for multiple comparisons is needed.

Method: Here we review all cluster-based correction for multiple comparison methods (cluster-height, cluster-size, cluster-mass, and threshold free cluster enhancement - TFCE), in conjunction with two computational approaches (permutation and bootstrap).

Results: Data driven Monte-Carlo simulations comparing two conditions within subjects (two sample Student's t-test) showed that, on average, all cluster-based methods using permutation or bootstrap alike control well the family-wise error rate (FWER), with a few caveats.

Conclusions: (i) A minimum of 800 iterations are necessary to obtain stable results; (ii) below 50 trials, bootstrap methods are too conservative; (iii) for low critical family-wise error rates (e.g. p=1%), permutations can be too liberal; (iv) TFCE controls best the type 1 error rate with an attenuated extent parameter (i.e. power<1).

Keywords: Cluster-based statistics; ERP; Family-wise error rate; Monte-Carlo simulations; Multiple comparison correction; Threshold free cluster enhancement.

Figures

Fig. 1
Fig. 1
Illustration of cluster-based methods applied to caricatured ERP data. Two effects were created, one transient effect (+25 μV) over 3 right posterior electrodes and one more sustained effect (+7 μV) over 8 electrodes. These effects are not meant to represent true EEG signal, but illustrate the different cluster attributes that are obtained on the basis of thresholded t values. From the observed t values, a binary ‘map’ is obtained (i.e. p < 0.05), and cluster attributes and TFCE data are computed via spatiotemporal clustering (3 first rows of the figure). The transformed data, to be thresholded, are presented for 2 electrodes (D12 and A30) and over the full space. Because the statistics are now based on cluster attributes, effect sizes can differ substantially from the original effects: (i) with cluster extent, effect-sizes are reversed with the sustained effect being stronger than the transient effect because it has a large support in space and time; (ii) cluster-height preserves effect-sizes but discards spatiotemporal information; (iii) with cluster-mass, effect-sizes are reversed but the difference between the sustained effect and the transient effect is attenuated compared to cluster-extend because cluster-mass accounts for height; (iv) with TFCE effect-sizes are preserved, and in contrast to cluster attributes, the shape of each effect is also preserved.
Fig. 2
Fig. 2
Type 1 FWER and percentages of agreement for a critical 5% FWE (cluster forming threshold p = 0.05). Results are presented per cluster statistic with curves showing the mean FWER across subjects with adjusted 95% confidence intervals. Boxplots show the median and inter-quartile range agreement between techniques (outliers marked with plus signs).
Fig. 3
Fig. 3
Type 1 FWER and percentages of agreement for a critical 1% FWE (cluster forming threshold p = 0.01). Results are presented per cluster statistic with curves showing the mean FWER across subjects with adjusted 95% confidence intervals. Boxplots show the median and inter-quartile range agreement between techniques (outliers marked with plus signs).
Fig. 4
Fig. 4
Type 1 FWER for a critical 5% FWE (cluster forming threshold p = 0.05) for each cluster statistic and technique as a function of the number of sampling iterations for the 7 sample sizes tested (n = [10 25 50 100 300 500 900] per group).
Fig. 5
Fig. 5
Type 1 FWER observed for a critical 1% FWE (cluster forming threshold p = 0.01) for each cluster statistic and technique as a function of the number of sampling iterations for the 7 sample sizes tested (n = [10 25 50 100 300 500 900] per group).
Fig. 6
Fig. 6
Type 1 FWER for cluster-mass (CM) and Threshold Free Cluster Enhancement (TFCE) using 4 combinations of extent and height. Boxes show for each subject the mean type 1 FWER and associated binomial 95% CI. The bottom right plots show bar graphs of the mean type 1 FWER across subjects and 95% CI, and the mean differences between each TFCE parameter set and cluster-mass type 1 FWER.

Similar articles

See all similar articles

Cited by 45 articles

See all "Cited by" articles

References

    1. Bullmore E.T., Suckling J., Overmeyer S., Rabe-Hesketh S., Taylor E., Brammer M.J. Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain. IEEE Trans Med Imaging. 1999;18(1):32–42. - PubMed
    1. Dunn O.J. Multiple comparisons among means. J Am Stat Assoc. 1961;56:52–64.
    1. Efron B., Tibshirani R.J. Chapman and Hall; New York: 1993. An introduction to the bootstrap.
    1. Gorgolewski K.J., Storkey A.J., Bastin M.E., Pernet C.R. Adaptive thresholding for reliable topological inference in single subject fMRI analysis. Front Hum Neurosci. 2012;6 - PMC - PubMed
    1. Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika. 1988;75(4):800–802.

Publication types

LinkOut - more resources

Feedback