The relation between statistical power and inference in fMRI

Abstract

Statistically underpowered studies can result in experimental failure even when all other experimental considerations have been addressed impeccably. In fMRI the combination of a large number of dependent variables, a relatively small number of observations (subjects), and a need to correct for multiple comparisons can decrease statistical power dramatically. This problem has been clearly addressed yet remains controversial-especially in regards to the expected effect sizes in fMRI, and especially for between-subjects effects such as group comparisons and brain-behavior correlations. We aimed to clarify the power problem by considering and contrasting two simulated scenarios of such possible brain-behavior correlations: weak diffuse effects and strong localized effects. Sampling from these scenarios shows that, particularly in the weak diffuse scenario, common sample sizes (n = 20-30) display extremely low statistical power, poorly represent the actual effects in the full sample, and show large variation on subsequent replications. Empirical data from the Human Connectome Project resembles the weak diffuse scenario much more than the localized strong scenario, which underscores the extent of the power problem for many studies. Possible solutions to the power problem include increasing the sample size, using less stringent thresholds, or focusing on a region-of-interest. However, these approaches are not always feasible and some have major drawbacks. The most prominent solutions that may help address the power problem include model-based (multivariate) prediction methods and meta-analyses with related synthesis-oriented approaches.

Fig 1. Examples of the sampling simulation.

Two different brain-behavior correlation distributions (full sample n = 10.000): (a) Weak Diffuse (WD) and (b) Strong Localized (SL). The scatter plots display the voxel (indicated by a green circle in the image) with the strongest absolute correlation between its brain activity (y-axis) and the behavioral variable (x-axis). The color bar indicates the effect size (Pearson’s r). From these two full samples, 4 random subsamples are drawn with two different sample sizes: (c) n = 30 from the WD full sample (d) n = 30 from the SL full sample (e) n = 150 from the WD full sample (f) n = 150 from the SL full sample. For each subsample a p < .01 uncorrected threshold is applied. The distribution of the maximum effect sizes (absolute values) obtained from the subsampling procedure are shown for the (g) WD scenario and (h) SL scenario. The thick black lines indicate the maximum correlation of the full sample. Small samples (n = 30) of the WD scenario show a large discrepancy (overestimation of the effect sizes and underestimation of the spatial extend of brain-behavior correlations) with the full sample effects, and the WD subsamples appear more similar to the SL scenario. Larger samples of the WD scenario (n = 150) show less, but still substantial discrepancy with the full sample effects. For the SL scenario there is a small discrepancy between subsamples of either sample size and the full sample effects.

Fig 2. Results of the sampling simulations.

Relation between sample size (n) and (a) average (solid line) and at least one (alo; dashed line) statistical power; (b) detected effect size (Pearson’s r) in the samples, and the full population effect size range (shown as a colored transparent bar); (c) Percentage of voxels below the threshold (p < .01); (d) mean dice coefficient: spatial overlap of significant (p < .01) voxels between two subsequent replications. SL; Strong Localized effects, WD; Weak Diffuse effects. The shaded grey area around the estimates reflects the 95% confidence intervals based on the sampling distribution.

Fig 3. Main effect of the HCP social cognition task.

(a) effects for the TOM>CON contrast in the full sample (n = 485). The colors reflect effect size (Cohen's d). (b) number of significant voxels and mean d in significant voxels as a function of subsample size. (c) Results of TOM>CON contrast for 16 random subsamples of n = 15. TOM; theory of Mind. CON; control condition.

Fig 4. Correlation between the HCP social cognition contrast and the personality trait agreeableness.

The top row is the full sample (n = 485), the scatterplot at the right displays the single strongest voxel (r = .25). The bottom row shows 8 random subsamples of n = 30. Each one shows a slice through the brain where effects are "detected" and the scatterplot shows the single most strongly correlated voxel.