Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 4 (3), 299-307

Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul Et Al. (2009)

Affiliations

Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul Et Al. (2009)

Matthew D Lieberman et al. Perspect Psychol Sci.

Abstract

Vul, Harris, Winkielman, and Pashler (2009), (this issue) claim that many brain-personality correlations in fMRI studies are "likely … spurious" (p. 274), and "should not be believed" (p. 285). Several of their conclusions are incorrect. First, they incorrectly claim that whole-brain regressions use an invalid and "nonindependent" two-step inferential procedure, a determination based on a survey sent to researchers that only included nondiagnostic questions about the descriptive process of plotting one's data. We explain how whole-brain regressions are a valid single-step method of identifying brain regions that have reliable correlations with individual difference measures. Second, they claim that large correlations from whole-brain regression analyses may be the result of noise alone. We provide a simulation to demonstrate that typical fMRI sample sizes will only rarely produce large correlations in the absence of any true effect. Third, they claim that the reported correlations are inflated to the point of being "implausibly high." Though biased post hoc correlation estimates are a well-known consequence of conducting multiple tests, Vul et al. make inaccurate assumptions when estimating the theoretical ceiling of such correlations. Moreover, their own "meta-analysis suggests that the magnitude of the bias is approximately .12-a rather modest bias.

Figures

Fig. 1
Fig. 1
A simulation of the number of high false-positive correlations (correlations above 0.8) that might reasonably occur in a typical whole-brain regression analysis. We conducted 1,000 simulated whole-brain regression analyses in which brain and covariate values were independent Gaussian random variables. The left panel shows a histogram of the number of simulated studies (y axis) that yielded a given number of tests in which r > 0.8 anywhere in the brain map (x axis). Studies with 10 subjects, as in Vul et al.’s simulation, yielded high numbers of false-positive tests (typically 15 to 25). Studies with 18 subjects (the mean of the criticized studies) yielded very few false-positive results. The right panel shows details of the histogram between 0 and 10 false-positive results. With 18 participants, 76% of studies showed no false-positive results at r > .8, 21% showed a single false-positive test, and 2% showed exactly two false-positive tests. These results are illustrative rather than exact; the actual false positive rate depends on details of the noise structure in the data and can be estimated using nonparametric methods on the full data set. The results presented here depend principally on the sample size (N), the number of effective independent tests (NEIT) performed in the whole-brain analysis, and standard assumptions of independence and normally distributed data. To estimate the NEIT, we used the p value thresholds for 11 independent whole-brain analyses reported in Nichols and Hayasaka (2003) that yield p < .05 with family-wise error-rate correction for multiple comparisons as assessed by Statistical Nonparametric Mapping software. We then equated this p value threshold to a Bonferroni correction based on an unknown number of independent comparisons and solved for the unknown NEIT for each study. Averaging over the 11 contrast maps yielded an average of 7,768 independent comparisons. Individual studies may vary substantially from this average. Dividing the number of voxels in each map by the NEIT for each study and averaging yielded a mean of 25.3 voxels per test; thus, each false-positive result can be thought of as a significant region encompassing 25 voxels.
Fig. 2
Fig. 2
Distribution of correlations in papers surveyed by Vul et al. but omitted from their meta-analysis. A: Independent correlations that were omitted from the Vul et al. meta-analysis. The dotted line indicates the mean of independent correlations (.57) that were included in their meta-analysis. Twelve of the 13 omitted independent correlations were higher than this mean. B: Nonindependent correlations that were omitted from the Vul et al. meta-analysis. The dotted line indicates the mean of nonindependent correlations (.69) that were included in their meta-analysis. Thirty-eight of the 41 omitted nonindependent correlations were lower than this mean.
Fig. 3
Fig. 3
Distribution of independent and nonindependent correlations uncorrected and corrected for restriction of range, based on papers included in the meta-analysis by Vul et al. A: A reconstruction of the correlations plotted in Figure 5 of Vul et al. Correlations are plotted as a percentage of total correlations of each type. In this display, nonindependent correlations (average r = .69) are inflated relative to the independent correlations (average r = .57) by an average of .12. B: A reanalysis of the data from the studies included in the meta-analysis by Vul et al. Independent correlations using a procedure likely to result in restricted range issues were corrected;, 52 correlations in the relevant papers that were omitted by Vul et al. were included, and 3 “correlations” that were not actually correlations were removed. In the reanalysis, the nonindependent correlations (average r = .69) are no longer observed to be inflated relative to independent correlations (average r = .70).

Comment on

Similar articles

See all similar articles

Cited by 37 PubMed Central articles

See all "Cited by" articles

LinkOut - more resources

Feedback