Analytic estimation of statistical significance maps for support vector machine based multi-variate image analysis and classification

Abstract

Multivariate pattern analysis (MVPA) methods such as support vector machines (SVMs) have been increasingly applied to fMRI and sMRI analyses, enabling the detection of distinctive imaging patterns. However, identifying brain regions that significantly contribute to the classification/group separation requires computationally expensive permutation testing. In this paper we show that the results of SVM-permutation testing can be analytically approximated. This approximation leads to more than a thousandfold speedup of the permutation testing procedure, thereby rendering it feasible to perform such tests on standard computers. The speedup achieved makes SVM based group difference analysis competitive with standard univariate group difference analysis methods.

Figure 1

Illustration of the SVM concept in two dimensions

Figure 2

Illustration of the permutation testing procedure

Figure 3

For most permutations the number of support vectors in the learnt model is almost equal to the total number of samples (a) simulated dataset(b) real dataset with Alzheimer’s patients and controls (c) real dataset with liars and truth tellers

Figure 4

Figure 5

Results of experiments with simulated data (a)A sagittal section through p-maps obtained from experimental and analytical permutation tests (b) A scatter plot of p-values from experimental and analytical p-value maps (c) Regions where simulated atrophy was introduced

Figure 6

Simulated data: Experimental and analytical p-value maps thresholded at arbitrary p-values (3D)

Figure 7

Results of experiments with simulated data (a)An axial section through p-maps obtained from experimental and analytical permutation tests (b) A scatter plot of p-values experimental and analytical p-value maps

Figure 8

Alzheimers disease: Experimental and analytical p-value maps thresholded at arbitrary p-values (3D)

Figure 9

Results of experiments with fMRI lie detection data (a)An axial section through p-maps obtained from experimental and analytical permutation tests (b) A scatter plot of p-values experimental and analytical p-value maps

Figure 10

fMRI lie detection data: Experimental and analytical p-value maps thresholded at arbitrary p-values (3D)

Figure 11

(Left) Bivariate pattern simulated using two features, (Right) Illustration of simulation procedure

Figure 12

(Top-Left) p-values generated by univariate t-tests,(Top-Right) p-values generated by SVM based permutation tests. (Bottom)

Figure 13

(Left) p-values generated using univariate tests which detect the effect and many false positives (Right) p-values generated using SVM based permutation tests

Figure 14

Approximation accuracy and number of permutations

Figure 15

Approximation accuracy and dimensionality

Figure 16

Approximation accuracy and dimensionality