Decoding word and category-specific spatiotemporal representations from MEG and EEG

Abstract

The organization and localization of lexico-semantic information in the brain has been debated for many years. Specifically, lesion and imaging studies have attempted to map the brain areas representing living versus nonliving objects, however, results remain variable. This may be due, in part, to the fact that the univariate statistical mapping analyses used to detect these brain areas are typically insensitive to subtle, but widespread, effects. Decoding techniques, on the other hand, allow for a powerful multivariate analysis of multichannel neural data. In this study, we utilize machine-learning algorithms to first demonstrate that semantic category, as well as individual words, can be decoded from EEG and MEG recordings of subjects performing a language task. Mean accuracies of 76% (chance=50%) and 83% (chance=20%) were obtained for the decoding of living vs. nonliving category or individual words respectively. Furthermore, we utilize this decoding analysis to demonstrate that the representations of words and semantic category are highly distributed both spatially and temporally. In particular, bilateral anterior temporal, bilateral inferior frontal, and left inferior temporal-occipital sensors are most important for discrimination. Successful intersubject and intermodality decoding shows that semantic representations between stimulus modalities and individuals are reasonably consistent. These results suggest that both word and category-specific information are present in extracranially recorded neural activity and that these representations may be more distributed, both spatially and temporally, than previous studies suggest.

Figure 1. Decoding framework utilizing amplitude-based feature extraction and SVMs

The amplitude at 6 post-stimulus time points are selected from each channel and concatenated into an initial feature vector. The feature vectors from all channels are concatenated into a final feature vector. A single feature vector represents the spatio-temporal dynamics of a single trial. A nonlinear SVM is trained on these feature vectors to discriminate between the two semantic classes (living vs. non-living objects) or between individual words. This results in a decision boundary by which new trials can be classified. In the multiclass case, multiple decision boundaries are generated to separate individual classes from each other.

Figure 2. Decoding accuracy when distinguishing between living and non-living objects or individual words

The bar graphs illustrate classifier accuracy for each subject when distinguishing between living and non-living object category (A–B) or between individual words (C–D) after averaging 5 trials. Inset panels illustrate mean decoding accuracy as a function of the number of trials averaged. Blue indicates the use of EEG features, red indicates MEG features, and green indicates that both EEG and MEG features were used. In both the main figure and insets, chance accuracy (0.5 for living/non-living and 0.2 for individual words) is shown as the horizontal black line and accuracies above the dashed line are statistically significant (permutation test, p<0.05). A–B) Data from all subjects show significant decoding ability in at least one set of features. In all cases, utilizing combined EEG and MEG features resulted in significant decode accuracies. C–D) When utilizing both EEG and MEG features, decoding performance when distinguishing individual words is statistically significant in all cases, and exceeds 95% accuracy in the SA task.

Figure 3. Classifier weights show important times and locations for decoding

A–B) SVM weights of the classifier trained on living and non-living object categories in MEG show areas of significant differences. Areas of dark red indicate biases in classification towards non-living objects, and blue denotes biases towards animals and living objects. Averaged weights across all subjects are shown at each sensor-time point for the (A) visual and (B) auditory tasks. Bilateral anterior temporal and inferior frontal differences are seen at 400–600ms during both the SV and SA tasks (white arrows). Left temporal-occipital differences showing larger responses to objects are apparent at 200ms (red arrows) with differences showing larger responses to living objects occurring at 400–700ms in both modalities (black arrows). C–D) Variance of SVM weights is shown at each time-sensor point indicating relative importance of each feature in individual word discrimination. Features with larger variance indicate larger separation between the SVM weights in that particular dimension and correlate with increased discrimination ability. C) Extracranial weights from the SV task indicate occipital significance around 300–400ms (black arrows) and inferior temporal significance at several times (white arrows). D) Weights from the SA task show bilateral anterior temporal and inferior frontal significance from 250–450ms (white arrows), and inferior occipital significance at 300 and 500ms (black arrow). Inferior parietal significance is also seen from 350–400ms (blue arrow).

Figure 4. Individual word decoding confusion matrices

A) Averaged confusion matrices for decoding all 10 individual words (averaging 5 trials) indicate the types of errors made. The vertical axis displays the actual stimulus word while the horizontal axis displays the word predicted by the classifier. The colors along any given row (actual word) indicate the proportion of trials of that word which were classified as each of the possible choices (predicted words). The diagonal elements display correctly classified trials. Words are sorted into small and large objects (divided by black lines), and living or non-living categories (blue and red text). These matrices demonstrate a significant ability to decode individual words without regard to large/small conditions. B) Within and between-category confusion rates are shown for the large/small and living/non-living object distinctions. In all cases, confusion rates between categories are statistically lower than confusion rates within each category.

Figure 5. Intermodality and intersubject classification shows word and category representation consistencies

Accuracies of intermodality and intersubject decoding are shown after averaging 10 trials. Chance accuracy is indicated by solid horizontal line and the statistically significant threshold is shown by dashed line (permutation test, p<0.05). A) Training on living/non-living object data from SV and testing on data from SA results in data from 3 of 9 subjects showing statistically significant decode ability while training on SA and testing on SV results in data from 5 of 9 subjects showing significant decode ability. This indicates supramodal semantic information is encoded within the classification models generated by the SVM. B) Training an SVM on living/non-living object data from all but one subject and testing on the final subject results in data from 5 of 9 and 9 of 9 subjects showing statistically significant decode within the SV and SA modalities respectively. C) Training an SVM on individual word representations from all but one subject and testing on the final subject results in data from 6 of 9 and 9 of 9 subjects showing statistically significant decode ability. This indicates intersubject consistency in the neural representation of these words.

Figure 6. Hierarchical tree decoding improves classification performance

A three-level hierarchical tree decoder was utilized to first decode the large/small distinction (utilizing amplitude and spectral features), then the living/non-living object category (utilizing 200–700ms amplitude features), and finally the individual word (utilizing 250–500ms amplitude features). Data from decoding of the SA task are shown (data from SV task shown in Supplementary Figure S3). A) Average accuracies at each branch of the tree are shown with corresponding colors. Accuracies remain above 80% for all branches. B) Accuracies at each level of the decoder are shown on a per subject basis with dotted lines indicating chance accuracy. C) Cumulative accuracies at each level decrease as errors propagate through levels of the tree, but remain above 60%. D) Performance of the hierarchical tree is a significant improvement (Wilcoxon sign-rank, p<0.05) over training a single multi-class SVM to discriminate between all 10 words.