Generic decoding of seen and imagined objects using hierarchical visual features

Nat Commun. 2017 May 22:8:15037. doi: 10.1038/ncomms15037.

Abstract

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Adult
  • Brain / physiology
  • Female
  • Humans
  • Image Processing, Computer-Assisted
  • Imagery, Psychotherapy
  • Imagination*
  • Male
  • Neural Networks, Computer
  • Pattern Recognition, Visual*
  • Photic Stimulation
  • Time Factors
  • Visual Perception / physiology*
  • Young Adult