Investigating audiovisual integration of emotional signals in the human brain

Prog Brain Res. 2006:156:345-61. doi: 10.1016/S0079-6123(06)56019-4.


Humans can communicate their emotional state via facial expression and affective prosody. This chapter reviews behavioural, neuroanatomical, electrophysiological and neuroimaging studies pertaining to audiovisual integration of emotional communicative signals. Particular emphasis will be given to neuroimaging studies using positron emission tomography (PET) or functional magnetic resonance imaging (fMRI). Conjunction analyses, interaction analyses, correlation analyses between haemodynamic responses and behavioural effects and connectivity analyses have been employed to analyse neuroimaging data. There is no general agreement as to which of these approaches can be considered "optimal" to classify brain regions as multisensory. We argue that these approaches provide complementing information as they assess different aspects of multisensory integration of emotional information. Assets and drawbacks of the different analysis types are discussed and demonstrated on the basis of one fMRI data set.

Publication types

  • Research Support, Non-U.S. Gov't
  • Review

MeSH terms

  • Brain / physiology*
  • Expressed Emotion / physiology*
  • Facial Expression*
  • Humans
  • Magnetic Resonance Imaging
  • Pattern Recognition, Visual / physiology*
  • Speech / physiology
  • Speech Perception / physiology*