Social cues, mentalizing and the neural processing of speech accompanied by gestures

Abstract

Body orientation and eye gaze influence how information is conveyed during face-to-face communication. However, the neural pathways underpinning the comprehension of social cues in everyday interaction are not known. In this study we investigated the influence of addressing vs. non-addressing body orientation on the neural processing of speech accompanied by gestures. While in an fMRI scanner, participants viewed short video clips of an actor speaking sentences with object- (O; e.g., shape) or person-related content (P; e.g., saying goodbye) accompanied by iconic (e.g., circle) or emblematic gestures (e.g., waving), respectively. The actor's body was oriented either toward the participant (frontal, F) or toward a third person (lateral, L) not visible. For frontal vs. lateral actor orientation (F>L), we observed activation of bilateral occipital, inferior frontal, medial frontal, right anterior temporal and left parietal brain regions. Additionally, we observed activity in the occipital and anterior temporal lobes due to an interaction effect between actor orientation and content of the communication (PF>PL)>(OF>OL). Our findings indicate that social cues influence the neural processing of speech-gesture utterances. Mentalizing (the process of inferring the mental state of another individual) could be responsible for these effects. In particular, socially relevant cues seem to activate regions of the anterior temporal lobes if abstract person-related content is communicated by speech and gesture. These new findings illustrate the complexity of interpersonal communication, as our data demonstrate that multisensory information pathways interact at both perceptual and semantic levels.