Neural basis of sound-symbolic pseudoword-shape correspondences

Deborah A Barany; Simon Lacey; Kaitlyn L Matthews; Lynne C Nygaard; K Sathian

doi:10.1016/j.neuropsychologia.2023.108657

Neural basis of sound-symbolic pseudoword-shape correspondences

Neuropsychologia. 2023 Sep 9:188:108657. doi: 10.1016/j.neuropsychologia.2023.108657. Epub 2023 Aug 3.

Authors

Deborah A Barany¹, Simon Lacey², Kaitlyn L Matthews³, Lynne C Nygaard⁴, K Sathian⁵

Affiliations

¹ Department of Kinesiology, University of Georgia and Augusta University/University of Georgia Medical Partnership, Athens, GA, 30602, USA.
² Department of Neurology, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Neural & Behavioral Sciences, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Psychology, Penn State College of Liberal Arts, University Park, PA, 16802, USA.
³ Department of Psychology, Emory University, Atlanta, GA, 30322, USA; Present address: Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO, 63130, USA.
⁴ Department of Psychology, Emory University, Atlanta, GA, 30322, USA.
⁵ Department of Neurology, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Neural & Behavioral Sciences, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Psychology, Penn State College of Liberal Arts, University Park, PA, 16802, USA. Electronic address: ksathian@pennstatehealth.psu.edu.

PMID: 37543139
PMCID: PMC10529692 (available on 2024-09-09)
DOI: 10.1016/j.neuropsychologia.2023.108657

Abstract

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Keywords: Language; Multisensory; Multivoxel pattern analysis; Sound symbolism; fMRI.

Publication types

Research Support, Non-U.S. Gov't
Research Support, N.I.H., Extramural

MeSH terms

Acoustic Stimulation / methods
Auditory Cortex*
Auditory Perception
Brain Mapping
Humans
Language
Magnetic Resonance Imaging / methods
Speech Perception*
Symbolism
Visual Perception

Grants and funding

R01 EY025978/EY/NEI NIH HHS/United States