A Protracted Sensitive Period Regulates the Development of Cross-Modal Sound-Shape Associations in Humans

Abstract

Humans preferentially match arbitrary words containing higher- and lower-frequency phonemes to angular and smooth shapes, respectively. Here, we investigated the role of visual experience in the development of audiovisual and audiohaptic sound-shape associations (SSAs) using a unique set of five groups: individuals who had suffered a transient period of congenital blindness through congenital bilateral dense cataracts before undergoing cataract-reversal surgeries (CC group), individuals with a history of developmental cataracts (DC group), individuals with congenital permanent blindness (CB group), individuals with late permanent blindness (LB group), and controls with typical sight (TS group). Whereas the TS and LB groups showed highly robust SSAs, the CB, CC, and DC groups did not-in any of the modality combinations tested. These results provide evidence for a protracted sensitive period during which aberrant vision prevents SSA acquisition. Moreover, the finding of a systematic SSA in the LB group demonstrates that representations acquired during the sensitive period are resilient to loss despite dramatically changed experience.

Keywords: cataract; cross-modal correspondence; multisensory development; open data; open materials; sound–shape symbolism; visual deprivation.

Fig. 1.

Haptic and visual shape stimuli used in the experiment. Object pairs A, B, C, and D were haptic forms, whereas object pair E was presented on a white background to participants with visual capabilities. Pair A consisted of 3-D models printed in acrylonitrile butadiene styrene polymer. The dimensions of the smallest bounding cuboid were 100 mm × 70 mm × 60 mm. Pair B consisted of flat shapes obtained by laser-cutting plywood. The shapes were about 6 mm thick, and the smallest bounding rectangle dimension was 120 mm × 70 mm. Pair C consisted of 3-D-printed acrylic disks, 40 mm in diameter and 7 mm thick. Pair D consisted of heart and star shapes made of wood, about 70 mm in diameter. Pair E consisted of visually presented shapes printed on white paper. The outlines of the shapes in pair E were exactly the same as those in pair B. Background colors in the figure are for denoting object classes and were not part of the experiment. Object colors of haptic stimuli were not visible to the participants and hence played no role in the task.

Fig. 2.

Mean responses in the (a) sound–haptic-shape-association (SSA-H) and (b) sound–visual-shape-association (SSA-V) conditions. Responses are shown separately for individuals with congenital cataracts (CC group), developmental cataracts (DC group), congenital permanent blindness (CB group), late permanent blindness (LB group), and typical sight (TS control group), with kernel density estimated with Gaussian kernels. The width of each plot indicates the density of the data, the red circles indicate group means, the white circles indicate individual data points (jittered for readability), and the error bars indicate 95% confidence intervals of the group means obtained by smoothed bootstrapping with Gaussian kernels. A value of 1 on the y-axis indicates a congruent SSA-H or SSA-V (kiki was represented with an angular shape and bouba with a round shape). A value of 0 indicates a incongruent SSA-H or SSA-V (kiki was represented with a round shape and bouba with an angular shape). The dotted line indicates chance-level performance. Only the CC, DC, and TS groups participated in the SSA-V condition. Black asterisks indicate significant differences between groups, and red asterisks indicate significant differences between group mean responses and chance (*p < .05, **p < .01, ***p < .001).