Sensory substitution reveals a manipulation bias

Abstract

Sensory substitution is a promising therapeutic approach for replacing a missing or diseased sensory organ by translating inaccessible information into another sensory modality. However, many substitution systems are not well accepted by subjects. To explore the effect of sensory substitution on voluntary action repertoires and their associated affective valence, we study deaf songbirds to which we provide visual feedback as a substitute of auditory feedback. Surprisingly, deaf birds respond appetitively to song-contingent binary visual stimuli. They skillfully adapt their songs to increase the rate of visual stimuli, showing that auditory feedback is not required for making targeted changes to vocal repertoires. We find that visually instructed song learning is basal-ganglia dependent. Because hearing birds respond aversively to the same visual stimuli, sensory substitution reveals a preference for actions that elicit sensory feedback over actions that do not, suggesting that substitution systems should be designed to exploit the drive to manipulate.

Fig. 1. Light-off stimuli are positive reinforcers of vocal pitch in deaf songbirds.

a Schematic of the experiment. A singing deaf bird inside a sound-isolation chamber (left) experiences a light-off (LO) stimulus for a duration in the range of 100–500 ms (right) when the pitch of one of its song syllables (red note) exceeds a given threshold (Credit: Sarah Steinbacher, MELS UZH). b Example picture of a pair of surgically removed cochleas. Complete deafness was confirmed by the presence of the osseous spiral lamina and by verification of an intact loop including the lagena. c, e Example song spectrograms in birds b2y2 (c) and b2p19 (e) with substituted feedback for low-pitched (c) and high-pitched (e) syllable renditions. The time points of pitch measurement are indicated by white dashed lines and the LO stimuli by green (c) and blue (e) bars. d, f Pitch values for syllable renditions without substitution (black dots) and with substitution (green dots: low-pitch subs, d); blue dots: high-pitch subs, (f)). The birds adapted the pitch in the direction of increasing LO rate. g Histograms of average daily pitch changes during substitution in birds with high-pitch substitution (subs high, blue, n = 5 birds, the first bar corresponds to b2p19 shown in (e) and (f)), low-pitch substitution (subs low, green, n = 5 birds, the 8^th bar corresponds to b2y2 shown in (c) and (d)), and in deaf control birds without substitution (unsubs, dark gray, n = 10 birds). The light gray bar to the left of each colored bar indicates the average daily pitch change in that bird during the last 5 baseline days. The asterisks indicate subs birds with significant pitch changes compared to controls (two-sample, two-sided t-test, p < 0.05). h Subs birds, as a population, adapt pitch in the direction of substituted feedback. Shown are the three fixed-effect terms of a mixed linear-effect model and their standard errors (282 observations from n = 10 subs and n = 10 unsubs birds). The bars indicate the daily change in pitch (d’/day) during baseline, during substitution in the direction of increasing light-off rate (subs, **** indicates nonzero fixed effect 0.19 d’/day, p = 3.0 × 10⁻⁶, SE = 0.04, tstat = 4.77, df = 279, confidence interval 0.11–0.27 d’/day, n = 20 birds), and in control (unsubs) birds.

Fig. 2. In hearing birds, the valence of light-off reinforcers is negative.

a, b Hearing birds change pitch in the direction of decreasing LO rate, here shown for a low-pitch light-off (LO low) bird (bird b2y2, a same bird as in Fig. 1c, d) and a LO high bird (bird p6s6, b). Legend as in Fig. 1d, f. c Histograms of average daily pitch changes in LO high (blue, n = 6), LO low (green, n = 6), and in hearing control (noLO) birds (gray, n = 12). The asterisks indicate subs birds with significant pitch changes compared to controls (two-sample, two-sided t-test, p < 0.05, see “Methods”). d The three mixed linear fixed effect terms and their standard errors. The bars indicate the daily change in pitch (d’/day) during baseline, during LO exposure in the direction of increasing LO rate (LO, ***indicates non-zero fixed effect −0.08, p = 1.4 × 10⁻⁴, df = 377, SE = 0.02, tstat = −3.85, confidence interval −0.12 to −0.04 d’/day, n = 24 birds), and in hearing control birds (noLO). e Average directed pitch changes over all days in LO (hearing, 144 days) vs subs (deaf, 102 days) birds. Hearing birds changed their pitch away from the LO zone (decreasing the number of renditions with LO) and deaf birds towards the LO zone (increasing the number of renditions with LO). The error bars indicate the standard errors of the mean. f The magnitude of average pitch change is larger in subs birds than in LO birds (0.16 d’/day, *indicates p = 0.01 for average magnitude, tstat = 2.73, df = 20, two-sample two-sided t-test, n = 12 LO and n = 10 subs birds), and much larger than in unsubs birds (0.22d’/day in time-matched periods, **** indicates p = 4 × 10⁻⁵, tstat = −5.35, df = 18, n = 20 birds, two-sample two-tailed t-test). The magnitude of average pitch change is larger by 0.10 d’/day for hearing LO than noLO birds (p = 0.02, tstat = −2.43, df = 22, two-sample two-tailed t-test, p-values are not adjusted for multiple comparisons. The error bars indicate standard errors of the mean and the dots represent individual birds (colors as in c).

Fig. 3. Within-syllable pitch trajectories show time-localized learning.

a, b median (solid line) and quantiles (shaded area) of within-syllable pitch during baseline (blue) and on the last day of substitution (last day subs, red) in a subs-high (a) and a subs-low bird (b). The two dashed vertical lines show the window within which we calculated pitch to determine whether to switch off the light or not. The bird numbers correspond to target syllable numbers in Fig. 1g. c, d Average spectrogram of the targeted syllable before (top row) and after (bottom row) the substitution paradigm. Same time axis as in (a). The horizontal black bar indicates the pitch calculation window. e, f Traces of normalized pitch differences between (the last day of) baseline and the last day of subs in deaf subs (e) and in hearing LO birds (f). Birds adapted pitch within about 10 ms of the pitch calculation time window (delimited by dashed vertical lines). The fine curves show pitch difference traces in individual birds, the thick curve their average, and the gray area indicates the average ± one standard deviation. Curves in birds that decreased pitch are flipped to make pitch changes positive.

Fig. 4. A manipulation bonus is compatible with valence inversion.

a Average change in singing rate during the last three days of light off in deaf (n = 10) and in hearing birds (n = 12) and in their time-matched controls (n = 12 deaf and n = 12 hearing birds). The change is reported relative to the average on the last three days of baseline (* indicates p = 0.02, tstat = −2.65, df = 20, two-sample two-tailed t-test). The error bars indicate standard errors of the mean and the dots indicate individual birds (blue/green for light off high/low and black for birds that were not subjected to light off events). b Hearing (n = 12, LO) birds trigger light off on average in <50% of cases whereas deaf (n = 10, subs) birds do so on average in more than 50%. The center line of the boxplot represents the median, the box bound extends to the 25^th and 75^th percentile and the whiskers extend to the minima and maxima excluding outliers indicated as a black cross (outlier are defined according to MATLAB’s default definition as values that are more than 1.5 times the interquartile range away from the top or bottom of the box). c We modeled a simple agent that maximizes total reward formed by the sum of the extrinsic reinforcement r (red), an exploration bonus E, and a manipulation bonus M given by impact. The agent’s greedy policy is to choose the action with maximal Q value (expected total reward). Deaf birds receive no auditory input (green cross). d Markov model of an agent that generates one syllable composed of three consecutive notes, each associated with six possible variants (actions). An action triggers one of three possible sensory states with probabilities depicted with gray shading. States 13–16 trigger light-off (red). e Example syllable generated by the model (the underlying action-state pairs are delimited in yellow in d). f Hearing birds trigger light-off on <50% of syllables for all choices of negative reinforcement r per LO. Deaf birds reach above the critical level of 50% LO contingency (green), which is not the case when the manipulation bonus is zero (M = 0, dashed line). g Simulated subs birds are more motivated to sing than their controls (unsubs), their mean Q value (green, arbitrary units) is above that of unsubs birds (dashed green). In hearing birds, the situation is reversed, they are less motivated than their controls. The blue dashed area indicates the plausible reinforcement-per-LO region that qualitatively matches our results. h Model neurons’ firing rates (in hearing birds) agree with reward prediction error coding seen in dopaminergic neurons. On aversively reinforced trials during Note 2 (modeling a LO event or an acoustic white-noise stimulus), the firing rate decreases (red), whereas on escape trials (no reinforcer, no LO), the firing rate increases (black). Error bars depict mean ± standard deviations (across simulated model birds).

Fig. 5. A basal ganglia pathway is necessary for adaptive responses to substituted feedback.

a Example sagittal brain section of a bird with lesion (yellow arrows) in Area X (dashed white ellipse). The lamina pallio-subpallialis (LPS) is indicated by the white dashed line. The scale bar on the bottom right indicates 500 µm. Anterior is towards the left, posterior to the right. b Pitch values of all syllable renditions in an example deaf bird with bilateral lesions in Area X. There is no clear adaptive response to substitution. c The average pitch changes (d’/day) in deaf birds with Area X lesions during baseline (light gray), during high-pitch substitution (blue), and during low-pitch substitution (green). d The two fixed-effect terms of a mixed linear effect model and their standard errors: the daily pitch changes (1) during baseline (left bar, p = 0.21, tstat = 1.27, df = 83, n = 5 birds), and (2) during substitution in the direction of increasing LO rate (right bar, p = 0.64, SE = 0.04, tstat = −0.47, df = 83, n = 5 birds) are not significantly different from zero (not indicated in the figure).