Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans

Abstract

Stimuli that animals encounter in the natural world are frequently time-varying and activate multiple sensory systems together. Such stimuli pose a major challenge for the brain: Successful multisensory integration requires subjects to estimate the reliability of each modality and use these estimates to weight each signal appropriately. Here, we examined whether humans and rats can estimate the reliability of time-varying multisensory stimuli when stimulus reliability changes unpredictably from trial to trial. Using an existing multisensory decision task that features time-varying audiovisual stimuli, we independently manipulated the signal-to-noise ratios of each modality and measured subjects' decisions on single- and multi-sensory trials. We report three main findings: (a) Sensory reliability influences how subjects weight multisensory evidence even for time-varying, stochastic stimuli. (b) The ability to exploit sensory reliability extends beyond human and nonhuman primates: Rodents and humans both weight incoming sensory information in a reliability-dependent manner. (c) Regardless of sensory reliability, most subjects are disinclined to make "snap judgments" and instead base decisions on evidence presented over the majority of the trial duration. Rare departures from this trend highlight the importance of using time-varying stimuli that permit this analysis. Taken together, these results suggest that the brain's ability to use stimulus reliability to guide decision-making likely relies on computations that are conserved across species and operate over a wide range of stimulus conditions.

Keywords: cue weighting; decision-making; multisensory integration; psychophysics; rodent; sensory reliability.

Figure 1

Rate discrimination decision task. (a) Schematic of rat behavioral setup. Rats were trained to perform a multisensory rate discrimination task via operant conditioning. Rats initiated trials by inserting their snouts into a central port (left), triggering an infrared sensor. After a randomized delay period, 1-s stimuli consisting of auditory and/or visual event streams were delivered through a speaker and LED panel (middle). Rats were required to remain in the central port until the end of the stimulus, and were provided with 20 μL of water reward when they selected the correct choice port (low-rate trials: left port; high-rate trials: right port). Trials in which the rat did not remain in the central port for the full stimulus were punished with a 4-s timeout period before allowing initiation of a new trial. (b) Example stimuli presented in human version of task. Time courses indicate arrival of events over the course of the 1-second stimuli. 10 ms events were separated by either short (60 ms) or long (120 ms) intervals. Top: Values on the ordinate indicate average luminance of events and background noise presented during high-reliability (black) and low-reliability (gray) visual trials. Bottom: Spectrograms indicate spectral power of sound pressure fluctuations during auditory stimulus presentation (color bar indicates signal power across frequency bands in units of dB SPL/Hz). Arrows indicate event (220 Hz tone) arrival times for low- (middle) and high- (left, right) reliability auditory stimuli. Auditory events in rat experiments consisted of white noise bursts (not shown). Note logarithmic scaling of frequency range. Multisensory trials consisted of visual (top) and auditory (bottom) stimuli presented together, and included different pairings of auditory and visual reliabilities. Multisensory trials included cue conflict levels ranging from −2 (left) to +2 (right) events/second. Auditory and visual event streams were generated independently on all multisensory trials.

Figure 2

Single sensory performance on rate discrimination task depends on sensory reliability. (a) Performance of an individual human subject, displayed as the proportion of high-rate decisions plotted against the trial-averaged event rate. Data are presented separately for each single sensory trial type. Lines indicate psychometric functions fit via maximum likelihood estimation. Data were combined across multiple behavioral sessions (2,161 trials). (b) Psychophysical thresholds obtained from seven human subjects for each single sensory trial type (low/high reliability auditory: blue/green; low/high reliability visual: gray/black). Symbols depict individual subjects. (c) Single sensory performance in an individual rat, pooled from two consecutive sessions (975 trials). (d) Single sensory thresholds obtained across cohort of 5 rat subjects (symbols). Thresholds in (b) and (d) were estimated from data combined across multiple behavioral sessions (humans/rats: 19,143/62,363 total single sensory trials). Star symbols indicate the example human and rat subjects used in (a) and (c). Error bars indicate standard errors in all panels.

Figure 3

Subjects weight auditory and visual evidence in proportion to sensory reliability. (a) Performance on multisensory trials in an individual human pooled over multiple sessions (values on abscissae indicate mean trial event rates averaged between auditory and visual stimuli). Colors indicate level of conflict between modalities (Δ = visual rate – auditory rate). Presented human data were obtained from the low-reliability visual/high-reliability auditory condition. (b) Same as (a) but for one rat. Data were obtained from the visual/high-reliability auditory condition. (c) Points of subjective equality (PSEs) from multisensory trials plotted as a function of conflict level for different pairings of auditory and visual stimulus reliabilities, shown for the same subject as in (a). Fitted lines were obtained via linear regression. Plotted data correspond to trials consisting of low- and high-reliability auditory stimuli paired with high- and low-reliability visual stimuli, respectively. Analogous fits were obtained for the other pairings of auditory and visual reliabilities presented to human subjects (see Figure 4b). (d) Same as (c) but for the single rat subject in (b). (e) Comparisons of the observed visual weights to the values predicted from the example human's single sensory thresholds. Data pertain to the same two multisensory trial types reported in (c). N = 3,861 trials. (f) same as (e) but for the rat in (b) and (d). N = 4,018 trials. Error bars indicate standard errors in all panels.

Figure 4

Reliability-based sensory weighting is observed consistently across subjects. Cue weights were estimated from data pooled over multiple behavioral sessions (humans/rats = 23,873/17,984 total multisensory trials). (a) Data points indicate the change in observed cue weights observed in seven individual human subjects, computed as the differences in subjects' visual cue weights between high-reliability visual/low-reliability auditory and low-reliability visual/high-reliability auditory trials. * indicates significant change in visual cue weights (p < 0.05, within-subjects one-tailed Z-tests). (b) Scatterplot compares observed visual cue weights (ordinate) to predicted values (abscissa) for all multisensory trial types in the individual human subjects. Legend indicates colors corresponding to each multisensory trial type. (c) Comparison of the observed visual cue weights (ordinate) to the PSE for unisensory auditory trials (abscissa). Color conventions are the same as in (b). (d) Same as (a) but for five individual rats. (e) Same as (b) but showing data for five rats. (f) Same as (c) but for five rats. Error bars indicate 95% confidence intervals in all panels.

Figure 5

Excess rates: Predictions and individual subject examples. (a) Simulated excess rate curve for a decision process that is based on all evidence presented over a 1000-ms trial duration. Solid curve reflects stimuli from 2000 simulated trials assigned to a “high rate decision” or “low rate decision” pool based on the value of a decision variable at the end of the trial. Dashed curve reflects a shuffle control (Materials and methods). (b) Same as (a) but for a decision process that only considers evidence arriving in the first 500 ms of the trial. (c) Excess rate results for a single human subject on auditory trials. Abscissae indicate centers of sliding windows (milliseconds preceding end of stimulus). Shaded regions provide confidence bounds (mean ± SE) on excess rate curves at each time point. Colors indicate reliability. Dashed lines: shuffle controls. (d) Results for a single rat on auditory trials. Conventions are the same as in (b). (e) Visual performance in a second human subject demonstrating an atypical evidence accumulation strategy. Note that the high-reliability visual trace is only elevated early in the trial for this subject.

Figure 6

Pooled data indicate that most subjects integrate sensory evidence over the entire course of the trial. (a) Pooled auditory data from all seven humans. Line/color conventions are the same as in Figure 5. (b) Pooled auditory data for the five rats. (c) Scatter plot with data for all subjects comparing the value of excess rates early in the trial (700 ms before stimulus offset) and late in the trial (300 ms before stimulus offset). Color indicates unisensory trial type. Blue: low-reliability auditory; green: high-reliability auditory; gray: low-reliability visual; black: high-reliability visual. Shape indicates species. Circles: humans. Squares: rats. Arrow highlights a single human subject with unusual behavior for high-reliability visual trials. This is the same subject represented in Figure 5e.