Multisensory decision-making in rats and humans

Abstract

We report a novel multisensory decision task designed to encourage subjects to combine information across both time and sensory modalities. We presented subjects, humans and rats, with multisensory event streams, consisting of a series of brief auditory and/or visual events. Subjects made judgments about whether the event rate of these streams was high or low. We have three main findings. First, we report that subjects can combine multisensory information over time to improve judgments about whether a fluctuating rate is high or low. Importantly, the improvement we observed was frequently close to, or better than, the statistically optimal prediction. Second, we found that subjects showed a clear multisensory enhancement both when the inputs in each modality were redundant and when they provided independent evidence about the rate. This latter finding suggests a model where event rates are estimated separately for each modality and fused at a later stage. Finally, because a similar multisensory enhancement was observed in both humans and rats, we conclude that the ability to optimally exploit sequentially presented multisensory information is not restricted to a particular species.

Figure 1.

Schematic of rate discrimination decision task and experimental setup. a, Each trial consists of a stream of events (auditory or visual) separated by long or short intervals (top). Events are presented in the presence of ongoing white noise. For easy trials, all interevent intervals are either long (left) or short (right). More difficult trials are generated by selecting interevent intervals of both values (middle). Values of interevent intervals (bottom) reflect those used for all human subjects. b, Example auditory and visual event streams for the synchronous condition. Peaks represent auditory or visual events. Red dashed lines indicate that auditory and visual events are simultaneous. c, Example auditory and visual event streams for the independent condition. d, Schematic drawing of rodent in operant conditioning apparatus. Circles are the “ports” where the animal pokes his nose to initiate stimuli or receive liquid rewards. The white rectangle is the panel of LEDs. The speaker is positioned behind the LEDs.

Figure 2.

Subjects' performance is better on multisensory trials. a, Performance of a single subject plotted as the fraction of responses judged as high against the event rate. Green trace, auditory only; black trace, visual only; blue trace, multisensory. Error bars indicate SEM (binomial distribution). Smooth lines are cumulative Gaussian functions fit via maximum-likelihood estimation. n = 7680 trials. b, Same as a but for one rat (rat 3). n = 12,459 trials. c, Scatter plot comparing the observed thresholds on the multisensory condition with those predicted by the single-cue conditions for all subjects. Circles, human subjects; squares, rats. Green symbols are for the example rat and human shown in a and b. Black solid line shows x = y; points above the line show a suboptimal improvement. Error bars indicate 95% CIs. Prop, Proportion.

Figure 3.

Subjects make decisions according to event rates, not event counts. a, Example subject. Abscissa indicates the number of event counts. Each colored line shows the subject's performance for trials where the event count on the abscissa was presented over the duration specified by the labels. For a given event count (11 events, black arrow) the subject's choices differed depending on trial duration. n = 3656 trials b, Same data and color conventions as in a except that the abscissa indicates event rate instead of count. For a given event rate (11 events/s, black arrow), the subject's choices were very similar for all trial durations. c, d, Data for four subjects; conventions are the same as in a and b. n = 9727 trials. Prop, Proportion.

Figure 4.

The multisensory enhancement is still present for the independent condition. a, Performance of a single subject. Conventions are the same as in Figure 2a. n = 4255 trials. b, A comparison of accuracy for matched trials (blue trace), bonus trials (cyan trace), and neutral trials (orange trace). Abscissa plots the rate of the auditory stimulus. Data are pooled across six humans. n = 1957 (matched condition), 2933 (bonus condition), and 3825 (neutral condition). c, Same as a, but for a single rat (rat 1). n = 13,116 trials. d, Same as b, but for a single rat. n = 3725 (matched condition), 244 (bonus condition), and 247 (neutral condition) e, Scatter plot for all subjects comparing the observed thresholds on the multisensory condition with the predicted thresholds. Conventions are the same as in Figure 2c. Error bars indicate 95% CIs. Prop, Proportion.

Figure 5.

Human subjects' performance is better on multisensory trials even when the synchronous and independent conditions are presented in alternating blocks. a, Scatter plot for all subjects comparing the observed thresholds on the multisensory condition with the predicted thresholds. Data from the synchronous (circles) and independent (triangles) conditions are shown together. Error bars indicate 95% CIs. b, Subjects perform better on bonus trials compared with the matched trials and slightly worse on neutral trials. Conventions are the same as in Figure 4c. n = 1390 (matched condition), 2591 (bonus condition), and 2464 (neutral condition). Prop, Proportion.

Figure 6.

Subjects' decisions reflect evidence accumulated over the course of the trial. a, Schematic of average stimulus frequencies for trials supporting opposing decisions. Top, Trials were selected if their average stimulus rate from 0 to 700 ms was 10 Hz (seven events over 700 ms). Trials were then grouped according to whether the subject chose low (red) or high (blue) on each trial. Average stimulus rate within the bin of interest (700–1000 ms; dashed lines) was then compared for stimuli preceding left and right choices. Bottom, Same as in top panel except that the window of interest occurred early in the trial (0–300 ms). b, Solid traces indicate difference in average event rate for trials that preceded left versus right choices for all time points in a trial. Color conventions are the same as in Figure 2, a and b. Thin lines indicate SEM computed via bootstrapping. Dashed traces indicate difference in average rate for trials assigned randomly to two groups. Data were pooled from six human subjects. Trial numbers differed slightly for each time point; ∼1800 trials were included at each point. c, Same as b but for an individual rat. Trial numbers differed slightly for each time point; ∼3200 trials were included at each point.