Supralinear and Supramodal Integration of Visual and Tactile Signals in Rats: Psychophysics and Neuronal Mechanisms

Abstract

To better understand how object recognition can be triggered independently of the sensory channel through which information is acquired, we devised a task in which rats judged the orientation of a raised, black and white grating. They learned to recognize two categories of orientation: 0° ± 45° ("horizontal") and 90° ± 45° ("vertical"). Each trial required a visual (V), a tactile (T), or a visual-tactile (VT) discrimination; VT performance was better than that predicted by optimal linear combination of V and T signals, indicating synergy between sensory channels. We examined posterior parietal cortex (PPC) and uncovered key neuronal correlates of the behavioral findings: PPC carried both graded information about object orientation and categorical information about the rat's upcoming choice; single neurons exhibited identical responses under the three modality conditions. Finally, a linear classifier of neuronal population firing replicated the behavioral findings. Taken together, these findings suggest that PPC is involved in the supramodal processing of shape.

Keywords: Bayesian; linearity; multimodal integration; mutual information; neuronal coding; posterior parietal cortex; psychophysics; rat; touch; vision.

Graphical abstract

Figure 1

Visual-Tactile Orientation Categorization Task (A) Photographs of the object examined by the rat. Shown are front-side view (left) and exactly front view (right), the latter approximating the perspective of the rat. (B) Schematic of the orientations of the stimuli and rule of the categorization task. Rats were trained to categorize orientations from 0° to 45° (solid red) in one category and orientations from 45° to 90° (solid green) in another. When tested with −45° to 0° and 90°–135° (red and green stippling, respectively), they immediately generalized the rule, suggesting that the categories corresponded to horizontal (H) and vertical (V); see Figure S2C. (C) Sequential steps in the behavioral task. Each trial started with a head poke that interrupted a light beam and triggered the opening of an opaque gate, followed by either visual, tactile, or visual-tactile access to the object. After probing the stimulus, the rat turned its head toward one spout, in this illustration left for vertical and right for horizontal. On the right side of the figure, the sequence of actuators and sensors is shown schematically. See Figure S1A for the experimental setup.

Figure 2

Supralinear Performance through the Merging of Touch and Vision (A) Three example rats that, notwithstanding differences in modality-specific acuity, each showed better performance on VT trials as compared to V and T trials. In this and all figures, green traces correspond to the T condition, blue to V, and red to VT. Black curves are predicted psychometric curves from the Bayesian cue-combination model. Error bars show 95% binomial confidence intervals. See Figure S2A for tests of significance. (B) Pale curves give the performance of 12 rats in each modality. Dark data points and curves show the average over all rats in each modality. Error bars show 95% binomial confidence interval. Inset shows the improved performance in the VT condition versus V alone (blue) and T alone (green), as a function of orientation, averaged across rats. (C) Comparison of the thresholds predicted by linear combination of V and T signals based on Bayesian cue combination model (ordinate) with measured thresholds on VT trials (abscissa). Each point corresponds to one rat. Error bars indicate 95% confidence intervals by bootstrapping. (D) Comparison of mutual information between behavioral choice and stimulus category in VT trials (ordinate) with the mutual information between behavioral choice and stimulus category summated from V and T trials (abscissa). Each point corresponds to one rat. Error bars indicate 95% confidence intervals by bootstrapping. (E) Synthesis of psychometric curve parameters under the three modality conditions. Each point corresponds to one rat. Abscissa denotes the rat’s point of subjective equivalence. The histograms are the summed distributions across individual rats, with bin size 2°. Ordinate denotes the maximum slope of the rat’s psychometric curve (slope is the change in proportion of trials in which orientation was judged as vertical per degree of angle change). The histograms are the summed distributions across individual rats. Circled points are the average values. (F) Same analysis as in (C) for six subjects (rats 2–7) tested in a block design where seven sessions with only V and T trials were followed by seven sessions with only VT trials. Most rats showed supralinear combination of V and T signals, indicating that sensory channel integration did not depend on the modality sequence of trials. Error bars show 95% confidence interval. (G) Response times averaged across all test sessions for all rats. Data from V, T, and VT trials are pooled. Data show mean ± SEM. See Figures S2E and S2F for interquartile ranges.

Figure 3

Encoding of Stimulus Orientation and Stimulus Category in Posterior Parietal Cortex (A) Upper: the target area, PPC (green), is located between visual cortex (blue) and the vibrissal region of somatosensory cortex, Bf. Lower: histological section demonstrating recording sites in the LPtA zone of PPC according to Paxinos and Watson (2007). (B) Upper left: example PPC neuron action potentials (150 single waveforms and their average, black). Upper right: interspike interval histogram. Lower: raster plot showing the unit’s firing on 250 randomly selected trials. Modality conditions (V, T, and VT) are labeled by color. Firing rate, computed in a window of 1 ms and smoothed with a Gaussian ($\sigma$ = 50 ms), is plotted with the same color code. Shaded bands are ±SEM. (C and D) Raster plots (left plots), peri-event time histograms (PETHs, middle plots, smoothed with a Gaussian kernel $\sigma$ = 50 ms), and angular tuning curves (right plots, separated by modality) of four example PPC neurons. Error trials and trials with 45° orientation are excluded. Trials are grouped by stimulus orientation (see color key). Tuning curves show average firing rate for trials grouped with the same stimulus orientation in a 300 ms window preceding the reward spout lick. Data points show the average firing rate per angle ±SEM. The curves are cumulative Gaussian fits (see STAR Methods).

Figure 4

Supramodal Responses in Posterior Parietal Cortex (A) Each point denotes the average firing rates of one neuron (total, 622 neurons) in a 300 ms window preceding the reward spout lick in V, T, and VT trials, separately. Points are clustered about a line (orange) corresponding to equal responses under the three modality conditions. Data are not separated according to orientation. (B) Separated by angles (colors), points indicate the firing rate of all neurons, again in the 300 ms window preceding the reward spout lick, in T (abscissa) versus V (ordinate) trials. R is the Pearson linear correlation coefficient. (C) Upper plots: mean firing rates on T (abscissa) versus V (ordinate) trials for each of the six tested orientations for two neurons. Linear regression coefficient and slope are reported on the plot. The neuron in the left panel was also illustrated in the lower plot of Figure 3C and the neuron in the right panel from the upper plot of Figure 3D. Data points show the average firing rate per angle ±SEM. Lower plots: histograms are the distributions of the linear regression coefficients for all neurons (left) and slopes for neurons with statistically significant regression coefficient (right). (D) Upper histogram: modality selectivity for a population of 622 PPC neurons. Positive index values correspond to higher firing rate for V trials and negative values to higher firing for T trials. Lower histogram: category selectivity for the same population of neurons. Positive index values correspond to higher firing rate for leftward choices and negative values to higher firing for rightward choices. See Figure S4 for time course of selectivity index measured for each neuron.

Figure 5

Joint Information about Stimulus Orientation and Category in Posterior Parietal Cortex Data points illustrate for 622 neurons the values of conditional mutual information computed from the start of the trial until the response lick. Information was measured between firing rate and stimulus angle conditional on stimulus category as well as the information between firing rate and category conditional on stimulus angle. Neurons numbered as 1–4 correspond to the same neurons of Figures 3C and 3D, top to bottom. Dashed ellipse in main plot highlights a small set of neurons that carried particularly high conditional category information; dashed ellipse in marginal histogram indicates the same set of neurons.

Figure 6

Choices Are Encoded by Neuronal Population Activity (A) Illustration of the linear discriminant classifier used to decode choices from the population activity. The clouds of circles and triangles show the sets of response population vectors produced in different trials of a single recording session. Circles are associated with turning to the left spout, triangles with the right spout. Symbols representing data from training trials have no border, while the symbols representing data from test trials have a black border and are grouped within the dashed boundaries. Since the test data fall on the correct side of the discrimination boundary (near the training trials of the same symbol), the performance in this schematic dataset would be perfect. (B) Actual trial-by-trial performance of the decoder for each session separately (ten sessions from five rats) is in gray. Black is average across sessions. Decoder performance is measured based on neuronal activity aligned to the time of first response lick; 100 ms windows shifted in 50 ms steps. True decoder performance surpasses 90% correct. (C) Average psychometric curves (see STAR Methods for the fit) based on decoding neuronal responses (solid lines) and the rats’ observed psychometric curves (dashed lines). Data are presented as mean ±SEM. See Figure S5B for individual sessions and modalities.