Distinct ventral stream and prefrontal cortex representational dynamics during sustained conscious visual perception

Abstract

Instances of sustained stationary sensory input are ubiquitous. However, previous work focused almost exclusively on transient onset responses. This presents a critical challenge for neural theories of consciousness, which should account for the full temporal extent of experience. To address this question, we use intracranial recordings from ten human patients with epilepsy to view diverse images of multiple durations. We reveal that, in sensory regions, despite dramatic changes in activation magnitude, the distributed representation of categories and exemplars remains sustained and stable. In contrast, in frontoparietal regions, we find transient content representation at stimulus onset. Our results highlight the connection between the anatomical and temporal correlates of experience. To the extent perception is sustained, it may rely on sensory representations and to the extent perception is discrete, centered on perceptual updating, it may rely on frontoparietal representations.

Keywords: CP: Neuroscience; discrete perception; distributed coding; neural adaptation; neural correlates of consciousness; perceptual awareness; representation; representational drift; time-consciousness; visual perception.

Figure 1.. Experimental setup and design, electrode locations, and single-electrode response dynamics showing substantial attenuation after the onset response

(A) Experimental setup and example images from the four categories. (B) Two target types (together 10% of trials): (1) clothing items and (2) blurring of the image. (C) Electrode locations (pooled across patients), colored by ROI. The same color scheme applies for all figures. See Tables S1C and S1D. (D) HFA response dynamics relative to the prestimulus baseline (positively responding electrodes). To highlight response dynamics, for each electrode, only trials from categories it was responsive to were included. Shaded area: SEM across electrodes (number shown in the inset). See also Figures S1A-S1E. (E) Relative attenuation in HFA responses from peak to 800–900 ms (peak minus end activity relative to peak; attenuation >100% when end activity is lower than baseline levels). Colored dots: single electrodes (same as in D), colored horizontal lines: mean across electrodes, white dots: median across electrodes, gray vertical bars: interquartile range, contour lines: kernel probability density estimate. Black horizontal lines and asterisks: significant post hoc differences between ROIs (Tukey-Kramer method). (F) Category selectivity dynamics in category-selective electrodes (η² expressed as percentage of explained variance from a one-way ANOVA between categories). Notations as in (D). See Figures S1F-S1H for electrode locations and single-electrode properties. (G) Relative attenuation in selectivity, higher numbers indicate stronger attenuation. Notations as in (E). (D–G) Image durations ≥900 ms.

Figure 2.. Multivariate state-space dynamics in sensory regions track the duration of the stimulus

(A) State-space trajectories per category (image durations ≥ 900 ms; first 3 principal components using all responsive electrodes, responses in each category averaged prior to principal-component analysis [PCA]; PCA was performed solely for visualization purposes). Trajectory lines are darker and thicker as time progress, dots are 5 ms apart. Insets: point-by-point distance of each trajectory from the (baseline) prestimulus state (computed using the full response, prior to PCA); colored vertical lines on the abscissa: peak distance times. See Figures S2 and S3E for extended analysis of state-space trajectories. (B) Dynamics of distance from baseline (face images, see also Figure S3A; for other categories, see Figure S3B; baseline subtracted for presentation purposes). Offsets marked by vertical lines with corresponding hues. Horizontal bars: time points of significant differences between durations (max-statistic permutations, p < 0.05; 1,500 vs. 900 ms, 900 vs. 300 ms; colors correspond to the shorter duration in the contrast). Traces are cropped 600 ms after stimulus offset (shortest inter-stimulus interval, ISI). Absolute distances are comparable within regions at different time points, not between regions, as magnitude is dependent on the number of electrodes. See also Figures S3C and S3D.

Figure 3.. Visual category representation is sustained and stable in sensory regions and transient in frontoparietal regions

(A) Schematic illustration of decoding for a single time point: colored dots represent single-trial responses; a gray bar represents the linear classifier. (B) Peak decoding. Significance computed by permutation testing. Gray horizontal lines: significance threshold (max-statistic permutation testing; threshold is higher for comparisons involving categories with less exemplars). (C) Decoding dynamics (face-watch; other comparisons and direct comparisons between regions: Figure S4). Dashed lines: stimulus onset and chance level. Red bars: significant clusters by cluster-based permutations, black bars: significant points by point-by-point permutation testing (FDR corrected). (D) Temporal generalization matrices (face-watch; other comparisons are shown for VT and Occ in Figures S5D and S5E). The diagonal (training and testing on the same time point) corresponds to the time courses in (C). Black contour: contiguous points significant by point-by-point permutation testing (FDR corrected). Right-side plots: mean generalization dynamics for 200 ms blocks of training time. Red bars: testing points significant for ≥50% of training points in the range. (B–D) All stimuli durations ≥900 ms. See Figures S5 and S6 for single-patient results, analysis of Occ and PFC subregions, and control analyses ruling out the contribution of ocular muscle activity to PFC decoding.

Figure 4.. Category information in visual, but not frontoparietal, regions tracks stimulus duration

(A) Decoding dynamics per duration (darker lines correspond to longer stimuli, offsets marked by corresponding vertical lines). Horizontal bars of corresponding color: significant decoding clusters (cluster permutation test); p values are indicated by bottom-right corner asterisks (corresponding to the cluster temporal order). Traces are cropped 600 ms after stimulus offset (shortest ISI). See also Figures S7A-7E. (B) Difference of decoding time courses (1,500–900 ms, dark lines; 900–300 ms, bright lines). Statistical testing and notations as in (A). (C) TGMs per duration (see Figure S7F for the other regions). Dashed lines: stimulus onset and offset and the diagonal (corresponding to the dynamics in A). Black contours: significant clusters; corresponding p values shown above each TGM. (D) Comparison between durations (notation and statistical testing as in C). (A–D) Occ and VT shown for patients S4-S10 (similar for S1-S3, Figure S7B). PFC is shown for S4-S10 as well (no responsive PFC electrodes for S1-S3). Par is shown for S1-S3 (not significant for S4-S10). See STAR Methods and Tables S1F and S1G for the rationale behind the split. All panels depict face-watch decoding (object-watch shown in Figure S7D).

Figure 5.. Exemplar information is sustained and stable in sensory regions and transient in frontoparietal regions

(A) Schematic illustration of item reliability (IR): for each image (red star), we compare the vector of dissimilarities with all other images in repetition 1 (full shapes) with the vector of dissimilarities with all other images in repetition 2 (empty shapes). (B) Geometry reliability (GR): we first compute the dissimilarities between all images in both repetitions, resulting in a symmetric matrix with four distinct representational structures (top). Pairs of geometries are averaged to yield geometries 1 and 2, and the two geometries are correlated. See STAR Methods for more details about both reliability metrics. (C) Models of potential category information in the representational geometry. All models assume that exemplars within each category are similar to each other and dissimilar to other categories. Three of the models add a hierarchy of similarity between categories (STAR Methods). (D–E) IR and GR dynamics. Colored lines: full representational geometry (see Figure S8 for control analyses); gray lines: after partialing out the model explaining the most category information (see Figure S10 for more details about the calculation and removal of other category models; Figure S11 for single-category results). Horizontal bars of the same color mark significant clusters (cluster permutation test); p values indicated by bottom-left corner asterisks (corresponding to the cluster temporal order). Dashed lines: stimulus onset and chance level (no single-item information). (F) IR temporal stability (GR: Figure S9B; removing category information: Figures S10D and S10E; other regions: Figure S9A; other controls: Figures S9C-S9E). Notations as in Figure 3D. (D–F) Images presented at least twice with duration ≥900 ms.