A hierarchy of temporal receptive windows in human cortex

Abstract

Real-world events unfold at different time scales and, therefore, cognitive and neuronal processes must likewise occur at different time scales. We present a novel procedure that identifies brain regions responsive to sensory information accumulated over different time scales. We measured functional magnetic resonance imaging activity while observers viewed silent films presented forward, backward, or piecewise-scrambled in time. Early visual areas (e.g., primary visual cortex and the motion-sensitive area MT+) exhibited high response reliability regardless of disruptions in temporal structure. In contrast, the reliability of responses in several higher brain areas, including the superior temporal sulcus (STS), precuneus, posterior lateral sulcus (LS), temporal parietal junction (TPJ), and frontal eye field (FEF), was affected by information accumulated over longer time scales. These regions showed highly reproducible responses for repeated forward, but not for backward or piecewise-scrambled presentations. Moreover, these regions exhibited marked differences in temporal characteristics, with LS, TPJ, and FEF responses depending on information accumulated over longer durations (approximately 36 s) than STS and precuneus (approximately 12 s). We conclude that, similar to the known cortical hierarchy of spatial receptive fields, there is a hierarchy of progressively longer temporal receptive windows in the human brain.

Figure 1.

Correlation analysis reveals that responses in many visual cortical areas are insensitive to time reversals: illustration from area MT+. A, Representative frames from the silent films (taken from City Lights). Observers viewed two presentations of each original movie (forward) and each time-reversed movie (backward). B, Average time courses (8 observers) sampled from MT+ for the two forward presentations (F1, F2) of the silent films. C, The average time courses in MT+ for the two backward presentations (B1, B2). D, Superimposed traces of the time-reversed version of the backward time course (rB) and the forward time course (F). Both time courses were shifted (Δt = 5 s) to correct for hemodynamic delay in the fMRI responses. E, Cross-correlations: forward 1 versus forward 2 (C_F1:F2, black); backward 1 versus backward 2 (C_B1:B2, red); forward versus backward (C_F:B, blue); reversed backward versus forward (C_rB:F, green). The correlation was computed at different lags (2 s intervals); the peak at lag 0 indicates reproducible responses that were time-locked to the movies. C_F:B, which is an estimate of the arbitrary correlation values that can be expected from such complex stimuli, is much lower than the other three cross-correlations. Error bars represent the SEM across nonoverlapping movie segments.

Figure 2.

Effect of time reversal across the cortical surface. A, Maps of the correlations between the two forward time courses (C_F1:F2, red) and the reversed-backward and forward time courses (C_rB:F, green). Regions in which the responses were time-reversible exhibit both high C_F1:F2 and high C_rB:F (overlap, orange). Correlation maps are shown on inflated (top) and unfolded (bottom) left and right hemispheres. The maps show only voxels for which the correlation exceeded a threshold value (0.3, chosen because it was above the highest C_F,B value exhibited by any voxel). C_rB:F was high in a number of posterior cortical regions, indicating that the responses to the backward films were a simple time reversal of the responses to the forward films. In other brain regions, we observed high C_F1:F2, but low C_rB:F (red). White outlines mark the main regions in which responses were not time reversible. Anatomical abbreviations: ITS, inferior temporal sulcus; LS, lateral sulcus; STS, superior temporal sulcus; TPJ, temporal parietal junction; CS, central sulcus; IPS, intraparietal sulcus. Several higher-order visual areas were functionally defined based on their responses to faces (red outlines), objects (blue outlines), and houses (green outlines). Functionally and anatomically defined cortical areas: V1, primary visual cortex; MT+, MT complex responsive to visual motion; PPA, parahippocampal place area; FFA, fusiform face area; LO, lateral occipital complex responsive to pictures of objects; STS-face, area in superior temporal sulcus responsive to faces. B, Reliability of response time courses for regions that exhibited time-reversible responses: MT+, V1, LO, and PPA. In each of these cortical regions, the values of C_rB:F and C_B1:B2 were similar to the C_F1:F2 values. Error bars represent the SEM across nonoverlapping movie segments. C, Brain regions in which responses were not time reversible (white outlines in A): precuneus, posterior LS, TPJ, FEFs, and the posterior STS. In each of these cortical regions, the values of C_B1:B2 and C_rB:F were much less than C_F1:F2.

Figure 3.

Coherence analysis and power spectra analysis. A, Coherence (correlation as a function of frequency) between the two presentations of the forward (c_F1:F2, black), backward (c_B1:B2, red), reversed-backward versus forward (c_rB:F, green), and forward versus backward (c_F:B, blue) silent films. The coherence was computed for each ROI, and averaged across regions that exhibited similar dependency on time scale in the correlation-based analyses: left, V1 and MT+; middle, LO, PPA, STS, and precuneus; right, LS, TPJ, and FEF. Error bars represent the SE across ROIs. The results of this analysis support the findings presented in Figure 2, B and C. Left, The coherence values of c_rB:F and c_B1:B2 were similar to the c_F1:F2 values across all frequencies. Right, The coherence values of c_rB:F and c_B1:B2 were much less than c_F1:F2 and closer to c_F:B across all frequencies. Middle, The coherence values of c_rB:F and c_B1:B2 were in between c_F1:F2 and c_F:B. B, Coherence between the two presentations of the uninterrupted forward films (c_F1:F2, black), and piecewise scrambled films at a long time scale (36 ± 4 s, red), intermediate time scale (12 ± 3 s, green), and short time scale (4 ± 1 s, blue). The results of this analysis support the findings presented in Figure 5. Left, No difference in the coherence values across conditions. Middle, Lower coherence values when the films clips were shuffled at a short time scale. Right, High coherence values only for the longest time scales. C, Power spectra of the responses were similar across all experimental conditions. Log of power spectrum for the forward (black), backward (gray), and randomly shuffled clips at a long time scale (36 ± 4 s, red), intermediate time scale (12 ± 3 s, green), and short time scale (4 ± 1 s, blue). Power spectra were computed separately for each subject and each ROI, and then averaged across subjects and ROIs. The results of this analysis support the findings presented in Figure 6A. The power spectra were similar for all five conditions, demonstrating that observed differences in correlation values across regions were not confounded with a decrease in the response amplitudes in any frequency band. A variant of this analysis was also performed, with similar results, in which the response time courses were first averaged across subjects before computing the power spectra.

Figure 4.

Reproducible eye movements regardless of time reversal. A, Cross-correlations of eye positions over time for the two forward presentations of the silent films. Red curve, Vertical eye position. Blue curve, Horizontal eye position. The correlation was computed at different lags (0.2 s intervals); the peak at lag 0 indicates a reproducible sequence of eye positions in the two presentations, that was time-locked to the movies. Error bars indicate SEM across observers. B, Cross-correlations for the two backward presentations. C, Cross-correlations for the reversed-backward and forward presentations. The high correlations indicate that the sequence of eye positions was similar (but reversed in order) during the forward and backward viewings. Correlation coefficients at zero lag corresponding to forward:forward, backward:backward, and forward:reversed-backward were not statistically distinguishable from one another C_F1:F2:C_B1:B2 horizontal eye position, p = 0.54; C_F1:F2:C_B1:B2 vertical eye position, p = 0.43; C_F1:F2:C_rB:F horizontal eye position, p = 0.22; C_F1:F2:C_rB:F vertical eye position. p = 0.3). D, Cross-correlations for the backward (not flipped) and forward presentations.

Figure 5.

Effect of scrambling at different time scales. A, The reliability of response time courses in each of several brain regions. Time courses were sampled from the same regions of interest as in Figure 2, B and C (V1, MT+, LO, PPA, FFA, STS, precuneus, LS, TPJ, and FEF). Black, Correlations for repeated presentations of the uninterrupted forward films (C_F1:F2). Red, Piecewise scrambled at a long time scale (36 ± 4 s). Green, Intermediate time scale (12 ± 3 s). Blue, Short time scale (4 ± 1 s). Asterisks denote statistically significant differences between a given time scale and the uninterrupted forward film (p < 0.05, one-tailed t test). Early visual areas (V1, MT+) exhibited no difference across conditions. High-order visual areas (LO, FFA, PPA, STS, precuneus) exhibited smaller correlation values when the films clips were scrambled at a short time scale. LS, TPJ, and FEF responses were reliable only for the longest time scales. B, Regions in which the responses were reliable (r >0.3) for all piecewise scrambled movies (at long, middle, and short time scales; e.g., MT and V1 above) are marked in blue. Regions in which the responses were reliable (r >0.3) only for the long and intermediate time scales, but not when the film clips were scrambled at a short time scale (e.g., LO, PPA, FFA, STS above) are marked in green. Regions in which the responses were reliable (r >0.3) only for the longest time scales (e.g., LS, TPJ, FEF above) are marked in red. Figure layout is identical to that in Figure 2A.

Figure 6.

Dissociation between reliability and response amplitude. A, Dynamic range of responses. SD of the fMRI responses (dynamic range) for forward, time-reversed, and scrambled films. Time courses were sampled from a subset of the same regions of interest (MT+, STS, LS, TPJ, FEF and precuneus) used in Figure 2, B and C. Inset, Time course of responses for the forward and reversed-backward silent films from FEF in one representative subject. Bars and error bars indicate the mean and SD across observers (n = 8 for the forward and backward films; n = 5 for the scrambled films). Note that the dynamic range of the fMRI responses within each brain region was similar for all five films. B, fMRI responses to the block-alternation control experiment, in which short movie clips were presented forward (red) and backward (blue) relative to blank screen. Time courses were sampled from a subset of the same regions of interest (MT+, STS, FEF, LS, TPJ and precuneus) used in Figure 2. Dotted lines mark the beginning and end of the epoch. In all regions the response modulations evoked by the backward movie clips were the same as or larger than those evoked by forward clips.