Categorical clustering of the neural representation of color

Abstract

Cortical activity was measured with functional magnetic resonance imaging (fMRI) while human subjects viewed 12 stimulus colors and performed either a color-naming or diverted attention task. A forward model was used to extract lower dimensional neural color spaces from the high-dimensional fMRI responses. The neural color spaces in two visual areas, human ventral V4 (V4v) and VO1, exhibited clustering (greater similarity between activity patterns evoked by stimulus colors within a perceptual category, compared to between-category colors) for the color-naming task, but not for the diverted attention task. Response amplitudes and signal-to-noise ratios were higher in most visual cortical areas for color naming compared to diverted attention. But only in V4v and VO1 did the cortical representation of color change to a categorical color space. A model is presented that induces such a categorical representation by changing the response gains of subpopulations of color-selective neurons.

Figure 1.

Stimuli. A, Color stimuli were square-wave spiral gratings, within a circular aperture (0.40 to 10 degree radius). Stimulus duration was 1 s, and the interstimulus interval was 2–5 s in steps of 1.5 s. B, Locations of the 12 different stimulus colors in DKL space. C, The same 12 colors in CIE (Commission internationale de l'eclairage) 1931 xyY space. Diagonal lines represent the axes of DKL space. Dashed triangle, Gamut of the LCD monitor used in the fMRI experiments.

Figure 2.

Schematic overview of the analysis procedure using simulated data. A, Voxel time courses were preprocessed (motion compensation, drift correction, high-pass filtering). B, Response amplitudes for each voxel were computed from the preprocessed voxel time courses, for each stimulus color, for each run, using linear regression (general linear model). C, Neural color space. A forward model (see Materials and Methods) was used to transform the data from the high-dimensional space of fMRI responses (dimensionality equal to the number of voxels) to a lower-dimensional space of six channels (each specified by a basis function for its color tuning curve). We refer to this six-dimensional space of channel responses as a neural color space; each stimulus color evokes a vector of six channel responses, i.e., a position in the six-dimensional space. The distances between colors in this space capture similarities between the corresponding patterns of activity. D, Voxel-based tuning curve. A von Mises function was fitted to the responses of each voxel to the 12 colors, separately for each task and visual area. The best-fit parameters of each tuning curve specified color preference, tuning width, and gain. E, Mean response and SNR. Response amplitudes were averaged across voxels, separately for each task and visual area. The SNR of each voxel was computed as the mean response of the voxel across stimulus colors divided by the SD across colors. The SNR of each visual area was computed by averaging SNRs across voxels. F, Two-dimensional visualization of the neural color space. The six-dimensional channel response matrices (C) were reduced by means of principal component analysis (PCA) to two dimensions for visualization. G, Clustering. The neural color spaces (C) were used to compute a categorical clustering index. The clustering index reflected the similarity between activity patterns evoked by stimulus colors within a perceptual category, compared to between-category colors. In addition, k-means and EM clustering algorithms were used to partition the neural color spaces into clusters. H, Hierarchical organization of the neural color space. The k-means and EM algorithms were used to compute hierarchical clustering, visualized by means of dendrograms.

Figure 3.

Psychophysics: color categorization. First row, Twelve stimulus colors used in the fMRI experiments. Second row, Sixty-four colors (taken from the same circular color space as the 12 colors in the first row) used in the psychophysics experiment. Rows S1, S2, S3, S4, and S5 represent the category boundaries for each subject, respectively. Subjects were asked to partition the 64 stimulus colors into five bins, according to what they felt was a natural division of the continuous color space into five categories. The color representing each category is the average of all the colors that the subject included in the category.

Figure 4.

Categorical clustering in neural color spaces. A, Visual area VO1. B, V4v. C, V3. Left, Diverted attention task. Right, Color-naming task. Channel responses were estimated from the fMRI measurements using the forward model and visualized in 2D plots using principal component analysis. Coordinates of each stimulus color in the plots correspond to the first two principal component scores of the channel responses (see Materials and Methods). Each individual point is the mean coordinate averaged across 100 bootstrapped estimates of the channel responses (see Materials and Methods). Errors bars represent the horizontal and vertical SDs of each coordinate.

Figure 5.

Color-category specific clustering. A, Categorical clustering indices for each visual area and for each of the two task conditions. Light gray bars, Diverted attention task; dark gray bars, color-naming task; dashed line, baseline categorical clustering index for a perfectly circular color space that shows no clustering at all. Asterisks indicate visual areas/tasks for which the index was significantly greater than expected from a circular color space. Error bars indicate the SD in clustering across bootstrapped channel responses. B, Adjusted Rand indices for each visual area and each task (same format as in A). The adjusted Rand index quantifies the correspondence between clustering in the neural color spaces and the perceptual categories. The solid line indicates the mean of the baseline distribution of adjusted Rand indices for a perfectly circular color space that shows no clustering at all, computed using a Monte Carlo simulation. The dashed lines indicate the 5th and 95th percentiles for the baseline distribution. Asterisks indicate visual areas/task for which the index was significantly greater than expected from a circular color space. Error bars indicate the SD in the adjusted Rand indices computed across bootstrapped channel responses.

Figure 6.

Hierarchical clustering. A, Dendrogram extracted from VO1 activity during the diverted attention task. The height of each pair of branches in the dendrogram (cluster tree) represents the similarity between the patterns of activity evoked by the two colors being connected. B, Dendrogram extracted from VO1 activity during the color-naming task. C, Dendrogram extracted from color categorization psychophysics (see Fig. 2). The height of each pair of branches represents the relative frequency with which subjects placed the two stimulus colors in the same category.

Figure 7.

Response amplitudes, gain, and voxel tuning width. A, Response amplitudes, averaged across all voxels within each visual area, for each of the two task conditions. Light gray bars, Diverted attention task; dark gray bars, color-naming task. Error bars indicate SDs across subjects and scanning sessions. Asterisks indicate visual areas for which the responses were significantly greater during the color-naming task compared to the diverted attention task. B, Estimated gain in each visual, for each task. Error bars indicate SDs across subjects and scanning sessions. C, Estimated average tuning width (full-width at half maximum, in degrees) for each visual area and for each task. Error bars indicate SDs across subjects and scanning sessions.

Figure 8.

A neural model for categorical clustering. A, Color is represented by the activity of broadly tuned neurons, each with a different preferred color. Each black curve is a simulated tuning curve. For the diverted attention task, all neurons have the same gain (peak height of the tuning curves). B, Circular neural color space extracted from the simulated responses in A. C, For the color-naming task, responses from some visual areas (e.g., V4v and VO1) are enhanced, but only for neurons that prefer colors near the category centers (e.g., red, green, blue, yellow). D, Neural color space extracted from the simulated responses in C, showing categorical clustering.