Local statistics in natural scenes predict the saliency of synthetic textures

Abstract

The visual system is challenged with extracting and representing behaviorally relevant information contained in natural inputs of great complexity and detail. This task begins in the sensory periphery: retinal receptive fields and circuits are matched to the first and second-order statistical structure of natural inputs. This matching enables the retina to remove stimulus components that are predictable (and therefore uninformative), and primarily transmit what is unpredictable (and therefore informative). Here we show that this design principle applies to more complex aspects of natural scenes, and to central visual processing. We do this by classifying high-order statistics of natural scenes according to whether they are uninformative vs. informative. We find that the uninformative ones are perceptually nonsalient, while the informative ones are highly salient, and correspond to previously identified perceptual mechanisms whose neural basis is likely central. Our results suggest that the principle of efficient coding not only accounts for filtering operations in the sensory periphery, but also shapes subsequent stages of sensory processing that are sensitive to high-order image statistics.

Fig. 1.

Local distributions of light in natural scenes. (A) Natural images are discretized into 16 equipopulated grayscale levels. Central pixels with intensity σ₀ are chosen randomly, and the distribution of intensities σ₁ at a radius R from the center is sampled. The distance R is represented by the scale bar in pixels. (B), (C), and (D) Histograms of pixel intensities at different distances from the center (shown here for ) if the central pixel σ₀ is black, gray, or white (black, gray, and white lines). Note the difference in histograms for black vs. white central pixel σ₀ for small R (see SI Appendix for P(σ₁) as a function of R).

Fig. 2.

Variability of local intensity histograms. (A) The three principal components of the ensemble of local intensity histograms P_R(σ₁|σ₀) (blue, green, and red), along with the fraction of variance explained by each (bar chart inset). Together, the three components explain ∼90% of the variance between histograms. (B) An orthogonal transformation rotates the three principal components, {v_j} → {w_j}, so that w₁ is as close as possible to a linear function of σ₁, and w₂ to a quadratic function of σ₁ (open blue circles = linear function, θ₁(σ₁); open green circles = quadratic function, θ₂(σ₁)). The three new axes {w_j} relate to variations in mean, variance, and blackshot of the intensity histogram. (C), (D), and (E) IID texture pairs. The intensity of each pixel is chosen independently according to the inset distributions which vary from the uniform distribution by adding or subtracting one principal component w_j from B. These additions vary the mean (C), variance (D) or blackshot (E) of the texture. Only IID textures that vary in at least one of these three ways can be reliably discriminated by humans (15).

Fig. 3.

Spatially correlated textures. (A) Correlations of different orders between four pixels. There are four mean pixel luminances (pink circles), six pairwise correlations (blue lines), four triplet correlations (green triangles), and one quadruplet (fourth-order) correlation (red square); translation invariance reduces the number independent quantities to 10 (numbers in parenthesis). (B) Examples of gliders and the textures they generate (see Materials and Methods). Both displayed textures have equally many white and black pixels, have no second- or third-order correlations, and a large fourth-order correlation. Gliders from Group 1 generate textures that are perceptually salient against a white binary noise background, while textures generated from gliders in Group 2 are not perceptually salient (16, 24). (C) An example of a distribution over binary patterns in a square glider. This distribution generates synthetic textures that have only fourth-order correlations (example texture in 3B, left). (D) To measure the fourth-order correlations in natural scenes we select patches of R × R pixels from whitened natural scenes binarized to have equally many white and black pixels. Each of the eight gliders in B (a square glider shown here in red) is scanned across a patch, and the histogram of binary patterns encountered by the glider is accumulated. (E) Histogram of binary patterns encountered by a square glider scanning a 64 × 64 patch from a natural image. (F) The information about texture in a 64 × 64 binary image patch that is contained in second-, third-, and fourth-order correlations, extracted with a square glider.

Fig. 4.

Fourth-order correlations and perceptual salience. (A) Decomposition of textural information into second (blue) , third (green), and fourth (red) order for the two groups of gliders and many spatial scales (central line = mean, thin surrounding lines = std across gliders). In large image patches there is significantly more information about texture in the correlations between four pixels arranged in the patterns from Group 1 gliders, which also generate perceptually salient textures. Group 1 and Group 2 gliders have similar amounts of I^(2,3). (B) Fourth-order correlations as measured by the parameter Eq. 1. Results at each R are averaged across Group 1 gliders (solid, circles) and Group 2 gliders (dashed, squares), and across many R × R texture patches. The shaded areas show the standard deviation of across texture patches for the two groups. As R increases the correlations within the perceptually salient gliders acquire high statistical significance. (C) The Jensen-Shannon distance, D_JS, between the distributions of sampled across many R × R image patches, for all pairs of gliders (arrangements of four pixels, see Fig. 3B). As R increases, the gliders cluster into two sets, which respectively generate the perceptually salient (Group 1) and nonsalient (Group 2) textures determined by psychophysical studies (17).