The statistics of local motion signals in naturalistic movies

Abstract

Extraction of motion from visual input plays an important role in many visual tasks, such as separation of figure from ground and navigation through space. Several kinds of local motion signals have been distinguished based on mathematical and computational considerations (e.g., motion based on spatiotemporal correlation of luminance, and motion based on spatiotemporal correlation of flicker), but little is known about the prevalence of these different kinds of signals in the real world. To address this question, we first note that different kinds of local motion signals (e.g., Fourier, non-Fourier, and glider) are characterized by second- and higher-order correlations in slanted spatiotemporal regions. The prevalence of local motion signals in natural scenes can thus be estimated by measuring the extent to which each of these correlations are present in space-time patches and whether they are coherent across spatiotemporal scales. We apply this technique to several popular movies. The results show that all three kinds of local motion signals are present in natural movies. While the balance of the different kinds of motion signals varies from segment to segment during the course of each movie, the overall pattern of prevalence of the different kinds of motion and their subtypes, and the correlations between them, is strikingly similar across movies (but is absent from white noise movies). In sum, naturalistic movies contain a diversity of local motion signals that occur with a consistent prevalence and pattern of covariation, indicating a substantial regularity of their high-order spatiotemporal image statistics.

Keywords: glider motion; local motion signals; non-Fourier motion; spatiotemporal image statistics.

Figure 1

A summary of calculation of the basic local motion scores. (A) The templates used to quantify each kind of local motion. (B) Details of the various motion score calculations. First, Weber contrast values in the solid-bordered checks are multiplied together. These products are then summed in an opponent fashion (scores from red-outlined configurations are added; scores from green-outlined configurations are subtracted) to generate a local score motion signal (see text for details). (C) How opponency removes spurious signals due to static luminance edges. Each subpanel diagrams the result of a computation of the local motion score when the template (stars) is positioned near a luminance edge. The four components of each subpanel correspond to the four components of the subpanels in B. For F and NF templates, left-oriented and right-oriented placements of the template each include the same number of dark and light checks. Thus, the left and right components of the calculation result in cancellation by their opponency. In contrast, for the G template, the left-oriented placement of the template contains one dark check, while the right-oriented placement contains two dark checks. Thus, the left-versus-right opponency does not result in cancellation. However, forward and backward placements of the template are matched in terms of the luminances of the checks that they contain, and therefore the forward-versus-backward opponency properly cancels the spurious motion signal. Note that for the F and NF templates, this second explicit stage of opponency has no effect. This is because of their symmetry: A left-to-right flip of the template is the same as a forward-to-backward flip (B).

Figure 2

A summary of calculation of the RMO and PMO local scores in a spatiotemporal ROI, using NF-S motion as an example; further details are in the text. (A) The library of eight template colorings consistent with NF-S motion. Note that all colorings have an even number of black checks. (The four rightmost colorings, marked in purple, are the templates used for pure NF-S, as the two-check F templates that they contain are inconsistent with F motion.) (B) Calculation of RMO and PMO scores for a 1×4×4 spatiotemporal ROI. For the RMO method (top), we consider all placements of the template within the ROI. There are six such placements (red dashes), and we tally the placements that yield colorings contained in the library of panel A, as these are the placements in which the black and white checks are consistent with NF-S motion. Checkmarks indicate the placements that result in colorings that are within the library; circles indicate the placements that result in colorings that are not in the library. Tallying the number of placements in the library yields a unidirectional RM motion score (in this case, right forward). Analogous scores are calculated by reversing the NF-S template in space (left forward) and time (right backward, left backward). These four unidirectional RM scores are combined in an opponent calculation to yield the RMO score for the ROI. For the PMO method (bottom) the entire ROI is treated as a whole. We determine the fewest number of checks that must be changed so that every placement of the template within the ROI yields a coloring that is in the library of panel A. In this case, changing two checks suffices: When these two checks are flipped in contrast (dashed arrows), all glider placements are in the library, and the resulting ROI is entirely consistent with NF-S motion. The tally of these changes yields the right, forward unidirectional PM signal. These four unidirectional PM signals are combined by an opponent computation to yield the PMO score for the ROI.

Figure 3

Prevalence of different kinds of motion signals is similar across movies. For each movie, SM scores (see Materials and methods) were calculated for each movie segment, and the distribution is summarized by the median (horizontal line), the interquartile range (heavy vertical line), the “whiskers” (thin vertical line, covering four times the interquartile range), and the outliers (individual symbols, outside the range of the whiskers). Values are normalized by SM motion scores obtained from movies of random pixels of similar segment length. Each motion was calculated with respect to its relevant template shape (see Figure 1A) in the YT plane (i.e., horizontal motion); each check corresponded to a single pixel in the discretization of the movie (256×256 pixels per frame, 24 frames per second). Movies were (1) The 39 Steps (1935), (2) Anna Karenina (1935), (3) A Night at the Opera (1935), and (4) Mr. and Mrs. Smith (2005).

Figure 4

Prevalence of different kinds of motion signals, analyzed at a coarse spatial scale, is similar across movies. For each movie, SM scores were calculated after downsampling each 16×16 block of pixels in the original movie to a single check. For other details see Figure 3.

Figure 5

The effect of binarization on local motion scores for (A) F, (B) NF-S, (C) NF-T, and (D) G motions. SM scores were calculated for each movie segment based on raw luminance values (abscissa), and also following binarization with the threshold set at the overall shot median luminance (ordinate). No spatial downsampling was applied. A random sample of 500 shots from each movie is presented here. Movies were color coded as follows: (red) The 39 Steps, (blue) Anna Karenina, (green) A Night at the Opera, and (cyan) Mr. and Mrs. Smith (2005). The black line is the line of identity.

Figure 6

Prevalence of different kinds of motion signals is similar across binarized movies. For each movie, data were first converted to −1 or +1 using a threshold equal to the median overall luminance value within each shot, and SM scores were then calculated. For other details, see Figure 3.

Figure 7

Prevalence of different kinds of motion signals, analyzed at a coarse spatial scale, is similar across binarized movies. For each movie, SM scores were calculated after downsampling each 16×16 block of pixels in the original movie to a single check, and then binarization. For other details see Figure 4.

Figure 8

Prevalence of different kinds of motion signals is similar across movies, as measured by the PMO score. The ROI consisted of a 4×4 block of checks in the YT plane (i.e., horizontal motion); each check corresponded to a single pixel in the discretization of the movie (256×256 pixels per frame, 24 frames per second). For other details see Figure 3.

Figure 9

Prevalence of different kinds of motion signals is similar across movies, as measured by the RMO score. The ROI consisted of a 4×4 block of checks in the YT (horizontal motion) plane. For other details see Figure 8.

Figure 10

Covariance patterns of motion scores: (A) PMO and (B) RMO. Within each scattergram, each point represents a pair of normalized motion scores determined from a single movie segment (“shot”). Axes range from 0 to 2 (PMO) and 0 to 1 (RMO). The number in each plot indicates the average ratio between the pair of motion scores; the two sloping lines in each plot indicate the wedge that contains 95% of the values. Large values of one motion score typically occur with large values of the other scores, but the ratios between a pair of scores can vary by up to a factor of two. (Pure NF-S and NF-T PMO scores are identical, and the two G motion RMO scores are identical; see Supplement S2.) Analysis was carried out in the YT plane (horizontal motion) at the maximum resolution of the database for The 39 Steps.