Toward a unified theory of efficient, predictive, and sparse coding

Abstract

A central goal in theoretical neuroscience is to predict the response properties of sensory neurons from first principles. To this end, "efficient coding" posits that sensory neurons encode maximal information about their inputs given internal constraints. There exist, however, many variants of efficient coding (e.g., redundancy reduction, different formulations of predictive coding, robust coding, sparse coding, etc.), differing in their regimes of applicability, in the relevance of signals to be encoded, and in the choice of constraints. It is unclear how these types of efficient coding relate or what is expected when different coding objectives are combined. Here we present a unified framework that encompasses previously proposed efficient coding models and extends to unique regimes. We show that optimizing neural responses to encode predictive information can lead them to either correlate or decorrelate their inputs, depending on the stimulus statistics; in contrast, at low noise, efficiently encoding the past always predicts decorrelation. Later, we investigate coding of naturalistic movies and show that qualitatively different types of visual motion tuning and levels of response sparsity are predicted, depending on whether the objective is to recover the past or predict the future. Our approach promises a way to explain the observed diversity of sensory neural responses, as due to multiple functional goals and constraints fulfilled by different cell types and/or circuits.

Keywords: efficient coding; information theory; neural coding; prediction; sparse coding.

Fig. 1.

Schematic of modeling framework. (A) A stimulus (stim.) (Upper) elicits a response in a population of neurons (Lower). We look for codes where the responses within a time window of length $\tau$ maximize information encoded about the stimulus at lag $\mathrm{\Delta }$, subject to a constraint on the information about past inputs, $C$. (B) For a given stimulus, the optimal code depends on three parameters: $\tau$, $\mathrm{\Delta }$, and $C$. Previous work on efficient temporal coding generally looked at $\tau >\mathrm{\hspace{0.17em}0}$ and $\mathrm{\Delta }<0$ (blue sphere). Recent work posited that neurons encode maximal information about the future ($\mathrm{\Delta }>0$) but only treated instantaneous codes $\tau \sim \mathrm{\hspace{0.17em}0}$ (red plane). Our theory is valid in all regimes, but we focus in particular on $\mathrm{\Delta }>0$ and $\tau >\mathrm{\hspace{0.17em}0}$ (black sphere). (C) We further explore how optimal codes change when there is a sparse latent structure in the stimulus (natural image patch; Right) vs. when there is none (filtered noise; Left).

Fig. 2.

Dependence of optimal code on decoding lag, $\mathrm{\Delta }$; code length, $\tau$; and coding capacity, $C$. (A) We investigated two types of code: instantaneous codes, where $\tau =\mathrm{\hspace{0.17em}0}$ (C and D), and temporal codes, where $\tau >\mathrm{\hspace{0.17em}0}$ (E and F). (B) Training stimuli (stim.) used in our simulations. Markov stimulus: future only depends on the present state. Two-timescale stimulus: sum of two Markov processes that vary over different timescales (slow stimulus component is shown in red). Inertial stimulus: future depends on present position and velocity. (C) Neural responses to probe stimulus (dashed lines) after optimization (opt.) with varying $\mathrm{\Delta }$ and $\tau =\mathrm{\hspace{0.17em}0}$. Responses are normalized by the final steady-state value. (D) Correlation (corr.) index after optimization with varying $\mathrm{\Delta }$ and $C$. This index measures the correlation between responses at adjacent time steps normalized by the stimulus correlation at adjacent time steps (i.e., $⟨r_t{r}_{t+1}⟩/⟨r_t^2⟩$ divided by $⟨x_t{x}_{t+1}⟩/⟨x_t^2⟩$). Values greater/less than one indicate that neurons temporally correlate (red)/decorrelate (blue) their input. Filled circles show the parameter values used in C. (E and F) Same as C and D but with code optimized for $\tau \gg 0$. Plots in E correspond to responses to probe stimulus (dashed lines) at varying coding capacity and fixed decoding lag (i.e., $\mathrm{\Delta }=3$; indicated by dashed lines in F).

Fig. 3.

Efficient coding of naturalistic stimuli. (A) Movies were constructed from a 10 $\times$ 10-pixel patch (red square), which drifted stochastically across static natural images. (B) Information encoded [i.e., reconstruction (recon.) quality] by neural responses about the stimulus at varying lag (i.e., reconstruction lag) after optimization with $\mathrm{\Delta }=-6$ (blue) and $\mathrm{\Delta }=1$ (red). (C) Spatiotemporal encoding filters for four example neurons after optimization with $\mathrm{\Delta }=-6$. (D) Same as C for $\mathrm{\Delta }=1$. (E) Directionality index of neural responses after optimization with $\mathrm{\Delta }=-6$ and $\mathrm{\Delta }=1$. The directionality index measures the percentage change in response to a grating stimulus moving in a neuron’s preferred direction vs. the same stimulus moving in the opposite direction.

Fig. 4.

Efficient coding of a “Gaussian-bump” stimulus. (A) Stimuli (stim.) consisted of Gaussian bumps that drifted stochastically along a single spatial dimension (dim.) (with circular boundary conditions). (B) Information encoded by neural responses about the stimulus at varying lag, ${\mathrm{\Delta }}_{\mathrm{test}}$, after optimization with varying ${\mathrm{\Delta }}_{\mathrm{train}}$. Black dots indicate the maximum for each column. (C) Response of example neuron to a test stimulus (Upper) and after optimization with $\mathrm{\Delta }=-2$ (blue), $\mathrm{\Delta }=0$ (green), and $\mathrm{\Delta }=2$ (red; Lower). (D) Spatiotemporal encoding filters for an example neuron after optimization with different $\mathrm{\Delta }$. (E) Circular correlation between the reconstructed speed of a moving Gaussian blob and its true speed vs. the circular correlation between the reconstructed position and its true position obtained from neural responses optimized with $\mathrm{\Delta }=\pm 2$ (red and blue curves). Curves were obtained by varying $\gamma$ in Eq. 3 to find codes with different coding capacities. (F) Linear reconstruction of the stimulus trajectory obtained from neural responses optimized with $\mathrm{\Delta }=\pm 2$ (red and blue curves). The full stimulus is shown in grayscale. While coding capacity was chosen to equalize the mean reconstruction error for both models (vertical dashed line in E), the reconstructed trajectory was much smoother after optimization with $\mathrm{\Delta }=2$ than with $\mathrm{\Delta }=-2$. (G) Response sparsity (defined as the negentropy of neural responses) vs. $\mathrm{\Delta }$ (dots indicate individual neurons; the line indicates population average). (H) Delay between stimulus presented at a neuron’s preferred location and each neuron’s maximum response vs. $\mathrm{\Delta }$.