Nonlinear Hebbian Learning as a Unifying Principle in Receptive Field Formation

Abstract

The development of sensory receptive fields has been modeled in the past by a variety of models including normative models such as sparse coding or independent component analysis and bottom-up models such as spike-timing dependent plasticity or the Bienenstock-Cooper-Munro model of synaptic plasticity. Here we show that the above variety of approaches can all be unified into a single common principle, namely nonlinear Hebbian learning. When nonlinear Hebbian learning is applied to natural images, receptive field shapes were strongly constrained by the input statistics and preprocessing, but exhibited only modest variation across different choices of nonlinearities in neuron models or synaptic plasticity rules. Neither overcompleteness nor sparse network activity are necessary for the development of localized receptive fields. The analysis of alternative sensory modalities such as auditory models or V2 development lead to the same conclusions. In all examples, receptive fields can be predicted a priori by reformulating an abstract model as nonlinear Hebbian learning. Thus nonlinear Hebbian learning and natural statistics can account for many aspects of receptive field formation across models and sensory modalities.

Fig 1. The effective Hebbian nonlinearity of plastic cortical networks.

(a) Receptive field development between an input layer of neurons with activities x_i, connected by synaptic projections w_i to a neuron with firing rate y, given by an f-I curve y = g(w^Tx)). Synaptic connections change according to a Hebbian rule Δw_i ∝ x_i h(y). (b) f-I curve (blue) of a GIF model [26] of a pyramidal neurons in response to step currents of 500 ms duration (dashed line: piece-wise linear fit, with slope a = 143 Hz/nA and threshold θ = 0.08 nA). (c) Plasticity function of the triplet STDP model [24] (blue), fitted to visual cortex plasticity data [27, 24], showing the weight change Δw_i as a function of the post-synaptic rate y, under a constant pre-synaptic stimulation x_i (dashed line: fit by quadratic function, with LTD factor b = 22.1 Hz). (d) The combination of the f-I curve and plasticity function generates the effective Hebbian nonlinearity (dashed line: quadratic nonlinearity with LTD threshold θ₁ = 0.08 nA, LTP threshold θ₂ = 0.23 nA).

Fig 2. Simple cell development from natural images regardless of specific effective Hebbian nonlinearity.

(a) Effective nonlinearity of five common models (arbitrary units): quadratic rectifier (green, as in cortical and BCM models, θ₁ = 1., θ₂ = 2.), linear rectifier (dark blue, as in L₁ sparse coding or networks with linear STDP, θ = 3.), Cauchy sparse coding nonlinearity (light blue, λ = 3.), L₀ sparse coding nonlinearity (orange, λ = 3.), and negative sigmoid (purple, as in ICA models). (b) Relative optimization value 〈F(w^Tx)〉 for each of the five models in a, for different preselected features w, averaged over natural image patches x. Candidate features are represented as two-dimensional receptive fields. For all models, the optimum is achieved at the localized oriented receptive field. Inset: Example of natural image and image patch (red square) used as sensory input. (c) Receptive fields learned in four trials for ten effective Hebbian functions f (from top: the five functions considered above, u³, − sin(u), u, (|u| − 2)₊, − cos(u)) (left column), and their opposites − f (right column). The first seven functions (above the dashed line) lead to localized oriented filters, while a sign-flip leads to random patterns. Linear or symmetric functions are exceptions and do not develop oriented filters (bottom rows).

Fig 3. Selectivity index for different nonlinearities f.

(a) Quadratic rectifier (small graphic, three examples with different LTP thresholds) with LTD threshold at θ₁ = 1: LTP threshold must be below 3.5 to secure positive selectivity index (green region, main Fig) and learn localized oriented receptive fields (inset). A negative selectivity index (red region) leads to a random connectivity pattern (inset) (b) Linear rectifier: activation threshold must be above zero. (c) Sigmoid: center must be below a = − 1.2 or, for a stronger effect, above a = +1.2. The opposite conditions apply to the negative sigmoid. (d) Cauchy sparse coding nonlinearity: positive but weak feature selectivity for any sparseness penalty λ > 0. Insets show the nonlinearities for different choices of parameters.

Fig 4. Optimal receptive field shapes in model networks induce diversity.

(a-f) Gray level indicates the optimization value for different lengths and widths (see inset in a) of oriented receptive fields for natural images, for the quadratic rectifier (left, see Fig 2a), linear rectifier (center) and L₀ sparse coding (right). Optima marked with a black cross. (a-c) Colored circles indicate the receptive fields of different shapes developed in a network of 50 neurons with lateral inhibitory connections. Insets on the right show example receptive fields developed during simulation. (d-f) Same for a network of 1000 neurons.

Fig 5. Diversity of receptive field size, position and orientation.

(a) The optimization value of localized oriented receptive fields, within a 16x16 pixel patch of sensors, as a function of size (see Methods), for five nonlinearities (colors as in Fig 2a). Optimal size is a receptive field of width around 3 to 4 pixels (filled triangles). (b) The optimization value as a function of position of the receptive field center, for a receptive field width of 4 pixels, indicates invariance to position within the 16x16 patch, except near the borders. (c) The optimization value as a function of orientation shows preference toward horizontal and vertical directions, for all five nonlinearities. (d) Receptive field position, orientation and length (colored bars) learned for 50 single-neuron trials. The color code indicates different orientations. (e) Receptive field positions and orientations learned in a 50 neuron network reveal diversification of positions, except at the borders. (f) With 1000 neurons, positions and orientations cover the full range of combinations (top). Selecting 50 randomly chosen receptive fields highlights the diversification of position, orientation and size (bottom). Receptive fields were learned through the quadratic rectifier nonlinearity (θ₁ = 1., θ₂ = 2.).

Fig 6. Receptive fields for non-whitened natural images.

(a-i) Receptive field obtained for network simulations with the quadratic rectifier (top), linear rectifier (center) and L₀ sparse coding (bottom). For few neurons (left and center), the principal components dominate the optimization and receptive fields are nonlocal, since they extend over most of the image patch. For an overcomplete network with 1000 neurons (right), lateral inhibition promotes diversity of receptive fields, including more localized ones. (insets) Sample receptive fields developed for each simulation.

Fig 7. Nonlinear Hebbian learning across sensory modalities.

(a) The auditory input is modeled as segments over time and frequency (red) of the spectrotemporal representation of speech signals. (b) The V2 input is assembled from the output of modeled V1 complex cells at different positions and orientations. Receptive fields are represented by bars with size proportional to the connection strength to the complex cell with the respective position and orientation. (c) Strabismic rearing is modeled as binocular stimuli with non-overlapping left and right eye input patches (red). (d-f) Statistical distribution (log scale) of the input projected onto three different features for speech (d), V2 (e) and strabismus (f). In all three cases, the learned receptive field (blue, inset) is characterized by a longer tailed distribution (arrows) than the random (red) and comparative (green) features. (g-i) Relative optimization value for five nonlinearities (same as in Fig 2), for the three selected patterns (insets). The receptive fields learned with the quadratic rectifier nonlinearity (θ₁ = 1., θ₂ = 2.) are the maxima among the three patterns, for all five nonlinearities, for all three datasets.