Developmental Emergence of Sparse Coding: A Dynamic Systems Approach

Abstract

During neocortical development, network activity undergoes a dramatic transition from largely synchronized, so-called cluster activity, to a relatively sparse pattern around the time of eye-opening in rodents. Biophysical mechanisms underlying this sparsification phenomenon remain poorly understood. Here, we present a dynamic systems modeling study of a developing neural network that provides the first mechanistic insights into sparsification. We find that the rest state of immature networks is strongly affected by the dynamics of a transient, unstable state hidden in their firing activities, allowing these networks to either be silent or generate large cluster activity. We address how, and which, specific developmental changes in neuronal and synaptic parameters drive sparsification. We also reveal how these changes refine the information processing capabilities of an in vivo developing network, mainly by showing a developmental reduction in the instability of network's firing activity, an effective availability of inhibition-stabilized states, and an emergence of spontaneous attractors and state transition mechanisms. Furthermore, we demonstrate the key role of GABAergic transmission and depressing glutamatergic synapses in governing the spatiotemporal evolution of cluster activity. These results, by providing a strong link between experimental observations and model behavior, suggest how adult sparse coding networks may emerge developmentally.

Figure 1

The stationary and cluster activity properties of a STP-RNN at P3. (a) Graph representing the spatially localized network of one excitatory, E, and one inhibitory, I, neuronal populations that are recurrently connected (RNN), and can receive inputs. (b) The E _r-I _r-plane of the RNN with dynamic synapses (STP-RNN) at postnatal day P3 (early period of physiological blindness). The colored regions show the fixed point (FP) domains of three possible operating regimes: unstable dynamics, inhibition-stabilized network (ISN), and Non-ISN; see Table 2 for details. (Inset) Zoom-in of the phase plane at lower activity ranges, overlaid by the FP of the STP-RNN. The ISN FP-domain is barely visible. The rest state and the vertical branch of the quasi $E_{\mathrm{r}}$ nullcline at $E_{\mathrm{r}}=0$ belong to the Non-ISN FP-domain. The horizontal branch of the quasi I _r- nullcline at $I_{\mathrm{r}}=0$, forming the bottom-border of the ISN (barely visible) and unstable regimes, belong to the unstable FP-domain (see Supplementary Methods). Only the non-negative branches of the quasi E _r- and I _r- nullclines (i.e. in ≥0 Hz ranges) were displayed (e.g. see Supplementary Fig. 4, for the negative branches of the quasi nullclines). (c) Cluster activity triggered by an impulse perturbation ($e_{\mathrm{E}}^{\mathrm{per}}$) to the E-population at time $t=0$ when the network was relaxed at the stable rest state. (Inset) Trajectories of cluster activity in E _r-I _r-plane. Simulations were performed for $e_{\mathrm{E}}^{\mathrm{per}}=e_{\mathrm{E}}(t=0)=30$ Hz. For all simulations, parameter values can be found in Table 1.

Figure 2

The transient unstable state hidden in fast (i.e. firing activity) dynamics of a STP-RNN at P3. (a) The same cluster activity as in Fig. 1c, but shown in terms of the sum activity $A_{\mathrm{sum}}=E_{\mathrm{r}}+I_{\mathrm{r}}$. (b) The time-evolution of synaptic efficacy of the recurrent excitatory connection $J_{\mathrm{E}}{x}_{\mathrm{E}}{u}_{\mathrm{E}}$, during the network activity shown in (a). (c) Activity-perturbation domains of STP-RNN. For the network relaxed at the rest state, setting the initial condition of the network activity at different E-and I-activity values revealed two different types of domains; amplification domain (cream-colored region): After perturbation, the sum activity was effectively amplified and cluster activity emerged; Non-amplification domain (black region): After perturbation, the sum activity monotonically decayed back to the rest state. Note, for the P3 network (mono-stable), both these domains are attraction domains of the rest state in the STP-RNN. (d) The E _r-I _r-plane and the E _r- (red) and I _r- nullclines (blue) of the STP-RNN with frozen synaptic efficacies (i.e. Frozen STP-RNN) at the rest state (see the time I in (a); $t=-150$ ms, relative to onset of input). Grey region: The attraction domain of the stable rest state in the Frozen STP-RNN. Purple region: The activities initiated here undergo an overall continuous growing (non-attraction domain). We call the border between these two regions the amplification-threshold. This border is approximately the same as the border between the two domains in (c). (e) Disappearance of the hidden unstable state during cluster activity. (I): Zoom-in of (d) at lower activity rates, overlaid by the FPs of the corresponding Frozen-RNN. (II-IV): Same as (I), but for synaptic efficacies frozen at different sample times (see (a)): (II): $t=75$ ms, (III): $t=175$ ms, (IV): $t=1000$ ms. Black dots show the unstable FPs. Dark-brown streamlines show, at each point, the local direction of sample trajectories in the corresponding Frozen-RNN.

Figure 3

Sparsification process in developing cortex. (a) Postnatal changes of the network’s stationary firing dynamics. The same format is used as in Fig. 1b; unstable dynamics (yellow region), ISN (light-yellow region), and Non-ISN (pink region). (b) The existence of a hidden unstable state (black dot) in the firing activity of developing networks. The same format is used as in Fig. 2e, panel I. The networks were frozen at the rest state ($t=-150$ ms). (c) Postnatal changes of the network’s transient firing dynamics where cluster activity was triggered by an impulse perturbation. The same format is used as in Fig. 1c. P3: early period of physiological blindness, P10: a few days before eye-opening, P14: the day after eye-opening, and P20: a few days after eye-opening. Green dots represent the stable FPs. (d) Developmental changes in the size of cluster activities, estimated qualitatively by $P{S}_{\mathrm{net}}^{\mathrm{amp}}$. (e) Developmental changes in the area of the FP-domain (AOD) of Non-ISN and ISN operating regimes, and the ratio AOD _ISN/Unstable of areas of ISN and unstable FP-domains. The developmental increase in AOD _ISN/Unstable indicates that the unstable FP-domain was effectively decreased, and replaced by the ISN FP-domain. For each panel, the values were normalized to the maximum value during all four stages. At each of these stages, the AOD s were computed based on the square E _r-I _r-plane with the lower-left corner located at the rest state (origin) and the upper-right corner (i.e. $[E_{\mathrm{r}}^{\max },I_{\mathrm{r}}^{\max }]$) at [10,10] Hz. Moreover, the developmental decrease in $AO{D}_{\mathrm{Non}-\mathrm{ISN}}$ just means that the E-activity rate after which the Non-ISN FP-domain starts in the E _r-I _r-plane, was shifted to higher levels. See Table 2 for details about technical terms.

Figure 4

Contribution of GABAergic transmission to cluster activity during course of development. (a) The effect of blocking GABAergic receptors (i.e. $J_{\mathrm{I}}=0$) on cluster activities, during development. The plots show that even with this blockage, cluster activity that emerges from the rest state can still converge back to this state. Solid lines: cluster activity before the blockage, Dashed lines: cluster activity after the blockage. (b) Zoom-in of P3 in (a) at lower activity rates. Note the increase in E- and I-activities after the blockage. (c) Normalized differences in cluster activity size before and after the blockage (after minus before), at the four developmental stages. The values were normalized to the maximum difference observed throughout P3 to P20. (d) Same as (c), but for cluster activity duration. (e) The effect of freezing the synaptic efficacies at the stable rest state, at P3. A threshold-crossing perturbation (see light-purple region in Fig. 2d) leads to run away activities; note the scale of the y-axis.

Figure 5

Impact of specific maturational processes on the sparsification process. We measured the change in network behavior when, virtually, we only mature a single parameter or small sets of similar parameters from P10 to P20. This enables us to indicate key parameters required for sparsification. (a) The plotted values of $rati{o}_{PS}$ measure the modification-induced changes in $P{S}_{\mathrm{net}}^{\mathrm{amp}}$, i.e. the size of simulated cluster activity, relative to the decrease when transitioning from P10 to P20. The dashed orange line at $rati{o}_{PS}=-100\%$ indicates the normal amount of decrease in $P{S}_{\mathrm{net}}^{\mathrm{amp}}$, as expected when transitioning from P10 to P20. (b) The plotted values of $rati{o}_{AOD}$ measure the modification-induced change in $AO{D}_{\mathrm{ISN}/\mathrm{Unstable}}$, i.e. the ratio of areas of ISN and unstable FP-domains, relative to the increase when transitioning from P10 to P20. The dashed orange line at $rati{o}_{AOD}=+100\%$ indicates the normal amount of increase in AOD _ISN/Unstable, as expected when transitioning from P10 to P20. See Methods for the formulas of $rati{o}_{PS}$ and $rati{o}_{AOD}$. For computing AOD _ISN/Unstable, we considered the E _r-I _r-plane plots with $[E_{\mathrm{r}}^{\max },I_{\mathrm{r}}^{\max }]$ = [10,10] Hz. Parameters combinations are $\mathrm{J}=\left\{J_{\mathrm{E}},J_{\mathrm{I}}\right\}$, ${\mathrm{\tau }}_{\mathrm{syn}}=\left\{{\tau }_{\mathrm{E}},{\tau }_{\mathrm{I}}\right\}$, ${\mathrm{STP}}_{\mathrm{all}}=\left\{{\mathrm{STP}}_{\mathrm{E}},{\mathrm{STP}}_{\mathrm{I}}\right\}$, ${\mathrm{STP}}_{\mathrm{E}}=\{U_{\mathrm{E}},{\tau }_{{\mathrm{r}}_{\mathrm{E}}},{\tau }_{{\mathrm{f}}_{\mathrm{E}}}\}$, ${\mathrm{STP}}_{\mathrm{I}}=\{U_{\mathrm{I}},{\tau }_{{\mathrm{r}}_{\mathrm{I}}},{\tau }_{{\mathrm{f}}_{\mathrm{I}}}\}$, $\mathrm{\theta }=\left\{{\theta }_{\mathrm{E}},{\theta }_{\mathrm{I}}\right\}$.