Deterministic networks for probabilistic computing

Abstract

Neuronal network models of high-level brain functions such as memory recall and reasoning often rely on the presence of some form of noise. The majority of these models assumes that each neuron in the functional network is equipped with its own private source of randomness, often in the form of uncorrelated external noise. In vivo, synaptic background input has been suggested to serve as the main source of noise in biological neuronal networks. However, the finiteness of the number of such noise sources constitutes a challenge to this idea. Here, we show that shared-noise correlations resulting from a finite number of independent noise sources can substantially impair the performance of stochastic network models. We demonstrate that this problem is naturally overcome by replacing the ensemble of independent noise sources by a deterministic recurrent neuronal network. By virtue of inhibitory feedback, such networks can generate small residual spatial correlations in their activity which, counter to intuition, suppress the detrimental effect of shared input. We exploit this mechanism to show that a single recurrent network of a few hundred neurons can serve as a natural noise source for a large ensemble of functional networks performing probabilistic computations, each comprising thousands of units.

Figure 1

Sources of noise (gray) for functional neural networks (black). Stars indicate intrinsically stochastic units. Open circles correspond to deterministic units. Intrinsic: Intrinsically stochastic units updating their states with a probability determined by their total synaptic input. Private: Deterministic units receiving private additive independent noise. Shared: Deterministic units receiving noise from a finite population of independent stochastic sources. Network: Deterministic units receiving quasi-random input generated by a finite recurrent network of deterministic units.

Figure 2

(a) Sampling error as measured by Kullback-Leibler divergence D_KL(p, p^*) between the empirical state distribution p of a sampling network and the state distribution p^* generated by the corresponding Boltzmann machine as a function of the sampling duration T for different sources of noise (legend, cf. Fig. 1). Error bands indicate mean ± SEM over 5 random network realizations. Inset: Same data as main panel in double-logarithmic representation. (b) Relative frequencies (vertical, log scale) of six exemplary states s (horizontal) for T = 10⁶ ms. Parameters: β = 1, M = 100, K = 200, N = 222, m = 6 (for details, see Supplementary Material).

Figure 3

Origin of shared-input correlations and their suppression by correlated presynaptic activity. A pair of neurons i and j receiving input from a finite population of noise sources (left) or a recurrent network (right). The input correlation $C_{ij}^{\mathrm{in}}$ decomposes into a contribution $C_{\mathrm{shared},ij}^{\mathrm{in}}$ resulting from shared noise sources (solid black lines) and a contribution $C_{\mathrm{corr},ij}^{\mathrm{in}}$ due to correlations between sources (dashed black lines). If Dale’s law is respected (neurons are either excitatory or inhibitory), shared-input correlations are always positive ($C_{\mathrm{shared},ij}^{\mathrm{in}}>0$). Left: In the shared -noise scenario, sources are by definition uncorrelated ($C_{\mathrm{corr},ij}^{\mathrm{in}}=0$) and cannot compensate for shared-input correlations. Right: In inhibition-dominated neural networks (network case), correlations between units arrange such that $C_{\mathrm{corr},ij}^{\mathrm{in}}$ is negative, thereby compensating for shared-input correlations such that the total input correlation $C_{ij}^{\mathrm{in}}$ approximately vanishes.

Figure 4

Sampling error D_KL(p, p^*) as a function of the number N of noise sources for different sources of noise (legend). Error bands indicate mean ± SEM over 5 random-network realizations. Inset: Dependence of average input correlation coefficient ρ of mutually unconnected sampling units on N. Black curve represents $~\mathrm{1/}N$ fit. Sampling duration T = 10⁵ ms. Remaining parameters as in Fig. 2.

Figure 5

Performance of a generative network trained on an imbalanced subset of the MNIST dataset for different noise sources (legend). (a) Left: Sketch of the network consisting of external noise inputs, input units, trained to represent patterns corresponding to handwritten digits, and label units trained to indicate the currently active pattern. Right: Network activity and trial-averaged relative activity of label units for intrinsic noise (black) and target distribution (yellow), with even digits occurring twice as often as odd digits. (b) Sampling error D_KL(p, p^*) between the empirical state distribution p of label units and the state distribution p^* of label units generated by the corresponding Boltzmann machine as a function of the number N of noise sources for shared and network case. Error bands indicate mean ± SEM over 20 trials with different initial conditions and noise realizations.

Figure 6

Sampling error D_KL(p, p^*) as a function of the entropy S of the target distribution for different sources of noise (same colors as in other figures). Error bands indicate mean ± SEM over 5 random network realizations. Vertical dashed gray line indicates maximal entropy, corresponding to a uniform target distribution. Inset: pairwise activity correlation coefficients in a Boltzmann machine for different entropies of the sampled state distribution. Sampling duration T = 10⁵ ms. Remaining parameters as in Fig. 2.

Figure 7

Sampling error D_KL(p, p^*) as a function of the sampling-network size M for different sources of noise (legend). Error bands indicate mean ± SEM over 5 random network realizations. Inset: Entropy of the sampled state distribution p as a function of M. Horizontal dashed dark gray line indicates entropy of uniform distribution, i.e., maximal entropy. Average weight in sampling networks: ${\mu }_{\mathrm{BM}}=-\mathrm{0.15/}\sqrt{M}$. Sampling duration T = 10⁵ ms. Remaining parameters as in Fig. 2.

Figure 8

Sampling in spiking networks with biologically plausible noise networks. Kullback-Leibler divergence D_KL(p, p^*) between the empirical state distribution p of a sampling network of spiking neurons and the state distribution p^* generated by the corresponding Boltzmann machine as a function of the number N of noise sources. Error bands indicate mean ± SEM over 10 random network realizations (see Supplementary Material).

Figure 9

Fit of error function to logistic function via Taylor expansion (purple) and L² difference of integrals (green). (A) Difference of logistic activation function and error function with adjusted σ via Eq. 7 (purple) and via Eq. 10 (green). Inset: activation functions. (B) L² difference of activation functions (Eq. 8) as a function of the strength of the Gaussian noise σ. Vertical bars indicate σ obtained via the respective method.

Algorithm 1

Training a fully visible Boltzmann machine via CD-1 to represent a particular distribution q^* over label units with one-hot encoding.