Optimal anticipatory control as a theory of motor preparation: A thalamo-cortical circuit model

Abstract

Across a range of motor and cognitive tasks, cortical activity can be accurately described by low-dimensional dynamics unfolding from specific initial conditions on every trial. These "preparatory states" largely determine the subsequent evolution of both neural activity and behavior, and their importance raises questions regarding how they are, or ought to be, set. Here, we formulate motor preparation as optimal anticipatory control of future movements and show that the solution requires a form of internal feedback control of cortical circuit dynamics. In contrast to a simple feedforward strategy, feedback control enables fast movement preparation by selectively controlling the cortical state in the small subspace that matters for the upcoming movement. Feedback but not feedforward control explains the orthogonality between preparatory and movement activity observed in reaching monkeys. We propose a circuit model in which optimal preparatory control is implemented as a thalamo-cortical loop gated by the basal ganglia.

Keywords: manifold; movement preparation; neural circuits; neural population dynamics; nullspace; optimal control; thalamo-cortical loop.

Figure 1

Preparation and execution of ballistic movements (A) Under a dynamical systems view of motor control (Shenoy et al., 2013), movement is generated by M1 dynamics. Prior to movement, the M1 population activity state $\mathit{x}\left(t\right)$ must be controlled into an optimal, movement-specific subspace in a phase of preparation; this requires internally generated control inputs $\mathit{u}\left(t\right)$. (B) Schematic state space trajectory during movement preparation and execution.

Figure 2

Movement generation in a network model of M1 (A) Schematics of our M1 model of motor pattern generation. The dynamics of an excitation-inhibition network (Hennequin et al., 2014) unfold from movement-specific initial conditions, resulting in firing rate trajectories (left; five neurons shown), which are linearly read out into joint torques (middle), thereby producing hand movements (right). The model is calibrated for the production of eight straight center-out reaches (20 cm length); firing rates and torques are shown only for the movement colored in black. To help visualize initial conditions, firing rates are artificially clamped for the first 100 ms. See also Figure S1. (B) Network activity and corresponding hand trajectories as in (A), for three different preparation lengths, under the naive feedforward strategy whereby a static input step (green) moves the fixed point of the dynamics to the desired initial condition.

Figure 3

Formalization of the optimal subspace hypothesis (A) Effect of three qualitatively different types of small perturbations of the initial condition (prospectively potent, prospectively readout-null, prospectively dynamic-null) on the three processing stages leading to movement (M1 activity, joint torques, and hand position), as shown in Figure 2A. Unperturbed traces are shown as solid lines, perturbed ones as dashed red lines. Only one example neuron (top) is shown for clarity. Despite all having the same size here (Euclidean norm), these three types of perturbation on the initial state have very different consequences. Left: “prospectively potent” perturbations result in errors at every stage. Middle: “prospectively readout-null” perturbations cause sizable changes in internal network activity but not in the torques. Right: “prospectively dynamic-null” perturbations are inconsequential at all stages. (B) Time course of the root-mean-square error in M1 activity across neurons and reach conditions, for the three different types of perturbations. (C) Same as (B) for the root-mean-square error in torques. (D) The motor potency of the top 20 most potent modes. In (A)–(C), signals are artificially held constant in the first 100 ms for visualization, and black scale bars denote 200 ms from movement onset. See also Supplemental Math Note S1 and Figure S4.

Figure 4

Optimal preparatory control (A) Dynamics of the model during optimal preparation and execution of a straight reach at a 144° angle. Optimal control inputs are fed to the cortical network during preparation and subsequently withdrawn to elicit movement. Top: firing rates of a selection of ten model neurons. Middle: generated torques (line), compared with targets (dots). Bottom: the prospective motor error $\mathcal{C}\left(\mathit{x}\left(t\right)\right)$ quantifies the accuracy of the movement if it were initiated at time t during the preparatory phase. Under the action of optimal control inputs, $\mathcal{C}\left(\mathit{x}\left(t\right)\right)$ decreases very fast, until it becomes small enough that an accurate movement can be triggered. The dashed line shows the evolution of the prospective cost for the naive feedforward strategy (see text). Gray lines denote the other seven reaches for completeness. (B) Hand trajectories for each of the eight reaches (solid), following optimal preparation over windows of 25 ms (left), 50 ms (center), and 200 ms (right). Dashed lines show the target movements. (C) Firing rate of a representative neuron in the model (left) and the two monkeys (center and right) for each movement condition (color-coded as in Figure 2B). Green bars mark the 500 ms preparation window, black scale bars indicate 20 Hz. (D) Evolution of the average across-movement variance in single-neuron preparatory activity in the model (left) and the monkeys (center and right). Black scale bars indicate 16 ${\text{Hz}}^2$. (E) Prospective motor error during preparation, averaged over the eight reaches, for different values of the energy penalty parameter λ. (F) The state of the cortical network is artificially set to deviate randomly from the target movement-specific initial state at time $t=0$, just prior to movement preparation. The temporal evolution of the squared Euclidean deviation from target (averaged over trials and movements) is decomposed into contributions from the ten most and ten least potent directions, color-coded by their motor potency as in Figure 3D. In (A)–(D) and (F), we used $\lambda =0.1$.

Figure 5

Optimal preparatory control benefits other models of movement generation (A) ISN model. Top: eigenvalues of the connectivity matrix. Middle: activity of 20 example neurons during optimal (LQR) and naive feedforward preparatory control and subsequent execution of one movement, with hand trajectories shown as an inset for all movements. The green bar marks the preparatory period. Bottom: prospective motor error under LQR (solid) and naive feedforward (dashed) preparation, for each movement. (B)–(D) Same as (A), for a representative instance of each of the three other network classes (see text). (E)–(G) Index of nonnormality (E), Frobenius norm of the connectivity matrix ${\Vert \mathit{W}\Vert }_{\text{F}}$ (F), and ${\mathcal{H}}_2$ norm (G) for the 10 network instances of each class (excluding chaotic networks for which these quantities are either undefined or uninterpretable). Dots are randomly jittered horizontally for better visualization. Both $\Vert {\mathit{W}}_{\text{F}}\Vert$ and the ${\mathcal{H}}_2$ norm are normalized by the ISN average. (H) Quantification of controllability in the various networks. Optimal control cost (${\int }_0^{\infty }\mathcal{C}\left(\mathit{x}\left(t\right)\right)dt$ in Equation 6) against associated control energy cost $\left({\int }_0^{\infty }\mathcal{R}\left(\mathit{u}\left(t\right)\right)dt\right)$, for different values of the energy penalty parameter λ, and for each network (same colors as in E–G). Note that the naive feedforward strategy corresponds to the limit of zero energy cost (horizontal asymptote). See also Figure S6.

Figure 6

Reorganization between preparatory and movement activity in model and monkey (A) Example single-neuron PSTHs in model (top) and monkey M1/PMd (middle and bottom), for each reach (cf. Figure 2B). (B) Fraction of variance (time and conditions) explained during movement preparation (left) and execution (right) by principal components calculated from preparatory (green) and movement-related (magenta) trial-averaged activity. Only the first ten components are shown for each epoch. Variance is across reach conditions and time in 300 ms prep. and move. windows indicated by green and magenta bars in (A). The three rows correspond to those of (A). (C) Alignment index calculated as in Elsayed et al. (2016), for the two monkeys (left) and the three classes of trained networks (colors as in Figures 5A–5C) under three different preparation strategies (LQR, instant, naive; right). Here, preparation is long enough (500 ms) that even the naive feedforward strategy leads to the correct movements in all networks. Hashed bars show the average alignment index between random-but-constrained subspaces drawn as in Elsayed et al. (2016). (D) Amplification factor, quantifying the growth of the centered population activity vector $\mathit{x}\left(t\right)-{\mathit{x}}_{\text{sp}}$ during the course of movement, relative to the pre-movement state (STAR Methods). It is shown here for the two monkeys (black and gray), as well as for the three classes of trained networks (solid) and their surrogate counterparts (dashed). Shaded region denotes ±1 SD around the mean across the ten instances of each network class. To isolate the autonomous part of the movement-epoch dynamics, here we set $\mathit{h}\left(t\right)=0$ in Equation 2. (E) Alignment index under the “instant” preparation strategy, for the original trained networks and their surrogates. In (C)–(E), error bars denote ±1 SD across the ten networks of each class.

Figure 7

Optimal movement preparation via a gated thalamo-cortical loop (A) Proposed circuit architecture for the optimal movement preparation (cf. text). (B) Cortical (top) and thalamic (upper middle) activity (ten example neurons), generated torques (lower middle), and prospective motor error (bottom) during the course of movement preparation and execution in the circuit architecture shown in (A). The prospective motor error for the naive strategy is shown as a dotted line as in Figure 4A. All black curves correspond to the same example movement (324° reach), and gray curves show the prospective motor error for the other seven reaches. (C) Hand trajectories (solid) compared with target trajectories (dashed) for the eight reaches, triggered after 100 ms (left), 200 ms (middle), and 600 ms (right) of motor preparation. See also Figure S3.

Figure 8

Testable prediction: Selective recovery from preparatory perturbations (A) Illustration of perturbation via “photoinhibition”: a subset (60%) of I neurons in the model are driven by strong positive input. (B) Left: firing rates (solid, perturbed; dashed, unperturbed) for a pair of targeted I cells (top), untargeted I cells (middle), and E cells (bottom). Green bars (1.6 s) mark the movement preparation epoch, and embedded turquoise bars (400 ms) denote the perturbation period. Right: histogram of firing rates observed at the end of the perturbation (turquoise) and at the same time in unperturbed trials (gray). Error bars show 1 SD across 300 experiments, each with a different random set of targeted I cells. (C) Prospective motor error, averaged across movements and perturbation experiments, in perturbed (turquoise) versus unperturbed (black) conditions. Subsequent hand trajectories are shown for one experiment of each condition (middle and bottom insets; dashed lines show target trajectories). These are compared with the reaches obtained by randomly shuffling final preparatory errors $\delta \mathit{x}$ across neurons and re-simulating the cortical dynamics thereafter (turquoise square mark). The purple line shows the performance of an optimal feedforward strategy, which pre-computes the inputs that would be provided to the cortex under optimal feedback control and subsequently replays those inputs at preparation onset without taking the state of the cortex into account anymore. (D) Magnitude of the deviation caused by the perturbation in the activity of the network projected into the coding subspace (left), the persistent subspace (center), and the remaining subspace (right). Lines denote the mean across perturbation experiments, and shadings indicate ±1 SD. Green and turquoise bars as in (B).