Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
, 38 (44), 9390-9401

Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces

Affiliations
Review

Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces

Chethan Pandarinath et al. J Neurosci.

Abstract

In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the "latent factors" underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain-machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.

Keywords: brain-machine interfaces; dynamical systems; machine learning; motor control; motor cortex; neural population dynamics.

Figures

Figure 1.
Figure 1.
Intuition for latent factors and dynamical systems. A, n(t) is a vector representing observed spiking activity. Each element of the vector captures the number of spikes a given neuron emits within a short time window around time t. n(t) can typically be captured by the neural state variable x(t), an abstract, lower-dimensional representation that captures the state of the network. Dynamics are the rules that govern how the state updates in time. For a completely autonomous dynamical system without noise, if the dynamics f(x) are known, then the upcoming states are completely predictable based on an initial state x(0). B, In a simple three-neuron example, the ensemble's activity at each point in time traces out a trajectory in a 3-D state space, where each axis represents the activity of a given neuron. Not all possible patterns of activity are observed, rather, activity is confined to a 2-D plane within the 3-D space. The axes of this plane represent the neural state dimensions. Adapted from Cunningham and Yu, 2014. C, Conceptual low-dimensional dynamical system: a 1-D pendulum. A pendulum released from point p1 or p2 traces out different positions and velocities over time, and the state of the system can be captured by two state variables (position and velocity). D, The evolution of the system over time follows a fixed set of dynamic rules, i.e., the pendulum's equations of motion. Knowing the pendulum's initial state [x(0), filled circles] and the dynamical rules that govern its evolution [f(x), gray vector flow-field] is sufficient to predict the system's state at all future time points.
Figure 2.
Figure 2.
Overview of results supporting the dynamical systems view of motor cortex. A, The neural state achieved during the delay period (green-red dots) predicts the subsequent trajectory of movement activity (green-red lines). Each dot/line is a single reach condition, recorded from a 108-condition task (inset). Adapted from Churchland et al., 2012. B, In dynamical systems, places where neighboring points in state space have very different dynamics are indications of “tangling”. Such regions would be highly sensitive to noise; small perturbations yield very different trajectories. C, Conceptual example illustrating tangling. Imagine a system that needs to produce two sine waves, one of which has double the frequency and is phase-shifted 1/4 of a cycle relative to the other. If it contains these sine waves with no additional dimensions, activity would trace out a figure 8, with a point of “high tangling” in the center. By adding in a third dimension, the system can move from a high tangling to a “low tangling” configuration, using the third dimension to separate the tangled points. Adapted from Russo et al., 2018. D, Although EMG often displays highly-tangled points (x-axis), MC's neural activity maintains low tangling (y-axis). E, Illustration of muscle-potent/muscle-null concept. Imagine a muscle that is driven with a strength equal to the sum of the firing rates of two units. If the units change in such a way that one unit's firing rate decreases as the other increases, then the overall drive to the muscle will remain the same (muscle-null). If, on the other hand, the neurons increase or decrease together, then the drive to the muscle will change (muscle-potent). In this way, neural activity can change in the muscle-null space while avoiding causing a direct change in the command to the muscles. Adapted from Kaufman et al., 2014. F, Neural activity in MC occupies a different set of dimensions during motor preparation than during movement. Red, Neural activity across different reach conditions in “preparatory” dimensions; green, neural activity across different reach conditions in “movement” dimensions. Adapted from Elsayed et al., 2016.
Figure 3.
Figure 3.
Applications of latent state and dynamics estimation methods to MC ensemble activity. A, Generative model of observed neural activity. Population spiking activity is assumed to reflect an underlying latent state x(t) whose temporal evolution follows consistent rules (dynamics). Firing rates for each neuron r(t) are derived from x(t), and observed spikes n(t) reflect a noisy sample from r(t). B, dPCA applied to trial-averaged MC activity during a delayed reaching task separates condition-invariant and condition-variant dimensions. Each bar shows the total variance captured by each dimension, with red portions denoting condition-invariant fraction, and blue portions denoting condition-variant fraction. Traces show projection onto first dimension found by dPCA. Each trace corresponds to a single condition (inset, kinematic trajectories with corresponding colors). Adapted from Kaufman et al., 2016. C, GPFA reveals single-trial state space trajectories during a delayed reaching task. Gray traces represent individual trials. Ellipses indicate across-trial variability of the neural state at reach target onset (red shading), go cue (green shading), and movement onset (blue shading). Adapted from Yu et al., 2009. D, SLDS enables segmentation of individual trials by their dynamics. Each horizontal trace represents a single trial for the first state dimension found by the SLDS. Trace coloring represents time periods with distinct (discrete) dynamics for each trial, recognized in an unsupervised fashion. Switching between dynamic states reliably follow target onset and precede movement onset, with time lags that are correlated with reaction time. Adapted from Petreska et al., 2011.
Figure 4.
Figure 4.
LFADS uses recurrent neural networks to infer precise estimates of single-trial population dynamics. A, A recurrent neural network (simplified) is a set of artificial neurons that implements a nonlinear dynamical system, with dynamics set by adjusting the weights of its recurrent connections. Conceptually, the RNN can be “unrolled” in time, where future states of the RNN are completely predicted based in an initial state g(0) and its learned recurrent connectivity (compare Fig. 3A). B, The SAE framework consists of an encoding network and decoding network. The encoder (RNN) compresses single-trial observed activity n(t) into a trial code g(0), which sets the initial state of the decoder RNN. The decoder attempts to re-create n(t) based only on g(0). To do so, the decoder must model the ensemble's dynamics using its recurrent connectivity. The output of the decoder is x(t), the latent factors, and r(t), the de-noised firing rates. C, The de-noised single-trial estimates produced by LFADS uncover known dynamic features (such as rotations; Fig. 2A) on single trials. D, Decoding the LFADS-de-noised rates using simple optimal linear estimation leads to vastly improved predictions of behavioral variables (hand velocities) over Gaussian smoothing, even with limited numbers of neurons. Adapted from Pandarinath et al., 2018.
Figure 5.
Figure 5.
Improving BMI performance and longevity by leveraging neural dynamics. A, Graphical model of decoder with dynamical smoothing. B, Illustration of smoothing latent state estimates using neural dynamics. The instantaneous estimate of the latent state (blue) is augmented by a dynamical prior (gray flow-field) to produce a smoother, de-noised estimate (orange). C, Smoothing using neural dynamics results in better closed-loop BMI performance than other approaches. Performance is achieved information bitrate. Adapted from Kao et al., 2015. D, Example of low-dimensional signals that can be used to augment intracortical BMIs. PCA applied to neural activity around the time of target selection identifies a putative “error signal”, allowing real-time detection and correction of user errors in a typing BMI. Adapted from Even-Chen et al., 2017. E, Remembering dynamics from earlier recording conditions can extend performance as neurons are lost. Performance measure is (off-line) mean velocity correlation. F, Comparison of closed-loop performance when 110 channels are “lost” shows a >3× improvement achieved by remembering dynamics. FIT-KF, state-of-the-art kinematic Kalman filter (Fan et al., 2014). Adapted from Kao et al., 2017. G, Dynamic neural stitching with LFADS. A single model was trained on 44 recording sessions. Each session used a 24-channel recording probe. Left, Recording locations in MC. Right, Single-trial reaches from an example session. Arc. Sp., arcuate spur; PCd, precentral dimple; CS, central sulcus. H, Neural state space trajectories inferred by LFADS. Each trace of a given color is from a separate recording session (44 traces per condition). Inferred trajectories are consistent across 5 months. jPC1 and jPC2 are the first two components identified by jPCA (Churchland et al., 2012). I, Using LFADS to align 5 months of data (“Stitched”) significantly improves decoding versus other tested methods. Adapted from Pandarinath et al., 2018. ***Significant improvement in median R2; P < 10−8, Wilcoxon signed-rank test.
Figure 6.
Figure 6.
Distribution alignment methods for stabilizing movement decoders across days and subjects. A, Data dimensionality is first reduced, and then low-dimensional projections are aligned onto a previously recorded movement distribution. B, KL-divergence provides a robust metric for alignment (displayed as a function of the angle used to rotate the data). Many local minima exist (points 1, 2, 3, 4), which makes alignment difficult. C, Prediction accuracy of 2-D kinematics for distribution alignment decoding and supervised methods. Left, Accuracy of DAD using movements from Subject M (DAD-M), from Subject C (DAD-C), and using movements from both Subjects M and C (DAD-MC). Right, Standard L2-regularized supervised decoder (Sup) and a combined decoder (Sup-DAD), which averages the results of the supervised and DAD decoders. All results are compared with an Oracle decoder (far right), which provides an upper bound for the best linear decoding performance for this task. Adapted from Dyer et al., 2017. D, A schematic of a generative adversarial network strategy for distribution alignment across multiple days: generator network (left) receives new data and learns a transformation of the data to match the prior (from a previous day).

Similar articles

See all similar articles

Cited by 4 articles

Publication types

LinkOut - more resources

Feedback