Decoding complete reach and grasp actions from local primary motor cortex populations

Abstract

How the activity of populations of cortical neurons generates coordinated multijoint actions of the arm, wrist, and hand is poorly understood. This study combined multielectrode recording techniques with full arm motion capture to relate neural activity in primary motor cortex (M1) of macaques (Macaca mulatta) to arm, wrist, and hand postures during movement. We find that the firing rate of individual M1 neurons is typically modulated by the kinematics of multiple joints and that small, local ensembles of M1 neurons contain sufficient information to reconstruct 25 measured joint angles (representing an estimated 10 functionally independent degrees of freedom). Beyond showing that the spiking patterns of local M1 ensembles represent a rich set of naturalistic movements involving the entire upper limb, the results also suggest that achieving high-dimensional reach and grasp actions with neuroprosthetic devices may be possible using small intracortical arrays like those already being tested in human pilot clinical trials.

Figure 1.

The monkeys used different grasping strategies characterized by varied patterns of grip aperture scaling and wrist motion to intercept and hold each of the objects. Monkeys were trained to intercept and hold objects of various shapes and sizes swinging toward them at the end of a string. Measurements of grip aperture, wrist pronation/supination, and wrist flexion/extension are shown for nine segments of data in which different objects were grasped. The objects are shown above the plots, and consistent scaling was used to preserve relative dimensions. All measurements were taken during a single session (C2).

Figure 2.

The dynamic grasping task elicits a wide range of upper limb movements. a, Movement was recorded by tracking 29 reflective markers attached to the monkey with mild water-soluble adhesive. b, Model of the hand and arm fit to the 3D position of the markers to calculate the joint angles for each frame. c, Distribution of measured grip apertures (top) and wrist angles (bottom). d, Projection of the position of the arm endpoint (proximal wrist marker) onto the coronal, sagittal, and transverse planes (dataset C1). Color saturation indicates the density of the points. Marginal distributions of density are shown along each axis. Crosses denote the mean and quartiles. The highest density of points (highest color saturation) was noted at the location where the monkeys typically rested their arm after each reach and grasp action.

Figure 3.

The dynamic grasping task essentially decouples the DoFs of the hand, wrist, and arm. a, Correlations display the relative coupling between DoFs, which are weaker as distance between joints increases. Values for eight representative parameters averaged across monkeys are shown. s., Shoulder; e., elbow; w., wrist; int./ext. rot., internal/external rotation; pron./sup., pronation/supination; uln./rad., ulnar/radial deviation; flex./ext., flexion/extension. b, Dimensionality estimation for reach and grasp actions. This plot shows the cumulative percentage of variance accounted for as a function of the number of PCs used to represent the joint angles for each of four sessions. c, The firing rate of individual M1 neurons was only moderately correlated with any given joint angle. Correlation coefficients (CC) between the firing rate of each neuron and each measured joint angle are divided into three groups: those related to the arm (shoulder and elbow, 4 DoFs), the wrist (3 DoFs), and the hand (18 DoFs).

Figure 4.

Neural correlations with multiple sets of kinematic parameters cannot be accounted for by correlations among the kinematics themselves. In all histograms, results are shown in solid gray for monkey C and in black outlines for monkey G. a, Coefficients of multiple correlation (R²) for each of the recorded neurons. These values can be interpreted as the fraction of variance in firing rates accounted for by linear relationships with the full set of kinematics (joint angles and joint angle velocities). Note that in all cases neural activity preceded movement, because the analysis was limited to predictive lags to approximate online decoding conditions. b–d, Semipartial correlation coefficients (sr²) for three sets of kinematic variables. These values can be interpreted as the amount of firing rate variance accounted for by one set of kinematic variables (arm, wrist, or hand) beyond that accounted for by the other two. Thus, b shows the variance in neuronal firing rates explained exclusively by arm kinematics after removing the variance explained by hand and wrist variables. c and d show similar distributions for wrist and hand joint angles. e, Overlap between groups of neurons that show statistically significant semipartial correlations >0.5 (representing >5% of the variance in firing rate) to arm, wrist, or hand kinematics. Values for monkeys C and G are shown separately in each partition. Overall, 86.7% of the neurons recorded in monkey C and 75.4% of the neurons recorded in monkey G displayed a set-specific relationship with at least one set group of kinematic variables. Note that the areas in the Venn diagram are not to scale. MAD, Median absolute deviation.

Figure 5.

Single-unit activity in M1 during naturalistic reaching and grasping movements. Raster plot showing the action potentials fired by 25 individual neurons over a span of 5.5 s. Raster plots are colored according to set specific relationships with kinematics (significant semipartial correlations >0.05; for details, see Results, Neuron tuning properties): blue, hand only; green, wrist only; red, hand only; orange, hand + wrist; magenta, hand + arm; black, hand + arm + wrist. The data shown corresponds to session G2. Of the 30 neurons recorded in M1, only the 25 shown had set-specific relationships to at least one set of kinematic parameters. During the time span shown, the monkey performed three separate reach-to-grasp movements targeting a small ball. Grip aperture measures are overlaid over the raster plot to highlight the moments when prehension occurs (troughs). The full posture of the arm is shown for four time points in the first movement.

Figure 6.

Most neurons were not preferentially related to specific subsets of kinematic parameters, even after removing the influence of correlations between kinematics. In all panels, results are shown in solid gray for monkey C and in black outlines for monkey G. a, Index of kinematic selectivity for arm joints (for details, see Results). These values are a ratio of the variance in firing rates exclusively represented by arm joints (semipartial correlation for arm kinematics) over the total set-specific variance (the sum of semipartial correlations for the arm, wrist, and hand). b, c, Index of kinematic selectivity for wrist and hand joints, respectively. MAD, Median absolute deviation.

Figure 7.

M1 neurons are simultaneously tuned to proximal and distal DoFs. Pairs of measured DoFs were discretized into 100 equally sized 2D bins spanning 99% of the data (after removing outliers at each extreme). Colored circles show the average firing rate for each bin (color was interpolated for the space between the circles). Each row shows the activity of a single M1 neuron. The first column shows firing rate as a function of shoulder elevation and grip aperture. The second and third columns have similar displays, using grip aperture versus wrist pronation/supination and shoulder elevation versus wrist pronation/supination.

Figure 8.

Optimal decoding was accomplished using populations of 30 neurons to reconstruct each DoF. a, Mean r (over all variables) between measured and decoded kinematics as a function of the number of neurons used to decode each DoF. Neurons were added to the models in order from most to least correlated with the DoF being decoded (following the “greedy”' selection method described in Results, Arm and hand movement reconstruction). Based on this graph, the final set of decoding models were constructed using 30 neurons for each DoF. The total number of neurons used for decoding in each session (pooled across all DoFs) is shown next to each plot as a fraction of the total number of neurons recorded. For details on the overlap of populations used to decode arm, wrist, and hand movements, see supplemental Figure 7 (available at www.jneurosci.org as supplemental material). b, Error distribution across all joint angles reconstructed across all test datasets using sets of 30 neurons (solid gray histogram for monkey C and outlined black histogram for monkey G). Triangles along the x-axis denote the range that encompasses 95% of the data for each distribution (green for monkey G and red for monkey C). More than 50% of the reconstructed joint angles were within 5° of the measured values. MAD, Median absolute deviation.

Figure 9.

Decoding continuous 25-dimensional movement. a, Examples of measured (ghost) and decoded (solid) arm postures from a reach and grasp trial (with each of the 25 joint angles decoded independently). b, Detailed view of measured (blue) and reconstructed (black) values for grip aperture and shoulder azimuth. c, Correlation coefficients between measured and decoded variables. Colored dots represent the values for each experimental session, and solid bars mark the mean over all sessions. Black asterisks represent chance levels of performance (95% confidence limit for correlations between reconstructed kinematics and temporally shifted kinematics; for details, see Results). In addition to joint angles, grip aperture, as well as the x, y, and z position of the endpoint of the arm were directly decoded. Decoding accuracy was above chance for every degree of freedom examined. MAE, Mean absolute error; In./Ex. Rot., internal/external rotation; Flex./Ext., flexion/extension; Ul./Rad., ulnar/radial deviation; Pron./Sup., pronation/supination; MCP, metacarpophalangeal; Ante./Retro., anteposition/retroposition; Rad. Ab./Ad., radial abduction/adduction; Palm. Ab./Ad., palmar abduction/adduction; PIP, proximal interphalangeal.