Decoding effector-dependent and effector-independent movement intentions from human parieto-frontal brain activity

Abstract

Our present understanding of the neural mechanisms and sensorimotor transformations that govern the planning of arm and eye movements predominantly come from invasive parieto-frontal neural recordings in nonhuman primates. While functional MRI (fMRI) has motivated investigations on much of these same issues in humans, the highly distributed and multiplexed organization of parieto-frontal neurons necessarily constrain the types of intention-related signals that can be detected with traditional fMRI analysis techniques. Here we employed multivoxel pattern analysis (MVPA), a multivariate technique sensitive to spatially distributed fMRI patterns, to provide a more detailed understanding of how hand and eye movement plans are coded in human parieto-frontal cortex. Subjects performed an event-related delayed movement task requiring that a reach or saccade be planned and executed toward one of two spatial target positions. We show with MVPA that, even in the absence of signal amplitude differences, the fMRI spatial activity patterns preceding movement onset are predictive of upcoming reaches and saccades and their intended directions. Within certain parieto-frontal regions we show that these predictive activity patterns reflect a similar spatial target representation for the hand and eye. Within some of the same regions, we further demonstrate that these preparatory spatial signals can be discriminated from nonspatial, effector-specific signals. In contrast to the largely graded effector- and direction-related planning responses found with fMRI subtraction methods, these results reveal considerable consensus with the parieto-frontal network organization suggested from primate neurophysiology and specifically show how predictive spatial and nonspatial movement information coexists within single human parieto-frontal areas.

Figure 1.

Experimental methods and example brain activity. A, Subject setup from side view. B, Top, Experimental apparatus and objects shown from the subject's point of view. The objects (white blocks) never changed position from trial-to-trial. Green star with dark shadow represents the fixation LED and its location in depth. Dashed line represents the arc of reachability with respect to the participant. The hand is positioned at its starting location. Bottom, Executed saccade and reach movements. Dashed lines represent eye position. C, Timing of one event-related trial. Trials began with the 3D objects being simultaneously illuminated while the subject maintained fixation (Preview phase; 6 s). Subjects were then instructed via headphones to perform one of four movements: touch the left object (“Touch Left”), touch the right object (“Touch Right”), saccade to the left object (“Look Left”), or saccade to the right object (“Look Right”). This cue initiated the Plan phase portion of the trial. Following a fixed delay interval (12 s), subjects were cued (“beep”) to perform the instructed hand movement (initiating the Execute phase). Two seconds after the Go cue, vision of the workspace was extinguished and participants waited for the following trial to begin (14 s, ITI). D, Averaged neural activity from left posterior IPS, pIPS, over the length of a single trial. Events in D are time-locked to correspond to events in C. MVPA was performed on single trials based on the windowed average of the percentage signal change corresponding to the three different time points denoted by each of the gray-shaded bars (each corresponding to activity elicited from the three distinct trial phases: Preview, Plan and Execute). The time points corresponding to the Plan phase (bordered in red) were of critical interest and provide the focus of our analyses.

Figure 2.

Parieto-frontal brain areas selected for MVPA. Cortical areas that exhibited larger responses during movement generation than the preceding visual phase [(Plan + Execute) > 2*(Preview)] are shown in orange/yellow activation. Results calculated across all subjects (Random Effects GLM) are displayed on one representative subject's inflated hemispheres. The general locations of the selected ROIs are outlined in circles (actual ROIs were anatomically defined separately in each subject). L, Left; R, Right (ROI acronyms are spelled out in main text). Sulcal landmarks are denoted by white lines (stylized according to the corresponding legend). LH, Left hemisphere; RH, right hemisphere; corr., corrected; df, degrees of freedom.

Figure 3.

Trial-related percentage signal change neural activity in the parieto-frontal regions used for MVPA. Activity in each plot is averaged across voxels within each ROI and across subjects. Vertical dashed lines correspond to the onset of the Preview, Plan, and Execute phases of each trial (from left to right). Shaded gray bars highlight the 2-volume (4 s) windows that were averaged and extracted for MVPA (a conventional univariate analysis of signal amplitude differences within these same time-windows is provided in Fig. 6). Note that time corresponds to imaging volumes (TR = 2) and not seconds.

Figure 4.

Decoding movement intentions across the parieto-frontal network. Decoding accuracies are shown for each time phase (Preview, Plan, and Execute) in each ROI. Classifier training was done on single trials and tested on the average activity patterns of the single trials for each condition in the independent test dataset. Importantly, accurate classification can only be attributed to the spatial response patterns of different planned movement types and not to the overall signal response amplitudes within each ROI (see Fig. 6). Note that decoding accuracies are color coded according to pairwise discriminations and not trial types. Error bars represent standard error of the mean (SEM) across subjects. Solid black lines are chance accuracy level (50%). Black asterisks assess statistical significance with t tests across subjects with respect to 50%. Red asterisks assess statistical significance based on a FDR correction of q ≤ 0.05 (critical p value of 0.012), based on all t tests performed.

Figure 5.

Cross trial-type decoding accuracies examining the degree of effector specificity and spatial specificity of the intended movements. Decoding accuracies are shown for each time phase (Preview, Plan, and Execute) in each ROI. Effector-across-space accuracies were computed from training classifiers on HandL versus EyeL trials and testing on HandR versus EyeR trials and then averaging these values with the opposite train-and-test ordering within each subject. Space-across-effector accuracies were computed from training classifiers on EyeL versus EyeR trials and testing on HandL versus HandR trials (again, averaging these values with the opposite train-and-test ordering within each subject). Error bars represent SEM across subjects. Solid black lines are chance accuracy level (50%). Black asterisks assess statistical significance with t tests across subjects with respect to 50%. Larger dark asterisks assess statistical significance based on a FDR correction of q ≤ 0.05 (critical p value of 0.010), based on all t tests performed.

Figure 6.

Few signal amplitude differences found within the parieto-frontal regions and time windows used for MVPA. Responses are averaged across voxels within each ROI and across subjects for the 2-volume averaged windows corresponding to Preview, Plan, and Execute phases. Note that very few statistically significant univariate differences are found throughout the parieto-frontal network. Errors bars represent SEM across subjects.

Figure 7.

Classifier decoding accuracies in nonbrain control regions. A, Nonbrain control ROIs defined in each subject (denoted in light orange; example subject shown). B, Classifier accuracies for the right ventricle (left) and outside the brain ROI (right). Error bars represent SEM across subjects. Solid lines show chance classification accuracy (50%). Note that no significant differences were found with t tests across subjects with respect to 50% chance. C, Percentage signal change activity from each selected region, averaged across subjects.

Figure 8.

Activation topography of effector selectivity (eye vs hand) during movement planning and execution defined with conventional subtraction analyses. Left, Brain areas that showed significant activation (RFX GLM, t₍₇₎ = 3, p < 0.01, cluster threshold corrected) during movement planning for the eye or hand independent of the spatial target location: [Plan(EyeL + EyeR) > Preview(EyeL + EyeR)] in yellow; [Plan(HandL + HandR) > Preview(HandL + HandR)] in red. Right, Brain areas that showed significant activation (at the same statistical threshold) for movement execution of the eye or hand: [Execute(EyeL + EyeR) > Preview(EyeL + EyeR)] in yellow; [Execute(HandL + HandR) > Preview(HandL + HandR)] in red. The overlap of eye and hand movement planning or execution is shown in orange.

Figure 9.

Activation topography of spatial selectivity (left vs right targets) for reach planning and execution defined with conventional subtraction analyses. Left, Brain areas that showed significant activation (RFX GLM, t₍₇₎ = 3, p < 0.01, cluster threshold corrected) during reach planning depending on the spatial target location [Plan(HandL) > Preview(HandL)] in yellow; [Plan(HandR) > Preview(HandR)] in red. Right, Brain areas that showed significant activation (at the same statistical threshold) for reach execution: [Execute(HandL) > Preview(HandL)] in yellow; [Execute(HandR) > Preview(HandR)] in red. The overlap of left and right reach planning or execution is shown in orange.

Figure 10.

Activation topography of spatial selectivity (left vs right targets) for saccade planning and execution defined with conventional subtraction analyses. Left, Brain areas that showed significant activation (RFX GLM, t₍₇₎ = 3, p < 0.01, cluster threshold corrected) during saccade planning depending on the spatial target location: [Plan(EyeL) > Preview(EyeL)] in yellow; [Plan(EyeR) > Preview(EyeR)] in red. Right, Brain areas that showed significant activation (at the same statistical threshold) for saccade execution: [Execute(EyeL) > Preview(EyeL)] in yellow; [Execute(EyeR) > Preview(EyeR)] in red. The overlap of left and right saccade planning or execution is shown in orange.