Eye-hand coordination during learning of a novel visuomotor task

Abstract

We investigated how gaze behavior and eye-hand coordination change when subjects learned a challenging visuomotor task that required acquisition of a novel mapping between bimanual actions and their visual sensory consequences. By applying isometric forces and torques to a rigid tool held freely between the two hands, subjects learned to control a cursor on a computer screen to hit successively displayed targets as quickly as possible. The learning occurred in stages that could be distinguished by changes in performance (target-hit rate) as well as by gaze behavior and eye-hand coordination. In a first exploratory stage, the hit rate was consistently low, the cursor position varied widely, and gaze typically pursued the cursor. In a second skill acquisition stage, the hit rate improved rapidly, and gaze fixations began to mark predictively desired cursor positions, indicating that subjects started to program spatially congruent eye and hand motor commands. In a third skill refinement stage, performance continued to improve gradually, and gaze shifted directly toward the target. We suggest that during the exploratory stage, the learner attempts to establish basic mapping rules between manual actions and eye-movement commands. In this process, subjects may establish correlations between hand motor commands and their visual sensory consequences, primarily in fovea-anchored, gaze-centered coordinates, and correlations between recent hand motor commands and eye motor commands. The established mapping rules are then implemented and refined in the skill acquisition and refinement stages.

Figure 1.

Apparatus and target distribution on screen. A, Tool used by the subject to control a cursor on the screen by bimanually applying forces and torques about the long axis of the tool. B, The corresponding arrows in A and B indicate the two different mapping rules relating hand forces and torques to cursor movements. C, Distribution of target positions (filled circles) in which consecutively appearing targets are connected with a line; the target shape (square) is shown to scale in the bottom right corner.

Figure 2.

Learning of the novel visuomotor task as reflected by an increasing hit rate and decreasing cursor path index. A, The hit rate as a function of time for one practice session by each of the eight subjects that displayed three learning stages. Data points indicate, for each target hit, the instantaneous hit rate. The thick gray curve shows the hit rate filtered with a moving median of ±100 s. The superimposed thin black lines shows the piecewise linear function fitted to the hit rate curves to distinguish transitions between learning stages. The shaded area represents the skill acquisition stage. B, Examples of sessions with only two learning stages (same format as in A). C, Superimposed thin lines show the filtered hit-rate curves on a normalized time base for the same eight subjects shown in A. The thick curve represents the mean values of these subject curves, and the shaded zone provides the range. Ex, Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage. D, Filtered cursor path index for individual subjects shown as a function of normalized time using the same format as in C. Note the logarithmic scale of the ordinate.

Figure 3.

Gaze and cursor movements for examples of target transitions during a practice session by one subject. A–D, Dashed and solid lines represent the positions of the cursor and gaze on the screen, respectively, and squares indicate target positions. The numbers in bold indicate successive gaze fixations, and the numbered circles indicate the cursor position at the start of the corresponding fixation. A, Exploratory stage. Starting at the center of the screen, the subject moves the cursor to the very first target (square). For this target transition, the path index was 63.4 (ratio between the distance traveled by the cursor and the intertarget distance). B, A 2 s epoch of the movements shown in A; the most right point of the cursor trajectory is marked by an asterisk, which corresponds to the asterisk inA. C, D, Example of target transitions during the skill acquisition and skill refinement stages, respectively, with path indices of 2.4 (C) and ∼1.2 (D). E, Positions of the target, cursor, and gaze together with cursor and gaze tangential velocity as a function of time during the movements shown in C. Thick-line segments of the cursor velocity trace mark scored hand submovements, the start and end of which are indicated by vertical lines.

Figure 4.

Occurrence of saccades and manual submovements. A, B, The number of saccades and number of manual (cursor) submovements per target transition. Note the logarithmic scale of the ordinate. C, D, The rate of saccades (saccades per second) and rate of submovement (submovements per second). E, The number of saccades that brought the gaze into the 3°radius target zone per target transition. In all panels, the superimposed thin lines plot, on the normalized time base, data from the eight subjects shown in Figure 2 A after filtering with a moving median of ±100s.The thick line gives the mean values for those subject curves, and the shaded zone provides the range. Learning stages are indicated as in Figure 2C.

Figure 5.

Postsaccadic fixations leading versus lagging the cursor position. A, Data points show, for a single subject (Fig. 2 A, asterisk), the difference between the time at which the distance between gaze and the cursor reached a minimum and the fixation onset (negative time implies cursor lead) for each relevant gaze fixation (see Materials and Methods) as a function of practice time. The curve shows the same data filtered with a moving median of ±100 s. B, The corresponding filtered curves obtained in all eight subjects shown in Figure 2 A (same format as in Fig. 2C). C, Distribution of this time difference including data from all relevant gaze fixations obtained from all subjects. Each of the four columns represents data from one of the four learning stages (Fig.2C). Negative times (cursor leading gaze) are indicated by the black areas of the histograms. Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

Figure 6.

Amplitudes of saccadic gaze displacement and cursor submovements. A, Data points indicate saccade displacements as a function of practice time for a single subject (Fig. 2 A, asterisk). The curve shows the same data filtered with a moving median of ±100 s. B, Corresponding filtered curves obtained for the eight subjects shown in Figure 2 A (same format as in Fig. 2C). C, Distributions of saccadic displacements including data obtained from all subjects. The black areas of the histograms indicate saccadic displacements for which the onset of the postsaccadic fixation lagged the position of the cursor (Fig. 5, negative times). D, Frequency distribution of cursor submovement displacements including data obtained from all subjects. C, D, Learning stages represented as in Figure 5C. Ex, Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

Figure 7.

Distance between gaze fixations and cursor and target positions and changes in these distances during a saccade. A, Data points indicate, for a single subject (Fig. 2 A, asterisk), the change in distance between cursor and gaze during each saccade plotted as a function of practice time (negative distance implies that gaze shifted toward the cursor). The curve shows the same data filtered with a moving median of ±100 s. B, Corresponding filtered curves obtained for the same eight subjects shown in Figure 2 A (same format as in Fig. 2C). C–F, Frequency distributions including data from all saccades or submovements obtained from all subjects showing gaze–cursor distance changes during saccades (C), distance between gaze and cursor at the onset of gaze fixations (D), gaze–target distance changes during saccades (negative distance implies that gaze shifted toward the target) (E), and distance between gaze and target at the onset of gaze fixations (F). Learning stages are represented as in Figure 5C. The black areas of the histograms in C and E refer to postsaccadic fixations that lagged the position of the cursor (Fig. 5, negative times). Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

Figure 8.

Direction relative to target and absolute direction (in screen coordinates) of saccades and cursor submovements. A, Data points indicate, as a function of practice time, direction relative to the target for the saccades generated by a single subject (Fig. 2 A, asterisk). Saccade direction (±180°) is provided in absolute values, where 0° indicates gaze shifts toward the target. The curve shows the corresponding data filtered with a moving median of ±100 s. B, Corresponding filtered curves obtained for the eight subjects shown in Figure 2 A (same format as in Fig. 2C). C, D, Distribution of the direction of saccades and cursor submovements relative to the target for all saccades and submovements obtained from all eight subjects. E, Superimposed thin lines plot absolute direction of cursor submovement relative to the target on the normalized time base for the same subjects after filtering with a moving median of ±100 s using the same format as in B. F, Absolute direction of saccades with reference to the target plotted against the corresponding direction of cursor submovements for the duration of the practice session. The curve represents the mean values obtained from the eight pairs of filtered curves representing saccade direction (B) and submovement direction (E) for each of the eight subjects. The dark and light gray zones represent the variability across subjects expressed as ±1 SEM and ±1 SD in the y-coordinate. G, H, Distribution of absolute directions (in screen coordinates) for all saccades and submovements obtained from all eight subjects. C, D, G, and H, Learning stages represented as in Figure 5C. Ex, Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.