Eye-hand coordination in object manipulation

Abstract

We analyzed the coordination between gaze behavior, fingertip movements, and movements of the manipulated object when subjects reached for and grasped a bar and moved it to press a target-switch. Subjects almost exclusively fixated certain landmarks critical for the control of the task. Landmarks at which contact events took place were obligatory gaze targets. These included the grasp site on the bar, the target, and the support surface where the bar was returned after target contact. Any obstacle in the direct movement path and the tip of the bar were optional landmarks. Subjects never fixated the hand or the moving bar. Gaze and hand/bar movements were linked concerning landmarks, with gaze leading. The instant that gaze exited a given landmark coincided with a kinematic event at that landmark in a manner suggesting that subjects monitored critical kinematic events for phasic verification of task progress and subgoal completion. For both the obstacle and target, subjects directed saccades and fixations to sites that were offset from the physical extension of the objects. Fixations related to an obstacle appeared to specify a location around which the extending tip of the bar should travel. We conclude that gaze supports hand movement planning by marking key positions to which the fingertips or grasped object are subsequently directed. The salience of gaze targets arises from the functional sensorimotor requirements of the task. We further suggest that gaze control contributes to the development and maintenance of sensorimotor correlation matrices that support predictive motor control in manipulation.

Fig. 1.

Apparatus and kinematically defined phases of the target contact task. A, An infrared-based eye-tracker was used to measure line of gaze of the right eye while the subject used the tips of the right index finger and thumb to grasp and move a bar in a vertical plane 39 cm in front of subject's eyes. Miniature electromagnetic sensors recorded the three-dimensional positions and orientations of the fingertips and the bar. For on-line monitoring of subjects' behavior and recording on video tape (SVHS), gaze position was superimposed on an image from a “scene” video camera that gave the subjects' view of the workspace via a one-way reflecting mirror. An electronic shutter was used to block the subjects' view of the scene. Between trials, subjects grasped a “parking bar” between the right index finger and thumb that was fixed on the tabletop; the grasp points were two small “bumps” on either side of the bar. B, Solid black line represents the position of the tip of the index finger or the left tip of the bar. During the (1) pre-reach phase the subject held the parking bar. The (2) reach phase began when the tip of the index finger had moved 2 cm from its parking position. The (3) grasp phase started when the tip of the index finger was <5 cm (dotted-line circle) from the forthcoming grasp site and ended when the bar began to move. In the (4) up phase the bar was moved toward the target. The (5) target phase began when the distance between the left tip of the bar and the target went below 3 cm (dotted-line circle) and lasted while the tip of the bar was within 3 cm of the center of target contact surface (dark plus sign). C, Solid black linerepresents the position of the centroid of the bar or the tip of the index finger. In the (6) down phase, which began when the bar exited the target zone, the bar was moved toward the table support. The (7) replace phase commenced when the vertical distance between the centroid of the bar and its final position on the support surface became <3 cm and ended when the bar was repositioned on the support surface. During the (8) reset phase, the hand was transported to the parking position. B–C, Arrowheadsdemarcate the end of each phase.

Fig. 2.

Accuracy of gaze measurements and saccade and fixation parameters based on 1316 recorded saccade-fixation episodes.A, Computed gaze position (G) in the horizontal (x) and vertical (y) dimensions together with the recorded point-of-regard signals (R) and pupil position signals (P) for an epoch comprising two saccades,S1 and S2. Gaze velocity in the work plane was assessed from the computed gaze position signals. The first and second maxima of the second time derivative of the gaze velocity signal defined the start and end of a saccade. B, Estimation of the error in gaze position measurements for the horizontal (x) and vertical (y) coordinates of the work plane.C, Histogram showing distribution of saccade durations.D, Histogram showing distribution of saccade amplitudes.E, Scatter plots showing the relationship between saccade amplitude and peak saccade velocity in centimeters per second referenced to the work plane (left ordinate) and in degrees per second (right ordinate). F, The relationship between saccade amplitude and saccade duration.G, Histogram showing distribution of fixation durations.B, D–F, Bottom andtop abscissas represent measurements scaled in distance on the work plane and in degrees of eye movement.

Fig. 3.

Gaze and hand movements for a single trial with the triangular obstacle. A, Behavior up until the bar contacts the target. B, Behavior from target contact until the end of the trial. A, B,Dashed line represents the position of the tip of the index finger, and the solid line represents gaze position. The numbered circles indicate successive gaze fixations, and the numbers attached to the path of finger movement indicate the finger movement during the corresponding fixation–saccade period. Consecutive fixation–saccade units and hand path units are represented in alternating colors of grayand black. The hand is outside the represented area of the workspace during fixation-saccades 0,1, and 16. The small finger movement occurring during the third fixation (A) is masked by the fixation symbol.

Fig. 4.

Gaze fixations in relation to landmarks for all three obstacle conditions. A, The left panels show the distribution of all gaze fixations, from all subjects and trials, from the time gaze left the fixation zone to the start of the target phase. The right panels show the corresponding distribution of gaze fixations after the start of the target phase. Black and gray circlesrepresent fixations inside and outside a 3° (2 cm in the work plane) radius of one of the five landmarks: grasp site, left tip of bar, target, support surface, and protruding point(s) of the obstacle. The area of each circle is proportional to the duration of the fixation; for calibration see the circles inset in the top right corner. In theleft and right panels, the solid contours of the bar represent its mean position at the start and end of the trial, respectively, and the dashed extensions depict the range of positions attributable to the inter-trial variation of the bar position. The curve represents the path of the tip of the bar en route to the target zone (left panels) and from the target zone to the support surface (right panels); bar position data were averaged across all trials for each of the obstacle conditions. B, Distribution of gaze fixations within landmark zones normalized for bar movement and movement of grasp site; dots representing fixation locations are not scaled for duration. The landmark zones (3° radius) are represented by dotted circles, and the center of the cross (thin solid lines) indicates the mean position of the grasp site on the bar. Fixations in the landmark zones were combined for all obstacle conditions and phases of the movement, except for fixations in the obstacle zone, which were taken from the triangular obstacle condition. The area enclosed by the smaller solid-line circles within the landmark zones captured 90% of the fixations. C, Location of the grasp site (○) and the mean gaze position of grasp site fixations (●) for each trial in bar coordinates (bottom panel). The scatter plot plots the horizontal (x) position of the grasp site against that of gaze (r = 0.76;p < 0.001; linear regression) (top panel).

Fig. 5.

Flow of gaze fixations between the defined landmarks for the no-obstacle (A), triangular obstacle (B), and rectangular obstacle (C) conditions. The width of eacharrow represents the proportion of all shifts between landmarks, and arrows accounting for 5% or more of the shifts are filled; shifts at proportions <3% are not shown. Left panels show gaze shifts between landmarks en route to the target (circles indicate start position of gaze when the shutter opened). Right panels show gaze shifts between landmarks away from the target en route to the support surface. The widths of arrows in the boxof the right panel in A provide calibration information. The number of fixation shifts between landmarks per trial was 3.8 ± 0.8 (mean ± SD) for the no-obstacle condition, and 5.0 ± 0.73 and 5.6 ± 1.2 for the triangular and rectangular obstacle conditions, respectively.

Fig. 6.

Spatiotemporal coordination of gaze and manipulatory actions based on data pooled across all subjects and trials with the triangular object. For all plots the common time base has been normalized such that each phase of each trial has been scaled to the median duration of that phase; the trial was initiated at time 0. The vertical lines mark the phase transitions in the task, and the phases are indicated above the top panels.A, Each graph plots the distance between gaze and a landmark (Lm) and the distance between the tip of the index finger (a), tip of the bar (c, d), or the bottom of the bar represented as the lowest point on the bar (e) and the same landmark. The gray dots represent gaze positions at the start of individual fixations, and thehorizontal lines represent fixation durations. Thesolid curve represents the median gaze position as a function of time. The dashed curve represents the median distance between the landmark and either the index finger or the bar. The horizontal rectangles represent the 2 cm (3°) landmark zones. B, Probabilities of fixation of the different landmarks (Lm). The thick curveshows the time-varying instantaneous probability of fixations within 3° of a given landmark (computed for 100 msec bins), and the contour of the gray area shows the corresponding data for fixations within 2° (1.4 cm) of the landmark. The thin curve represents the probability that a fixation within 3° of the landmark had occurred at any previous time during the trial.C, D, Median fixation duration and cumulative median number of fixations per trial as a function of normalized time.

Fig. 7.

Gaze fixation characteristics and duration of movement phases of the target contact task for each obstacle condition based on data combined from all subjects and all trials.White, gray, and stripedbars refer to the no-obstacle, rectangular, and triangular obstacle conditions, respectively (see key attop). A, Bar height indicates the probability of fixation occurring within 3° of the landmark during a trial. B, Total fixation durations per trial for each landmark zone. C, Number of fixations per landmark for trials in which the landmark zone was fixated. D, Durations of individual fixations for each landmark zone.E, Duration of each phase of the trial for the three obstacle conditions. A–D, Fixations of the obstacle before and after the up phase are shown separately. Fixations indicated for the support surface occurred after the up phase.B–E, Box plots illustrate median, quartiles (25th and 75th percentiles), and 5th and 95th percentiles.

Fig. 8.

Time relation between kinematic events associated with various landmarks and gaze entering and exiting the corresponding landmark zones (Lm) represented as cumulative frequency distributions. A, Gaze entering and exiting the grasp site zone relative to index finger contacting the bar.B, Gaze entering and exiting the obstacle zone with reference to the time at which the tip of the bar passed closest to the obstacle during the up phase. C, Gaze entering and exiting the target zone referenced to when the target switch was released. D, Gaze entering and exiting the obstacle zone referenced to the time at which the tip of the bar passed closest to the obstacle during the down phase. E, Gaze entering and exiting the landmark zone of the support surface referenced to when the bar contacted the support surface. A–E,Dashed and solid line curves refer to gaze entering and exiting the landmark zone, respectively. Thin curves represent no-obstacle condition, and the thick curves represent data for the two obstacle conditions combined. Negative time values represent gaze lead before the kinematic event. Gaze landmark zones were defined as gaze fixations being within 3° of the landmark.

Fig. 9.

Estimates of error of saccadic gaze shifts directed to the target based on saccades from the grasp site, tip of the bar, obstacle, and target landmarks. Data from all gaze shifts to within 3 cm (4.4°) of the centroid of all fixations within the 2 cm (3°) target landmark zone that is included. A, Resultant distance between the fixation at the end of the saccade and the target fixation centroid as a function of saccade amplitude. Saccades ∼10 cm originated from the grasp site or tip of the bar, saccades ∼5 cm from the obstacle, and the small amplitude saccades are refixations within the target landmark zone. C,D, The x and y components of these distance as a function of saccade amplitude. Positivex-values indicate that the fixation was to the right of the centroid, and negative y-values indicate that the fixation was below the centroid (compare Fig. 4). D, Fixation duration as a function of distance from the fixation to the centroid. Note the log scale of the ordinate. A–D, Data from all three obstacle conditions combined. Bottom andtop abscissas represent distance in the work plane and in degrees of eye movement, respectively. For each graph, the slope of the linear regression line was different from zero (p < 0.001 in all four instances).

Fig. 10.

Cumulative frequency histograms of travel distances of the tip of the bar during the target phase with free eye movements (Free gaze), without eye movements (Gaze fixation), and with vision occluded after an initial 3 sec viewing time before action. Data were combined from all four trials that each subject performed with the rectangular object under each visual condition. The dotted vertical lineindicates the maximum of travel distance observed with free gaze.