Multisensory integration in dynamical behaviors: maximum likelihood estimation across bimanual skill learning

Abstract

Optimal integration of different sensory modalities weights each modality as a function of its degree of certainty (maximum likelihood). Humans rely on near-optimal integration in decision-making tasks (involving e.g., auditory, visual, and/or tactile afferents), and some support for these processes has also been provided for discrete sensorimotor tasks. Here, we tested optimal integration during the continuous execution of a motor task, using a cyclical bimanual coordination pattern in which feedback was provided by means of proprioception and augmented visual feedback (AVF, the position of both wrists being displayed as the orthogonal coordinates of a single cursor). Assuming maximum likelihood integration, the following predictions were addressed: (1) the coordination variability with both AVF and proprioception available is smaller than with only one of the two modalities, and should reach an optimal level; (2) if the AVF is artificially corrupted by noise, variability should increase but saturate toward the level without AVF; (3) if the AVF is imperceptibly phase shifted, the stabilized pattern should be partly adapted to compensate for this phase shift, whereby the amount of compensation reflects the weight assigned to AVF in the computation of the integrated signal. Whereas performance variability gradually decreased over 5 d of practice, we showed that these model-based predictions were already observed on the first day. This suggests not only that the performer integrated proprioceptive feedback and AVF online during task execution by tending to optimize the signal statistics, but also that this occurred before reaching an asymptotic performance level.

Figure 1.

Experimental setup and protocol. a, Target movement: the performer had to continuously move both wrists back and forth, being required to maintain 90° out of phase (a quarter of the cycle) between them. b, Sketch of the experimental setup. c, During some trials, AVF was provided on a computer screen in front of the performer, by displaying the position of both wrists as the orthogonal coordinates of a single cursor. Perfect 90°-out-of-phase cyclical movement corresponds to a circular trajectory on the screen. d, Experimental protocol. The daily experimental session was divided into three parts, delimited by the vertical lines. Part I consisted of a single block (white) of four trials during which subjects had to perform in-phase cyclical movements, without AVF. In parts II and III, the target movement was the 90° out of phase. Part II consisted of four “learning” (red–magenta) or “phase-shift” (gray) blocks, each containing eight trials during which subject received AVF during four randomly selected trials (50%). In the “phase-shift” blocks, an artificial phase shift was inserted between the actual movement and the visual feedback, when present. Randomly inserted between these four blocks, part II also contained a “shakers” block (blue–orange) with four trials during which the subject received AVF and four trials without, and had shakers fixed onto the dorsal and palmar section (flexors and extensors) of both wrists. Finally, part III was done on days 1, 3, and 5 and consisted of three “visual noise” blocks, each containing seven trials during which AVF was provided but corrupted by noise, whose level was randomly selected for each trial.

Figure 2.

Alternative predictions in the reference frame of the visual feedback (see Fig. 1c) due to the introduction of phase shift in the visual display. Either the phase shift is not compensated, such that the performer always executes the 90°-out-of-phase movement while the visual display is skewed depending on the introduced phase shift; or the phase shift is entirely compensated, such that the performer adapts the executed pattern in opposition to the introduced phase shift and the visual display is always a circle. Half compensation lies in between.

Figure 3.

Evolution of task stability across practice and types of task and feedback. The error bars represent the between-subjects SE. a, SD (std) of the phase difference between both wrists across the 5 d of practice for the in-phase movement (black), the “feedforward only” (orange), the intact proprioception+feedforward (red), the “vision+feedforward” (blue), and the proprioception+vision+feedforward (magenta) conditions. The dotted lines are best exponential fits (Eq. 4). The model-based prediction on the optimal level to reach with the three feedback modalities is displayed in green (dashed). It has been calculated from estimates of the single-modality levels from the actual measurements in the three other conditions, according to Equation 10. b, Model error (ME): difference between actual task variability with the three feedback modalities (the magenta curve in a) and the model-based prediction (the dashed green curve in a). The dotted line is the best exponential fit (Eq. 4). c, Parameters of the exponential fits: asymptote (ς_ϕ,∞), amplitude (ς_ϕ,1), and time constant (δ) for the 5 conditions displayed in a, and the ME displayed in b. The dots represent the group average, and the error bar the mean ± SE for individual fits. The asterisks highlight the amplitude fits that are not significantly different from zero (t tests, p > 0.18).

Figure 4.

Evolution of the weights assigned to each modality [proprioception (P), augmented visual feedback (V), and feedforward (FF)] across the 5 d of practice. These weights are inferred from the actual measurements, according to Equation 8.

Figure 5.

SD (std) of the phase difference between both wrists for the “visual noise” blocks. The “vision+feedforward” (V+FF), vision+proprioception+feedforward (V+P+FF), and proprioception+feedforward (P+FF) levels reached on the corresponding days are displayed. Between the V+P+FF and P+FF conditions, the SDs reached with the seven noise levels (nl) are displayed. The error bars represent the between-subjects SE. The dotted curves represent model-based predictions, the performance reaching the same level as without AVF when the noise is augmented.

Figure 6.

Average movement trajectory and phase difference when a phase shift is introduced in the visual display. a, Average trajectories and corresponding visual display (AVF), for the six phase shifts introduced (from yellow to red) and without phase shift (black). All cycles—delimited by the maxima of the right wrist position—have been resampled by 360 equally time-spaced points, then averaged, and are displayed in the reference frame of the AVF (see Fig. 1c). b, Corresponding mean phase difference between both wrists (black). The graph represents the extent to which the participants modified the phase relationship between their wrists when faced with phase shifts in the visual display. The dashed gray line represents the model-based prediction according to the weights computed from Equation 8 and Figure 3a. The asterisks denote the phase shifts for which the measured compensation was not statistically different from this model-based prediction (t tests, all p > 0.05). c, SD (std) of the same variable. The error bars in b and c represent the between-subjects SE.