Attention, learning, and the value of information

Abstract

Despite many studies on selective attention, fundamental questions remain about its nature and neural mechanisms. Here I draw from the animal and machine learning fields that describe attention as a mechanism for active learning and uncertainty reduction and explore the implications of this view for understanding visual attention and eye movement control. I propose that a closer integration of these different views has the potential greatly to expand our understanding of oculomotor control and our ability to use this system as a window into high level but poorly understood cognitive functions, including the capacity for curiosity and exploration and for inferring internal models of the external world.

Figure 1

Current approach to attention research A Cortical areas investigated in relation to attention Lateral view of the macaque monkey cortex showing some of the areas that have been investigated in relation with attention, including primary visual cortex (V1), area V4, the middle temporal area (MT), and two sensorimotor areas, the lateral intraparietal area (LIP) and the frontal eye field (FEF). B Normalization model of attention The model includes two populations of cells: feature selective neurons that are sensitive to stimulus location and features (e.g., orientation) and respond to a stimulus with both excitatory and suppressive drives (black panels, lower row) and attention neurons that are selective only for location and provide a selective multiplicative gain (gray, top row). Reproduced with permission from(Reynolds and Heeger, 2009) C Reward sensitive target selection activity in a decision task. Monkeys were trained to direct gaze to one of two possible targets for receipt of a juice reward, and the targets were placed so as to fall inside or opposite the receptive field of an intraparietal cell (dashed oval). Traces show the average responses of a cell population, aligned on the time of target presentation and the monkeys’ subsequent choice. The neurons encoded the direction of the chosen saccade, responding more for saccades directed toward or away from the receptive field (blue vs. green). Directional selectivity however was not constant but depended on the local probability of reward, becoming stronger in proportion to the probability of reward for an receptive field-directed saccade (dotted, thin solid and thick solid traces show progressively larger differences in expected reward). Reproduced with permission from (Sugrue et al., 2004).

Figure 2

Attention as information selection. A Gaze behavior in naturalistic tasks where a subject fills a kettle for preparing tea (top) or prepares a peanut butter sandwich (bottom). Gaze is directed to task relevant locations that reduce the subject’s uncertainty, and precede the skeletal actions. Reproduced with permission from (Land, 2009). B Three putative attentional mechanisms that assign associability according to the reliability (left), uncertainty (middle) or reward probability (right) predicted by a cue.

Figure 3

Dopamine neuron responses in an information choice task A On each trial after achieving central fixation monkeys viewed a target prefacing an informative (green) or uninformative (orange) cue. Single target trials (top and bottom) were interleaved with 2-target trials where monkeys were free to select the target they wished to view. If monkeys shifted gaze to the informative target (green) they were shown two subsequent cues that were consistently associated with, respectively, a large or small water reward. If monkeys shifted gaze to the uninformative target (orange) they were shown two other cues that were inconsistently associated with the large or small rewards (50% predictive validity). The large and small rewards were equally likely to occur, so that the informative and uninformative targets had equal expected rewards. B Neural responses of DA cells on the information choice task The traces show average activity in a population of DA cells, aligned on the time of target presentation, appearance of the reward cues and delivery of the final reward. At the time of target presentation the neurons had stronger responses when the display contained an informative target (dark and light red traces) than when it only contained the uninformative target (blue). After the information was revealed (cue) DA neurons had the expected reward prediction response. At the time of cue presentation they had excitatory and inhibitory responses to, respectively, the high and low reward predictive pattern, and small excitatory responses to the uncertain pattern announcing a 50% probability of reward. At the time of the reward the neurons had excitatory and inhibitory responses upon receipt of, respectively, the large and small reward, but only if this reward was unpredicted (i.e., upon selection of the uninformative cue). Reproduced with permission from (Bromberg-Martin and Hikosaka, 2009).

Figure 4

Lateral intraparietal neurons combine responses to visual selection and visuo-manual associations. A Search task An array of several figure-8 placeholders remained stable on the screen at all times. To begin a trial monkeys directed their eye to the central fixation point (dot) and grabbed two response bars. The search display was then revealed, and contained a cue (a right or left-facing letter “E”) that appeared at an unpredictable location in among letter-like distractors. Monkeys were trained to continue holding central fixation, and release a bar held in the right or left hand to indicate whether the “E” was facing, respectively, to the right or to the left. B A parietal neuron that was sensitive only to cue location The panels show the activity of a lateral intraparietal neuron aligned on the time of target onset. In each row of action potential, the time of the manual release marked by a black dot. Left and right panels are sorted according to the location of the “E”. Blue and red traces refer to trials in which the “E” required release of, respectively, the left or right bar. C A neuron sensitive to both cue location and manual release The neuron encoded “E” location but was modulated by the manual release, responding more strongly if the monkey released the left rather than the right bar. Reproduced with permission from (Oristaglio et al., 2006).

Figure 5

Pavlovian attention in the lateral intraparietal area A Behavioral task. Each trial has a 50% prior probability of ending in a reward. After monkeys achieved central fixation a peripheral cue was flashed for 300 ms either inside the neuron’s receptive field (dark oval, illustrated) or at the opposite location. Cues were abstract colored patterns that signaled with certainty whether the trial will receive a reward (CS+) or no reward (CS−). After a 600 ms delay period a second target appeared unpredictably at the same or opposite location relative to the CS and monkeys have to make an immediate saccade to this target to receive the outcome announced by the CS. An error trial is immediately repeated until correctly completed, so that monkeys have to perform each trial to progress in the task. B Parietal responses to the reward cues. When a CS appeared in the receptive field the population of cells showed transient and sustained responses that were selective for cue value, being stronger for a positive cue predicting a reward (CS+, blue) relative to a negative cue predicting no reward (CS−, red). The stars show time bins with a significant difference between the two conditions. The bottom dashed line shows the pre-cue level of activity. Shading shows the standard error of the mean. C CS-evoked responses were spatially specific. The dark traces in each panel show responses when the CS appeared in the receptive field and the gray traces, responses when the CS appeared at the opposite location. (The dark traces are the same as, respectively, the blue and red traces in panel A, but are shown on an expanded vertical axis.) Responses evoked by a receptive field cue are higher than (CS+) or lower than (CS−) those at the opposite, non-stimulated location, showing that they reflected a spatial bias not a global change in motivation. D Saccadic effects of CS− cues Eye movements in a representative session on unrewarded trials when the saccade target was spatially congruent with a CS−. The location of the CS and target is normalized as if falling horizontally on the right (coordinates of (1,0)) and each gray dot shows the endpoint of a single saccade in these normalized coordinates. The bottom panel shows saccades that followed highly familiar, overlearned CS− (corresponding to the neural responses shown in panel B). The top panel shows responses on trial with newly-learned CS− that were introduced and trained within a single session. Measurement of anticipatory licking showed that monkeys learned the value of the novel CS within the first 5–10 trials, and data collection began after this learning was complete. Presentation of a CS− impaired saccade accuracy if the target happened to be congruent with the CS− location, and the impairment was stronger for overlearned relative to newly-learned CS−. E Overtraining produces plastic changes in the visual response Bottom-up responses to the trained CS were tested in a separate control condition where the previously trained CS were flashed as task-irrelevant probes. In this condition a first predictive CS and the saccade target appeared opposite the receptive field (top panel). Simultaneous with presentation of the saccade target a previously trained CS (the probe) was flashed briefly in the receptive field. The probes had prior reward associations but did not predict reward on these trials. For an overtrained pattern, the bottom-up response remained selective to previous reward associations (bottom left). This value dependent visual response produced differential interference with the saccade, as shown in the bottom right panel. Saccade reaction times (RT) were longer in the presence of a positive relative to a negative probe (blue vs. red), reflecting the stronger interference by the positive pattern. Note that RT were longer on unrewarded relative to rewarded trials, showing that monkeys correctly inferred reward probability based on the first predictive CS (that had appeared opposite the receptive field) and not based on the irrelevant probe. Modified with permission from (Peck et al., 2009).