A number of studies have shown that stationary backgrounds compromise smooth pursuit eye movements. It has been suggested that poor attentional selection of the pursuit target was responsible for reductions of pursuit gain. To quantify the detrimental effects of attention, we instructed observers to either pay attention to background objects or to ignore them. The to-be-attended object was indicated by peripheral or central cues. Strong reductions of pursuit gain occurred when the following conditions were met: (a) the subject payed attention to the object (b) a salient event was present, for instance the onset of the target or cue and (c) the attended target produced retinal motion. Removing any of the three conditions resulted in no or far smaller decreases of pursuit gain. Further, decreases in pursuit gain were present with perceptual discrimination and simple manual detection.