Observers searched for a target among distractors while the display items traded places every 110 ms. Search was slower when the target was always relocated to a position previously occupied by a distractor than when the items remained in place, showing the importance of memory for locations in a visual search task. Experiment 2 repeated a previous study in which items could move to any location within the display, but used a larger range of set sizes than tested in the earlier study. A cost in search times to relocating items was found at the larger set sizes, most likely reflecting that the probability that the target would replace a distractor increased with the set size. The findings provide strong evidence for the role of memory for locations within trials in a visual search task.