We investigated the effects of age, background babble, and acoustic distortion of the word itself on serial position memory in a series of experiments involving six different auditory environments (quiet, and 12-talker background babble presented between, overlapping, or concurrent with word presentation or with two kinds of distortion applied to the words). To control for hearing, the level of babble or distortion was adjusted so that younger and older adults could hear the words equally well. Although the presence of continuous and word-flanking background babble adversely affected memory in the early serial positions in both age groups, only older adults' memory was adversely affected in the later serial positions. Moreover, younger adults' memory was not affected by acoustic word distortion, whereas one of the two types of temporal distortion adversely affected memory for later serial positions in older adults. The exact pattern of impairment and its interaction with age suggests that memory in older adults is more affected than that in younger adults in complex listening situations because they either need more time or have to employ more attentional resources to segregate different auditory streams, thereby depleting the pool of resources available for memory encoding.