Declines in neural processing speed have been proposed to underlie a broad range of cognitive deficits in older adults. However, the impact of delays in neural processing during stimulus encoding on working memory (WM) performance is not well understood. In the current study, we assessed the influence of aging on the relationship between neural measures of processing speed and WM performance during a selective delayed-recognition task for color and motion stimuli, while electroencephalography (EEG) was recorded in young and older adults. A latency delay was observed for the selection negativity (SN) and alpha band activity (measures of attentional allocation) in older adults during WM encoding of both motion and color stimuli, with the latency and magnitude of the SN predicting subsequent recognition performance. Furthermore, an age-related delay in the N1 latency occurred specifically during the encoding of color stimuli. These results suggest that the presence of both generalized feature-based and feature-specific deficits in the speed of selective encoding of information contributes to WM performance deficits in older adults.