It has been well established that working memory abilities decrease with advancing age; however, the specific time point in the adult life span at which this deficit begins and the rate at which it advances are still controversial. There is no agreement on whether working memory declines equally for visuospatial and verbal information, and the literature disagrees on how task difficulty may influence this decay. We addressed these questions in a lifespan sample of 1,500 participants between 21 and 80 years old. The n-back task was used, with letters and circles presented at different positions around an imaginary circle, to evaluate working memory in the verbal and visuospatial domains, respectively. The participants' task was to judge whether the current stimulus matched a stimulus that was shown n trials prior. Both domains were evaluated in two levels of difficulty: 1-back and 2-back. The comparison across decades showed that discrimination in the visuospatial and 1-back tasks started to decline earlier in women than in men; however, discrimination was equal between the sexes in the verbal and 2-back tasks. Performance on tasks in the visuospatial domain exhibited more pronounced decline than in those in the verbal domain. The rate of decline in working memory accuracy was superior in 2-back tasks than in 1-back tasks, independent of the domain. These results revealed that the effects of aging on working memory are less dependent on the type of information and more reliant on the resources demanded by the task.