Cognitive decline is a common occurrence of the natural aging process in animals and studying age-related changes in gene expression in the brain might shed light on disrupted molecular pathways that play a role in this decline. The fruit fly is a useful neurobiological model for studying aging due to its short generational time and relatively small brain size. We investigated age-dependent changes in the Drosophila melanogaster whole-brain transcriptome by comparing 5-, 20-, 30- and 40-day-old flies of both sexes. We used RNA-Sequencing of dissected brain samples followed by differential expression, temporal clustering, co-expression network and gene ontology enrichment analyses. We found an overall decline in expression of genes from the mitochondrial oxidative phosphorylation pathway that occurred as part of aging. We also detected, in females, a pattern of continuously declining expression for many neuronal function genes, which was unexpectedly reversed later in life. This group of genes was highly enriched in memory-impairing genes previously identified through an RNAi screen. We also identified deficits in short-term olfactory memory performance in older flies of both sexes, some of which matched the timing of certain changes in the brain transcriptome. Our study provides the first transcriptome profile of aging brains from fruit flies of both sexes, and it will serve as an important resource for those who study aging and cognitive decline in this model.