Cognitive aging processes are underpinned by multiple processes including genetic factors. The brain-derived neurotrophic factor (BDNF) has been suggested to be involved in age-related cognitive decline in otherwise healthy individuals. The gender-specific role of the BDNF gene in cognitive aging remains unclear. The identification of genetic biomarkers might be a useful approach to identify individuals at risk of cognitive decline during healthy aging processes. The aim of this study was to investigate the associations between three single-nucleotide polymorphisms (SNPs) in the BDNF gene and domains of cognitive functioning in normal cognitive aging. The sample, comprising 369 participants (M = 72.7 years, SD = 4.45 years), completed an extensive neuropsychological test battery measuring memory, motor function, and perceptual speed. The relationships between the SNPs rs6265, rs7103411, and rs7124442 and cognitive domains were examined. While significant main effects of BDNF SNPs on cognitive function were found for the association between rs7103411 and memory performance, gender-specific analyses revealed for females significant main effects of rs7103411 for memory and of rs6265 for perceptual speed independent of the APOE*E4 status and education. The finding for the association between rs6265 and perceptual speed in females remained significant after Bonferroni correction for multiple comparisons. None of the analyses showed significant results for males. This study is the first to implicate that the SNPs rs6265 and rs7103411 affect cognitive function in the elderly in a gender-specific way.