Losses of working and long-term memory are hallmarks of human aging and may signal impending neurodegenerative disease. The maintenance of neural elements in brain systems that support memory, such as synapse formation in prefrontal cortex and hippocampus, are critical for cognitive health in aging. This paper reviews the biological basis for androgens as neuroprotectants or neuromodulators in aging and the importance of androgens on the brain systems important for memory. We relate biological effects to cognitive outcomes in elderly men under a variety of androgen conditions. In brief, androgen deprivation causes significant loss of synapses in the hippocampus in rodent and nonhuman primates, increases amyloid deposition in human and rodent models and causes changes in neurotransmission in prefrontal cortex in rodent models. Recent work suggests that these changes modify age-related cognitive loss, particularly to memory in men. In addition, the conversion of testosterone to its androgen metabolites or to estradiol may play a special role in the preservation of memory in aging. This paper reviews discrepancies between studies using animal models and studies of human cognition, and suggests new directions that are likely to be fruitful in the future for understanding the role of androgens in brain aging. This review suggests that studies of low androgen levels in older men may not index the same biological mechanisms and behavioral effects as the studies of gonadectomy in animal models.