Social, physical and intellectual activities are thought to facilitate cognitive performance and slow the rate of age associated cognitive decline, but little is known about this association in younger adulthood. We used multiple regression to test the association between two kinds of activity at 36 years-physical exercise and spare-time activity-and verbal memory at 43 and 53 years in 1919 males and females enrolled in the MRC National Survey of Health and Development (the British 1946 birth cohort). Both kinds of activities were significantly and positively associated with memory performance at 43 years, after controlling for sex, education, occupational social class, IQ at 15 years, and recurrent ill health and significant mental distress. Furthermore, physical exercise at 36 years (but not spare-time activity) was associated with a significantly slower rate of decline in memory from 43 to 53 years, after controlling for the same factors, with evidence that continuing physical exercise after 36 years was important for protection. We conclude that physical exercise and spare-time activity are significantly associated with benefit to memory in midlife, although these two kinds of voluntary activity may exert their effects on cognition via different paths.