Although caffeine can enhance cognitive function acutely, long-term effects of consumption of caffeine-containing beverages such as tea and coffee are uncertain. Data on 4,809 participants aged 65 and older from the Cardiovascular Health Study (CHS) were used to examine the relationship of consumption of tea and coffee, assessed by food frequency questionnaire, on change in cognitive function by gender. Cognitive performance was assessed using serial Modified Mini-Mental State (3MS) examinations, which were administered annually up to 9 times. Linear mixed models were used to estimate rates of change in standard 3MS scores and scores modeled using item response theory (IRT). Models were adjusted for age, education, smoking status, clinic site, diabetes, hypertension, stroke, coronary heart disease, depression score, and APOE genotype. Over the median 7.9 years of follow-up, participants who did not consume tea or coffee declined annually an average of 1.30 points (women) and 1.11 points (men) on standard 3MS scores. In fully adjusted models using either standard or IRT 3MS scores, we found modestly reduced rates of cognitive decline for some, but not all, levels of coffee and tea consumption for women, with no consistent effect for men. Caffeine consumption was also associated with attenuation in cognitive decline in women. Dose-response relationships were not linear. These longitudinal analyses suggest a somewhat attenuated rate of cognitive decline among tea and coffee consumers compared to non-consumers in women but not in men. Whether this association is causal or due to unmeasured confounding requires further study.