Physical fitness has salutary psychological and physical effects in older adults by promoting neuroplasticity and adaptation to stress. In aging, however, the effects of fitness on the hypothalamic-pituitary-adrenal (HPA) axis are mixed. We investigated the association between cardiorespiratory fitness and HPA activity in healthy elderly men (n = 22, mean age 68 y; smokers, obese subjects, those taking drugs or reporting recent stressful events were excluded), by measuring in saliva: i) daily pattern of cortisol secretion (6 samples: 30' post-awakening, and at 12.00, 15.00, 18.00, 21.00, 24.00 h); and ii) the cortisol response to a mental challenge. Cardiorespiratory fitness (VO2max) was estimated using the Rockport Walking Test and the participants were assigned to high-fit (HF, ≥60°, n = 10) and low-fit (LF, ≤35°, n = 12) groups according to age-specific percentiles of VO2max distribution in the general population. At all daytimes, basal cortisol levels were lower in the HF than the LF group, most notably in the evening and midnight samples, with a significant main effect of physical fitness for cortisol levels overall; the area-under-the-curve for total daily cortisol output was significantly smaller in the HF group. Among the subjects who responded to mental stress (baseline-to-peak increment >1.5 nmol/L; n = 13, 5 LF, 8 HF), the amplitude of cortisol response and the steepness of recovery decline displayed an increasing trend in the HF subjects, although between-group differences failed to reach the threshold for significance. In conclusion, cardiorespiratory fitness in healthy aging men is negatively correlated with daily cortisol output and contributes to buffering the HPA dysregulation that occurs with advancing age, thus possibly playing a beneficial role in contrasting age-related cognitive and physical decline.