The objective of this prospective randomized controlled study was to compare the long-term effects of a structured physical activity intervention with those of aerobic exercises alone, in a cohort of elderly patients who had undergone elective cardiac surgery, and who were classified as frail at the end of rehabilitation based on their Short Physical Performance Battery (SPPB) score. At the end of rehabilitation, 140 frail elderly patients were randomly allocated either to the intervention group (IG) or to the control group (CG). CG participants received the usual aerobic exercise prescription, while IG participants were also taught additional exercises for strength, flexibility, balance and coordination. The improvement in SPPB score after 1 year was the outcome of the study. IG showed a significant improvement in SPPB score (9.0 ± 1.1 vs. 7.7 ± 1.4, p < 0.001), while no significant change was found in CG (7.7 ± 1.6 vs. 7.6 ± 1.5, p = 0.252). IG also showed a significantly higher proportion of participants who improved their SPPB score of at least 1 point (70 vs. 37%, p < 0.001). In conclusions, our structured physical activity intervention significantly improves the SPPB score in frail elderly patients who have undergone elective cardiac surgery. An intervention that improves the SPPB score might delay the occurrence of mobility disability.