The programming effect of birth weight on later body composition has become of increasing interest in recent years. This programming effect is affected by factors such as gender. Physical fitness could be another factor of influence. This study aimed to examine whether handgrip strength (HG) or cardiovascular fitness (CVF) modify the associations between birth weight and body composition in adolescents. A sample of 1,740 (942 females) adolescents aged 13 to 18.5 years, born at more than 35 weeks of gestation, from the AVENA study was studied. Waist circumference was measured, percentage body fat was calculated from skinfold thicknesses and fat free mass was derived by subtracting fat mass from total body weight. HG and CVF were assessed using the HG test and the 20 m shuttle run test, respectively. Birth weight was positively associated with fat free mass in females with high (above the median) CVF (P < 0.001), but not in those with low (below the median) CVF. In contrast, birth weight was positively associated with total and central adiposity in males with low HG (both P = 0.002), but not in those with high HG. These results suggest that the programming effect of birth weight on later body composition is dependent on gender and fitness level. A high birth weight may have a healthy programming effect on body composition in those adolescents with a high fitness level, being associated with increased fat free mass levels in females and not showing the increased adiposity levels observed in unfit males.