Early-life experiences can dramatically affect adult traits. However, the evolutionary origins of such early-life effects are debated. The predictive adaptive response hypothesis argues that adverse early environments prompt adaptive phenotypic adjustments that prepare animals for similar challenges in adulthood. In contrast, the developmental constraints hypothesis argues that early adversity is generally costly. To differentiate between these hypotheses, we studied two sets of wild female baboons: those born during low-rainfall, low-quality years and those born during normal-rainfall, high-quality years. For each female, we measured fertility-related fitness components during years in adulthood that matched and mismatched her early conditions. We found support for the developmental constraints hypothesis: females born in low-quality environments showed greater decreases in fertility during drought years than females born in high-quality environments, even though drought years matched the early conditions of females born in low-quality environments. Additionally, we found that females born in low-quality years to high-status mothers did not experience reduced fertility during drought years. These results indicate that early ecological adversity did not prepare individuals to cope with ecological challenges in later life. Instead, individuals that experienced at least one high-quality early environment--either ecological or social--were more resilient to ecological stress in later life. Together, these data suggest that early adversity carries lifelong costs, which is consistent with the developmental constraints hypothesis.