The rate at which a species responds to natural selection is a central predictor of the species' ability to adapt to environmental change. It is well-known that spatially-structured environments slow the rate of adaptation due to increased intra-genotype competition. Here, we show that this effect magnifies over time as a species becomes better adapted and grows faster. Using a reaction-diffusion model, we demonstrate that growth rates are inextricably coupled with effective spatial scales, such that higher growth rates cause more localized competition. This has two effects: selection requires more generations for beneficial mutations to fix, and spatially-caused genetic drift increases. Together, these effects diminish the value of additional growth rate mutations in structured environments.