Many studies document negative inbreeding effects on individuals, and conservation efforts to preserve rare species routinely employ strategies to reduce inbreeding. Despite this, there are few clear examples in nature of inbreeding decreasing the growth rates of populations, and the extent of population-level effects of inbreeding in the wild remains controversial. Here, we take advantage of a long-term dataset of 26 reintroduced Alpine ibex (Capra ibex ibex) populations spanning nearly 100 years to show that inbreeding substantially reduced per capita population growth rates, particularly for populations in harsher environments. Populations with high average inbreeding (F ≈ 0.2) had population growth rates reduced by 71% compared with populations with no inbreeding. Our results show that inbreeding can have long-term demographic consequences even when environmental variation is large and deleterious alleles may have been purged during bottlenecks. Thus, efforts to guard against inbreeding effects in populations of endangered species have not been misplaced.