The colonization of novel habitats involves complex interactions between founder events, selection, and ongoing migration, and can lead to diverse evolutionary outcomes from local extinction to adaptation to speciation. Although there have been several studies of the demography of colonization of remote habitats, less is known about the demographic consequences of colonization of novel habitats within a continuous species range. Populations of the Eastern Fence Lizard, Sceloporus undulatus, are continuously distributed across two dramatic transitions in substrate color in southern New Mexico and have undergone rapid adaptation following colonization of these novel environments. Blanched forms inhabit the gypsum sand dunes of White Sands and melanic forms are found on the black basalt rocks of the Carrizozo lava flow. Each of these habitats formed within the last 10,000 years, allowing comparison of genetic signatures of population history for two independent colonizations from the same source population. We present evidence on phenotypic variation in lizard color, environmental variation in substrate color, and sequence variation for mitochondrial DNA and 19 independent nuclear loci. To confirm the influence of natural selection and gene flow in this system, we show that phenotypic variation is best explained by environmental variation and that neutral genetic variation is related to distance between populations, not partitioned by habitat. The historical demography of colonization was inferred using an Approximate Bayesian Computation (ABC) framework that incorporates known geological information and allows for ongoing migration with the source population. The inferences differed somewhat between mtDNA and nuclear markers, but overall provided strong evidence of historical size reductions in both white sand and black lava populations at the time of colonization. Populations in both novel habitats appear to have undergone partial but incomplete recovery from the initial bottleneck. Both ABC analyses and measures of mtDNA sequence diversity also suggested that population reductions were more severe in the black lava compared to the white sands habitat. Differences observed between habitats may be explained by differences in colonization time, habitat geometry, and strength or response to natural selection for substrate matching. Finally, effective population size reductions in this system appear to be more dramatic when colonization is accompanied by a change in selection regime. Our analyses are consistent with a demographic cost of adaptation to novel environments and show that it is possible to infer aspects of the historical demography of local adaptation even in the presence of ongoing gene flow.