Colonizing species may often encounter strong selection during the initial stages of adaptation to novel environments. Such selection is particularly likely to act on traits expressed early in development since early survival is necessary for the expression of adaptive phenotypes later in life. Genetic studies of fitness under field conditions, however, seldom include the earliest developmental stages. Using a new set of recombinant inbred lines, we present a study of the genetic basis of fitness variation in Arabidopsis thaliana in which genotypes, environments, and geographic location were manipulated to study total lifetime fitness, beginning with the seed stage. Large-effect quantitative trait loci (QTLs) for fitness changed allele frequency and closely approached 90% in some treatments within a single generation. These QTLs colocated with QTLs for germination phenology when seeds were dispersed following a schedule of a typical winter annual, and they were detected in two geographic locations at different latitudes. Epistatically interacting loci affected both fitness and germination in many cases. QTLs for field germination phenology colocated with known QTLs for primary dormancy induction as assessed in laboratory tests, including the candidate genes DOG1 and DOG6. Therefore fitness, germination phenology, and primary dormancy are genetically associated at the level of specific chromosomal regions and candidate loci. Genes associated with the ability to arrest development at early life stages and assess environmental conditions are thereby likely targets of intense natural selection early in the colonization process.