The presence or absence of epistasis, or gene interaction, is explicitly assumed in many evolutionary models. Although many empirical studies have documented a role of epistasis in population divergence under laboratory conditions, there have been very few attempts at quantifying epistasis in the native environment where natural selection is expected to act. In addition, we have little understanding of the frequency with which epistasis contributes to the evolution of natural populations. In this study we used a quantitative genetic design to quantify the contribution of epistasis to population divergence for fitness components of a native annual legume, Chamaecrista fasciculata. The design incorporated the contrast of performance of F2 and F3 segregating progeny of 18 interpopulation crosses with the F1 and their parents. Crosses were conducted between populations from 100 m to 2000 km apart. All generations were grown for two seasons in the natural environment of one of the parents. The F1 often outperformed the parents. This F1 heterosis reveals population structure and suggests that drift is a major contributor to population differentiation. The F2 generation demonstrated that combining genes from different populations can sometimes have unexpected positive effects. However, the F3 performance indicated that combining genes from different populations decreased vigor beyond that due to the expected loss of heterozygosity. Combined with previous data, our results suggest that both selection and drift contribute to population differentiation that is based on epistatic genetic divergence. Because only the F3 consistently expressed hybrid breakdown, we conclude that the epistasis documented in our study reflects interactions among linked loci.