The transition to selfing in Capsella rubella accompanies its recent divergence from the ancestral outcrossing C. grandiflora species about 100,000 years ago. Whether the change in mating system was accompanied by the evolution of additional reproductive barriers that enforced species divergence remained unknown. Here, we show that C. rubella and C. grandiflora are reproductively separated by an endosperm-based, non-reciprocal postzygotic hybridization barrier. While hybridizations of C. rubella maternal plants with C. grandiflora pollen donors resulted in complete seed abortion caused by endosperm cellularization failure, the reciprocal hybridization resulted in the formation of small seeds with precociously cellularized endosperm. Strikingly, the transcriptomic response of both hybridizations mimicked respectively the response of paternal and maternal excess hybridizations in Arabidopsis thaliana, suggesting unbalanced genome strength causes hybridization failure in both species. These results provide strong support for the theory that crosses between plants of different mating systems will be unbalanced, with the outcrosser behaving like a plant of increased ploidy, evoking a response that resembles an interploidy-type seed failure. Seed incompatilibity of C. rubella pollinated by C. grandiflora followed the Bateson-Dobzhansky-Muller model, involving negative genetic interaction of multiple paternal C. grandiflora loci with at least one maternal C. rubella locus. Given that both species only recently diverged, our data suggest that a fast evolving mechanism underlies the post-zygotic hybridization barrier(s) separating both species.