The X chromosome plays a central role in the evolution of reproductive isolation, but few studies have examined the genetic basis of X-linked incompatibilities during the early stages of speciation. We report the results of a large experiment focused on the reciprocal introgression of the X chromosome between two species of house mice, Mus musculus and M. domesticus. Introgression of the M. musculus X chromosome into a wild-derived M. domesticus genetic background produced male-limited sterility, qualitatively consistent with previous experiments using classic inbred strains to represent M. domesticus. The genetic basis of sterility involved a minimum of four X-linked factors. The phenotypic effects of major sterility QTL were largely additive and resulted in complete sterility when combined. No sterility factors were uncovered on the M. domesticus X chromosome. Overall, these results revealed a complex and asymmetric genetic basis to X-linked hybrid male sterility during the early stages of speciation in mice. Combined with data from previous studies, we identify one relatively narrow interval on the M. musculus X chromosome involved in hybrid male sterility. Only a handful of spermatogenic genes are within this region, including one of the most rapidly evolving genes on the mouse X chromosome.