In Batesian mimicry, animals avoid predation by resembling distasteful models. In the swallowtail butterfly Papilio polytes, only mimetic-form females resemble the unpalatable butterfly Pachliopta aristolochiae. A recent report showed that a single gene, doublesex (dsx), controls this mimicry; however, the detailed molecular mechanisms remain unclear. Here we determined two whole-genome sequences of P. polytes and a related species, Papilio xuthus, identifying a single ∼130-kb autosomal inversion, including dsx, between mimetic (H-type) and non-mimetic (h-type) chromosomes in P. polytes. This inversion is associated with the mimicry-related locus H, as identified by linkage mapping. Knockdown experiments demonstrated that female-specific dsx isoforms expressed from the inverted H allele (dsx(H)) induce mimetic coloration patterns and simultaneously repress non-mimetic patterns. In contrast, dsx(h) does not alter mimetic patterns. We propose that dsx(H) switches the coloration of predetermined wing patterns and that female-limited polymorphism is tightly maintained by chromosomal inversion.