Chromosomal fusion plays a recurring role in the evolution of adaptations and reproductive isolation among species, yet little is known of the evolutionary drivers of chromosomal fusions. Because sex chromosomes (X and Y in male heterogametic systems, Z and W in female heterogametic systems) differ in their selective, mutational, and demographic environments, those differences provide a unique opportunity to dissect the evolutionary forces that drive chromosomal fusions. We estimate the rate at which fusions between sex chromosomes and autosomes become established across the phylogenies of both fishes and squamate reptiles. Both the incidence among extant species and the establishment rate of Y-autosome fusions is much higher than for X-autosome, Z-autosome, or W-autosome fusions. Using population genetic models, we show that this pattern cannot be reconciled with many standard explanations for the spread of fusions. In particular, direct selection acting on fusions or sexually antagonistic selection cannot, on their own, account for the predominance of Y-autosome fusions. The most plausible explanation for the observed data seems to be (a) that fusions are slightly deleterious, and (b) that the mutation rate is male-biased or the reproductive sex ratio is female-biased. We identify other combinations of evolutionary forces that might in principle account for the data although they appear less likely. Our results shed light on the processes that drive structural changes throughout the genome.