In addition to initial sex determination, genes on the sex chromosomes are theorized to play a particularly important role in phenotypic differences between males and females. Sex chromosomes in many species display molecular signatures consistent with these theoretical predictions, particularly through sex-specific gene expression. However, the phenotypic implications of this molecular signature are unresolved, and the role of the sex chromosomes in quantitative genetic studies of phenotypic sex differences is largely equivocal. In this article, we examine molecular and phenotypic data in the light of theoretical predictions about masculinization and feminization of the sex chromosomes. Additionally, we discuss the role of genetic and regulatory complexities in the genome–phenotype relationship, and ultimately how these affect the overall role of the sex chromosomes in sex differences.