Genes on the X chromosome are extensively expressed in the human brain. However, little is known for the X chromosome's impact on the brain anatomy, microstructure, and functional networks. We examined 1045 complex brain imaging traits from 38,529 participants in the UK Biobank. We unveiled potential autosome-X chromosome interactions while proposing an atlas outlining dosage compensation for brain imaging traits. Through extensive association studies, we identified 72 genome-wide significant trait-locus pairs (including 29 new associations) that share genetic architectures with brain-related disorders, notably schizophrenia. Furthermore, we found unique sex-specific associations and assessed variations in genetic effects between sexes. Our research offers critical insights into the X chromosome's role in the human brain, underscoring its contribution to the differences observed in brain structure and functionality between sexes.