Human brain wrinkling has been implicated in neurodevelopmental disorders and yet its origins remain unknown. Polymer gel models suggest that wrinkling emerges spontaneously due to compression forces arising during differential swelling, but these ideas have not been tested in a living system. Here, we report the appearance of surface wrinkles during the in vitro development and self-organization of human brain organoids in a micro-fabricated compartment that supports in situ imaging over a timescale of weeks. We observe the emergence of convolutions at a critical cell density and maximal nuclear strain, which are indicative of a mechanical instability. We identify two opposing forces contributing to differential growth: cytoskeletal contraction at the organoid core and cell-cycle-dependent nuclear expansion at the organoid perimeter. The wrinkling wavelength exhibits linear scaling with tissue thickness, consistent with balanced bending and stretching energies. Lissencephalic (smooth brain) organoids display reduced convolutions, modified scaling and a reduced elastic modulus. Although the mechanism here does not include the neuronal migration seen in in vivo, it models the physics of the folding brain remarkably well. Our on-chip approach offers a means for studying the emergent properties of organoid development, with implications for the embryonic human brain.