Human visual cortex contains discrete areas that respond selectively to specific object categories such as faces, bodies, and places. A long-standing question is whether these areas are shaped by genetic or environmental factors. To address this question, here we analyzed functional MRI data from an unprecedented number (n = 424) of monozygotic (MZ) and dizygotic (DZ) twins. Category-selective maps were more identical in MZ than DZ twins. Within each category-selective area, distinct subregions showed significant genetic influence. Structural MRI analysis revealed that the 'genetic voxels' were predominantly located in regions with higher cortical curvature (gyral crowns in face areas and sulcal fundi in place areas). Moreover, we found that cortex was thicker and more myelinated in genetic voxels of face areas, while it was thinner and less myelinated in genetic voxels of place areas. This double dissociation suggests a differential development of face and place areas in cerebral cortex.