In humans and other Old World primates, much of visual cortex comprises a set of retinotopic maps, embedded in a cortical sheet with well known, identifiable folding patterns. However, the relationship between these two prominent cortical variables has not been comprehensively studied. Here, we quantitatively tested this relationship using functional and structural magnetic resonance imaging in monkeys and humans. We found that the vertical meridian of the visual field tends to be represented on gyri (convex folds), whereas the horizontal meridian is preferentially represented in sulci (concave folds), throughout visual cortex in both primate species. This relationship suggests that the retinotopic maps may constrain the pattern of cortical folding during development.