Both cortical folding and structural connection patterns are more elaborated during the evolution of primate neocortex. For instance, cortical gyral shapes and structural connection patterns in humans are more complex and variable than those in chimpanzees and macaques. However, the intrinsic relationship between gyral folding and structural connection and their coevolution patterns across primates remain unclear. Here, our qualitative and quantitative analyses of in vivo diffusion tensor imaging (DTI) and structural magnetic resonance imaging (MRI) data consistently demonstrate that structural fiber connection patterns closely follow gyral folding patterns in the direction "tangent" to the cortical sphere, and this close relationship is well preserved in the neocortices of macaque, chimpanzee, and human brains, despite the progressively increasing complexity and variability of cortical folding and structural connection patterns. The findings suggest a hypothesis that a common axonal fiber pushing mechanism sculpts the curved patterns of gyri in the tangent direction during primate brain evolution. Our DTI/MRI data analysis provides novel insights into the structural architecture of primate brains, a new viewpoint of the relationship between cortical morphology and connection, and a basis for future elucidation of the functional implication of coevolution of cortical folding and structural connection patterns.