The mammalian brain is composed of numerous functionally distinct structures that vary in size within and between clades, reflecting selection for sensory and cognitive specialization. Primates represent a particularly interesting case in which to examine mosaic brain evolution since they exhibit marked behavioural variation, spanning most social structures, diets and activity periods observed across mammals. Although studies have consistently demonstrated a trade-off between visual and olfactory specialization in primates, studies of some regions (for example, the neocortex) have produced conflicting results. Here, we analyse the socioecological factors influencing the relative size of 33 brain regions, using updated statistical techniques and data from more species and individuals than previous studies. Our results confirm that group-living species and those with high-quality diets have expanded olfactory or visual systems, depending on whether they are nocturnal or diurnal. Conversely, regions associated with spatial memory are expanded in solitary species and those with low-quality diets, suggesting a trade-off between visual processing and spatial memory. Contrary to previous work, we show that diet quality predicts relative neocortex size at least as well as, if not better than, social complexity. Overall, our results demonstrate that primate brain structure is largely driven by selection on sensory and cognitive specializations that develop in response to divergent socioecological niches.