Emerging evidence suggests sex differences in the early origins of adult metabolic disease, but this has been little investigated in developing countries. We investigated sex-specific associations between low birth weight (LBW; <2.5 kg) and adult-onset diabetes in 12,525 participants from the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil). Diabetes was defined by self-reported information and laboratory measurements. In confounder-adjusted analyses, LBW (vs. 2.5-4 kg) was associated with higher prevalence of diabetes in women (Prevalence Ratio (PR) 1.54, 95% CI: 1.32-1.79), not in men (PR 1.06, 95% CI: 0.91-1.25; Pheterogeneity = 0.003). The association was stronger among participants with maternal diabetes (PR 1.60, 95% CI: 1.35-1.91), than those without (PR 1.15, 95% CI: 0.99-1.32; Pheterogeneity = 0.03). When jointly stratified by sex and maternal diabetes, the association was observed for women with (PR 1.77, 95% CI: 1.37-2.29) and without (PR 1.45, 95% CI: 1.20-1.75) maternal diabetes. In contrast, in men, LBW was associated with diabetes in participants with maternal diabetes (PR 1.45, 95% CI: 1.15-1.83), but not in those without (PR 0.92, 95% CI: 0.74-1.14). These sex-specific findings extended to continuous measures of glucose homeostasis. LBW was associated with higher diabetes prevalence in Brazilian women, and in men with maternal diabetes, suggesting sex-specific intrauterine effects on adult metabolic health.