Prolactin (PRL) has roles in various physiological functions. Although experimental studies showed that PRL has both beneficial and adverse effects on type 2 diabetes mellitus, clinical findings in subjects with hyperprolactinemia indicate adverse effects on glucose metabolism. However, effects of PRL within the physiological range in human are controversial. A population-based study of 370 Japanese men enrolled in the 2014 Iwaki study (aged 52.0 ± 14.8 years). In this cross-sectional study, associations between serum PRL levels and homeostatic model assessment (HOMA) indices representing glucose metabolism in a physiological setting were examined using multivariable regression analysis. Although univariate linear regression analyses showed significant associations between serum PRL levels and HOMA indices, adjustment with multiple factors made the association with HOMA-ß (insulin secretion) insignificant, while those with HOMA-R (insulin resistance) remained significant (ß = 0.084, p = 0.035). Non-linear regression analyses showed a regression curve with a peak at serum PRL level, 12.4 ng/mL and a positive association of serum PRL level with HOMA-R below the peak (ß = 0.119, p = 0.004). Higher serum PRL levels within the physiological range seem to be associated with insulin resistance in men.