Estrogen and androgen are thought to influence breast cancer risk. The actions of estrogens and androgens are mediated through the respective receptors. In this study we examine the association of the Xb1 polymorphism of estrogen receptor alpha (ESR1) and the CAG repeat of the androgen receptor (AR) gene with risk of breast cancer in women living in the Southwestern United States. Cases (N = 1169 non-Hispanic white (NHW) and 576 Hispanic) with first primary breast cancer were matched to controls (N = 1330 NHW and 725 Hispanic) by location (Arizona, Colorado, New Mexico, or Utah) and 5-year age group. Detailed weight history was obtained along with other diet and lifestyle information. Neither the ESR1 nor the AR polymorphisms evaluated were associated independently with breast cancer risk in either Hispanic or NHW women. However, among Hispanic women taking hormone replacement therapy (HRT), there was a 40% reduced risk of breast cancer among women with an X allele (95% CI 0.39, 0.94). Also Hispanic women with the xx genotype had a significant reduced risk of breast cancer in the presence of weight gain prior to age 50 if post-menopausal or prior to diagnosis if pre-menopausal (P interaction 0.02 and <0.01 respectively). These results suggest differences in risk factors for NHW and Hispanic women. However, they provide only minor support for the role of the AR and ESR1 gene in the etiology of breast cancer.