The insulin/insulin-like growth factor (IGF) system and related pathways such as growth hormone, and leptin signaling have a key role in cancer development. It is unclear how germline variation in these pathways affects breast cancer risk. We conducted gene-based analyses of 184 genes in the insulin/IGF, growth hormone, and leptin pathways to identify genetic variation associated with risk of breast cancer overall, and for estrogen receptor (ER) subtypes. Tag single-nucleotide polymorphisms (SNPs) for each gene were selected and genotyped on a customized Illumina SNP array. Imputation was carried out using 1000 Genomes haplotypes. The analysis included 91,627 SNPs genotyped or imputed in 3,663 breast cancer cases, (1,983 ER-positive and 1,098 ER-negative) and 4,687 controls from the African American Breast Cancer Epidemiology and Risk consortium, a collaborative project of four large studies of breast cancer in African-American women (Carolina Breast Cancer Study, Black Women's Health Study, Women's Circle of Health Study, and Multiethnic Cohort). We used a multi-locus adaptive joint test to determine the association of each gene with overall breast cancer and ER subtypes. The most significant gene associations (P ≤ 0.01) were BAIAP2 and CALM2 for overall breast cancer; BAIAP2 and CSNK2A1 for ER+ breast cancer; and BRAF, BAD, and MAPK3 for ER- breast cancer. The association of BAD with ER- breast cancer was explained by a two-SNP risk model; all other associations were best explained by one-SNP risk models. In total, six genes and seven SNPs had suggestive associations with overall breast cancer or ER subtypes in African-American women.