Adrenal gland function is mediated through secreted hormones, which play a vital role in the autonomic and hypothalamic-pituitary-adrenal (HPA)-axis-mediated stress response. The genetic underpinnings of the stress response can be approached using a quantitative trait locus (QTL) analysis. This method has been used to investigate genomic regions associated with variation in complex phenotypes, but it has not been used to explore the structure of the adrenal. We used QTL analyses to identify candidate genes underlying adrenal weight and adrenal cortical zone and medulla widths. We used 64 BXD recombinant inbred (RI) strains of mice (n = 528) and 2 parental strains (C57BL/6J and DBA/2J; n = 20) to measure adrenal weights and adrenal zone widths. For adrenal weight, we found significant QTLs on chromosome 3 for females (Fawq1) and Chr 4 for males (Mawq1) and suggestive QTLs on Chrs 1, 3, 10, and 14 for females and Chrs 2, 4, 10, 17, and X for males. We identified a significant QTL on Chr 10 (Mawdq1) and a suggestive QTL on Chr 13 for male adrenal total width. For male adrenal medulla width, we found a significant QTL on Chr 5 (Mmwdq1) and a suggestive QTL on Chr 1. We also identified significant QTLs on Chrs 10 (Mxwdq1) and 14 (Mxwdq2) for male X-zone width. There are 113 genes that mapped within the significant QTL intervals, and we identified 4 candidate genes associated with adrenal structure and/or function. In summary, this study is an important first step for detecting genetic factors influencing the structure of the adrenal component of the HPA axis using QTL analyses, which may relate to adrenal function and provide further insights into elucidating genes critical for stress-related phenotypes.