Low bone mass and an increased risk of fracture are predictors of osteoporosis. Individuals who share the same bone-mineral density (BMD) vary in their fracture risk, suggesting that microstructural architecture is an important determinant of skeletal strength. Here, we utilized the rich diversity of the Collaborative Cross mice to identify putative causal genes that contribute to the risk of fractures. Using microcomputed tomography, we examined key structural features that pertain to bone quality in the femoral cortical and trabecular compartments of male and female mice. We estimated the broad-sense heritability to be 50-60% for all examined traits, and we identified five quantitative trait loci (QTL) significantly associated with six traits. We refined each QTL by combining information inferred from the ancestry of the mice, ranging from RNA-Seq data and published literature to shortlist candidate genes. We found strong evidence for new candidate genes, particularly Rhbdf2, whose close association with the trabecular bone volume fraction and number was strongly suggested by our analyses. We confirmed our findings with mRNA expression assays of Rhbdf2 in extreme-phenotype mice, and by phenotyping bones of Rhbdf2 knockout mice. Our results indicate that Rhbdf2 plays a decisive role in bone mass accrual and microarchitecture.