Sex steroids are chief regulators of gender differences in the skeleton, and male gender is one of the strongest protective factors against osteoporotic fractures. This advantage in bone strength relies mainly on greater cortical bone expansion during pubertal peak bone mass acquisition and superior skeletal maintenance during aging. During both these phases, estrogens acting via estrogen receptor-α in osteoblast lineage cells are crucial for male cortical and trabecular bone, as evident from conditional genetic mouse models, epidemiological studies, rare genetic conditions, genome-wide meta-analyses, and recent interventional trials. Genetic mouse models have also demonstrated a direct role for androgens independent of aromatization on trabecular bone via the androgen receptor in osteoblasts and osteocytes, although the target cell for their key effects on periosteal bone formation remains elusive. Low serum estradiol predicts incident fractures, but the highest risk occurs in men with additionally low T and high SHBG. Still, the possible clinical utility of serum sex steroids for fracture prediction is unknown. It is likely that sex steroid actions on male bone metabolism rely also on extraskeletal mechanisms and cross talk with other signaling pathways. We propose that estrogens influence fracture risk in aging men via direct effects on bone, whereas androgens exert an additional antifracture effect mainly via extraskeletal parameters such as muscle mass and propensity to fall. Given the demographic trends of increased longevity and consequent rise of osteoporosis, an increased understanding of how sex steroids influence male bone health remains a high research priority.