Traditionally, testosterone and estrogen have been considered to be male and female sex hormones, respectively. However, estradiol, the predominant form of estrogen, also plays a critical role in male sexual function. Estradiol in men is essential for modulating libido, erectile function, and spermatogenesis. Estrogen receptors, as well as aromatase, the enzyme that converts testosterone to estrogen, are abundant in brain, penis, and testis, organs important for sexual function. In the brain, estradiol synthesis is increased in areas related to sexual arousal. In addition, in the penis, estrogen receptors are found throughout the corpus cavernosum with high concentration around neurovascular bundles. Low testosterone and elevated estrogen increase the incidence of erectile dysfunction independently of one another. In the testes, spermatogenesis is modulated at every level by estrogen, starting with the hypothalamus-pituitary-gonadal axis, followed by the Leydig, Sertoli, and germ cells, and finishing with the ductal epithelium, epididymis, and mature sperm. Regulation of testicular cells by estradiol shows both an inhibitory and a stimulatory influence, indicating an intricate symphony of dose-dependent and temporally sensitive modulation. Our goal in this review is to elucidate the overall contribution of estradiol to male sexual function by looking at the hormone's effects on erectile function, spermatogenesis, and libido.