Observations on the role of ovarian hormones in breast cancer growth, as well as interest in contraception, stimulated research into the biology of estrogens. The identification of the classical receptors ERα and ERβ and the transmembrane receptor GPER and the resolution of the structure of the ligand bound to its receptor established the principal molecular mechanisms of estrogen action. The presence of estrogen-like compounds in many plants used in traditional medicine or ingested as food ingredients, phytoestrogens, as well as the estrogenic activities of many industrial pollutants and pesticides, xenoestrogens, have prompted investigations into their role in human health. Phyto- and xenoestrogens bind to the estrogen receptors with a lower affinity than the endogenous estrogens and can compete or substitute the hormone. Xenoestrogens, which accumulate in the body throughout life, are believed to increase breast cancer risk, especially in cases of prenatal and prepuberal exposure whereas the role of phytoestrogens is still a matter of debate. At present, the application of phytoestrogens appears to be limited to the treatment of post-menopausal symptoms in women where the production of endogenous estrogens has ceased. In this review we discuss chemistry, structure and classification, estrogen signaling and the consequences of the interactions of estrogens, phytoestrogens and xenoestrogens with their receptors, the complex interactions of endogenous and exogenous ligands, the evaluation of the health risks related to xenoestrogens, and the perspectives toward the synthesis of potent third generation selective estrogen receptor modulators (SERMs).