The field of behavioral neuroendocrinology has generated thousands of studies that indicate differences in brain structure and reactivity to gonadal steroids that produce sex-specific patterns of social behavior. However, rapidly emerging evidence shows that genetic polymorphisms and resulting differences in the expression of neuroactive peptides and receptors as well as early-life experience and epigenetic changes are important modifiers of social behavior. Furthermore, due to its inherent complexity, the neurochemical mechanisms underlying sex differences in social behavior are usually studied in a tightly regulated laboratory setting rather than in complex environments. Importantly, specific hormones may elicit a range of different behaviors depending on the cues present in these environments. For example, individuals exposed to a psychosocial stressor may respond differently to the effects of a gonadal steroid than those not exposed to chronic stress. The objective of this review is not to re-examine the activational effects of hormones on sex differences in social behavior but rather to consider how genetic and environmental factors modify the effects of hormones on behavior. We will focus on estrogen and its receptors but consideration is also given to the role of androgens. Furthermore, we have limited our discussions to the importance of oxytocin and vasopressin as targets of gonadal steroids and how these effects are modified by genetic and experiential situations. Taken together, the data clearly underscore the need to expand research initiatives to consider gene-environment interactions for better understanding of the neurobiology of sex differences in social behavior.