Sex differences have been found in the anatomy of brains of a wide variety of vertebrates including humans. Common lore tells us that sex differences in the brain cause sex differences in behavior. This review entertains the possibility that sex differences in the brain may also do the exact opposite. Specifically, sex differences may allow males and females to display remarkably similar behaviors, despite major differences in their physiological and hormonal conditions. First, the difficulties in interpreting the relationship between structure and function will be illustrated by discussing the role of the sexually dimorphic medial preoptic area (MPOA) in male sexual behavior and parental behavior. Second, the sexually dimorphic vasopressin innervation of the brain will be presented as a system that appears to promote as well as prevent sex differences in behavior. Finally, basic and clinical aspects of sex differences in human brains will be discussed.