Sex steroid action is critical to form sexually dimorphic nuclei, although it is not fully understood. We previously reported that masculinization of the principal nucleus of the bed nucleus of the stria terminalis (BNSTp), which is larger and has more neurons in males than in females, involves aromatized testosterone that acts via estrogen receptor-α (ERα), but not estrogen receptor-β (ERβ). Here, we examined sex steroid action on the formation of the anteroventral periventricular nucleus (AVPV) that is larger and has more neurons in females. Morphometrical analysis of transgenic mice lacking aromatase, ERα, or ERβ genes revealed that the volume and neuron number of the male AVPV were significantly increased by deletion of aromatase and ERα genes, but not the ERβ gene. We further examined the AVPV and BNSTp of androgen receptor knockout (ARKO) mice. The volume and neuron number of the male BNSTp were smaller in ARKO mice than those in wild-type mice, while no significant effect of ARKO was found on the AVPV and female BNSTp. We also examined aromatase, ERα, and AR mRNA levels in the AVPV and BNSTp of wild-type and ARKO mice on embryonic day (ED) 18 and postnatal day (PD) 4. AR mRNA in the BNSTp and AVPV of wild-type mice was not expressed on ED18 and emerged on PD4. In the AVPV, the aromatase mRNA level was higher on ED18, although the ERα mRNA level was higher on PD4 without any effect of AR gene deletion. Aromatase and ERα mRNA levels in the male BNSTp were significantly increased on PD4 by AR gene deletion. These results suggest that estradiol signaling via ERα during the perinatal period and testosterone signaling via AR during the postnatal period are required for masculinization of the BNSTp, whereas the former is sufficient to defeminize the AVPV.