Several childhood factors are reported to be associated with a homosexual orientation in men, including gender nonconformity and rejection by parents and peers. The purpose of this study was to explore the associations between these childhood factors and attachment anxiety (the tendency to experience anxiety regarding potential loss and rejection in close relationships) and attachment avoidance (the tendency to avoid versus seek out closeness in relationships) in gay and bisexual men. A community sample of 191 gay and bisexual men completed questionnaires and an attachment interview. Gender nonconformity was significantly associated with paternal, maternal, and peer rejection in childhood. In addition, paternal and peer rejection, but not maternal rejection, independently predicted attachment anxiety. Peer rejection and, to a lesser extent, paternal rejection mediated the association between gender nonconformity and attachment anxiety. Finally, peer rejection mediated the association between paternal rejection and attachment avoidance. Findings highlight the role of gender nonconformity in contributing to childhood rejection and the importance of peer relationships in the socialization of gay men.