Research has established the broad mammalian developmental plan that genes on the sex chromosomes influence gonad development which determines gonadal hormone production (or its absence) leading to modification of the genitalia and simultaneously biasing the nervous system to organize adult sexual behavior. This might be considered the "gonad to hormones to behavior" model. It is clear, however, that although this model generally works well it is incomplete. The model does not account for behavioral influences attributed to the environment or to genetic but nongonadal or hormonal factors. In this essay we probe those areas of sexual development that are neither differentiated by hormones nor activated by them. The concept of the environment used for our discussion is very broad; it incorporates considerations of both the molar and the molecular levels. The general sense of the word "environment" as something exterior to the person is retained, even if that something influences intraperson processes. In addition, we focus directly on molecular events themselves. Here the "environment" involved can be that within a DNA segment. We also expand the notion of "biologically based sex differences." Although many, and perhaps most, important sex differences arise from gonadal and hormonal development, also important are sex differences which are neither gonadal nor hormonal. All these factors affect the internal workings of the individual and intervene in structuring how the social environment might or might not modify sexual behavior. This discourse calls attention to features that are central to the so-called nature-nurture discussion.