It is well-established that genetic factors account for large proportions of individual differences in multiple cognitive abilities. It is also well-established that individual differences in performance on many different cognitive ability measures are strongly correlated. Recent empirical investigations, however, have suggested two interesting qualifications to these well-established findings: Genetic variance in cognitive abilities is higher in richer home environments (gene-by-environment interaction), and common variance in different cognitive abilities is lower at higher levels of overall ability (nonlinear factor structure). Although they have been investigated independently, these two phenomena may interact, because richer environments are routinely associated with higher ability levels. Using simulation we demonstrate how un-modeled nonlinear factor structure can obscure interpretation of gene-by-environment interaction. We then reanalyze data from the National Collaborative Perinatal Project, previously used by Turkheimer et al. (2003; Psychol Science), with a two-step method to model both phenomena.