Comparing genetic and phenotypic similarity among unrelated individuals seems a promising way to quantify the genetic component of traits while avoiding the problematic assumptions plaguing twin- and other kin-based estimates of heritability. One approach uses a Genetic Relatedness Estimation through Maximum Likelihood (GREML) model for individuals who are related at less than 0.025 to predict their phenotypic similarity by their genetic similarity. Here we test the key underlying assumption of this approach: that genetic relatedness is orthogonal to environmental similarity. Using data from the Health and Retirement Study (and two other surveys), we show two unrelated individuals may be more likely to have been reared in a similar environment (urban versus nonurban setting) if they are genetically similar. This effect is not eliminated by controls for population structure. However, when we include this environmental confound in GREML models, heritabilities do not change substantially and thus potential bias in estimates of most biological phenotypes is probably minimal.