Early life conditions are related to cognitive development and abilities in childhood and cognitive function in adulthood. However, the association between early life conditions and cognitive change in old age is unknown. The authors examined the relation between socioeconomic position (SEP) and cognitive milieu in childhood and change in cognitive function in a population-based sample of 4,398 community-dwelling adults (62.1% female; 61.7% Black) aged 65 years or older from Chicago, Illinois. Baseline data were collected in 1993-1997. Change in cognitive function was assessed by means of a global cognitive index derived from measures of memory, perceptual speed, and overall cognitive function administered at three in-home assessments between 1993 and 2003, with an average of 5.3 years of follow-up. After results were controlled for age, sex, race, and education, interactions of time with childhood SEP (beta = -0.003, t = -1.0, p = 0.32) and childhood cognitive milieu (beta = -0.0008, t = -0.5, p = 0.62) were nonsignificant, indicating that early life conditions were not related to cognitive change. SEP (beta = 0.034, t = 2.4, p = 0.01) and cognitive milieu (beta = 0.017, t = 2.2, p = 0.03) were associated with absolute level of cognitive function, with better performance being related to higher SEP and a better cognitive milieu. A better SEP and a more stimulating cognitive milieu in childhood have small but significant effects on absolute level of cognitive function; however, they do not seem to protect against cognitive decline in old age.