The developmental origins of the health and disease hypothesis suggests that fetal growth restriction (FGR) is a risk factor for several chronic diseases of adulthood. However, most supporting studies use birth weight as a proxy measure of FGR. To examine the relationship between birth weight and FGR, the present study used serial prenatal ultrasound to identify periods of FGR during gestation, and related these periods to birth size and shape. The data in this study included serial prenatal ultrasounds performed on 1,349 high-risk Scandinavian women enrolled in the National Institute of Child Health and Human Development Study of Successive Small for Gestational Age Births. Fetal growth velocity between ultrasounds was used to identify periods of isolated FGR, and these were studied in relation to anthropometry at birth. FGR was identified in 184 subjects. A control group of 384 subjects without FGR was also identified. Infants with first-trimester FGR (n = 20) had the highest birth weight, ponderal index, and subscapular skinfold thickness. Infants with second-trimester FGR (n = 37) had the highest arm fat percentage. Infants with early third-trimester FGR (n = 55) had the lowest mean birth weight and ponderal index. When infant gender, gestational age, maternal body mass index, and smoking were controlled, birth weight was predicted only by third-trimester FGR (not first- or second-trimester FGR), and arm fat percent was predicted only by second-trimester FGR. These results suggest that birth weight is not a valid indicator of FGR occurring before the third trimester. Body composition may be a more sensitive marker of early FGR.