Exposure to maternal anxiety predicts offspring brain development. However, because children's brains are commonly assessed years after birth, the timing of such maternal influences in humans is unclear. This study aimed to examine the consequences of antenatal and postnatal exposure to maternal anxiety upon early infant development of the hippocampus, a key structure for stress regulation. A total of 175 neonates underwent magnetic resonance imaging (MRI) at birth and among them 35 had repeated scans at 6 months of age. Maternal anxiety was assessed using the State-Trait Anxiety Inventory (STAI) at week 26 of pregnancy and 3 months after delivery. Regression analyses showed that antenatal maternal anxiety did not influence bilateral hippocampal volume at birth. However, children of mothers reporting increased anxiety during pregnancy showed slower growth of both the left and right hippocampus over the first 6 months of life. This effect of antenatal maternal anxiety upon right hippocampal growth became statistically stronger when controlling for postnatal maternal anxiety. Furthermore, a strong positive association between postnatal maternal anxiety and right hippocampal growth was detected, whereas a strong negative association between postnatal maternal anxiety and the left hippocampal volume at 6 months of life was found. Hence, the postnatal growth of bilateral hippocampi shows distinct responses to postnatal maternal anxiety. The size of the left hippocampus during early development is likely to reflect the influence of the exposure to perinatal maternal anxiety, whereas right hippocampal growth is constrained by antenatal maternal anxiety, but enhanced in response to increased postnatal maternal anxiety.