Canonical transforming growth factor-beta (TGF-β) signaling exerts neuroprotection and influences memory formation and synaptic plasticity. It has been considered as a new target for the prevention and treatment of depression. This study aimed to examine its modulatory role in linking prenatal maternal depressive symptoms and the amygdala volumes from birth to 6 years of age. We included mother-child dyads (birth: n = 161; 4.5 years: n = 131; 6 years: n = 162) and acquired structural brain images of children at these three time points. Perinatal maternal depressive symptoms were assessed using the Edinburgh Postnatal Depression Scale (EPDS) questionnaire to mothers at 26 weeks of pregnancy and 3 months postpartum. Our findings showed that the genetic variants of TGF-β type I transmembrane receptor (TGF-βRI) modulated the association between prenatal maternal depressive symptoms and the amygdala volume consistently from birth to 6 years of age despite a trend of significance at 4.5 years of age. Children with a lower gene expression score (GES) of TGF-βRI exhibited larger amygdala volumes in relation to greater prenatal maternal depressive symptoms. Moreover, children with a lower GES of the TGF-β type II transmembrane receptor (TGF-βRII), Smad4, and Smad7 showed larger amygdala volumes at 6 years of age in relation to greater prenatal maternal depressive symptoms. These findings support the involvement of the canonical TGF-β signaling pathway in the brain development of children in the context of in utero maternal environment. Such involvement is age-dependent.