Organogenesis accompanies the establishment of the vascular system which begins with sprouting angiogenesis. Vascular endothelial growth factor (VEGF) provides the primary stimulation in the vascular sprouting process but the negative regulation of this process remains unclear. This study examined the role of the transforming growth factor-beta (TGF-beta) superfamily in vascular sprouting using a three-dimensional dorsal aorta culture system, in which the dissected tissue was embedded in type I collagen gel. The cultures were maintained under hypoxic conditions to enhance the expression of Flk-1, a receptor for VEGF, thereby ensuring the responsibility to VEGF. Under the culture conditions employed, the dorsal aorta formed many cord-like structures in response to VEGF. To examine the role of TGF-beta in vascular sprouting, each member of the TGF-beta superfamily was applied to this culture system. TGF-beta1, as well as TGF-beta2 and TGF-beta 3, inhibited capillary formation. Likewise, activin A, another member of TGF-beta superfamily, also abolished vascular sprouting, but bone morphogenetic protein 2 did not noticeably change the morphology. Both neutralizing anti-TGF-beta1 antibody and TGF-beta type I receptor (ALK5) inhibitor partially reversed the inhibitory effect of TGF-beta1. Furthermore, down-regulation of ALK5 with small interfering RNA rather than activin receptor-like kinase-1 (ALK1) reversed the effect of TGF-beta1. These data suggest that TGF-beta superfamily may act as an inhibitor of vascular sprouting mainly through ALK5 signaling pathway. We propose that VEGF may antagonize the TGF-beta autoregulatory action to initiate vascular sprouting.