Human epithelial ovarian cancer is the most lethal female cancer. Hormones and growth factors, including the TGF-beta superfamily, have been suggested to play a role in ovarian tumorigenesis. The biological effects of TGF-beta superfamily are mediated by type I and type II serine/threonine kinase receptors and by intracellular Smad proteins. Recently, we have cloned four transcripts of human activin receptor-like kinase 7 (ALK7), a type I receptor for Nodal. In this study, we have investigated the role of Nodal and ALK7 in four ovarian cancer cell lines, OV2008, C13*, A2780-s, and A2780-cp. Overexpression of Nodal resulted in a significant decrease in the number of metabolically active cells. This effect was mimicked by a constitutively active ALK7 (ALK7-ca) but blocked by dominant negative mutants of ALK7, Smad2, or Smad3. Transient transfection of Nodal and ALK7-ca significantly decreased X-linked inhibitor of apoptosis protein (Xiap) expression, activated both caspase-3 and caspase-9, and increased apoptosis as determined by Hoechst nuclear staining and flow cytometry. In addition, Nodal and ALK7-ca also inhibited cell proliferation as measured by 5-bromo-2'-deoxyuridine (BrdU) assays. Interestingly, the effects of Nodal and ALK7-ca were more potent in chemosensitive A2780-s cells than in its chemoresistant counterpart, A2780-cp cells. These findings demonstrate that Nodal induces apoptosis and inhibits proliferation via ALK7 and Smad2/3 and that the effect of Nodal-ALK7 on apoptosis may be mediated in part by the down-regulation of Xiap and activation of caspase-9 and caspase-3.