Müllerian inhibiting substance (MIS), a transforming growth factor-beta family member, causes regression of the Müllerian duct in male embryos. MIS overexpression in transgenic mice ablates the ovary, and MIS inhibits the growth of ovarian cancer cell lines in vitro, suggesting a key role for this hormone in postnatal development of the ovary. This report describes a mechanism for MIS-mediated growth inhibition in both a human epithelial ovarian cancer cell line and a cell line derived from normal ovarian surface epithelium, which is the origin of human epithelial ovarian cancers. MIS-treated cells accumulated in the G(1) phase of the cell cycle and subsequently underwent apoptosis. MIS up-regulated the cyclin-dependent kinase inhibitor p16 through an MIS type II receptor-mediated mechanism and inhibited growth in the absence of detectable or inactive Rb protein. Prolonged treatment with MIS down-regulated the Rb-related protein p130 and increased the Rb family-regulated transcription factor E2F1, overexpression of which inhibited growth. These findings demonstrate that p16 is required for MIS-mediated growth inhibition in ovarian epithelial cells and tumor cells and suggest that up-regulation of E2F1 also plays a role in this process.