We recently identified missense mutations at amino acid residues 526 and 537 located within the highly conserved subdomain XI of the transforming growth factor beta type II receptor (TbetaR-II) serine-threonine kinase in two human squamous carcinoma cell lines. These cell lines are resistant to transforming growth factor beta-mediated inhibition of growth. Moreover, treatment with transforming growth factor beta fails to increase the levels of type 1 plasminogen activator inhibitor and fibronectin synthesis. To test the effects of the mutations on receptor function, mutant TbetaR-II cDNAs were expressed in TbetaR-II-deficient T47D cells. Cyclin A promoter activity was reduced by 50% in cells expressing wild-type TbetaR-II but increased 2-fold in cells transfected with either of the two mutant receptors. Conversely, plasminogen activator inhibitor type 1 promoter activity was increased 6-fold in cells transfected with wild-type receptor but not with either of the two mutant receptors. Moreover, the activity of both mutant serine-threonine kinases was strongly reduced compared to that of the wild-type receptor. Thus, the amino acid residues at positions 526 and 537 seem to be essential for kinase function and signaling activity of the TbetaR-II.