The androgen-signaling pathway is important in the growth and progression of prostate cancer. Androgen ablation therapy, which may result in programmed cell death, is often used to treat advanced prostate cancer. The growth-promoting effects of androgen are mediated mostly through the androgen receptor (AR). Transforming growth factor beta (TGF-beta) plays critical roles in controlling prostate cell proliferation, differentiation, and apoptosis. Normal transcripts and proteins of TGF-beta receptors are frequently lost in prostate cancer cells, especially in advanced stages of the disease. However, the mechanisms by which TGF-beta inhibits proliferation and induces apoptosis in prostate cancer cells is not clear. We investigated the molecular mechanism by which TGF-beta inhibits transcriptional activation mediated by AR. Using transient transfection systems, we demonstrated that Smad3 specifically represses transcriptional activation mediated by AR on two natural androgen-responsive promoters. This repression is transmitted through TGF-beta signaling and can be regulated by other Smad proteins. A protein-protein interaction between AR and Smad3 was identified in vitro and in vivo, and the transcription activation domain of AR and the MH2 of Smad3 were identified as being responsible for binding. Additional functional experiments showed that the repression of AR by Smad3 is mediated solely through the MH2 domain. These results provide fresh insight for understanding the mechanism by which TGF-beta regulates the androgen-signaling pathway in prostate cancer cells.