There is increasing evidence that growth and differentiation of prostatic carcinoma cells may be modulated not only by androgens and growth factors but also by vitamin D, retinoids, and phenylacetate (PA). The latter agonists may have a role in the prevention and therapy of prostate cancer but their exact therapeutic potential remains unclear. Since both retinoids and vitamin D act via nuclear receptors, the same way androgens do, we studied the interactions of these compounds with androgen-induced proliferation and differentiation using LNCaP cells as a model of androgen-responsive tumor cells. PA was included because of its suspected different mode of action [H3]-thymidine incorporation was used as a measure of proliferative activity, secretion of prostate-specific antigen (PSA) as a measure of differentiated function. The present data show that 1alpha,25-dihydroxycholecalciferol (VD3), all-trans retinoic acid (atRA), 9-cis retinoic acid (9cRA), and PA stimulated LNCaP cell-differentiated function in the presence or absence of androgens. The effects on cell growth were more complicated. In the absence of androgens growth stimulatory effects were observed for the retinoids and under some conditions for VD3. These effects were limited, however, and tended to be more pronounced at low cell densities. In the presence of androgens nearly exclusively growth inhibitory effects were observed. On a molar basis VD3 was the most effective antiproliferative agonist (ED50 = 10(-9) M). It completely neutralized the stimulatory effects of androgens. Growth inhibition was not due to a decrease in the concentration of androgen receptor: whereas atRA, 9cRA, and PA did not alter androgen receptor levels, VD3 provoked a twofold increase. Neither in the presence nor in the absence of androgens did we observe any cooperativity in the growth stimulatory or inhibitory effects of VD3, atRA, or 9cRA. To test whether treatment with any of the studied agonists resulted in a phenotypic reversion and sustained growth arrest, LNCaP cells were pretreated with VD3, atRA, 9cRA, or PA for 6-12 days and reseeded at equal densities as untreated cells. In all cases tested [3H]-thymidine incorporation was restored within 6 days suggesting that none of these compounds caused irreversible growth inhibition.