Epidemiological evidence has shown that men consuming a low-fat, high-fiber diet containing high amounts of plant products have a lower risk of prostate cancer than men consuming a Western diet. One of the main differences between these two diets is the type of dietary fat, including dietary sterols. This study was undertaken to compare the effect of two dietary sterols on prostate cancer cells in vitro. beta-Sitosterol (SIT), the most common plant sterol, and cholesterol, an animal sterol, were compared for effect on LNCaP cell growth, differentiation, apoptosis, and sphingomyelin cycle intermediates. Cells were treated for up to seven days with sterols delivered by a cyclodextrin vehicle. Compared with cholesterol, SIT (16 microM) decreased growth by 24% and induced apoptosis fourfold, which was accompanied by cell rounding and a 50% increase in ceramide production. No effect was observed on differentiation as measured by prostate-specific antigen and prostatic acid phosphatase, although total acid phosphatase increased with SIT treatment for up to seven days. The results suggest that the decrease in cell number and increase in apoptosis associated with SIT treatment are mediated by activating the sphingomyelin cycle.