The malignant transformation of human prostatic epithelium is associated with the loss of androgen receptor (AR) in the surrounding stroma. However, the function and mechanisms of AR signaling in prostate cancer (PCa) stroma remain elusive. Here we report, by using proteomics pathway array analysis (PPAA), that androgen and its receptor inhibit the proliferation of prostate stromal cells through transcriptional suppression of cyclin B1, and we confirmed our findings at mRNA and protein levels using AR-negative or -positive primary prostate stromal cells. Furthermore, AR showed a negative correlation with cyclin B1 expression in stroma of human PCa samples in vivo. Mechanistically, we identify cyclin B1 as a bona fide AR target gene in prostate stromal cells. The negative regulation of cyclin B1 by AR is mediated through switching between E2F1 and E2F4 on the promoter of cyclin B1. E2F1 binds to the cyclin B1 promoter and maintains its expression and subsequent cell cycle progression in AR-negative stromal cells or AR-positive stromal cells when androgens are depleted. Upon stimulation with androgen in AR-positive stromal cells, E2F1 is displaced from the binding site by AR and replaced with E2F4, leading to the recruitment of the silencing mediator for retinoid and thyroid hormone receptor (SMRT)/histone deacetylase 3 (HDAC3) corepressor complex and repression of cyclin B1 at the chromatin level. The switch between E2F1 and E2F4 at the E2F binding site of the cyclin B1 promoter coincides with an androgen-dependent interaction between AR and E2F1 as well as the cytoplasmic-to-nuclear translocation of E2F4. Thus, we identified a novel mechanism for E2F factors in the regulation of cell cycle gene expression and cell cycle progression under the control of AR signaling.