Androgen deprivation therapy (ADT) is a cornerstone treatment for locally advanced or metastatic prostate cancer (PCa). However, its potential effects on the tumor immune microenvironment (TIM) of PCa patients and the underlying mechanism remain largely unclear. To explore the effects of ADT on PCa TIM, RNA sequencing was performed on six paired pre-ADT biopsy and post-ADT PCa lesions, and five paired paracancerous benign tissues from patients receiving neoadjuvant ADT with locally advanced PCa. Bioinformatics methods including ESTIMATE and ssGSEA were used to evaluate the stromal immune score and immune cell infiltration in PCa and paracancerous tissues. Weighted correlation network analysis was used to screen hub genes in the ADT-induced immune remodeling process. The results showed differences exist between PCa and paracancerous tissues in response to ADT. Compared with paracancerous tissues, the immune remodeling effect of ADT in PCa was more intense. ZFP36, JUNB, and SOCS3 served as hub genes in the ADT-induced immune remodeling process and were associated with PSA recurrent-free survival in the TCGA and our neoadjuvant ADT cohort. To investigate the joint action of the above three hub genes, an immune signature score was constructed. The results showed that immune signature score-based immune subtypes reveal the heterogeneity of the immune microenvironment of PCa and showed significant differences in patient prognosis, tumor immune infiltration, mutation burden, and landscape.