Prostate cancer (PC) is the second most common cancer in men. Family history is the major risk factor for PC. Only two susceptibility genes were identified in PC, BRCA2 and HOXB13. A comprehensive search of germline variants for patients with PC has not been reported in Japanese families. In this study, we conducted exome sequencing followed by Sanger sequencing to explore responsible germline variants in 140 Japanese patients with PC from 66 families. In addition to known susceptibility genes, BRCA2 and HOXB13, we identified TRRAP variants in a mutually exclusive manner in seven large PC families (three or four patients per family). We also found shared variants of BRCA2, HOXB13, and TRRAP from 59 additional small PC families (two patients per family). We identified two deleterious HOXB13 variants (F127C and G132E). Further exploration of the shared variants in rest of the families revealed deleterious variants of the so-called cancer genes (ATP1A1, BRIP1, FANCA, FGFR3, FLT3, HOXD11, MUTYH, PDGFRA, SMARCA4, and TCF3). The germline variant profile provides a new insight to clarify the genetic etiology and heterogeneity of PC among Japanese men.