Multiple fibroblast growth factor (FGF) axis alterations are known to occur in prostate cancer. Here we simultaneously profiled key components of this axis to determine their relevance in disease progression. An optimized immunohistochemistry protocol was used in expression analysis of FGF2, FGF8, FGFR1, FGFR4, and Sef (similar expression to FGF) in a single TMA of prostate cancer. FGF ligands and receptors were overexpressed in cancers compared to benign samples (p < 0.0001), while Sef expression was reduced (p < 0.0001). There was a positive association between higher grades and increased FGFR4 (p = 0.02), FGF2, and FGF8 (p = 0.002 and p < 0.0001). Sef expression was progressively lower with increasing grade (p = 0.005). Clinical stage was positively associated with FGF2, FGF8, and FGFR4 expression (p = 0.005, 0.03, and 0.012) but not with FGFR1 or Sef expression. Only reduced Sef was associated with bone metastasis (p = 0.02) and was also predictive of subsequent metastasis in initially localized tumours (p = 0.004). Down-regulation of Sef and increased FGFR4 were also the only independent variables associated with disease-specific survival (HR 1.73, p = 0.04 and HR 0.56, p = 0.01). In in vitro studies, silencing Sef enhanced the cell response to FGFs (p < 0.001) and substantially mitigated the effectiveness of an FGFR1 inhibitor. Conversely, increased Sef blocked the response to FGFs and had a comparable suppressive effect to the inhibitor. This study demonstrates that increased FGFR4 and reduced Sef may be critical FGF alterations associated with prostate cancer progression. Sef may also have a role in the tumour response to FGFR inhibition and warrants further investigation in this context.