Prostate cancer can be treated with external beam radiation therapy (EBRT) alone or combined with high-dose-rate brachytherapy (HDR-BT+), usually with additional hormonal treatment (HT). The aim of this long-term cross-sectional study was to compare patient-reported sexual function, hormonal symptoms and quality of life (QoL) after EBRT-only or HDR-BT+. Methods: In 2016, minimum 5 years after treatment, men treated with HDR-BT+ (n = 248) or EBRT-only (n = 91) responded to a questionnaire containing the Expanded Prostate Cancer Index Composite (EPIC)-26 and the Short Form-12 (SF-12). Results: Median age at time of answering the questionnaire was 74 years (range 54-86). The majority in the HDR-BT+ group received HT for ≥ 2 years compared to 1/3rd in the EBRT-only group. In crude analyses, the sexual domain summary score (DSS) was significantly higher in the HDR-BT+ compared to the EBRT-only group (32.3 vs. 24.1). All sexual items were significantly better in the HDR-BT+ group except sexual problem. Significantly less fatigue was seen in the HDR-BT+ group despite longer duration of HT. Physical and mental health were similar in both groups. When adjusted for possible confounders, the differences between treatment groups were no longer statistically significant. Only age was significantly associated with lower sexual scores. Low hormonal DSS was significantly associated with decreased QoL. Conclusion: Patients treated with HDR-BT+ had not worse long-term sexual function than men treated with EBRT-only. However, this difference disappeared when adjusted for possible confounders. Thus, the interplay of age, HT and fatigue indicates a strong impact on both QoL and sexual function.