To evaluate intensively the quality of the sediments contaminated by heavy metals and metalloids in the Xiangjiang River, 52 surface sediment samples were collected at 13 sites and different combinations of empirical and theoretical sediment quality indexes, the consensus-based sediment quality guidelines, sediment toxicity degree, and equilibrium partitioning method were applied. The average contents of Cd, Pb, Cu, Zn, Hg, Cr, and As in the sampled surface sediments were significantly higher than the background values of trace elements in soils of Hunan Province, China. Moreover, speciation fraction analyses revealed that Cd, Hg, and Pb in the sediments were dominated by the more bioavailable organic or exchangeable fractions, whereas the major species of As and Cr were the less bioavailable residual fractions after strong acid treatment. In addition, all indexes showed that these metals posed a median-high degree of toxic risk to benthic organisms in sediments from nearly all of the sampling sites along the Xiangjiang River. Cd, followed by Cu and Pb, erected the most severe ecological risk. Pearson correlation and linear regression analyses between the mean PEC quotients, sediment toxicity degree, interstitial water criteria toxic units, and sediment pollution index showed that these indexes were relatively consistent to assess the quality of sediments contaminated by heavy metals and metalloids in the Xiangjiang River. Our results will facilitate the proposal of proper sediment quality guidelines for the Xiangjiang River.