This paper aims to develop a reliable psychoacoustic evaluation tool for measuring snoring sounds. Thirteen snoring sound samples were assessed by 25 listeners, in terms of psychoacoustic metrics (loudness, sharpness, roughness, fluctuation strength, and annoyance) using a 7-point semantic differential scale with bipolar adjective pairs. The accuracy of this study was quantified by receiver operating characteristic curves, together with Pearson's product-moment and Spearman's rank correlation coefficients. Results consistently show that loudness, annoyance, and roughness are the best three metrics for classifying apneic and benign snorers as they can achieve high diagnostic accuracy and good correlation with apnea-hypopnea index, body mass index, and neck circumference. With these encouraging results, further research and development of an effective psychoacoustic evaluation tool is promising.