Over the past years there has been increasing evidence that the CIE color rendering index R(a) fails to correspond to the perceived color quality of many light sources, especially some Light-Emitting-Diodes. Several proposals to update, complement or even replace the CIE R(a) have therefore been made. The performance of thirteen color quality metrics was evaluated by calculating the average correlation of the metric predictions with the visual scaling of the perceived color quality obtained in several psychophysical studies. Two aspects of perceived color quality were investigated, appreciation (preference or attractiveness) and naturalness. The memory color quality metric (S(a)) of Smet et al. was found to correlate highly with perceived appreciation (r = 0.88). It was found to be statistically better (p<0.0001) at it than all other metrics. The CIE R(a) performed the worst. A metric that combines the gamut area index (GAI) and the CIE R(a) using an arithmetic mean correlated highly with the perceived naturalness of a light source (r = 0.85). It was found to be statistically better at predicting naturalness than all other metrics (p<0.0001). A negative correlation was found, between the capabilities of a light source's ability to predict appreciation and naturalness, indicating that a complete description of the color quality of a light source probably requires more than one metric.