To assess the validity of performance measures used in a nationwide pay-for-performance (P4P) project on hip and knee replacement, we analyzed hospital performance data from a Medicare P4P initiative and compared them to publicly available outcomes data. Overall, the ability to measure hospital quality was poor. A hospital's ranking on the composite score was primarily determined by process measures. A higher composite quality score was not associated with lower rates of complications or mortality. The current Medicare P4P quality measure has limited validity because of poor discrimination, lack of measure balance, and lack of correlation with important clinical outcomes.