Improving the reliability of physician performance assessment: identifying the "physician effect" on quality and creating composite measures

Abstract

Background: The proliferation of efforts to assess physician performance underscore the need to improve the reliability of physician-level quality measures.

Objective: Using diabetes care as a model, to address 2 key issues in creating reliable physician-level quality performance scores: estimating the physician effect on quality and creating composite measures.

Design: Retrospective longitudinal observational study.

Subjects: A national sample of physicians (n = 210) their patients with diabetes (n = 7574) participating in the National Committee on Quality Assurance-American Diabetes Association's Diabetes Provider Recognition Program.

Measures: Using 11 diabetes process and intermediate outcome quality measures abstracted from the medical records of participants, we tested each measure for the magnitude of physician-level variation (the physician effect or "thumbprint"). We then combined measures with a substantial physician effect into a composite, physician-level diabetes quality score and tested its reliability.

Results: We identified the lowest target values for each outcome measure for which there was a recognizable "physician thumbprint" (ie, intraclass correlation coefficient > or =0.30) to create a composite performance score. The internal consistency reliability (Cronbach's alpha) of the composite score, created by combining the process and outcome measures with an intraclass correlation coefficient > or =0.30, exceeded 0.80. The standard errors of the composite case-mix adjusted score were sufficiently small to discriminate those physicians scoring in the highest from those scoring in the lowest quartiles of the quality of care distribution with no overlap.

Conclusions: We conclude that the aggregation of well-tested quality measures that maximize the "physician effect" into a composite measure yields reliable physician-level quality of care scores for patients with diabetes.