Improving the reliability of physician "report cards"

Abstract

Background: Performance measures are widely used to profile primary care physicians (PCPs) but their reliability is often limited by small sample sizes. We evaluated the reliability of individual PCP profiles and whether they can be improved by combining measures into composites or by profiling practice groups.

Methods: We performed a cross-sectional analysis of electronic health record data for patients with diabetes (DM), congestive heart failure (CHF), ischemic vascular disease (IVD), or eligible for preventive care services seen by a PCP within a large, integrated health care system between April 2009 and May 2010. We evaluated performance on 14 measures of DM care, 9 of CHF, 7 of IVD, and 4 of preventive care.

Results: There were 51,771 patients observed by 163 physicians in 17 clinics. Few PCPs (0%-60%) could be profiled with 80% reliability using single process or intermediate-outcome measures. Combining measures into single-disease composites improved reliability for DM and preventive care with 74.5% and 76.7% of PCPs having sufficient panel sizes, but composites remained unreliable for CHF and IVD. A total of 85.3% of PCPs could be reliably profiled using a single overall composite. Aggregating PCPs into practice groups (3 to 21 PCPs per group) did not improve reliability in most cases because of little between-group practice variation.

Conclusions: Single measures rarely differentiate between individual PCPs or groups of PCPs reliably. Combining measures into single-disease or multidisease composites can improve reliability for some common conditions, but not all. Assessing PCP practice groups within a single health care system, rather than individual PCPs, did not substantially improve reliability.

Figure 1. PCP Quality Scores

Reliability-adjusted primary care physician (PCP) quality scores with confidence intervals for single-disease and overall composite measures. Score reflects the expected PCP performance on an average difficulty measure for a 65 year-old female patient with one comorbidity.

Figure 1. PCP Quality Scores

Reliability-adjusted primary care physician (PCP) quality scores with confidence intervals for single-disease and overall composite measures. Score reflects the expected PCP performance on an average difficulty measure for a 65 year-old female patient with one comorbidity.

Figure 1. PCP Quality Scores

Reliability-adjusted primary care physician (PCP) quality scores with confidence intervals for single-disease and overall composite measures. Score reflects the expected PCP performance on an average difficulty measure for a 65 year-old female patient with one comorbidity.