Risk adjustment for comparing hospital quality with surgery: how many variables are needed?

Abstract

Background: The American College of Surgeons' National Surgical Quality Improvement Program (ACS NSQIP) will soon be reporting procedure-specific outcomes, and hopes to reduce the burden of data collection by collecting fewer variables. We sought to determine whether these changes threaten the robustness of the risk adjustment of hospital quality comparisons.

Study design: We used prospective, clinical data from the ACS NSQIP from 2005 to 2007 (184 hospitals, 74,887 patients). For the 5 general surgery operations in the procedure-specific NSQIP, we compared the ability of the full model (21 variables), an intermediate model (12 variables), and a limited model (5 variables) to predict patient outcomes and to risk-adjust hospital outcomes.

Results: The intermediate and limited models were comparable with the full model in all analyses. In the assessment of patient risk, the limited and full models had very similar discrimination at the patient level (C-indices for all 5 procedures combined of 0.93 versus 0.91 for mortality and 0.78 versus 0.76 for morbidity) and showed good calibration across strata of patient risk. In assessing hospital-specific outcomes, results from the limited and full-risk models were highly correlated for both mortality (range 0.94 to 0.99 across the 5 operations) and morbidity (range 0.96 to 0.99).

Conclusions: Procedure-specific hospital quality measures can be adequately risk-adjusted with a limited number of variables. In the context of the ACS NSQIP, moving to a more limited model will dramatically reduce the burden of data collection for participating hospitals.

Figure 1

Comparison ofhospital ratio of observed to expe cted outcomes (O/E) ratios created using the intermediate versus full model for (A) mortality and (B) morbidity for all 5 procedures combined.

Figure 2

Comparison ofhospital observed to ex pected outcomes (O/E) ratios created using the limited versus full model for (A) mortality and (B) morbidity for all 5 procedures combined.