Reliability of surgical outcomes for predicting future hospital performance

Abstract

Background: Because of small sample sizes and low event rates, risk-adjusted surgical outcomes often do not meet reliability benchmarks for distinguishing hospital performance. Nonetheless, it is unclear whether these measures may still be useful for predicting future hospital surgical performance.

Methods: We used national Medicare data to analyze patients undergoing colectomy from 2007 to 2010 (n=462,959 patients). We first quantified 2007-2008 outcome reliability (ability to differentiate quality differences) and ranked hospitals based on their 2007-2008 risk-adjusted outcome rates. To assess the ability of adjusted outcomes to predict true performance, we evaluated future (2009-2010) outcomes across quintiles of past performance. We then systematically sampled 2007-2008 cases to evaluate performance prediction when hospitals' past performance was measured with progressively lower reliability levels.

Results: Outcomes in 2007-2008 were good predictors of outcomes in the next 2 years (2009-2010), but predictive strength depended upon reliability. With progressive sampling of 2007-2008 caseloads, outcome reliability and predictive strength decreased. With 100% sampling of 2007-2008 caseloads, the worst versus best hospital quintile based on past performance had 1.52 [95% confidence interval (CI), 1.44-1.60] times the odds of mortality and 1.50 (95% CI, 1.44-1.56) times the odds of complications in 2009-2010. With 10% sampling, outcome reliability was well below commonly accepted benchmarks, but the worst quintile of hospitals in 2007-2008 still had 1.12 (95% CI, 1.06-1.19) times the odds of mortality and 1.16 (95% CI, 1.11-1.21) times the odds of complications in 2009-2010 compared with the best quintile of hospitals.

Conclusions: Even at very low reliability levels, risk-adjusted outcome measures may distinguish best and worst hospitals' surgical performance. This study suggests that commonly accepted reliability thresholds may be too high, especially in the context of selective referral.

Figure 1

Risk-adjusted outcome rates in 2009-10 across quintiles of past hospital performance derived from different hospital caseload samples.

Figure 1

Risk-adjusted outcome rates in 2009-10 across quintiles of past hospital performance derived from different hospital caseload samples.

Figure 2

Proportional declines in outcome mean square root reliability (x-axis) and top-bottom quintile discrimination in adjusted 2009-10 outcome rates (y-axis) at different levels of caseload sampling. Proportional changes depicted to account for different starting reliability levels for each outcome*. Each point represents the change in outcome mean square root reliability and quintile discrimination at a given caseload sample size relative to 100% sampling. The connecting lines represent the particular outcomes from which the points were derived.