The Hospital Compare mortality model and the volume-outcome relationship

Abstract

Objective: We ask whether Medicare's Hospital Compare random effects model correctly assesses acute myocardial infarction (AMI) hospital mortality rates when there is a volume-outcome relationship.

Data sources/study setting: Medicare claims on 208,157 AMI patients admitted in 3,629 acute care hospitals throughout the United States.

Study design: We compared average-adjusted mortality using logistic regression with average adjusted mortality based on the Hospital Compare random effects model. We then fit random effects models with the same patient variables as in Medicare's Hospital Compare mortality model but also included terms for hospital Medicare AMI volume and another model that additionally included other hospital characteristics.

Principal findings: Hospital Compare's average adjusted mortality significantly underestimates average observed death rates in small volume hospitals. Placing hospital volume in the Hospital Compare model significantly improved predictions.

Conclusions: The Hospital Compare random effects model underestimates the typically poorer performance of low-volume hospitals. Placing hospital volume in the Hospital Compare model, and possibly other important hospital characteristics, appears indicated when using a random effects model to predict outcomes. Care must be taken to insure the proper method of reporting such models, especially if hospital characteristics are included in the random effects model.

Figure 1

Example of How Hospital Compare Shrinks Predictions Based on Acute Myocardial Infarction Volume Notes. This figure compares the 166 hospitals with between 11 and 13 patients a year (1-a-month) and the 86 hospitals with at least 250 patients a year (on the order of 1-a-day). The three pictures on the left are for 1-a-month. The three pictures on the right are for 250-a-year. O/E is on the outside. P/E is on the inside. O/E and P/E appear worse for 1-a-month than for 250-a-year. Medicare reports to the public the shrunken P/E rate times the national rate, not the O/E rate. For the higher volume hospitals, with 250 cases per year, we observe far less shrinkage, and the O/E and P/E ratios both appear below 1. For the 1-a-month hospitals, we observe a median O/E>1 but a P/E very near to 1.

Figure 2

How Much Does the Hospital Compare Model Emphasize an Individual Hospital's Observed Mortality Rate When Calculating the Predicted Mortality That Is Provided to the Public? Notes. The figure provides the value of λ where P/E=λ (O/E)+(1−λ)(E/E). Because Hospital Compare does not include volume or any hospital characteristics in the model, the (1−λ) term is multiplied by 1, which represents the national average or typical hospital death rate. As hospital acute myocardial infarction (AMI) volume increases, λ increases, suggesting an increasing emphasis on the hospitals own O/E. As hospital AMI volume decreases, λ decreases, and the Hospital Compare model emphasizes the national mortality rate (E/E) rather than the hospitals own observed mortality rate (O/E) when making the prediction P/E.

Figure 3

Comparison of Standard Logistic Regression Observed/Expected Mortality Ratios versus Three Versions of the Hospital Compare Random Effects Model Predicted/Expected Mortality Ratios Notes. Both O/E and P/E would be multiplied by the national acute myocardial infarction mortality rate to get an adjusted rate which could be reported to the public. Note that for the lower two quintiles of hospitals by volume (the smallest 40 percent of hospitals), there is a great discrepancy between the average O/E mortality rate ratios based on the standard O/E logit model (thin black line) and the average P/E mortality rate ratios based on the random effects model used by Medicare's Hospital Compare (the dashed line). However, when size (the thick black line) or size and other hospital characteristics (the thick gray line) are added to the present Hospital Compare model, the P/E values become almost identical to the standard O/E results.