Ranking USRDS provider specific SMRs from 1998-2001

Abstract

Provider profiling (ranking/percentiling) is prevalent in health services research. Bayesian models coupled with optimizing a loss function provide an effective framework for computing non-standard inferences such as ranks. Inferences depend on the posterior distribution and should be guided by inferential goals. However, even optimal methods might not lead to definitive results and ranks should be accompanied by valid uncertainty assessments. We outline the Bayesian approach and use estimated Standardized Mortality Ratios (SMRs) in 1998-2001 from the United States Renal Data System (USRDS) as a platform to identify issues and demonstrate approaches. Our analyses extend Liu et al. (2004) by computing estimates developed by Lin et al. (2006) that minimize errors in classifying providers above or below a percentile cut-point, by combining evidence over multiple years via a first-order, autoregressive model on log(SMR), and by use of a nonparametric prior. Results show that ranks/percentiles based on maximum likelihood estimates of the SMRs and those based on testing whether an SMR = 1 substantially under-perform the optimal estimates. Combining evidence over the four years using the autoregressive model reduces uncertainty, improving performance over percentiles based on only one year. Furthermore, percentiles based on posterior probabilities of exceeding a properly chosen SMR threshold are essentially identical to those produced by minimizing classification loss. Uncertainty measures effectively calibrate performance, showing that considerable uncertainty remains even when using optimal methods. Findings highlight the importance of using loss function guided percentiles and the necessity of accompanying estimates with uncertainty assessments.

Fig. 1

π_k (0.8|Y) versus ${\tilde{P}}_k\left(0.8\right)$ for 1998. Optimal percentiles and posterior probabilities computed with the single year model (ϕ ≡ 0) and the AR(1) model (ϕ = 0.90) Two curves don't cross at γ = 0.8. The line for fully informative data, i.e., when there is no uncertainty associated with ranking results is given as reference

Fig. 2

Comparison of 1998 $\tilde{P}\left(0.8\right)$ with NPML and Gaussian prior. Circles represent 40 dialysis centers evenly spread across percentiles estimated with NPML prior. The percentiles of the same center are connected

Fig. 3

π_k(0.8) versus estimated percentiles by three ranking methods using the 1998 data: ${\tilde{P}}_k\left(\gamma \right)$, MLE-based and Z-score-based. For small dialysis centers (fewer than 5 patients in 1998), the symbol “-” represents the MLE-based percentiles, the symbol “1” the Z-score-based percentiles and the symbol “ ^ ” the ${\tilde{P}}_k\left(\gamma \right)$

Fig. 4

SEL-based percentiles for 1998. For each display, the Y-axis is 100 × ${\stackrel{‒}{P}}_k$ with its 95% probability interval. The X-axis for the upper left panel is $\widehat{P}$, for the upper right is percentiles based on ρ^pm, for the lower left is percentiles based on the ρ^mle and for the lower right is percentiles based on Z-scores testing ρ_k = 1

Fig. 5

Estimated priors for θ = log(ρ) using the 1998 data. The solid curve is a smoothed NPML using the “density” function in R with adjustment parameter = 10. The dashed curve is Gaussian using posterior medians for (μ, τ); the dotted curve is a mixture of Gaussians with (μ, τ) sampled from their MCMC computed joint posterior distribution