Statistical benchmarks for health care provider performance assessment: a comparison of standard approaches to a hierarchical Bayesian histogram-based method

Abstract

Objective: Examine how widely used statistical benchmarks of health care provider performance compare with histogram-based statistical benchmarks obtained via hierarchical Bayesian modeling.

Data sources: Publicly available data from 3,240 hospitals during April 2009-March 2010 on two process-of-care measures reported on the Medicare Hospital Compare website.

Study design: Secondary data analyses of two process-of-care measures comparing statistical benchmark estimates and threshold exceedance determinations under various combinations of hospital performance measure estimates and benchmarking approaches.

Principal findings: Statistical benchmarking approaches for determining top 10 percent performance varied with respect to which hospitals exceeded the performance benchmark; such differences were not found at the 50 percent threshold. Benchmarks derived from the histogram of provider performance under hierarchical Bayesian modeling provide a compromise between benchmarks based on direct (raw) estimates, which are overdispersed relative to the true distribution of provider performance and prone to high variance for small providers, and posterior mean provider performance, for which over-shrinkage and under-dispersion relative to the true provider performance distribution is a concern.

Conclusions: Given the rewards and penalties associated with characterizing top performance, the ability of statistical benchmarks to summarize key features of the provider performance distribution should be examined.

Keywords: Bayesian statistics; hierarchical model; provider profiling; public reporting; statistical benchmark.

Figure 1

Illustration Using Simulated Data of the True Distribution of Provider-Specific Performance (solid line) versus Distribution (gray) Based on (a) Posterior Means, (b) Raw Estimates, and (c) Histogram Method