Does voluntary reporting bias hospital quality rankings?

J Surg Res. 2010 Jun 15;161(2):190-4. doi: 10.1016/j.jss.2009.07.033. Epub 2009 Aug 22.


Background: The Leapfrog Group, a large consortium of private payers, collects voluntary, self-reported quality information from hospitals for several different procedures. Critics argue that voluntary reporting leads to over-representation of high quality hospitals.

Materials and methods: We used data from 1169 hospitals participating in the 2008 Leapfrog Survey for six high-risk procedures. We first ranked the hospitals who voluntarily submitted data to Leapfrog using a composite quality measure (a combination of mortality and hospital volume). We then created a similar set of rankings for a national sample of hospitals from the 2006 Nationwide Inpatient Sample (NIS). Next, we created four groups (quartiles) of hospital performance based on the national sample of hospitals. Finally, we compared the Leapfrog hospitals with the national sample of hospitals. If voluntary reporting resulted in bias, we would expect the Leapfrog hospitals to cluster in the top quartiles for each procedure.

Results: The distribution of the Leapfrog hospitals within the NIS performance quartiles varied extensively across procedures. For abdominal aortic aneurysm repair and coronary artery bypass grafting, there was a clustering of hospitals in the two worst quartiles (64% and 65%, respectively). Conversely, for aortic valve replacement, pancreatectomy, percutaneous coronary interventions, and esophagectomy, there were 52%, 56%, 57%, and 61% of hospitals in the top half, respectively.

Conclusions: In the Leapfrog Survey, voluntary reporting did not lead to a systematic over-representation of high quality hospitals. There should be little concern that hospital quality rankings would have significant bias introduced by voluntary participation of hospitals.

Publication types

  • Research Support, N.I.H., Extramural

MeSH terms

  • Bias
  • Coronary Artery Bypass / mortality
  • Coronary Artery Bypass / statistics & numerical data
  • Data Collection*
  • Esophagectomy / mortality
  • Esophagectomy / statistics & numerical data
  • Heart Valve Prosthesis Implantation / mortality
  • Heart Valve Prosthesis Implantation / statistics & numerical data
  • Hospital Mortality*
  • Hospitals / standards*
  • Humans
  • Inpatients*
  • Outcome Assessment, Health Care*
  • Pancreatectomy / mortality
  • Pancreatectomy / statistics & numerical data
  • Research Design
  • United States
  • Voluntary Programs*