Using estimated true safety event rates versus flagged safety event rates: does it change hospital profiling and payment?

Abstract

Objective: To assess whether use of the AHRQ Patient Safety Indicator (PSI) composite measure versus modified composite measures leads to changes in hospital profiles and payments.

Data sources/study setting: Retrospective analysis of 2010 Veterans Health Administration discharge data.

Study design: We used the AHRQ PSI software (v4.2) to obtain PSI-flagged events and composite scores for all 151 hospitals in the database (n = 517,814 hospitalizations). We compared the AHRQ PSI composite to two modified composites that estimated "true safety events" from previous chart abstraction findings: one with modified numerators based on the positive predictive value (PPV) of each PSI, and one with similarly modified numerators but whose denominators were based on the expected fraction of PSI-eligible cases that remained after removing those PSIs that were present-on-admission (POA).

Principal findings: Although a small percentage (5-6 percent) of hospitals changed outlier status based on modified PSI composites, some of these changes were substantial; 30 and 19 percent of hospitals changed ≥20 ranks after adjustment for PPVs and POA flags, respectively. We estimate that 33 percent of hospitals would see a change of at least 10 percent in performance payments.

Conclusions: Changes in hospital profiles and payments would be substantial for some hospitals if the PSI composite score used weights reflecting the relative prevalence of true versus flagged events.

Keywords: AHRQ Patient Safety Indicators (PSIs); Patient safety measurement; hospital profiling; quality of care.

Figure 1A

Comparison of Hospital Ranks Based on the Original AHRQ Patient Safety Composite Score§ and the VA-PPV-Adjusted PSI Composite Score** Note. §Original numerator weights of individual PSIs built in the PSI software version 4.2. (AHRQ quality indicators); **VA-PPV-Adjusted numerator weight for PSI_i = (original weight for PSI_i × PPV_i)/∑ (original weight for PSI_i × PPV_i). This simulates the current VA scenario, in that the VA has not yet incorporated POA flags into administrative data. Hospitals are arrayed on the horizontal axis from best performer (rank 1) to worst performer using the original PSI composite score.

Figure 1B

Comparison of Hospital Ranks Based on the Original AHRQ Patient Safety Composite§ and the CMS-PPV-Adjusted PSI Composite Excluding POA §§ Note. §Original numerator weights of individual PSIs built in the PSI software version 4.2. (AHRQ quality indicators); §§CMS-PPV-Adjusted excluding POA numerator weight for PSI_i = (original weight for PSI_i × PPV excluding POA_i)/∑ (original weight for PSI_i × PPV excluding POA_i). This simulates the current CMS scenario, because CMS has incorporated POA flags into Medicare claims data. Hospitals are arrayed on the horizontal axis from best performer (rank 1) to worst performer using the original PSI composite score.

Figure 2A

Comparison of the Percentage of Payment Pool Based on the Original AHRQ Patient Safety Composite§ and the VA-PPV-Adjusted PSI Composite** Note. §Original numerator weights of individual PSIs built in the PSI software version 4.2. (AHRQ quality indicators); **VA-PPV-Adjusted numerator weight for PSI_i = (original weight for PSI_i × PPV_i)/∑ (original weight for PSI_i × PPV_i). This simulates the current VA scenario, in that the VA has not yet incorporated POA flags into administrative data. Hospitals are arrayed on the horizontal axis from best performer (rank 1) to worst performer using the original PSI composite score.

Figure 2B

Comparison of the Percentage of Payment Pool Based on the Original AHRQ Patient Safety Composite§ and the CMS-PPV-Adjusted PSI Composite Excluding POA§§ Note. §Original numerator weights of individual PSIs built in the PSI software version 4.2. (AHRQ quality indicators); §§CMS-PPV-Adjusted excluding POA numerator weight for PSI_i = (original weight for PSI_i × PPV excluding POA_i)/∑ (original weight for PSI_i × PPV excluding POA_i). This simulates the current CMS scenario, because CMS has incorporated POA flags into Medicare claims data. Hospitals are arrayed on the horizontal axis from best performer (rank 1) to worst performer using the original PSI composite score.