Spending Carveouts Substantially Improve the Accuracy of Performance Measurement in Shared Savings Arrangements: Findings From Simulation Analysis of Medicaid ACOs

Abstract

Accuracy of spending-based provider performance metrics is limited by random variation and components of spending that are uncontrollable by providers. Such components vary according to the care management focus and operational maturity of each provider group. This study uses data from New Jersey Medicaid accountable care organizations (ACOs) to examine how carving out uncontrollable components of spending affects the accuracy of performance measures in shared savings arrangements. Spending on injury care, custodial care in facilities (CCF), and amounts above $100 000 per patient are used as examples of potentially uncontrollable spending. Data from 7 applicant Medicaid ACOs are used to conduct Monte Carlo simulations examining the effects of carving out each type of uncontrollable spending under the assumption that controllable spending is reduced by 5%. The simulations show that failure to carve out uncontrollable injury care spending adds -3 to +1 percentage points of bias to the measurement of the true average savings rate (ASR) of 5% and can increase mean squared error (MSE) by a factor of up to 3. Failure to carve out uncontrollable CCF spending generates bias ranging from -4 to +9 percentage points and increases MSE by factors of 8 or more. Failure to carve out uncontrollable spending above $100 000 per person generates bias ranging from -5 to +5 percentage points and increases MSE by factors of 13 or more. Compared with the main modeling reported above, sensitivity analyses find even greater distortions in measured performance when uncontrollable spending is not carved out of the ASR calculation.

Keywords: Medicaid; Monte Carlo method; accountable care organization; financial; incentive; reimbursement; risk sharing; shared savings.

Figure 1.

Distribution of the average savings rate with and without a carveout of injury spending. Source. NJ Medicaid Management Information System. Note. Box-whisker plots for ACOs 1 through 7 using data derived from 1000 iterations of the Monte Carlo simulation model described in the text. The model assumes that the average savings rate (ASR) for all spending excluding injury care is 0.05. ACO = accountable care organization.

Figure 2.

Distribution of the average savings rate with and without a carveout of spending for CCF. Source. NJ Medicaid Management Information System. Note. Box-whisker plots for ACOs 1 through 7 using data derived from 1000 iterations of the Monte Carlo simulation model described in the text. The model assumes that the average savings rate (ASR) for all spending excluding CCF is 0.05. ACO = accountable care organization; CCF = custodial care in facilities.

Figure 3.

Distribution of the average savings rate with and without a carveout of individual spending amounts above $100 000. Source. NJ Medicaid Management Information System. Note. Box-whisker plots for ACOs 1 through 7 using data derived from 1000 iterations of the Monte Carlo simulation model described in the text. The model assumes that the average savings rate (ASR) for all spending excluding individual spending amounts above $100 000 is 0.05. ACO = accountable care organization.