Identifying low acuity Emergency Department visits with a machine learning approach: The low acuity visit algorithms (LAVA)

Objective: To improve the performance of International Classification of Disease (ICD) code rule-based algorithms for identifying low acuity Emergency Department (ED) visits by using machine learning methods and additional covariates.

Data sources: We used secondary data on ED visits from the National Hospital Ambulatory Medical Survey (NHAMCS), from 2016 to 2020.

Study design: We established baseline performance metrics with seven published algorithms consisting of International Classification of Disease, Tenth Revision codes used to identify low acuity ED visits. We then trained logistic regression, random forest, and gradient boosting (XGBoost) models to predict low acuity ED visits. Each model was trained on five different covariate sets of demographic and clinical data. Model performance was compared using a separate validation dataset. The primary performance metric was the probability that a visit identified by an algorithm as low acuity did not experience significant testing, treatment, or disposition (positive predictive value, PPV). Subgroup analyses assessed model performance across age, sex, and race/ethnicity.

Data collection: We used 2016-2019 NHAMCS data as the training set and 2020 NHAMCS data for validation.

Principal findings: The training and validation data consisted of 53,074 and 9542 observations, respectively. Among seven rule-based algorithms, the highest-performing had a PPV of 0.35 (95% CI [0.33, 0.36]). All model-based algorithms outperformed existing algorithms, with the least effective-random forest using only age and sex-improving PPV by 26% (up to 0.44; 95% CI [0.40, 0.48]). Logistic regression and XGBoost trained on all variables improved PPV by 83% (to 0.64; 95% CI [0.62, 0.66]). Multivariable models also demonstrated higher PPV across all three demographic subgroups.

Conclusions: Machine learning models substantially outperform existing algorithms based on ICD codes in predicting low acuity ED visits. Variations in model performance across demographic groups highlight the need for further research to ensure their applicability and fairness across diverse populations.