A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission

Velibor V Mišić; Kumar Rajaram; Eilon Gabel

doi:10.1038/s41746-021-00468-7

A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission

NPJ Digit Med. 2021 Jun 14;4(1):98. doi: 10.1038/s41746-021-00468-7.

Authors

Velibor V Mišić¹, Kumar Rajaram², Eilon Gabel³

Affiliations

¹ Decisions, Operations and Technology Management, Anderson School of Management, University of California Los Angeles, Los Angeles, CA, USA. velibor.misic@anderson.ucla.edu.
² Decisions, Operations and Technology Management, Anderson School of Management, University of California Los Angeles, Los Angeles, CA, USA.
³ Department of Anesthesiology and Perioperative Medicine, University of California Los Angeles, Los Angeles, CA, USA.

Abstract

The interest in applying machine learning in healthcare has grown rapidly in recent years. Most predictive algorithms requiring pathway implementations are evaluated using metrics focused on predictive performance, such as the c statistic. However, these metrics are of limited clinical value, for two reasons: (1) they do not account for the algorithm's role within a provider workflow; and (2) they do not quantify the algorithm's value in terms of patient outcomes and cost savings. We propose a model for simulating the selection of patients over time by a clinician using a machine learning algorithm, and quantifying the expected patient outcomes and cost savings. Using data on unplanned emergency department surgical readmissions, we show that factors such as the provider's schedule and postoperative prediction timing can have major effects on the pathway cohort size and potential cost reductions from preventing hospital readmissions.