APLUS: A Python library for usefulness simulations of machine learning models in healthcare

J Biomed Inform. 2023 Mar:139:104319. doi: 10.1016/j.jbi.2023.104319. Epub 2023 Feb 13.


Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.

Keywords: Clinical workflows; Discrete-event simulation; Machine learning; Model deployment; Usefulness assessment; Utility.

Publication types

  • Research Support, U.S. Gov't, Non-P.H.S.
  • Research Support, Non-U.S. Gov't

MeSH terms

  • Computer Simulation
  • Delivery of Health Care*
  • Humans
  • Language
  • Machine Learning*
  • Workflow