Generalization of Machine Learning Approaches to Identify Notifiable Conditions from a Statewide Health Information Exchange

Gregory P Dexter; Shaun J Grannis; Brian E Dixon; Suranga N Kasthurirathne

Generalization of Machine Learning Approaches to Identify Notifiable Conditions from a Statewide Health Information Exchange

AMIA Jt Summits Transl Sci Proc. 2020 May 30:2020:152-161. eCollection 2020.

Authors

Gregory P Dexter¹, Shaun J Grannis^{1

2}, Brian E Dixon^{1

3}, Suranga N Kasthurirathne^{1

3}

Affiliations

¹ Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, IN, USA.
² Indiana University School of Medicine, Indianapolis, IN, USA.
³ Indiana University Richard M. Fairbanks School of Public Health, Indianapolis, IN, USA.

PMID: 32477634
PMCID: PMC7233074

Abstract

Healthcare analytics is impeded by a lack of machine learning (ML) model generalizability, the ability of a model to predict accurately on varied data sources not included in the model's training dataset. We leveraged free-text laboratory data from a Health Information Exchange network to evaluate ML generalization using Notifiable Condition Detection (NCD) for public health surveillance as a use case. We 1) built ML models for detecting syphilis, salmonella, and histoplasmosis; 2) evaluated generalizability of these models across data from holdout lab systems, and; 3) explored factors that influence weak model generalizability. Models for predicting each disease reported considerable accuracy. However, they demonstrated poor generalizability across data from holdout lab systems being tested. Our evaluation determined that weak generalization was influenced by variant syntactic nature of free-text datasets across each lab system. Results highlight the need for actionable methodology to generalize ML solutions for healthcare analytics.

Keywords: Generalizability; Machine learning; Notifiable condition detection; Public health surveillance.