Machine learning-assisted screening for cognitive impairment in the emergency department

doi:10.1111/jgs.17491

Randomized Controlled Trial

. 2022 Mar;70(3):831-837.

doi: 10.1111/jgs.17491. Epub 2021 Oct 13.

Machine learning-assisted screening for cognitive impairment in the emergency department

Simon R Yadgir¹, Collin Engstrom^{1

2}, Gwen Costa Jacobsohn¹, Rebecca K Green¹, Courtney M C Jones^{3

4}, Jeremy T Cushman^{3

4

5}, Thomas V Caprio⁶, Amy J H Kind^{7

8

9}, Michael Lohmeier¹, Manish N Shah^{1

7

10}, Brian W Patterson^{1

11

12}

Affiliations

¹ BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, USA.
² Department of Computer Science, Winona State University, Rochester, MN, USA.
³ Department of Emergency Medicine, University of Rochester Medical Center, Rochester, New York, USA.
⁴ Department of Public Health Sciences, University of Rochester Medical Center, Rochester, New York, USA.
⁵ Department of Health Policy and Management, School of Public Health, University of Maryland, College Park, Maryland, USA.
⁶ Division of Geriatrics, Department of Medicine, University of Rochester Medical Center, Rochester, New York, USA.
⁷ Division of Geriatrics and Gerontology, Department of Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA.
⁸ William S. Middleton Veterans Affairs Geriatrics Research, Education, and Clinical Center, Madison, Wisconsin, USA.
⁹ UW Center for Health Disparities Research, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin, USA.
¹⁰ Department of Population Health Sciences, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin, USA.
¹¹ Health Innovation Program, University of Wisconsin-Madison, Madison, Wisconsin, USA.
¹² Department of Industrial and Systems Engineering, Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, Wisconsin, USA.

PMID: 34643944
PMCID: PMC8904269
DOI: 10.1111/jgs.17491

Randomized Controlled Trial

Machine learning-assisted screening for cognitive impairment in the emergency department

Simon R Yadgir et al. J Am Geriatr Soc. 2022 Mar.

. 2022 Mar;70(3):831-837.

doi: 10.1111/jgs.17491. Epub 2021 Oct 13.

Authors

Affiliations

¹ BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, USA.
² Department of Computer Science, Winona State University, Rochester, MN, USA.
³ Department of Emergency Medicine, University of Rochester Medical Center, Rochester, New York, USA.
⁴ Department of Public Health Sciences, University of Rochester Medical Center, Rochester, New York, USA.
⁵ Department of Health Policy and Management, School of Public Health, University of Maryland, College Park, Maryland, USA.
⁶ Division of Geriatrics, Department of Medicine, University of Rochester Medical Center, Rochester, New York, USA.
⁷ Division of Geriatrics and Gerontology, Department of Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA.
⁸ William S. Middleton Veterans Affairs Geriatrics Research, Education, and Clinical Center, Madison, Wisconsin, USA.
⁹ UW Center for Health Disparities Research, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin, USA.
¹⁰ Department of Population Health Sciences, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin, USA.
¹¹ Health Innovation Program, University of Wisconsin-Madison, Madison, Wisconsin, USA.
¹² Department of Industrial and Systems Engineering, Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, Wisconsin, USA.

PMID: 34643944
PMCID: PMC8904269
DOI: 10.1111/jgs.17491

Abstract

Background/objectives: Despite a high prevalence and association with poor outcomes, screening to identify cognitive impairment (CI) in the emergency department (ED) is uncommon. Identification of high-risk subsets of older adults is a critical challenge to expanding screening programs. We developed and evaluated an automated screening tool to identify a subset of patients at high risk for CI.

Methods: In this secondary analysis of existing data collected for a randomized control trial, we developed machine-learning models to identify patients at higher risk of CI using only variables available in electronic health record (EHR). We used records from 1736 community-dwelling adults age > 59 being discharged from three EDs. Potential CI was determined based on the Blessed Orientation Memory Concentration (BOMC) test, administered in the ED. A nested cross-validation framework was used to evaluate machine-learning algorithms, comparing area under the receiver-operator curve (AUC) as the primary metric of performance.

Results: Based on BOMC scores, 121 of 1736 (7%) participants screened positive for potential CI at the time of their ED visit. The best performing algorithm, an XGBoost model, predicted BOMC positivity with an AUC of 0.72. With a classification threshold of 0.4, this model had a sensitivity of 0.73, a specificity of 0.64, a negative predictive value of 0.97, and a positive predictive value of 0.13. In a hypothetical ED with 200 older adult visits per week, the use of this model would lead to a decrease in the in-person screening burden from 200 to 77 individuals in order to detect 10 of 14 patients who would fail a BOMC.

Conclusion: This study demonstrates that an algorithm based on EHR data can define a subset of patients at higher risk for CI. Incorporating such an algorithm into a screening workflow could allow screening efforts and resources to be focused where they have the most impact.

Keywords: cognitive impairment; delirium; dementia; emergency medicine; machine learning.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest

The authors have no conflicts.

Figures

**Figure 1.**
Model training framework flow diagram. AUC: Area under the curve.

See this image and copyright information in PMC

Comment in

Automating cognitive impairment screening in emergency departments: A small step forward.
Hirshon JM. Hirshon JM. J Am Geriatr Soc. 2022 Mar;70(3):695-697. doi: 10.1111/jgs.17654. Epub 2022 Jan 18. J Am Geriatr Soc. 2022. PMID: 35043418 No abstract available.

Cited by

Hematological Malignancies in Older Patients: Focus on the Potential Role of a Geriatric Assessment Management.
Caserta S, Cancemi G, Loreta S, Allegra A, Stagno F. Caserta S, et al. Diagnostics (Basel). 2024 Jun 29;14(13):1390. doi: 10.3390/diagnostics14131390. Diagnostics (Basel). 2024. PMID: 39001280 Free PMC article. Review.
Community screening for dementia among older adults in China: a machine learning-based strategy.
Zhang Y, Xu J, Zhang C, Zhang X, Yuan X, Ni W, Zhang H, Zheng Y, Zhao Z. Zhang Y, et al. BMC Public Health. 2024 May 1;24(1):1206. doi: 10.1186/s12889-024-18692-7. BMC Public Health. 2024. PMID: 38693495 Free PMC article.
Broad Trends in Public Health and Epidemiology Informatics.
Diallo G, Bordea G, Samieri C; Section Editors for the IMIA Yearbook Section on Public Health and Epidemiology Informatics. Diallo G, et al. Yearb Med Inform. 2023 Aug;32(1):264-268. doi: 10.1055/s-0043-1768754. Epub 2023 Dec 26. Yearb Med Inform. 2023. PMID: 38147868 Free PMC article.
Automating risk stratification for geriatric syndromes in the emergency department.
Haimovich AD, Shah MN, Southerland LT, Hwang U, Patterson BW. Haimovich AD, et al. J Am Geriatr Soc. 2024 Jan;72(1):258-267. doi: 10.1111/jgs.18594. Epub 2023 Oct 9. J Am Geriatr Soc. 2024. PMID: 37811698
Self-supervised machine learning using adult inpatient data produces effective models for pediatric clinical prediction tasks.
Lemmon J, Guo LL, Steinberg E, Morse KE, Fleming SL, Aftandilian C, Pfohl SR, Posada JD, Shah N, Fries J, Sung L. Lemmon J, et al. J Am Med Inform Assoc. 2023 Nov 17;30(12):2004-2011. doi: 10.1093/jamia/ocad175. J Am Med Inform Assoc. 2023. PMID: 37639620 Free PMC article.

See all "Cited by" articles

References

1. LaMantia M, Stump TE, Messina FC, Miller DK,Callahan CM. Emergency Department Use Among Older Adults With Dementia. Alzheimer Dis Assoc Disord 2016;30(1):35–40. DOI: 10.1097/WAD.0000000000000118. - DOI - PMC - PubMed
1. Han JH, Bryce SN, Ely EW et al. The effect of cognitive impairment on the accuracy of the presenting complaint and discharge instruction comprehension in older emergency department patients. Ann Emerg Med 2011;57(6):662–671.e2. DOI: 10.1016/j.annemergmed.2010.12.002. - DOI - PMC - PubMed
1. Hustey FM, Meldon SW. The Prevalence and Documentation of Impaired Mental Status in Elderly Emergency Department Patients. Ann Emerg Med 2002;39(3):248–253. DOI: 10.1067/mem.2002.122057. - DOI - PubMed
1. Han JH, Suyama J. Delirium and Dementia. Clin Geriatr Med 2018;34(3):327–354. DOI: 10.1016/j.cger.2018.05.001. - DOI - PubMed
1. Kakuma R, Galbaud du Fort G, Arsenault L et al. Delirium in Older Emergency Department Patients Discharged Home: Effect on Survival. J Am Geriatr Soc 2003;51(4):443–450. DOI: 10.1046/j.1532-5415.2003.51151.x. - DOI - PubMed

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Medical
- MedlinePlus Health Information

[1] LaMantia M, Stump TE, Messina FC, Miller DK,Callahan CM. Emergency Department Use Among Older Adults With Dementia. Alzheimer Dis Assoc Disord 2016;30(1):35–40. DOI: 10.1097/WAD.0000000000000118. - DOI - PMC - PubMed

[2] LaMantia M, Stump TE, Messina FC, Miller DK,Callahan CM. Emergency Department Use Among Older Adults With Dementia. Alzheimer Dis Assoc Disord 2016;30(1):35–40. DOI: 10.1097/WAD.0000000000000118. - DOI - PMC - PubMed

[3] Han JH, Bryce SN, Ely EW et al. The effect of cognitive impairment on the accuracy of the presenting complaint and discharge instruction comprehension in older emergency department patients. Ann Emerg Med 2011;57(6):662–671.e2. DOI: 10.1016/j.annemergmed.2010.12.002. - DOI - PMC - PubMed

[4] Han JH, Bryce SN, Ely EW et al. The effect of cognitive impairment on the accuracy of the presenting complaint and discharge instruction comprehension in older emergency department patients. Ann Emerg Med 2011;57(6):662–671.e2. DOI: 10.1016/j.annemergmed.2010.12.002. - DOI - PMC - PubMed

[5] Hustey FM, Meldon SW. The Prevalence and Documentation of Impaired Mental Status in Elderly Emergency Department Patients. Ann Emerg Med 2002;39(3):248–253. DOI: 10.1067/mem.2002.122057. - DOI - PubMed

[6] Hustey FM, Meldon SW. The Prevalence and Documentation of Impaired Mental Status in Elderly Emergency Department Patients. Ann Emerg Med 2002;39(3):248–253. DOI: 10.1067/mem.2002.122057. - DOI - PubMed

[7] Han JH, Suyama J. Delirium and Dementia. Clin Geriatr Med 2018;34(3):327–354. DOI: 10.1016/j.cger.2018.05.001. - DOI - PubMed

[8] Han JH, Suyama J. Delirium and Dementia. Clin Geriatr Med 2018;34(3):327–354. DOI: 10.1016/j.cger.2018.05.001. - DOI - PubMed

[9] Kakuma R, Galbaud du Fort G, Arsenault L et al. Delirium in Older Emergency Department Patients Discharged Home: Effect on Survival. J Am Geriatr Soc 2003;51(4):443–450. DOI: 10.1046/j.1532-5415.2003.51151.x. - DOI - PubMed

[10] Kakuma R, Galbaud du Fort G, Arsenault L et al. Delirium in Older Emergency Department Patients Discharged Home: Effect on Survival. J Am Geriatr Soc 2003;51(4):443–450. DOI: 10.1046/j.1532-5415.2003.51151.x. - DOI - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Machine learning-assisted screening for cognitive impairment in the emergency department

Affiliations

Machine learning-assisted screening for cognitive impairment in the emergency department

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Comment in

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Medical

Abstract

Conflict of interest statement

Figures

Comment in

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Medical