Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning

Robert Y Lee; Lyndia C Brumback; William B Lober; James Sibley; Elizabeth L Nielsen; Patsy D Treece; Erin K Kross; Elizabeth T Loggers; James A Fausto; Charlotta Lindvall; Ruth A Engelberg; J Randall Curtis

doi:10.1016/j.jpainsymman.2020.08.024

Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning

J Pain Symptom Manage. 2021 Jan;61(1):136-142.e2. doi: 10.1016/j.jpainsymman.2020.08.024. Epub 2020 Aug 25.

Affiliations

¹ Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Division of Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine, Harborview Medical Center, University of Washington, Seattle, Washington, USA.
² Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Department of Biostatistics, University of Washington, Seattle, Washington, USA.
³ Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, Washington, USA; Department of Bioinformatics and Medical Education, University of Washington, Seattle, Washington, USA.
⁴ Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Division of Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine, Harborview Medical Center, University of Washington, Seattle, Washington, USA; Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, Washington, USA.
⁵ Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Clinical Research Division, Fred Hutchinson Cancer Research Center, Seattle, Washington, USA; Seattle Cancer Care Alliance, Seattle, Washington, USA.
⁶ Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Department of Family Medicine, University of Washington, Seattle, Washington, USA.
⁷ Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA.
⁸ Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Division of Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine, Harborview Medical Center, University of Washington, Seattle, Washington, USA; Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, Washington, USA; Department of Bioethics and Humanities, University of Washington, Seattle, Washington, USA. Electronic address: jrc@u.washington.edu.

Abstract

Context: Goals-of-care discussions are an important quality metric in palliative care. However, goals-of-care discussions are often documented as free text in diverse locations. It is difficult to identify these discussions in the electronic health record (EHR) efficiently.

Objectives: To develop, train, and test an automated approach to identifying goals-of-care discussions in the EHR, using natural language processing (NLP) and machine learning (ML).

Methods: From the electronic health records of an academic health system, we collected a purposive sample of 3183 EHR notes (1435 inpatient notes and 1748 outpatient notes) from 1426 patients with serious illness over 2008-2016, and manually reviewed each note for documentation of goals-of-care discussions. Separately, we developed a program to identify notes containing documentation of goals-of-care discussions using NLP and supervised ML. We estimated the performance characteristics of the NLP/ML program across 100 pairs of randomly partitioned training and test sets. We repeated these methods for inpatient-only and outpatient-only subsets.

Results: Of 3183 notes, 689 contained documentation of goals-of-care discussions. The mean sensitivity of the NLP/ML program was 82.3% (SD 3.2%), and the mean specificity was 97.4% (SD 0.7%). NLP/ML results had a median positive likelihood ratio of 32.2 (IQR 27.5-39.2) and a median negative likelihood ratio of 0.18 (IQR 0.16-0.20). Performance was better in inpatient-only samples than outpatient-only samples.

Conclusion: Using NLP and ML techniques, we developed a novel approach to identifying goals-of-care discussions in the EHR. NLP and ML represent a potential approach toward measuring goals-of-care discussions as a research outcome and quality metric.

Keywords: Natural language processing; electronic health record; goals of care; machine learning; medical informatics; quality improvement.

Publication types

Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't

MeSH terms

Electronic Health Records*
Humans
Machine Learning
Natural Language Processing*
Palliative Care
Patient Care Planning

Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning

Authors

Affiliations

Abstract

Publication types

MeSH terms

Grants and funding