Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning

J Pain Symptom Manage. 2021 Jan;61(1):136-142.e2. doi: 10.1016/j.jpainsymman.2020.08.024. Epub 2020 Aug 25.


Context: Goals-of-care discussions are an important quality metric in palliative care. However, goals-of-care discussions are often documented as free text in diverse locations. It is difficult to identify these discussions in the electronic health record (EHR) efficiently.

Objectives: To develop, train, and test an automated approach to identifying goals-of-care discussions in the EHR, using natural language processing (NLP) and machine learning (ML).

Methods: From the electronic health records of an academic health system, we collected a purposive sample of 3183 EHR notes (1435 inpatient notes and 1748 outpatient notes) from 1426 patients with serious illness over 2008-2016, and manually reviewed each note for documentation of goals-of-care discussions. Separately, we developed a program to identify notes containing documentation of goals-of-care discussions using NLP and supervised ML. We estimated the performance characteristics of the NLP/ML program across 100 pairs of randomly partitioned training and test sets. We repeated these methods for inpatient-only and outpatient-only subsets.

Results: Of 3183 notes, 689 contained documentation of goals-of-care discussions. The mean sensitivity of the NLP/ML program was 82.3% (SD 3.2%), and the mean specificity was 97.4% (SD 0.7%). NLP/ML results had a median positive likelihood ratio of 32.2 (IQR 27.5-39.2) and a median negative likelihood ratio of 0.18 (IQR 0.16-0.20). Performance was better in inpatient-only samples than outpatient-only samples.

Conclusion: Using NLP and ML techniques, we developed a novel approach to identifying goals-of-care discussions in the EHR. NLP and ML represent a potential approach toward measuring goals-of-care discussions as a research outcome and quality metric.

Keywords: Natural language processing; electronic health record; goals of care; machine learning; medical informatics; quality improvement.

Publication types

  • Research Support, N.I.H., Extramural
  • Research Support, Non-U.S. Gov't

MeSH terms

  • Electronic Health Records*
  • Humans
  • Machine Learning
  • Natural Language Processing*
  • Palliative Care
  • Patient Care Planning