Developing and evaluating an automated appendicitis risk stratification algorithm for pediatric patients in the emergency department

J Am Med Inform Assoc. 2013 Dec;20(e2):e212-20. doi: 10.1136/amiajnl-2013-001962. Epub 2013 Oct 15.


Objective: To evaluate a proposed natural language processing (NLP) and machine-learning based automated method to risk stratify abdominal pain patients by analyzing the content of the electronic health record (EHR).

Methods: We analyzed the EHRs of a random sample of 2100 pediatric emergency department (ED) patients with abdominal pain, including all with a final diagnosis of appendicitis. We developed an automated system to extract relevant elements from ED physician notes and lab values and to automatically assign a risk category for acute appendicitis (high, equivocal, or low), based on the Pediatric Appendicitis Score. We evaluated the performance of the system against a manually created gold standard (chart reviews by ED physicians) for recall, specificity, and precision.

Results: The system achieved an average F-measure of 0.867 (0.869 recall and 0.863 precision) for risk classification, which was comparable to physician experts. Recall/precision were 0.897/0.952 in the low-risk category, 0.855/0.886 in the high-risk category, and 0.854/0.766 in the equivocal-risk category. The information that the system required as input to achieve high F-measure was available within the first 4 h of the ED visit.

Conclusions: Automated appendicitis risk categorization based on EHR content, including information from clinical notes, shows comparable performance to physician chart reviewers as measured by their inter-annotator agreement and represents a promising new approach for computerized decision support to promote application of evidence-based medicine at the point of care.

Keywords: Electronic Health Record; Information Extraction; Natural Language Processing; Pediatric Appendicitis Score; Risk Stratification.

Publication types

  • Evaluation Study
  • Research Support, N.I.H., Extramural
  • Research Support, Non-U.S. Gov't

MeSH terms

  • Abdominal Pain / etiology*
  • Algorithms*
  • Appendicitis / diagnosis*
  • Artificial Intelligence
  • Child
  • Electronic Health Records*
  • Emergency Service, Hospital
  • Humans
  • Natural Language Processing*
  • Risk Assessment / methods