Validation of GPT-4 for clinical event classification: A comparative analysis with ICD codes and human reviewers

Yichen Wang; Yuting Huang; Induja R Nimma; Songhan Pang; Maoyin Pang; Tao Cui; Vivek Kumbhari

doi:10.1111/jgh.16561

Validation of GPT-4 for clinical event classification: A comparative analysis with ICD codes and human reviewers

J Gastroenterol Hepatol. 2024 Apr 16. doi: 10.1111/jgh.16561. Online ahead of print.

Authors

Yichen Wang¹, Yuting Huang², Induja R Nimma³, Songhan Pang⁴, Maoyin Pang², Tao Cui⁵, Vivek Kumbhari²

Affiliations

¹ Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA.
² Division of Gastroenterology and Hepatology, Mayo Clinic, Jacksonville, Florida, USA.
³ Department of Medicine, Mayo Clinic, Jacksonville, Florida, USA.
⁴ College of Arts and Sciences, University of Virginia, Charlottesville, Virginia, USA.
⁵ Research Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, Florida, USA.

PMID: 38627920
DOI: 10.1111/jgh.16561

Abstract

Background and aim: Effective clinical event classification is essential for clinical research and quality improvement. The validation of artificial intelligence (AI) models like Generative Pre-trained Transformer 4 (GPT-4) for this task and comparison with conventional methods remains unexplored.

Methods: We evaluated the performance of the GPT-4 model for classifying gastrointestinal (GI) bleeding episodes from 200 medical discharge summaries and compared the results with human review and an International Classification of Diseases (ICD) code-based system. The analysis included accuracy, sensitivity, and specificity evaluation, using ground truth determined by physician reviewers.

Results: GPT-4 exhibited an accuracy of 94.4% in identifying GI bleeding occurrences, outperforming ICD codes (accuracy 63.5%, P < 0.001). GPT-4's accuracy was either slightly lower or statistically similar to individual human reviewers (Reviewer 1: 98.5%, P < 0.001; Reviewer 2: 90.8%, P = 0.170). For location classification, GPT-4 achieved accuracies of 81.7% and 83.5% for confirmed and probable GI bleeding locations, respectively, with figures that were either slightly lower or comparable with those of human reviewers. GPT-4 was highly efficient, analyzing the dataset in 12.7 min at a cost of 21.2 USD, whereas human reviewers required 8-9 h each.

Conclusion: Our study indicates GPT-4 offers a reliable, cost-efficient, and faster alternative to current clinical event classification methods, outperforming the conventional ICD coding system and performing comparably to individual expert human reviewers. Its implementation could facilitate more accurate and granular clinical research and quality audits. Future research should explore scalability, prompt and model tuning, and ethical implications of high-performance AI models in clinical data processing.

Keywords: Artificial intelligence; Disease classification; GPT‐4; Medical records systems, computerized; Natural language processing.

Grants and funding

Dalio Philanthropies