Cross-Institutional Validation of a novel LLM-Based Cardiac Event Extraction framework from Electronic Health Records

Abstract

Purpose: Cardiac toxicity is a significant concern for cancer patients undergoing radiotherapy. The complexity of evaluating cardiac events, the time required to parse the Electronic Health Record (EHR), and the need for large datasets make cardiotoxicity studies time- and resource-intensive, limiting progress of cardio-oncology research. Large-language models (LLMs) have the potential to automatically identify cardiac events. This work aimed to develop and validate a novel LLM-based automated cardiac event identification framework using a two-institution dataset.

Methods: 411 lung and breast cancer patients from two institutions were analyzed. [Institution 1] data (n=266) were divided into a development cohort (lung, n=178) and an internal validation cohort (breast, n=88). External validation used [Institution 2]data (lung, n=145). Cardiac events were physician-adjudicated from the entire EHR patient history. Extracted EHR data comprised structured problem lists and unstructured clinical notes. We introduced the Two-phase Reasoning for Automated Cardiac Event Recognition (TRACER) framework, which combines structured term matching with LLM-based analysis of unstructured notes (including specialty-filtering, temporally aware queries, and few-shot examples). Following initial evaluation of six candidate models, the three top-performing open-source LLMs (DeepSeek-R1, Llama-3.3, Mistral-Large) were validated. Performance was evaluated against physician-adjudicated ground truth using accuracy and processing time.

Results: Of 411 patients, 220 had at least one cardiac event. The top three models achieved mean accuracies of 79.4%, 81.0%, and 79.3% for the development, internal validation, and external validation cohorts, respectively. DeepSeek-R1 achieved the highest accuracy on internal cohorts (83.4-85.2%), while Llama-3.3 reached 85.5% accuracy on external validation. TRACER processing time was 20-42 seconds per patient (2.3-4.8 hours total) versus 2 hours per patient for manual review (822 person-hours total).

Conclusion: TRACER, a locally deployed LLM framework, accurately extracted cardiac events across institutions and disease sites. This approach enables scalable cardio-oncology research by substantially reducing the resources required for cardiac event identification.

Keywords: Cardiac event identification; Cardio-oncology; Electronic health records (EHR); Large Language Models (LLMs); lung cancer; radiation therapy.