MedFactEval and MedAgentBrief: A Framework and Workflow for Generating and Evaluating Factual Clinical Summaries

Françcois Grolleau; Emily Alsentzer; Timothy Keyes; Philip Chung; Akshay Swaminathan; Asad Aali; Jason Hom; Tridu Huynh; Thomas Lew; April Liang; Weihan Chu; Natasha Steele; Christina Lin; Jingkun Yang; Kameron Black; Stephen Ma; Fateme N Haredasht; Nigam H Shah; Kevin Schulman; Jonathan H Chen

doi:10.1142/9789819824755_0027

MedFactEval and MedAgentBrief: A Framework and Workflow for Generating and Evaluating Factual Clinical Summaries

Pac Symp Biocomput. 2026:31:388-399. doi: 10.1142/9789819824755_0027.

Authors

Françcois Grolleau¹, Emily Alsentzer², Timothy Keyes³, Philip Chung⁴, Akshay Swaminathan², Asad Aali⁵, Jason Hom⁶, Tridu Huynh⁶, Thomas Lew⁶, April Liang⁶, Weihan Chu⁶, Natasha Steele⁶, Christina Lin⁶, Jingkun Yang⁶, Kameron Black⁶, Stephen Ma⁶, Fateme N Haredasht⁷, Nigam H Shah⁸, Kevin Schulman⁹, Jonathan H Chen¹⁰

Affiliations

¹ Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA, grolleau@stanford.edu.
² Department of Biomedical Data Science, Stanford University, Stanford, CA, USA.
³ Stanford Health Care, Palo Alto, CA, USA.
⁴ Department of Anesthesiology and Pain Medicine, Stanford Medicine, Stanford, CA, USA.
⁵ Department of Radiology, Stanford University, Stanford, CA, USA.
⁶ Department of Medicine, Stanford University, Stanford, CA, USA.
⁷ Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.
⁸ Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA2Department of Biomedical Data Science, Stanford University, Stanford, CA, USA7Department of Medicine, Stanford University, Stanford, CA, USA.
⁹ Stanford Clinical Excellence Research Center, Stanford University, Stanford, CA, USA7Department of Medicine, Stanford University, Stanford, CA, USA.
¹⁰ Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA6Stanford Clinical Excellence Research Center, Stanford University, Stanford, CA, USA7Department of Medicine, Stanford University, Stanford, CA, USA.

Abstract

Evaluating factual accuracy in Large Language Model (LLM)-generated clinical text is a critical barrier to adoption, as expert review is unscalable for the continuous quality assurance these systems require. We address this challenge with two complementary contributions. First, we introduce MedFactEval, a framework for scalable, fact-grounded evaluation where clinicians define high-salience key facts and an "LLM Jury"-a multi-LLM majority vote-assesses their inclusion in generated summaries. Second, we present MedAgentBrief, a model-agnostic, multi-step workflow designed to generate high-quality, factual discharge summaries. To validate our evaluation framework, we established a gold-standard reference using a seven-physician majority vote on clinician-defined key facts from inpatient cases. The MedFactEval LLM Jury achieved almost perfect agreement with this panel (Cohen's κ = 81%), a performance statistically non-inferior to that of a single human expert (κ = 67%, P < 0.001). Our work provides both a robust evaluation framework (MedFactEval) and a high-performing generation workflow (MedAgentBrief), offering a comprehensive approach to advance the responsible deployment of generative AI in clinical workflows.

MeSH terms

Computational Biology / methods
Electronic Health Records / statistics & numerical data
Humans
Natural Language Processing
Workflow*

Grants and funding

R01 AI178121/AI/NIAID NIH HHS/United States