Multidimensional evaluation of large language models in radiology report readability

Yunhai Mao; Chunyan Wang; Yuxin Li; Wei Wang; Mengchao Zhang

doi:10.1038/s41746-026-02589-3

Multidimensional evaluation of large language models in radiology report readability

NPJ Digit Med. 2026 Apr 1. doi: 10.1038/s41746-026-02589-3. Online ahead of print.

Authors

Yunhai Mao^#¹, Chunyan Wang^#¹, Yuxin Li¹, Wei Wang¹, Mengchao Zhang²

Affiliations

¹ Department of Radiology, the Third Hospital of Jilin University, Changchun, China.
² Department of Radiology, the Third Hospital of Jilin University, Changchun, China. zhangmengchao@jlu.edu.cn.

^# Contributed equally.

PMID: 41922696
DOI: 10.1038/s41746-026-02589-3

Abstract

This study systematically investigated the influence of demographic characteristics on the readability of patient-centric radiology reports and compared the performance of different large language models (LLMs) in generating patient-centered reports. Adopting a sequential two-stage design, the research first conducted a retrospective evaluation involving 320 radiology reports followed by a clinical setting validation with 800 patients. Results suggested that all three LLMs significantly improved the readability of radiology reports (P < 0.05), with DeepSeek-R1 showing potentially superior performance within this specific cohort. Demographic analysis revealed significant interactive effects: higher education and older age (within consistent educational levels) were associated with better comprehension. Clinical setting validation further indicated that reading simplified reports suggesting the potential to significantly improved patients' subjective and objective comprehension while significantly alleviating medical anxiety (P < 0.05). However, limitations persist, including inconsistent model outputs, missing anatomical details, and comprehension variances driven by demographic factors. Consequently, LLMs should be integrated as auxiliary communication tools for radiologists rather than standalone solutions, necessitating personalized interventions tailored to specific demographic profiles.