Exploration of the assessment of clinical decision-making capabilities in Clinical Oncology based on generative large language models

Li Zhao; Chunyan Yang; Chunhui Chen; Tingting Wang; Nana Xiang; Lei Peng; Shaohua Zhang

doi:10.1016/j.ejso.2025.111175

Exploration of the assessment of clinical decision-making capabilities in Clinical Oncology based on generative large language models

Eur J Surg Oncol. 2025 Nov 6;52(1):111175. doi: 10.1016/j.ejso.2025.111175. Online ahead of print.

Authors

Li Zhao¹, Chunyan Yang¹, Chunhui Chen¹, Tingting Wang¹, Nana Xiang², Lei Peng³, Shaohua Zhang⁴

Affiliations

¹ Department of Oncology, Beijing Anzhen Nanchong Hospital of Capital Medical University, Nanchong Central Hospital, Nanchong, 637200, China.
² Department of Urology, Beijing Anzhen Nanchong Hospital of Capital Medical University, Nanchong Central Hospital, Nanchong, 637200, China.
³ Institute of Motor Robotics, South China Hospital, Shenzhen University, Shenzhen, 518000, China.
⁴ Institute of Motor Robotics, South China Hospital, Shenzhen University, Shenzhen, 518000, China. Electronic address: zhangshaohua@szu.edu.cn.

PMID: 41232145
DOI: 10.1016/j.ejso.2025.111175

Abstract

Background: With advancements in large language model (LLM) technology, generative artificial intelligence (AI) has shown transformative potential in healthcare, particularly in optimizing clinical workflows through data integration and semantic reasoning for clinical decision support (CDS). However, existing AI models primarily rely on logical deductions from standardized guidelines, and their effectiveness in complex, high-risk clinical scenarios remains to be further validated. This study evaluates the CDS efficacy of DeepSeek R1 and ChatGPT-4 o1 in real-world oncology diagnosis and treatment settings, assessing the accuracy and adaptability of their recommendations through a multidimensional framework.

Methods: A time-sequenced, chain-structured clinical question set was developed based on real oncology cases. Responses from DeepSeek R1 and ChatGPT-4 o1 were independently generated and blindly evaluated for accuracy and feasibility by four oncology experts. Statistical analysis and visualization were performed using Prism GraphPad 10.0.

Results: Both DeepSeek R1 and ChatGPT-4 o1 demonstrated overall competent performance in oncology CDS, with no significant differences compared to human clinicians. Evaluations using a clinical decision quality assessment scale indicated robust performance for both models. Subgroup analysis revealed that DeepSeek R1 outperformed in medical humanistic care, while ChatGPT-4 o1 excelled in readability. No statistical differences were observed between the two models in knowledge accuracy, test rationality, or medication standardization.

Conclusion: Generative AI models such as DeepSeek R1 and ChatGPT-4 o1 exhibit comprehensive capabilities in oncology CDS comparable to clinicians, suggesting potential clinical utility. However, AI reliability in complex cases requires improvement and cannot yet replace physicians' expertise. Future research should prioritize multimodal knowledge integration and ethical oversight to enhance AI's role in optimizing diagnostic efficiency and humanistic care quality.

Keywords: Artificial intelligence; Clinical decision support; Diagnosis and treatment; Large language models; Oncology.