ChatGPT's ability to comprehend and answer cirrhosis related questions in Arabic

Jamil S Samaan; Yee Hui Yeo; Wee Han Ng; Peng-Sheng Ting; Hirsh Trivedi; Aarshi Vipani; Ju Dong Yang; Omer Liran; Brennan Spiegel; Alexander Kuo; Walid S Ayoub

doi:10.1016/j.ajg.2023.08.001

ChatGPT's ability to comprehend and answer cirrhosis related questions in Arabic

Arab J Gastroenterol. 2023 Aug;24(3):145-148. doi: 10.1016/j.ajg.2023.08.001. Epub 2023 Sep 4.

Authors

Affiliations

¹ Karsh Division of Gastroenterology and Hepatology, Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
² Bristol Medical School, University of Bristol, Bristol, UK.
³ School of Medicine, Tulane University, New Orleans, LA, USA.
⁴ Karsh Division of Gastroenterology and Hepatology, Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA; Comprehensive Transplant Center, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
⁵ Karsh Division of Gastroenterology and Hepatology, Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA; Comprehensive Transplant Center, Cedars-Sinai Medical Center, Los Angeles, CA, USA; Samuel Oschin Comprehensive Cancer Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
⁶ Department of Psychiatry and Behavioral Sciences, Cedars-Sinai, Los Angeles, CA, USA; Division of Health Services Research, Department of Medicine, Cedars-Sinai, Los Angeles, CA, USA.
⁷ Karsh Division of Gastroenterology and Hepatology, Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA; Division of Health Services Research, Department of Medicine, Cedars-Sinai, Los Angeles, CA, USA.
⁸ Karsh Division of Gastroenterology and Hepatology, Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA; Comprehensive Transplant Center, Cedars-Sinai Medical Center, Los Angeles, CA, USA. Electronic address: walid.ayoub@cshs.org.

PMID: 37673708
DOI: 10.1016/j.ajg.2023.08.001

Abstract

Background and study aims: Cirrhosis is a chronic progressive disease which requires complex care. Its incidence is rising in the Arab countries making it the 7th leading cause of death in the Arab League in 2010. ChatGPT is a large language model with a growing body of literature demonstrating its ability to answer clinical questions. We examined ChatGPT's accuracy in responding to cirrhosis related questions in Arabic and compared its performance to English.

Materials and methods: ChatGPTs responses to 91 questions in Arabic and English were graded by a transplant hepatologist fluent in both languages. Accuracy of responses was assessed using the scale: 1. Comprehensive, 2. Correct but inadequate, 3. Mixed with correct and incorrect/outdated data, and 4. Completely incorrect.Accuracy of Arabic compared to English responses was assessed using the scale: 1. Arabic response is more accurate, 2. Similar accuracy, 3. Arabic response is less accurate.

Results: The model provided 22 (24.2%) comprehensive, 44 (48.4%) correct but inadequate, 13 (14.3%) mixed with correct and incorrect/outdated data and 12 (13.2%) completely incorrect Arabic responses. When comparing the accuracy of Arabic and English responses, 9 (9.9%) of the Arabic responses were graded as more accurate, 52 (57.1%) similar in accuracy and 30 (33.0%) as less accurate compared to English.

Conclusion: ChatGPT has the potential to serve as an adjunct source of information for Arabic speaking patients with cirrhosis. The model provided correct responses in Arabic to 72.5% of questions, although its performance in Arabic was less accurate than in English. The model produced completely incorrect responses to 13.2% of questions, reinforcing its potential role as an adjunct and not replacement of care by licensed healthcare professionals. Future studies to refine this technology are needed to help Arabic speaking patients with cirrhosis across the globe understand their disease and improve their outcomes.

Keywords: Arabic; ChatGPT; Cirrhosis; Disparity; Large language model.