Assessing the diagnostic capacity of artificial intelligence chatbots for dysphonia types: Model development and validation

S Saeedi; M Aghajanzadeh

doi:10.1016/j.anorl.2025.01.001

Assessing the diagnostic capacity of artificial intelligence chatbots for dysphonia types: Model development and validation

Eur Ann Otorhinolaryngol Head Neck Dis. 2025 Feb 18:S1879-7296(25)00025-0. doi: 10.1016/j.anorl.2025.01.001. Online ahead of print.

Authors

S Saeedi¹, M Aghajanzadeh²

Affiliations

¹ Independent Researcher in Laryngology, Voice Pathology, and Speech-Language Pathology, Tehran, Iran.
² Department of Speech Therapy, School of Rehabilitation, Tehran University of Medical Sciences, 11489 Tehran, Iran. Electronic address: mahshid_aghajanzade@yahoo.com.

PMID: 39971639
DOI: 10.1016/j.anorl.2025.01.001

Abstract

Purpose: User-friendly artificial intelligence (AI) chatbots are increasingly being explored to assist healthcare teams in their decision-making processes. As accurate diagnosis in all medical fields is vital in treatment planning, this research seeks to explore the function of two specific AI chatbots, ChatGPT and Perplexity AI, in distinguishing the various types of dysphonia (organic, functional, and neurological).

Material and methods: In experiment 1, a script combining voice self-assessments plus the acoustic analysis, and in experiment 2, only the acoustic analysis of 37 dysphonic patients was fed into the ChatGPT and Perplexity AI chatbots specifying the type and asked to develop a complex AI-based model to determine dysphonia type. Then, the same process was redone with data from a sample of 27 other patients as a test.

Results: Although ChatGPT could not analyze the data and only provided guidance, the Cohen's Kappa agreement between experts' diagnoses and Perplexity AI diagnoses in experiment 1 (P=0.773) and experiment 2 (P=0.067) lacked statistically significance.

Conclusion: Regarding the preliminary poor performance of AI chatbots in differential diagnosis of dysphonia type, it is not currently recommended to use them in clinical settings. However, modifications in AI chatbots in the future might provide more promising results in determining the dysphonia type. Further research is needed to shed light on AI chatbots ability in voice clinics.

Keywords: ChatGPT; Diagnosis; Functional dysphonia; Neurological dysphonia; Organic dysphonia; Perplexity AI.