Evaluating the clinical utility of multimodal large language models in rare maculopathy

Melanie D Tran; Evan Walker; Ines D Nagel; Nehal Nailesh Mehta; Jesse Most; Henry A Ferreyra; Lesley A Everett; Paul Yang; Mark E Pennesi; Shyamanga Borooah

doi:10.1038/s41598-025-29299-2

Evaluating the clinical utility of multimodal large language models in rare maculopathy

Sci Rep. 2025 Dec 3;16(1):54. doi: 10.1038/s41598-025-29299-2.

Authors

Melanie D Tran^{1

2}, Evan Walker³, Ines D Nagel^{1

3}, Nehal Nailesh Mehta^{1

3}, Jesse Most^{1

2}, Henry A Ferreyra^{1

3}, Lesley A Everett⁴, Paul Yang⁴, Mark E Pennesi^{4

5}, Shyamanga Borooah^{6

7}

Affiliations

¹ Retinal Division, Jacobs Retina Center, Shiley Eye Institute, University of California San Diego, 9415 Campus Point Dr, La Jolla, CA, 92093, USA.
² School of Medicine, University of California San Diego, La Jolla, CA, 92037, USA.
³ Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, San Diego, CA, USA.
⁴ Casey Eye Institute, Oregon Health and Science University, Portland, OR, USA.
⁵ Retina Foundation of the Southwest, Dallas, TX, USA.
⁶ Retinal Division, Jacobs Retina Center, Shiley Eye Institute, University of California San Diego, 9415 Campus Point Dr, La Jolla, CA, 92093, USA. sborooah@health.ucsd.edu.
⁷ Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, San Diego, CA, USA. sborooah@health.ucsd.edu.

Abstract

This study aimed to assess how multimodal large language models (MLLM) diagnose and differentiate Pentosan Polysulfate (PPS) Maculopathy from other phenotypic mimics. A retrospective review of clinical records and multimodal retinal imaging was conducted with patients from the Shiley Eye Institute and Casey Eye Institute. Four MLLMs (ChatGPT-4o, Claude 3.5 Sonnet, Google Gemini 1.5 Pro, Perplexity Llama 3.1 Sonar/Default) along with human retinal specialists answered prompts based on retinal imaging and demographic data. Performance was evaluated using accuracy, sensitivity and specificity estimates. The study included 126 eyes from 63 patients, with 36 eyes with PPS maculopathy, 50 eyes with Stargardt disease, and 40 eyes with PRPH2-associated multifocal pattern dystrophy. MLLMs showed improved accuracy and sensitivity when answer choices were restricted, with ChatGPT consistently performing best when all imaging modalities were prompted together. The inclusion of demographic data further enhanced performance in prompts with limited answer choices. Human retinal specialist evaluations aligned with MLLM performance trends and also improved with demographic data. While MLLMs show diagnostic potential, further refinement is needed before clinical implementation. These findings highlight the importance of prompt design and demographic data to optimize MLLM performance with retinal imaging modalities.

Keywords: Pattern dystrophy; Pentosan polysulfate sodium; Stargardt.

MeSH terms

Adult
Aged
Diagnosis, Differential
Female
Humans
Large Language Models
Macular Degeneration* / diagnosis
Macular Degeneration* / diagnostic imaging
Male
Middle Aged
Multimodal Imaging / methods
Retina / diagnostic imaging
Retina / pathology
Retrospective Studies
Sensitivity and Specificity
Stargardt Disease / diagnosis

Abstract

MeSH terms

Grants and funding