Evaluation of the accuracy and readability of large language model responses on menopause and hormone therapy

Jana Karam; Chrisandra Shufelt; Nancy Safwan; Ekta Kapoor; Monica M Christmas; Stephanie S Faubion

doi:10.1097/GME.0000000000002695

Evaluation of the accuracy and readability of large language model responses on menopause and hormone therapy

Menopause. 2025 Dec 2. doi: 10.1097/GME.0000000000002695. Online ahead of print.

Authors

Jana Karam^{1

2}, Chrisandra Shufelt^{1

2

3}, Nancy Safwan^{1

2}, Ekta Kapoor^{2

3

4}, Monica M Christmas⁵, Stephanie S Faubion^{1

2

3}

Affiliations

¹ Division of General Internal Medicine, Mayo Clinic, Jacksonville, FL.
² Mayo Clinic Center for Women's Health, Rochester, MN.
³ Women's Health Research Center, Mayo Clinic, Rochester, MN.
⁴ Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Mayo Clinic, Rochester, MN.
⁵ Department of Obstetrics and Gynecology, University of Chicago, Chicago, IL.

PMID: 41329562
DOI: 10.1097/GME.0000000000002695

Abstract

Objective: Generative artificial intelligence is rapidly evolving and is now being explored in health care to support patient and clinician education. This study evaluated the accuracy, completeness, and readability of four large language models (LLMs): ChatGPT 3.5, Gemini, ChatGPT 4.0, and OpenEvidence in answering questions about menopause and hormone therapy.

Methods: A total of 35 questions (20 patient-level, 15 clinician-level) were entered into each LLM. OpenEvidence was only used for clinician-level questions. Four blinded expert reviewers rated responses as accurate and complete, accurate but incomplete, or inaccurate. Readability of patient-level responses was assessed using the Flesch Reading Ease Score (FRES) and word count. Analysis used ANOVA for readability, odds ratios for accuracy comparisons.

Results: For patient-level questions, ChatGPT 3.5 achieved the highest accuracy (70%), followed by ChatGPT 4.0 (60%) and Gemini (30%); Gemini had significantly lower odds of accuracy compared with ChatGPT 3.5 (OR=0.18, 95% CI=0.05-0.71; P=0.014). FRES scores differed significantly (P<0.001): Gemini scored 38.9±7.3 ("difficult"), ChatGPT 3.5 scored 31.0±11.2, and ChatGPT 4.0 scored 26.5±8.6 (both "very difficult"). For clinician-level questions, ChatGPT 4.0 achieved the highest accuracy (67%), followed by ChatGPT 3.5 and OpenEvidence (60% each) and Gemini (47%); no significant differences were observed among models (all P>0.05).

Conclusion: LLMs demonstrated limited accuracy and frequent incorrect or incomplete responses to menopause-related queries, highlighting the need to improve model performance to ensure accurate and reliable information for both patients and clinicians.

Keywords: Artificial intelligence; Clinician education; Large language models; Menopause education; Patient education..