Chat Generative Pretraining Transformer Answers Patient-focused Questions in Cervical Spine Surgery

Tejas Subramanian; Kasra Araghi; Troy B Amen; Austin Kaidi; Branden Sosa; Pratyush Shahi; Sheeraz Qureshi; Sravisht Iyer

doi:10.1097/BSD.0000000000001600

Chat Generative Pretraining Transformer Answers Patient-focused Questions in Cervical Spine Surgery

Clin Spine Surg. 2024 Mar 21. doi: 10.1097/BSD.0000000000001600. Online ahead of print.

Authors

Tejas Subramanian^{1

2}, Kasra Araghi¹, Troy B Amen¹, Austin Kaidi¹, Branden Sosa², Pratyush Shahi¹, Sheeraz Qureshi^{1

2}, Sravisht Iyer^{1

2}

Affiliations

¹ Department of Orthopedic Surgery, Hospital for Special Surgery.
² Weill Cornell Medicine, New York, NY.

PMID: 38531823
DOI: 10.1097/BSD.0000000000001600

Abstract

Study design: Review of Chat Generative Pretraining Transformer (ChatGPT) outputs to select patient-focused questions.

Objective: We aimed to examine the quality of ChatGPT responses to cervical spine questions.

Background: Artificial intelligence and its utilization to improve patient experience across medicine is seeing remarkable growth. One such usage is patient education. For the first time on a large scale, patients can ask targeted questions and receive similarly targeted answers. Although patients may use these resources to assist in decision-making, there still exists little data regarding their accuracy, especially within orthopedic surgery and more specifically spine surgery.

Methods: We compiled 9 frequently asked questions cervical spine surgeons receive in the clinic to test ChatGPT's version 3.5 ability to answer a nuanced topic. Responses were reviewed by 2 independent reviewers on a Likert Scale for the accuracy of information presented (0-5 points), appropriateness in giving a specific answer (0-3 points), and readability for a layperson (0-2 points). Readability was assessed through the Flesh-Kincaid grade level analysis for the original prompt and for a second prompt asking for rephrasing at the sixth-grade reading level.

Results: On average, ChatGPT's responses scored a 7.1/10. Accuracy was rated on average a 4.1/5. Appropriateness was 1.8/3. Readability was a 1.2/2. Readability was determined to be at the 13.5 grade level originally and at the 11.2 grade level after prompting.

Conclusions: ChatGPT has the capacity to be a powerful means for patients to gain important and specific information regarding their pathologies and surgical options. These responses are limited in their accuracy, and we, in addition, noted readability is not optimal for the average patient. Despite these limitations in ChatGPT's capability to answer these nuanced questions, the technology is impressive, and surgeons should be aware patients will likely increasingly rely on it.