ChatGPT in head and neck scientific writing: A precautionary anecdote

Robin T Wu; Rushil R Dang

doi:10.1016/j.amjoto.2023.103980

ChatGPT in head and neck scientific writing: A precautionary anecdote

Am J Otolaryngol. 2023 Nov-Dec;44(6):103980. doi: 10.1016/j.amjoto.2023.103980. Epub 2023 Jul 6.

Authors

Robin T Wu¹, Rushil R Dang²

Affiliations

¹ Department of Plastic and Reconstruction Surgery, Chang Gung Memorial Hospital, Linkou, Taiwan; Division of Plastic and Reconstructive Surgery, Department of Surgery, Stanford University Hospital, Stanford, CA, USA.
² Department of Plastic and Reconstruction Surgery, Chang Gung Memorial Hospital, Linkou, Taiwan; Former fellow, Maxillofacial Oncology and Reconstructive Surgery, Department of Oral and Maxillofacial Surgery, Boston Medical Center, Boston, MA, USA. Electronic address: rushilrd@bu.edu.

PMID: 37459740
DOI: 10.1016/j.amjoto.2023.103980

Abstract

Purpose: To evaluate the accuracy of ChatGPT references in scientific writing relevant to head and neck surgery.

Materials and methods: Five commonly researched keywords relevant to head and neck surgery were selected (osteoradionecrosis of the jaws, oral cancer, adjuvant therapy for oral cancer, TORS, and free flap reconstruction in oral cancer). The AI chatbot was then asked to provide ten complete citations for each of the keywords. Two independent authors reviewed the results for accuracy and assigned each article a numerical score based on pre-selected criteria.

Results: Among 50 total references provided by ChatGPT, only five (10 %) were found to have the correct title, journal, authors, year of publication, and DOI. Merely 14 % of the presented references had correct DOI. References regarding free flap reconstruction for oral cancer were the least accurate from all the five categories, with no correct DOI. Complete inter-rater agreement was noted while evaluating the citations.

Conclusion: Only 10 % of the articles provided by ChatGPT, relevant to head and neck surgery, were correct. A high degree of academic hallucination was noted.

Keywords: AI; ChatGPT; Head and neck surgery; Oral cancer.

MeSH terms

Combined Modality Therapy
Head*
Humans
Mouth Neoplasms*
Neck
Writing