Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms

Luigi Angelo Vaira; Jerome R Lechien; Vincenzo Abbate; Fabiana Allevi; Giovanni Audino; Giada Anna Beltramini; Michela Bergonzani; Paolo Boscolo-Rizzo; Gianluigi Califano; Giovanni Cammaroto; Carlos M Chiesa-Estomba; Umberto Committeri; Salvatore Crimi; Nicholas R Curran; Francesco di Bello; Arianna di Stadio; Andrea Frosolini; Guido Gabriele; Isabelle M Gengler; Fabio Lonardi; Fabio Maglitto; Miguel Mayo-Yáñez; Marzia Petrocelli; Resi Pucci; Alberto Maria Saibene; Gianmarco Saponaro; Alessandro Tel; Franco Trabalzini; Eleonora M C Trecca; Valentino Vellone; Giovanni Salzano; Giacomo De Riu

doi:10.1007/s00405-024-08710-0

Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms

Eur Arch Otorhinolaryngol. 2024 May 4. doi: 10.1007/s00405-024-08710-0. Online ahead of print.

Authors

Luigi Angelo Vaira^#^{1

2}, Jerome R Lechien^#^{3

4}, Vincenzo Abbate⁵, Fabiana Allevi⁶, Giovanni Audino⁵, Giada Anna Beltramini^{7

8}, Michela Bergonzani⁹, Paolo Boscolo-Rizzo¹⁰, Gianluigi Califano¹¹, Giovanni Cammaroto¹², Carlos M Chiesa-Estomba¹³, Umberto Committeri⁵, Salvatore Crimi¹⁴, Nicholas R Curran¹⁵, Francesco di Bello¹¹, Arianna di Stadio¹⁶, Andrea Frosolini¹⁷, Guido Gabriele¹⁷, Isabelle M Gengler¹⁵, Fabio Lonardi¹⁸, Fabio Maglitto¹⁹, Miguel Mayo-Yáñez²⁰, Marzia Petrocelli²¹, Resi Pucci²², Alberto Maria Saibene²³, Gianmarco Saponaro²⁴, Alessandro Tel²⁵, Franco Trabalzini²⁶, Eleonora M C Trecca^{27

28}, Valentino Vellone²⁹, Giovanni Salzano^#⁵, Giacomo De Riu^#³⁰

Affiliations

¹ Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Viale San Pietro 43/B, 07100, Sassari, Italy. lavaira@uniss.it.
² PhD School of Biomedical Science, Biomedical Sciences Department, University of Sassari, Sassari, Italy. lavaira@uniss.it.
³ Department of Laryngology and Bronchoesophagology, EpiCURA Hospital, Mons School of Medicine, UMONS. Research Institute for Health Sciences and Technology, University of Mons (UMons), Mons, Belgium.
⁴ Department of Otolaryngology-Head Neck Surgery, Elsan Polyclinic of Poitiers, Poitiers, France.
⁵ Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy.
⁶ Maxillofacial Surgery Department, ASSt Santi Paolo e Carlo, University of Milan, Milan, Italy.
⁷ Department of Biomedical, Surgical and Dental Sciences, University of Milan, Milan, Italy.
⁸ Maxillofacial and Dental Unit, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milan, Italy.
⁹ Maxillo-Facial Surgery Division, Head and Neck Department, University Hospital of Parma, Parma, USA.
¹⁰ Department of Medical, Surgical and Health Sciences, Section of Otolaryngology, University of Trieste, Trieste, Italy.
¹¹ Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy.
¹² ENT Department, Morgagni Pierantoni Hospital, AUSL Romagna, Forlì, Italy.
¹³ Department of Otorhinolaryngology-Head and Neck Surgery, Hospital Universitario Donostia, San Sebastian, Spain.
¹⁴ Operative Unit of Maxillofacial Surgery, Policlinico San Marco, University of Catania, Catania, Italy.
¹⁵ Department of Otolaryngology-Head and Neck Surgery, University of Cincinnati Medical Center, Cincinnati, OH, USA.
¹⁶ Otolaryngology Unit, GF Ingrassia Department, University of Catania, Catania, Italy.
¹⁷ Department of Maxillofacial Surgery, University of Siena, Siena, Italy.
¹⁸ Department of Maxillofacial Surgery, University of Verona, Verona, Italy.
¹⁹ Maxillo-Facial Surgery Unit, University of Bari "Aldo Moro", Bari, Italy.
²⁰ Otorhinolaryngology, Head and Neck Surgery Department, Complexo Hospitalario Universitario A Coruña (CHUAC), A Coruña, Galicia, Spain.
²¹ Maxillofacial Surgery Operative Unit, Bellaria and Maggiore Hospital, Bologna, Italy.
²² Maxillofacial Surgery Unit, San Camillo-Forlanini Hospital, Rome, Italy.
²³ Otolaryngology Unit, Santi Paolo e Carlo Hospital, Department of Health Sciences, University of Milan, Milan, Italy.
²⁴ Maxillo-Facial Surgery Unit, IRCSS "A. Gemelli" Foundation-Catholic University of the Sacred Heart, Rome, Italy.
²⁵ Clinic of Maxillofacial Surgery, Department of Head and Neck Surgery and Neuroscience, University Hospital of Udine, Udine, Italy.
²⁶ Department of Otorhinolaryngology, Head and Neck Surgery, Meyer Children's Hospital, Florence, Italy.
²⁷ Department of Otorhinolaryngology and Maxillofacial Surgery, IRCCS Hospital Casa Sollievo Della Sofferenza, San Giovanni Rotondo, Foggia, Italy.
²⁸ Department of Otorhinolaryngology, University Hospital of Foggia, Foggia, Italy.
²⁹ Maxillofacial Surgery Unit, "S. Maria" Hospital, Terni, Italy.
³⁰ Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Viale San Pietro 43/B, 07100, Sassari, Italy.

^# Contributed equally.

PMID: 38703195
DOI: 10.1007/s00405-024-08710-0

Abstract

Background: The widespread diffusion of Artificial Intelligence (AI) platforms is revolutionizing how health-related information is disseminated, thereby highlighting the need for tools to evaluate the quality of such information. This study aimed to propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms.

Methods: The QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed by 27 reviewers from 25 academic centers worldwide. Construct validity, internal consistency, inter-rater and test-retest reliability were assessed to validate the tool.

Results: The validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach's alpha = 0.837). The Interclass Correlation Coefficient was 0.983 (95% CI 0.973-0.991; F (29,542) = 68.3; p < 0.001), indicating excellent reliability. Test-retest reliability analysis revealed a moderate-to-strong correlation with a Pearson's coefficient of 0.876 (95% CI 0.859-0.891; p < 0.001).

Conclusions: The QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.

Keywords: AI; Artificial intelligence; ChatGPT; Head and neck surgery; Health-related information quality; Machine learning; Maxillofacial surgery; Natural language processing; Neural networks; Otorhinolaryngology.