Accuracy, comprehensiveness and understandability of AI-generated answers to questions from people with COPD: the AIR-COPD Study

Mattia Nigro; Greta E Behring; Andrea Aliverti; Alessandra Angelucci; Anita Kay Simonds; Antonio Anzueto; Peter Martin Calverley; Francesco Amati; Anna Stainer; Apostolos Bossios; Hilary Pinnock; Jeanette Boyd; Pippa Powell; Stefano Aliberti; AIR-COPD Task Force

doi:10.1186/s12931-025-03438-9

Accuracy, comprehensiveness and understandability of AI-generated answers to questions from people with COPD: the AIR-COPD Study

Respir Res. 2025 Dec 16;27(1):19. doi: 10.1186/s12931-025-03438-9.

Authors

Mattia Nigro^{1

2}, Greta E Behring^{1

2}, Andrea Aliverti³, Alessandra Angelucci³, Anita Kay Simonds^{4

5}, Antonio Anzueto⁶, Peter Martin Calverley⁷, Francesco Amati^{1

2}, Anna Stainer^{1

2}, Apostolos Bossios^{8

9

10}, Hilary Pinnock¹¹, Jeanette Boyd¹², Pippa Powell¹², Stefano Aliberti^{13

14}; AIR-COPD Task Force

Collaborators

AIR-COPD Task Force:
Antonio Spanevello, Marco Vanetti, David M G Halpin, Khanh Le Quoc Tran, Maria Elia Gomez-Merino, Deepak Muthreja, Boudewijn J H Dierick, Elene Khurtsidze, Vishakha Kalpesh Kapadia, Amalia Panagiotou, Miguel Gallego, Stavros Tryfon, Efthymia Papadopoulou, Carlos Figueiredo, Shailesh Balasaheb Kolekar, Pradeesh Sivapalan, Pedro J Marcos

Affiliations

¹ Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072, Milan, Italy.
² IRCCS Humanitas Research Hospital, Respiratory Unit, Via Manzoni 56, 20089, Rozzano, Milan, Italy.
³ Dipartimento Di Elettronica, Informazione E Bioingegneria, Politecnico Di Milano, Piazza Leonardo Da Vinci 32, 20133, Milan, Italy.
⁴ Sleep and Ventilation Unit, Royal Brompton and Harefield Hospital (Guys and St Thomas' NHS Foundation Trust), London, UK.
⁵ National Heart and Lung Institute, Imperial College London, London, UK.
⁶ South Texas Veterans Health Care System, University of Texas, San Antonio, TX, USA.
⁷ Institute of Life Course and Medical Sciences, University of Liverpool, Liverpool, UK.
⁸ Karolinska Severe Asthma Centre, Department of Respiratory Medicine and Allergy, Karolinska University Hospital, Stockholm, Sweden.
⁹ Division of Lung and Airway Research, Institute of Environmental Medicine, Karolinska Institutet, Stockholm, Sweden.
¹⁰ Lung Laboratory, Centre for Molecular Medicine, Karolinska University Hospital, Stockholm, Sweden.
¹¹ Usher Institute, The University of Edinburgh, Edinburgh, UK.
¹² European Lung Foundation, Sheffield, UK.
¹³ Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072, Milan, Italy. stefano.aliberti@hunimed.eu.
¹⁴ IRCCS Humanitas Research Hospital, Respiratory Unit, Via Manzoni 56, 20089, Rozzano, Milan, Italy. stefano.aliberti@hunimed.eu.

Abstract

Background: Chronic obstructive pulmonary disease (COPD) remains an underestimated and underdiagnosed condition due to low disease awareness. Generative Artificial Intelligence (AI) chatbots are convenient and accessible sources of medical information, but evaluation of the quality of answers provided by patient-generated questions about COPD has not been performed to date.

Objective: To assess and compare accuracy, comprehensiveness, understandability and reliability of different AI chatbots in response to patient-generated questions on the clinical management of COPD.

Methods: A cross-sectional study was conducted in collaboration with the European Respiratory Society (ERS), the European Lung Foundation (ELF), and the ERS CONNECT Clinical Research Collaboration (CRC). Fifteen real questions formulated by ELF COPD patient representatives were divided into three difficulty tiers (easy, medium, difficult) and submitted to ChatGPT (version 3.5), Bard, and Copilot. Experts assessed accuracy and comprehensiveness on a 0–10 scale; patients assessed understandability using the same scale. Reliability was assessed by two investigators. Reviewers were blinded to which AI system generated the answers, and only those who completed all evaluations were included in the analysis.

Results: ChatGPT responses were the most reliable (14/15), followed by Copilot (12/15) and Bard (11/15). ChatGPT scored higher for accuracy (8.0 [7.0 – 9.0]) and comprehensiveness (8.0 [6.8 – 9.0]) than Bard (6.0 [5.0 – 8.0] and 6.0 [5.0 – 7.0]) and Copilot (6.0 [5.0 – 7.3] and 6.0 [5.0 – 8.0]) (both P < 0.001). Understandability was similar across all software (ChatGPT: 8.0 [8.0–10.0]; Bard: 9.0 [8.0–10.0]; Copilot: 9.0 [8.0–10.0]) (P = 0.53). No significant effect was detected according to the difficulty of the question.

Conclusion: Our findings suggest that AI chatbots, particularly ChatGPT, can provide accurate, comprehensive and understandable answers to patients’ questions.

Keywords: AI; Accuracy; Artificial intelligence; COPD; Comprehensiveness; Disease awareness; Reliability; Understandability.