A mixed methods crossover randomized controlled trial exploring the experiences, perceptions, and usability of artificial intelligence (ChatGPT) in health sciences education

Mirella Veras; Joseph-Omer Dyer; Holly Shannon; Bryce J M Bogie; Morgan Ronney; Harmehr Sekhon; Derek Rutherford; Paulo Goberlânio Barros Silva; Dahlia Kairy

doi:10.1177/20552076241298485

A mixed methods crossover randomized controlled trial exploring the experiences, perceptions, and usability of artificial intelligence (ChatGPT) in health sciences education

Digit Health. 2024 Dec 5:10:20552076241298485. doi: 10.1177/20552076241298485. eCollection 2024 Jan-Dec.

Authors

Mirella Veras^{1

2}, Joseph-Omer Dyer^{3

4}, Holly Shannon⁵, Bryce J M Bogie⁶, Morgan Ronney⁷, Harmehr Sekhon^{8

9}, Derek Rutherford¹⁰, Paulo Goberlânio Barros Silva¹¹, Dahlia Kairy^{6

12}

Affiliations

¹ Department of Physical Therapy, College of Rehabilitation Sciences, University of Manitoba, Winnipeg, MB, Canada.
² Department of Health Sciences, Carleton University, Ottawa, ON, Canada.
³ École de Réadaptation, Faculté de Médecine, Université de Montréal, Montréal, QC, Canada.
⁴ Groupe Interdisciplinaire de Recherche sur la Cognition et le Raisonnement Professionnel, Faculty of Medicine, Université de Montréal, Montréal, QC, Canada.
⁵ Department of Neuroscience, Carleton University, Ottawa, ON, Canada.
⁶ Department of Cellular and Molecular Medicine, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada.
⁷ Teaching and Learning Services, Carleton University, Ottawa, ON, Canada.
⁸ Department of Medicine, Division of Geriatric Medicine, McGill University, and St Mary's Research Centre, Montréal, QC, Canada.
⁹ The Centre for Addiction and Mental Health, Toronto, ON, Canada.
¹⁰ School of Physiotherapy, Dalhousie University, Halifax, NS, Canada.
¹¹ Centro Universitário Christus, Fortaleza, Ceará, Brazil.
¹² Institut Universitaire sur la Réadaptation en Déficience Physique de Montréal, Centre Intégré Universitaire de Santé et Services Sociaux du Centre-Sud-de-l'Île-de-Montréal, Montréal, QC, Canada.

Abstract

Background: Generative artificial intelligence (AI) integrated programs such as Chat Generative Pre-trained Transformers (ChatGPT) are becoming more widespread in educational settings, with mounting ethical and reliability concerns regarding its usage. This paper explores the experiences, perceptions, and usability of ChatGPT in undergraduate health sciences students.

Methods: Twenty-seven students at Carleton University (Canada) were enrolled in a crossover randomized controlled trial study from a Health Sciences course during the Fall 2023 academic term. The intervention condition involved the use of ChatGPT-3.5, whereas the control condition involved using conventional web-based tools. Technology usability was compared between ChatGPT-3.5 and the traditional tools using questionnaires. Focus group discussions were conducted with seven students to further elaborate on student perceptions and experiences. Reflexive thematic analysis was employed to identify themes from the focus group data.

Results: Easiness of learnability for personal use and a perception of quick learnability towards ChatGPT-3.5 were significantly higher, compared to conventional online tools from the Systems Usability Scale. Qualitative results highlighted strong benefits of ChatGPT-3.5, such as being a tool for increased overall productivity and brainstorming. However, students identified challenges associated with reliability and accuracy, and concerns about academic integrity.

Conclusions: Despite the benefits and positive usability of ChatGPT-3.5 identified by students, an explicit need for the development of policies, procedures and regulations remains. An established framework of best practices for the usage of AI within health science education is necessary. This will ensure accountability of users and lead to a more effective integration of AI technologies into academic settings.

Keywords: AI; ChatGPT; artificial intelligence; crossover RCT; education; health sciences; learning outcomes; perceptions; randomized controlled trial; usability.