Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

C Yoonhee Ryder; Nicole M Mott; Christopher L Gross; Chioma Anidi; Leul Shigut; Serena S Bidwell; Erin Kim; Yimeng Zhao; Blessing Ngoin Ngam; Mark J Snell; B Joon Yu; Pawel Forczmanski; Deborah M Rooney; David R Jeffcoach; Grace J Kim

doi:10.1016/j.jsurg.2023.10.007

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

J Surg Educ. 2024 Feb;81(2):267-274. doi: 10.1016/j.jsurg.2023.10.007. Epub 2023 Dec 29.

Authors

Affiliations

¹ University of Michigan Medical School, Ann Arbor, Michigan.
² University of Florida College of Medicine, Gainesville, Florida.
³ Department of Surgery, Soddo Christian General Hospital, Soddo, Ethiopia.
⁴ Department of Surgery, Mbingo Baptist Hospital, Mbingo, Cameroon.
⁵ Department of Surgery, University of Michigan, Ann Arbor, Michigan.
⁶ Department of Computer Science and Information Technology, West Pomeranian University of Technology in Szczecin, Szczecin, Poland.
⁷ Department of Learning Sciences, University of Michigan, Ann Arbor, Michigan.
⁸ Department of Surgery, Community Regional Medical Center, Fresno, California.
⁹ Department of Surgery, University of Michigan, Ann Arbor, Michigan. Electronic address: gracejk@med.umich.edu.

PMID: 38160118
DOI: 10.1016/j.jsurg.2023.10.007

Abstract

Objective: Laparoscopic surgical skill assessment and machine learning are often inaccessible to low-and-middle-income countries (LMIC). Our team developed a low-cost laparoscopic training system to teach and assess psychomotor skills required in laparoscopic salpingostomy in LMICs. We performed video review using AI to assess global surgical techniques. The objective of this study was to assess the validity of artificial intelligence (AI) generated scoring measures of laparoscopic simulation videos by comparing the accuracy of AI results to human-generated scores.

Design: Seventy-four surgical simulation videos were collected and graded by human participants using a modified OSATS (Objective Structured Assessment of Technical Skills). The videos were then analyzed via AI using 3 different time and distance-based calculations of the laparoscopic instruments including path length, dimensionless jerk, and standard deviation of tool position. Predicted scores were generated using 5-fold cross validation and K-Nearest-Neighbors to train classifiers.

Setting: Surgical novices and experts from a variety of hospitals in Ethiopia, Cameroon, Kenya, and the United States contributed 74 laparoscopic salpingostomy simulation videos.

Results: Complete accuracy of AI compared to human assessment ranged from 65-77%. There were no statistical differences in rank mean scores for 3 domains, Flow of Operation, Respect for Tissue, and Economy of Motion, while there were significant differences in ratings for Instrument Handling, Overall Performance, and the total summed score of all 5 domains (Summed). Estimated effect sizes were all less than 0.11, indicating very small practical effect. Estimated intraclass correlation coefficient (ICC) of Summed was 0.72 indicating moderate correlation between AI and Human scores.

Conclusions: Video review using AI technology of global characteristics was similar to that of human review in our laparoscopic training system. Machine learning may help fill an educational gap in LMICs where direct apprenticeship may not be feasible.

Keywords: artificial intelligence; global surgery; laparoscopic salpingostomy; machine learning; surgical education.

MeSH terms

Artificial Intelligence
Clinical Competence
Computer Simulation
Educational Measurement / methods
Female
Humans
Internship and Residency*
Laparoscopy* / education