Validation of a natural language processing algorithm using national reporting data to improve identification of anesthesia-related ADVerse evENTs: The "ADVENTURE" study

Abstract

Background: Reporting and analysis of adverse events (AE) is associated with improved health system learning, quality outcomes, and patient safety. Manual text analysis is time-consuming, costly, and prone to human errors. We aimed to demonstrate the feasibility of novel machine learning and natural language processing (NLP) approaches for early predictions of adverse events and provide input to direct quality improvement and patient safety initiatives.

Methods: We used machine learning to analyze 9559 continuously reported AE by clinicians and healthcare systems to the French National Health accreditor (HAS) between January 1, 2009, and December 31, 2020 . We validated the labeling of 135,000 unique de-identified AE reports and determined the associations between different system's root causes and patient consequences. The model was validated by independent expert anesthesiologists.

Results: The machine learning (ML) and Artificial Intelligence (AI) model trained on 9559 AE datasets accurately categorized 8800 (88%) of reported AE. The three most frequent AE types were "difficult orotracheal intubation" (16.9% of AE reports), "medication error" (10.5%), and "post-induction hypotension" (6.9%). The accuracy of the AI model reached 70.9% sensitivity, 96.6% specificity for "difficult intubation", 43.2% sensitivity, and 98.9% specificity for "medication error."

Conclusions: This unsupervised ML method provides an accurate, automated, AI-supported search algorithm that ranks and helps to understand complex risk patterns and has greater speed, precision, and clarity when compared to manual human data extraction. Machine learning and Natural language processing (NLP) models can effectively be used to process natural language AE reports and augment expert clinician input. This model can support clinical applications and methodological standards and used to better inform and enhance decision-making for improved risk management and patient safety.

Trial registration: The study was approved by the ethics committee of the French Society of Anesthesiology (IRB 00010254-2020-20) and the CNIL (CNIL: 118 58 95) and the study was registered with ClinicalTrials.gov (NCT: NCT05185479).

Keywords: Adverse events; Artificial intelligence; Natural language processing; Patient safety; Quality improvement.