Using Active Learning to Identify Health Information Technology Related Patient Safety Events

Abstract

The widespread adoption of health information technology (HIT) has led to new patient safety hazards that are often difficult to identify. Patient safety event reports, which are self-reported descriptions of safety hazards, provide one view of potential HIT-related safety events. However, identifying HIT-related reports can be challenging as they are often categorized under other more predominate clinical categories. This challenge of identifying HIT-related reports is exacerbated by the increasing number and complexity of reports which pose challenges to human annotators that must manually review reports. In this paper, we apply active learning techniques to support classification of patient safety event reports as HIT-related. We evaluated different strategies and demonstrated a 30% increase in average precision of a confirmatory sampling strategy over a baseline no active learning approach after 10 learning iterations.

Keywords: Patient safety; active learning; health information technology; human-in-the-loop; machine learning; patient safety event reports.

Fig. 1

Active learning experiment workflow shows an iterative loop requiring human annotation feedback to update a machine learning model until an externally defined stopping criteria is met.

Fig. 2

Black solid circles indicate the unlabeled (circles with “?”) samples selected under the two different uncertainty sampling strategies for a binary classification task (“L” circles as “Likely” and “U” circles as “Unlikely”).

Fig. 3

After each iteration (vertical axis), the number of reports where the predicted label switched from "Likely" to "Unlikely" (left sub-graph) and from "Unlikely" to "Likely" (right sub-graph) for each AL strategy.