Comparing the Effectiveness of Alerts and Dynamically Annotated Visualizations (DAVs) in Improving Clinical Decision Making

Abstract

Objective: The aim of this study was to compare the effectiveness of two types of real-time decision support, an interrupting pop-up alert and a noninterrupting dynamically annotated visualization (DAV), in reducing clinically inappropriate diagnostic imaging orders.

Background: Alerts in electronic health record software are frequently disregarded due to high false-alarm rates, interruptions, and uncertainty about what triggered the alert. In other settings, providing visualizations and improving understandability of the guidance has been shown to improve overall decision making.

Method: Using a between-subject design, we examined the effect of two forms of decision support, alerts and DAVs, on reducing the proportion of inappropriate diagnostic imaging orders for 11 patients in a simulated environment. Nine attending and 11 resident physicians with experience using an electronic health record were randomly assigned to the form of decision support. Secondary measures were self-reported understandability, algorithm transparency, and clinical relevance.

Results: Fewer inappropriate diagnostic imaging tests were ordered with DAVs than with alerts (18% vs. 34%, p < .001). The DAV was rated higher for all three secondary measures (p < .001) for all participants.

Conclusion: DAVs were more effective than alerts in reducing inappropriate imaging orders and were preferred for all patient scenarios, especially scenarios where guidance was ambiguous or based on inaccurate information.

Application: Creating visualizations that are permanently displayed and vary in the strength of their guidance can mitigate the risk of system performance degradation due to incomplete or incorrect data. This interaction paradigm may be applicable for other settings with high false-alarm rates or where there is a need to reduce interruptions during decision making.

Keywords: alarm; alert; alert fatigue; decision support system; electronic health record; evidence-based medicine; interruption; naturalistic decision making; visualization.