An event-based approach for comparing the performance of methods for prospective medical product monitoring

Abstract

Background: Prospective medical product monitoring is intended to alert stakeholders about whether and when safety problems are identifiable in longitudinal electronic healthcare data. Little attention has been given to how to compare methods in this setting.

Purpose: To explore aspects of prospective monitoring that should be considered when comparing method performance and to develop a metric that explicitly accounts for these considerations.

Methods: We reviewed existing metrics and propose an event-based approach that classifies exposed outcomes according to whether a prior alert was generated.

Results: In comparing performance of methods for prospective monitoring, three factors must be considered: (1) accuracy in alerting; (2) timeliness of alerting; and (3) the trade-offs between the costs of false negative and false positive alerting. Traditional scenario-based measures of accuracy, such as sensitivity and specificity, which classify only at the end of monitoring, fail to appreciate timeliness of alerting and impose fixed tradeoffs between false positive versus false negative consequences. We provide an expression that summarizes event-based sensitivity (the proportion of exposed events that occur after alerting among all exposed events in scenarios with true safety issues) and event-based specificity (the proportion of exposed events that occur in the absence of alerting among all exposed events in scenarios with no true safety issues) by taking an average weighted by relative costs of false positive and false negative alerting.

Conclusions: The proposed approach explicitly accounts for accuracy in alerting, timeliness in alerting, and the trade-offs between the costs of false negative and false positive alerting.

Figure 1

Cumulative numbers of exposed events and alerting indications for two hypothetical monitoring methods across eight hypothetical monitoring scenarios (i.e. medical product-outcome pairs) Each cell indicates the number of cumulative hypothetical exposed events observed at each of ten monitoring periods across eight hypothetical scenarios (i.e. product-outcome pairs). Scenarios 1-4 are ones in which true safety issues exist such that alerts are true positives. Scenarios 5-8 are ones in which no true safety issues exist such that alerts would be false positives. The arrows indicate when each of two hypothetical monitoring methods generated alerts in each scenario. Neither method generated an alert in scenarios 1, 2, 5, 6, 7, and 8. Both methods generated alerts in Scenarios 3 and 4, but at different times.

Figure 2

Behaviors of three metrics at varying parameter values Plotted is the event-based performance (EBP) metric with w = 0.20 (red; left vertical axis), the diagnostic odds ratio (DOR; green; right vertical axis), and mean average precision (blue; left vertical axis) across values of sensitivity (horizontal axis) when prevalence (i.e. proportion of all scenarios that contain true safety issues) = 0.50 and specificity = 0.80 (thin solid lines), prevalence = 0.50 and specificity = 0.99 (dotted lines), and prevalence = 0.01 and specificity = 0.80 (think solid lines). The thin solid and thick solid lines are superimposed for EBP and for DOR.