Behavior of prediction performance metrics with rare events

Abstract

Objective: Area under the receiver operating characteristic curve (AUC) is commonly reported alongside prediction models for binary outcomes. Recent articles have raised concerns that AUC might be a misleading measure of prediction performance in the rare event setting. This setting is common since many events of clinical importance are rare. We aimed to determine whether the bias and variance of AUC are driven by the number of events or the event rate. We also investigated the behavior of other commonly used measures of prediction performance, including positive predictive value, accuracy, sensitivity, and specificity.

Study design and setting: We conducted a simulation study to determine when or whether AUC is unstable in the rare event setting by varying the size of datasets used to train and evaluate prediction models. This plasmode simulation study was based on data from the Mental Health Research Network; the data contained 149 predictors and the outcome of interest, suicide attempt, which had event rate 0.92% in the original dataset.

Results: Our results indicate that poor AUC behavior-as measured by empirical bias, variability of cross-validated AUC estimates, and empirical coverage of confidence intervals-is driven by the number of events in a rare-event setting, not event rate. Performance of sensitivity is driven by the number of events, while that of specificity is driven by the number of nonevents. Other measures, including positive predictive value and accuracy, depend on the event rate even in large samples.

Conclusion: AUC is reliable in the rare event setting provided that the total number of events is moderately large; in our simulations, we observed near zero bias with 1000 events.

Plain language summary: Predicting self-harm or suicidal behavior is medically important for guiding clinicians in providing care to patients. Several research teams have developed and evaluated suicide risk prediction models based on health records data. Part of evaluating these models is calculating area under the receiver operating characteristic curve (AUC) and other prediction performance metrics. Self-harm and suicide are rare events. Recent research has raised concerns with using AUC in rare-event settings. We aimed to determine whether having a sufficiently large dataset could remove these concerns. In our experiments, we found that AUC can be used without concern in settings with 1000 events or more. Thus, AUC is a valid measure of suicide risk prediction model performance in many large healthcare databases.

Keywords: Area under the receiver operating characteristic curve; Classification; Machine learning; Model evaluation; Prediction; Rare outcome.