Computationally efficient confidence intervals for cross-validated area under the ROC curve estimates

Electron J Stat. 2015;9(1):1583-1607. doi: 10.1214/15-EJS1035.

Abstract

In binary classification problems, the area under the ROC curve (AUC) is commonly used to evaluate the performance of a prediction model. Often, it is combined with cross-validation in order to assess how the results will generalize to an independent data set. In order to evaluate the quality of an estimate for cross-validated AUC, we obtain an estimate of its variance. For massive data sets, the process of generating a single performance estimate can be computationally expensive. Additionally, when using a complex prediction method, the process of cross-validating a predictive model on even a relatively small data set can still require a large amount of computation time. Thus, in many practical settings, the bootstrap is a computationally intractable approach to variance estimation. As an alternative to the bootstrap, we demonstrate a computationally efficient influence curve based approach to obtaining a variance estimate for cross-validated AUC.

Keywords: AUC; ROC; binary classification; confidence intervals; cross-validation; influence curve; influence function; machine learning; model selection; variance estimation.