Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival

Abstract

Random forests are a popular nonparametric tree ensemble procedure with broad applications to data analysis. While its widespread popularity stems from its prediction performance, an equally important feature is that it provides a fully nonparametric measure of variable importance (VIMP). A current limitation of VIMP, however, is that no systematic method exists for estimating its variance. As a solution, we propose a subsampling approach that can be used to estimate the variance of VIMP and for constructing confidence intervals. The method is general enough that it can be applied to many useful settings, including regression, classification, and survival problems. Using extensive simulations, we demonstrate the effectiveness of the subsampling estimator and in particular find that the delete-d jackknife variance estimator, a close cousin, is especially effective under low subsampling rates due to its bias correction properties. These 2 estimators are highly competitive when compared with the .164 bootstrap estimator, a modified bootstrap procedure designed to deal with ties in out-of-sample data. Most importantly, subsampling is computationally fast, thus making it especially attractive for big data settings.

Keywords: VIMP; bootstrap; delete-d jackknife; permutation importance; prediction error; subsampling.

FIGURE 10

Assessing asymptotic normality of VIMP from RF-C, RSF, and RSF competing risk simulations. Figure displays bias of standardized VIMP quantiles compared to standard normal quantiles. Values are displayed for 5,10,25,50,75,90,95 percentile values.

FIGURE 11

Results from competing risk simulations showing performance of subsampling, delete-d jackknife, and .164 bootstrap. Left and right hand side figures are based on subsampling rates b = n^1/2 and b = n^3/4. Top and middle figures display bias and standardized MSE for estimating VIMP standard error. Bottom figure displays coverage for VIMP 90% asymptotic normal confidence intervals. Results have been stratified into 6 groups based on 10, 25, 50, 75, and 90th percentiles of true finite VIMP.

FIGURE 1

Bias and standardized mean-squared-error (SMSE) performance for estimating VIMP standard error from RF-R (RF-regression) simulations. In total there are 240 variables (12 simulations, p = 20 variables in each simulation). These 240 variables have been stratified into 6 groups based on 10, 25, 50, 75, and 90th percentiles of true finite VIMP. Extreme right boxplots labeled “ALL” display performance for all 240 variables simultaneously.

FIGURE 2

Results from RF-R simulations but with increased subsampling rate b = n^3/4. Notice the improvement in bias and SMSE for the subsampling estimator.

FIGURE 3

Assessing asymptotic normality of VIMP from RF-R simulations. Left-hand figure displays normal quantile plots for standardized VIMP for each of the 12 simulations. Right-hand figure displays bias of VIMP quantiles compared to standard normal quantiles for all 240 variables from all 12 simulations.

FIGURE 4

Coverage of VIMP 90% asymptotic normal confidence intervals from RF-R simulations. Left and right hand side figures based on subsampling rates b = n^1/2 and b = n^3/4 respectively. Confidence regions for the 240 variables from the 12 simulation experiments have been stratified into 6 groups based on 10, 25, 50, 75, and 90th percentiles of true finite VIMP values.

FIGURE 5

OOB misclassification error rate versus OOB normalized Brier score for Wisconsin breast cancer data (obtained from the mlbench R-package). Note the fluctuations in misclassification error even after 20,000 trees in contrast to the stable behavior of the Brier score.

FIGURE 6

Results from RF-C (RF-classification) simulations showing performance of subsampling, delete-d jackknife, and .164 bootstrap. Left and right hand side figures are based on subsampling rates b = n^1/2 and b = n^3/4. Top and middle figures display bias and standardized MSE for estimating VIMP standard error. Bottom figure displays coverage for VIMP 90% asymptotic normal confidence intervals. Results have been stratified into 6 groups based on 10, 25, 50, 75, and 90th percentiles of true finite VIMP.

FIGURE 7

Delete-d jackknife 95% asymptotic normal confidence intervals from RSF analysis of systolic heart failure data.

FIGURE 8

Results from RSF simulations showing performance of subsampling, delete-d jackknife, and .164 bootstrap. Left and right hand side figures are based on subsampling rates b = n^1/2 and b = n^3/4. Top and middle figures display bias and standardized MSE for estimating VIMP standard error. Bottom figure displays coverage for VIMP 90% asymptotic normal confidence intervals. Results have been stratified into 6 groups based on 10, 25, 50, 75, and 90th percentiles of true finite VIMP.

FIGURE 9

Results from variable selection experiment using RF-R simulations of Section 4. Displayed are the true positive rate (TPR) and true negative rate (TNR) for variables selected using 100(1 − α)% delete-d jackknife confidence regions where α = .25, .1, .01, .001. Top and bottom figures are based on subsampling rates b = n^1/2 and b = n^3/4.