Introduction to vimp

Brian D. Williamson

2021-06-13

Introduction

vimp is a package that computes nonparametric estimates of variable importance and provides valid inference on the true importance. The package supports flexible estimation of variable importance based on the difference in nonparametric \(R^2\), classification accuracy, and area under the receiver operating characteristic curve (AUC). These quantities are all nonparametric generalizations of the usual measures in simple parametric models (e.g., linear models). For more details, see the accompanying manuscripts Williamson, Gilbert, Carone, et al. (2020), Williamson, Gilbert, Simon, et al. (2020), and Williamson and Feng (2020).

Variable importance estimates may be computed quickly, depending on the techniques used to estimate the underlying conditional means — if these techniques are slow, then the variable importance procedure will be slow.

The code can handle arbitrary dimensions of features, and may be used to estimate the importance of any single feature or group of features for predicting the outcome. The package also includes functions for cross-validated importance.

The author and maintainer of the vimp package is Brian Williamson. The methods implemented here have also been implemented in Python under the package vimpy.

Installation

A stable version of the package may be downloaded and installed from CRAN. Type the following command in your R console to install the stable version of vimp:

install.packages("vimp")

A development version of the package may be downloaded and installed from GitHub using the devtools package. Type the following command in your R console to install the development version of vimp:

# only run if you don't have devtools
# previously installed
# install.packages("devtools")
devtools::install_github("bdwilliamson/vimp")

Quick Start

This section should serve as a quick guide to using the vimp package — we will cover the main functions for estimating \(R^2\)-based variable importance using a simulated data example. More details are given in the next section.

First, load the vimp package:

library("vimp")

Next, create some data:

# -------------------------------------------------------------
# problem setup
# -------------------------------------------------------------
# set up the data
set.seed(5678910)
n <- 1000
p <- 2
s <- 1 # desire importance for X_1
x <- data.frame(replicate(p, runif(n, -1, 1)))
y <- (x[,1])^2*(x[,1]+7/5) + (25/9)*(x[,2])^2 + rnorm(n, 0, 1)
# set up folds for hypothesis testing
folds <- sample(rep(seq_len(2), length = length(y)))

This creates a matrix of covariates x with two columns, a vector y of normally-distributed outcome values, and a set of folds for a sample of n = 100 study participants.

The workhorse function of vimp, for \(R^2\)-based variable importance, is vimp_rsquared. There are two ways to compute variable importance: in the first method, you allow vimp to run regressions for you and return variable importance; in the second method (discussed in “Using precomputed regression function estimates in vimp”), you run the regressions yourself and plug these into vimp. I will focus on the first method here. The basic arguments are

This second-to-last argument, SL.library, determines the estimators you want to use for the conditional mean of Y given X. Estimates of variable importance rely on good estimators of the conditional mean, so we suggest using flexible estimators and model stacking to do so. One option for this is the SuperLearner package; load that package using

library("SuperLearner")
# load specific algorithms
library("ranger")

The code

est_1 <- vimp_rsquared(Y = y, X = x, indx = 1, run_regression = TRUE,
SL.library = c("SL.ranger", "SL.mean"), V = 2, env = environment())

uses the Super Learner to fit the required regression functions, and computes an estimate of variable importance for the importance of \(X_1\). We can visualize the estimate, standard error, and confidence interval by printing or typing the object name:

est_1
## Variable importance estimates:
##       Estimate   SE         95% CI         VIMP > 0 p-value  
## s = 1 0.01985866 0.04777691 [0, 0.1134997] FALSE    0.3388317
print(est_1)
## Variable importance estimates:
##       Estimate   SE         95% CI         VIMP > 0 p-value  
## s = 1 0.01985866 0.04777691 [0, 0.1134997] FALSE    0.3388317

This output shows that we have estimated the importance of \(X_1\) to be 0.02, with a 95% confidence interval of [0, 0.113].

Detailed guide

In this section, we provide a fuller example of estimating \(R^2\)-based variable importance in the context of the South African heart disease study data (Hastie, Tibshirani, and Friedman 2009).

Often when working with data we attempt to estimate the conditional mean of the outcome \(Y\) given features \(X\), defined as \(\mu_P(x) = E_P(Y \mid X = x)\).

There are many tools for estimating this conditional mean. We might choose a classical parametric tool such as linear regression. We might also want to be model-agnostic and use a more nonparametric approach to estimate the conditional mean. However,

Once we have a good estimate of the conditional mean, it is often of scientific interest to understand which features contribute the most to the variation in \(\mu_P\). Specifically, we might consider \[\mu_{P, s}(x) = E_P(Y \mid X_{(-s)} = x_{(-s)}),\] where for a vector \(v\) and a set of indices \(s\), \(v_{-(s)}\) denotes the elements of \(v\) with index not in \(s\). By comparing \(\mu_{P, s}\) to \(\mu_P\) we can evaluate the importance of the \(s\)th element (or group of elements).

Assume that our data are generated according to the mechanism \(P_0\). We define the population \(R^2\) value of a given regression function \(\mu\) as \(R^2(\mu, P_0) = 1 - \frac{E_{P_0}\{Y - \mu(X)\}^2}{var_{P_0}(Y)}\), where the numerator of this expression is the population mean squared error and the denominator is the population variance. We can then define a nonparametric measure of variable importance, \[\psi_{0, s} = R^2(\mu_{P_0}, P_0) - R^2(\mu_{P_0,s}, P_0),\] which is the proportion of the variability in the outcome explained by including \(X_j\) in our chosen estimation technique.

This document introduces you to the basic tools in vimp and how to apply them to a dataset. I will explore one method for obtaining variable estimates using vimp: you only specify a library of candidate estimators for the conditional means \(\mu_{P_0}\) and \(\mu_{P_0, s}\); you allow vimp to obtain the optimal estimates of these quantities using the SuperLearner (van der Laan, Polley, and Hubbard 2007), and use these estimates to obtain variable importance estimates. A second method (using precomputed estimates of the regression functions) exists and is described in “Using precomputed regression function estimates in vimp”.

A look at the South African heart disease study data

Throughout this document I will use the South African heart disease study data (Hastie, Tibshirani, and Friedman 2009), freely available from the Elements of Statistical Learning website. Information about these data is available here.

# read in the data from the Elements website
library("RCurl")
heart_data <- read.csv(text = getURL("http://web.stanford.edu/~hastie/ElemStatLearn/datasets/SAheart.data"), header = TRUE, stringsAsFactors = FALSE)
# minor data cleaning
heart <- heart_data[, 2:dim(heart_data)[2]]
heart$famhist <- ifelse(heart$famhist == "Present", 1, 0)
# sample-splitting folds for hypothesis testing
heart_folds <- sample(rep(seq_len(2), length = dim(heart)[1]))

In addition to the indicator of myocardial infarction chd, the outcome of interest, there are measurements on two groups of variables. First are behavioral features: cumulative tobacco consumption, current alcohol consumption, and type A behavior (a behavioral pattern linked to stress). Second are biological features: systolic blood pressure, low-density lipoprotein (LDL) cholesterol, adiposity (similar to body mass index), family history of heart disease, obesity, and age.

Since there are nine features and two groups, it is of interest to determine variable importance both for the nine individual features separately and for the two groups of features.

A first approach: linear regression

Suppose that I believe that a linear model truly describes the relationship between the outcome and the covariates in these data. In that case, I would be justified in only fitting a linear regression to estimate the conditional means; this means that in my importance analysis, I should also use only linear regression. Note that because I chose to fit the regression functions first and then plug them into vimp, I need to explicitly specify the folds used for hypothesis testing. The analysis is achieved by the following:

X <- heart[, -ncol(heart)]
lm_vim_sbp <- vim(Y = heart$chd, X = X, indx = 1, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
lm_vim_tob <- vim(Y = heart$chd, X = X, indx = 2, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
## Warning in vim(Y = heart$chd, X = X, indx = 2, run_regression = TRUE, SL.library
## = "SL.lm", : Original estimate < 0; returning zero.
lm_vim_ldl <- vim(Y = heart$chd, X = X, indx = 3, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
## Warning in vim(Y = heart$chd, X = X, indx = 3, run_regression = TRUE, SL.library
## = "SL.lm", : Original estimate < 0; returning zero.
lm_vim_adi <- vim(Y = heart$chd, X = X, indx = 4, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
## Warning in vim(Y = heart$chd, X = X, indx = 4, run_regression = TRUE, SL.library
## = "SL.lm", : Original estimate < 0; returning zero.
lm_vim_fam <- vim(Y = heart$chd, X = X, indx = 5, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
lm_vim_tpa <- vim(Y = heart$chd, X = X, indx = 6, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
lm_vim_obe <- vim(Y = heart$chd, X = X, indx = 7, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
## Warning in vim(Y = heart$chd, X = X, indx = 7, run_regression = TRUE, SL.library
## = "SL.lm", : Original estimate < 0; returning zero.
lm_vim_alc <- vim(Y = heart$chd, X = X, indx = 8, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")
## Warning in vim(Y = heart$chd, X = X, indx = 8, run_regression = TRUE, SL.library
## = "SL.lm", : Original estimate < 0; returning zero.
lm_vim_age <- vim(Y = heart$chd, X = X, indx = 9, run_regression = TRUE, SL.library = "SL.lm", type = "r_squared")

# make a table with the estimates using the merge_vim() function
library("dplyr")
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
lm_mat <- merge_vim(lm_vim_sbp, lm_vim_tob, lm_vim_ldl, lm_vim_adi,
                lm_vim_fam, lm_vim_tpa, lm_vim_obe, lm_vim_alc, lm_vim_age)
# print out the matrix
lm_mat
## Variable importance estimates:
##       Estimate   SE         95% CI                   VIMP > 0 p-value   
## s = 9 0.13103142 0.06487123 [0.003886152, 0.2581767]  TRUE    0.02169846
## s = 6 0.06811560 0.06509051 [0.000000000, 0.1956907] FALSE    0.14767086
## s = 1 0.05216276 0.06436184 [0.000000000, 0.1783096] FALSE    0.20883764
## s = 5 0.04812513 0.06276984 [0.000000000, 0.1711517] FALSE    0.22163233
## s = 2 0.00000000 0.06322478 [0.000000000, 0.1239183] FALSE    0.58737127
## s = 3 0.00000000 0.06355300 [0.000000000, 0.1245616] FALSE    0.58097486
## s = 4 0.00000000 0.06460803 [0.000000000, 0.1266294] FALSE    0.61381924
## s = 7 0.00000000 0.06458195 [0.000000000, 0.1265783] FALSE    0.60874835
## s = 8 0.00000000 0.06447907 [0.000000000, 0.1263767] FALSE    0.66773221

Building a library of learners

In general, we don’t believe that a linear model truly holds. Thinking about potential model misspecification leads us to consider other algorithms. Suppose that I prefer to use generalized additive models (Hastie and Tibshirani 1990) to estimate \(\mu_{P_0}\) and \(\mu_{P_0, s}\), so I am planning on using the gam package. Suppose that you prefer to use the elastic net (Zou and Hastie 2005), and are planning to use the glmnet package.

The choice of either method is somewhat subjective, and I also will have to use a technique like cross-validation to determine an optimal tuning parameter in each case. It is also possible that neither additive models nor the elastic net will do a good job estimating the true conditional means!

This motivates using SuperLearner to allow the data to determine the optimal combination of base learners from a library that I define. These base learners are a combination of different methods (e.g., generalized additive models and elastic net) and instances of the same method with different tuning parameter values (e.g., additive models with 3 and 4 degrees of freedom). The Super Learner is an example of model stacking, or model aggregation — these approaches use a data-adaptive combination of base learners to make predictions.

For instance, my library could include the elastic net, random forests (Breiman 2001), and gradient boosted trees (Friedman 2001) as follows:

# create a function for boosted stumps
SL.gbm.1 <- function(..., interaction.depth = 1) SL.gbm(..., interaction.depth = interaction.depth)

# create GAMs with different degrees of freedom
SL.gam.3 <- function(..., deg.gam = 3) SL.gam(..., deg.gam = deg.gam)
SL.gam.4 <- function(..., deg.gam = 4) SL.gam(..., deg.gam = deg.gam)
SL.gam.5 <- function(..., deg.gam = 5) SL.gam(..., deg.gam = deg.gam)

# add more levels of alpha for glmnet
create.SL.glmnet <- function(alpha = c(0.25, 0.5, 0.75)) {
  for (mm in seq(length(alpha))) {
    eval(parse(file = "", text = paste('SL.glmnet.', alpha[mm], '<- function(..., alpha = ', alpha[mm], ') SL.glmnet(..., alpha = alpha)', sep = '')), envir = .GlobalEnv)
  }
  invisible(TRUE)
}
create.SL.glmnet()

# add tuning parameters for randomForest
create.SL.randomForest <- function(tune = list(mtry = c(1, 5, 7), nodesize = c(1, 5, 10))) {
  tuneGrid <- expand.grid(tune, stringsAsFactors = FALSE)
  for (mm in seq(nrow(tuneGrid))) {
    eval(parse(file = "", text = paste("SL.randomForest.", mm, "<- function(..., mtry = ", tuneGrid[mm, 1], ", nodesize = ", tuneGrid[mm, 2], ") SL.randomForest(..., mtry = mtry, nodesize = nodesize)", sep = "")), envir = .GlobalEnv)
  }
  invisible(TRUE)
}
create.SL.randomForest()

# create the library
learners <- c("SL.glmnet", "SL.glmnet.0.25", "SL.glmnet.0.5", "SL.glmnet.0.75",
              "SL.randomForest", "SL.randomForest.1", "SL.randomForest.2", "SL.randomForest.3",
              "SL.randomForest.4", "SL.randomForest.5", "SL.randomForest.6", "SL.randomForest.7",
              "SL.randomForest.8", "SL.randomForest.9",
              "SL.gbm.1")

Now that I have created the library of learners, I can move on to estimating variable importance.

Estimating variable importance for a single variable

The main function for R-squared-based variable importance in the vimp package is the vimp_rsquared() function. There are five main arguments to vimp_rsquared():

The main arguments differ if precomputed regression function estimates are used; please see “Using precomputed regression function estimates in vimp” for further discussion of this case.

Suppose that the first feature that I want to estimate variable importance for is family history of heart disease, fam. Then supplying vimp_rsquared() with

means that:

The call to vimp_rsquared() looks like this:

vimp_rsquared(Y = heart$chd, X = X,
    indx = 5, run_regression = TRUE, SL.library = learners, V = 5)

While this is the preferred method for estimating variable importance, using a large library of learners may cause the function to take time to run. Usually this is okay — in general, you took a long time to collect the data, so letting an algorithm run for a few hours should not be an issue.

However, for the sake of illustration, I can estimate varibable importance for family history only using only using a small library, a small number of cross-validation folds in the Super Learner, and a small number of cross-fitting folds as follows (again, I suggest using a larger number of folds and a larger library in practice):

# small learners library
learners.2 <- c("SL.ranger")
# small number of cross-fitting folds
V <- 2
# small number of CV folds for Super Learner
sl_cvcontrol <- list(V = 2)

# now estimate variable importance
start_time <- Sys.time()
fam_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 5, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
end_time <- Sys.time()

This code takes approximately 2.153 seconds to run on a (not very fast) PC. I can display these estimates:

fam_vim
## Variable importance estimates:
##       Estimate SE         95% CI         VIMP > 0 p-value  
## s = 5 0        0.05859551 [0, 0.1148451] FALSE    0.9894053

The object returned by vimp_rsquared() also contains fitted values from using SuperLearner(); I access these using $full_fit and $red_fit. For example,

head(fam_vim$full_fit)
## [[1]]
##   [1] 0.6500000 0.4880000 0.2380000 0.6960000 0.4080000 0.7860000 0.3680000
##   [8] 0.8540000 0.1960000 0.3960000 0.2460000 0.1920000 0.1500000 0.5080000
##  [15] 0.1120000 0.1220000 0.2160000 0.6080000 0.0460000 0.2600000 0.5780000
##  [22] 0.2480000 0.0040000 0.5360000 0.4360000 0.5140000 0.0800000 0.2920000
##  [29] 0.5300000 0.6300000 0.5460000 0.1380000 0.2760000 0.4340000 0.0020000
##  [36] 0.7120000 0.1180000 0.0060000 0.5780000 0.4920000 0.7980000 0.2140000
##  [43] 0.2660000 0.1660000 0.4980000 0.1840000 0.3380000 0.0360000 0.6440000
##  [50] 0.1980000 0.7460000 0.3820000 0.6100000 0.4840000 0.1640000 0.0560000
##  [57] 0.5220000 0.0380000 0.6200000 0.3820000 0.4400000 0.0100000 0.3960000
##  [64] 0.3600000 0.2000000 0.2880000 0.1660000 0.6000000 0.0060000 0.0100000
##  [71] 0.0080000 0.1580000 0.2240000 0.2540000 0.1520000 0.2840000 0.1840000
##  [78] 0.3900000 0.2280000 0.4060000 0.2580000 0.3100000 0.3300000 0.1840000
##  [85] 0.3260000 0.2980000 0.3540000 0.4400000 0.5780000 0.2860000 0.2280000
##  [92] 0.3900000 0.7520000 0.5753333 0.6300000 0.2300000 0.0880000 0.6640000
##  [99] 0.5940000 0.6740000 0.0700000 0.2220000 0.1660000 0.3840000 0.0600000
## [106] 0.0920000 0.0460000 0.2240000 0.3260000 0.4380000 0.3140000 0.3260000
## [113] 0.5640000 0.3800000 0.3160000 0.3160000
## 
## [[2]]
##   [1] 0.2180000 0.6960000 0.0960000 0.4640000 0.2480000 0.3960000 0.2640000
##   [8] 0.6960000 0.2740000 0.1260000 0.2040000 0.3120000 0.2940000 0.2660000
##  [15] 0.0160000 0.1880000 0.4280000 0.1300000 0.4760000 0.4400000 0.2840000
##  [22] 0.2180000 0.0820000 0.3640000 0.3640000 0.0440000 0.3100000 0.1580000
##  [29] 0.8380000 0.2260000 0.2400000 0.6080000 0.3120000 0.3900000 0.6500000
##  [36] 0.1700000 0.0120000 0.4460000 0.5220000 0.1200000 0.3260000 0.2560000
##  [43] 0.0060000 0.6480000 0.6620000 0.4000000 0.0340000 0.0280000 0.4060000
##  [50] 0.0760000 0.8220000 0.2980000 0.5480000 0.4320000 0.4540000 0.1580000
##  [57] 0.4120000 0.8100000 0.4720000 0.5440000 0.1340000 0.4660000 0.4540000
##  [64] 0.0900000 0.2540000 0.4440000 0.7100000 0.2680000 0.3060000 0.3720000
##  [71] 0.2380000 0.5300000 0.4180000 0.2820000 0.0960000 0.4420000 0.3140000
##  [78] 0.0860000 0.7680000 0.3840000 0.3740000 0.5460000 0.2800000 0.6320000
##  [85] 0.3040000 0.3860000 0.6740000 0.5120000 0.0220000 0.3300000 0.5540000
##  [92] 0.5540000 0.4280000 0.0820000 0.4720000 0.3800000 0.6240000 0.3242857
##  [99] 0.1980000 0.1200000 0.4160000 0.3600000 0.1060000 0.3100000 0.2680000
## [106] 0.4020000 0.4140000 0.5300000 0.1200000 0.0940000 0.2220000 0.0780000
## [113] 0.2440000 0.1360000 0.7020000 0.3540000
## 
## [[3]]
##   [1] 0.6680 0.4520 0.4660 0.4360 0.0600 0.8020 0.2040 0.4640 0.3520 0.3660
##  [11] 0.1160 0.1840 0.1140 0.6840 0.3860 0.1840 0.2300 0.4860 0.4120 0.6440
##  [21] 0.2100 0.4800 0.1780 0.3920 0.1000 0.1380 0.3560 0.7040 0.4460 0.1040
##  [31] 0.6980 0.4680 0.6360 0.4420 0.2640 0.1280 0.5580 0.5680 0.1920 0.7180
##  [41] 0.1200 0.4520 0.0360 0.7620 0.5500 0.1172 0.7060 0.4440 0.0160 0.2660
##  [51] 0.2560 0.4960 0.1520 0.6480 0.2280 0.3300 0.1680 0.6960 0.4580 0.4040
##  [61] 0.4420 0.6840 0.4320 0.5940 0.0140 0.2880 0.4980 0.6440 0.1500 0.1100
##  [71] 0.6900 0.7340 0.0440 0.2960 0.0100 0.1420 0.0600 0.4700 0.2880 0.5500
##  [81] 0.4040 0.2600 0.6240 0.5200 0.5300 0.0300 0.2420 0.4840 0.1640 0.4220
##  [91] 0.1500 0.2140 0.3020 0.0880 0.6500 0.7800 0.2940 0.5520 0.5380 0.5820
## [101] 0.0720 0.2160 0.4160 0.2620 0.5680 0.6320 0.5480 0.2480 0.1220 0.0620
## [111] 0.0300 0.1580 0.2040 0.3340 0.5580
## 
## [[4]]
##   [1] 0.1920 0.3040 0.0380 0.5300 0.4340 0.7640 0.5540 0.4420 0.6700 0.2500
##  [11] 0.3280 0.3020 0.3140 0.3140 0.2340 0.0925 0.0585 0.1980 0.1285 0.6560
##  [21] 0.3880 0.5700 0.3560 0.4860 0.4280 0.1420 0.1280 0.4960 0.2800 0.4920
##  [31] 0.7200 0.1980 0.1700 0.1240 0.5200 0.1380 0.2000 0.8740 0.1500 0.3580
##  [41] 0.0160 0.2260 0.3480 0.4480 0.2180 0.3140 0.0260 0.1135 0.6100 0.1440
##  [51] 0.2360 0.2500 0.2920 0.2280 0.4720 0.5320 0.6440 0.0640 0.0800 0.2800
##  [61] 0.4460 0.4480 0.2880 0.0160 0.6780 0.3000 0.5800 0.4340 0.3080 0.3880
##  [71] 0.6940 0.7080 0.0765 0.2920 0.5200 0.3485 0.2320 0.0280 0.4920 0.0650
##  [81] 0.1160 0.3880 0.6120 0.5740 0.1340 0.4460 0.6580 0.4560 0.3080 0.5000
##  [91] 0.2800 0.4320 0.5080 0.4880 0.4580 0.3540 0.3420 0.6060 0.6760 0.5520
## [101] 0.4620 0.5760 0.4960 0.5360 0.4940 0.5620 0.1900 0.4360 0.2405 0.0340
## [111] 0.0260 0.0280 0.0260 0.1540 0.1340
head(fam_vim$red_fit)
## [[1]]
##   [1] 0.60330973 0.43751573 0.27145889 0.45316608 0.48293878 0.54442920
##   [7] 0.43720725 0.59464975 0.31213881 0.42809690 0.35664473 0.25326986
##  [13] 0.14032718 0.42222073 0.15371601 0.17191766 0.25320961 0.56254273
##  [19] 0.13203266 0.22741957 0.58414740 0.35785656 0.04634136 0.50408532
##  [25] 0.36434641 0.52636593 0.20288595 0.32883113 0.51962167 0.63835113
##  [31] 0.55999084 0.17067117 0.42430050 0.52292780 0.04509573 0.56713693
##  [37] 0.18702838 0.05183927 0.42591517 0.46276960 0.48536039 0.36263132
##  [43] 0.21988527 0.25043230 0.55279700 0.25081339 0.20088510 0.07670994
##  [49] 0.47575660 0.32232616 0.51383800 0.44866140 0.53789616 0.48331356
##  [55] 0.16787089 0.15518653 0.36922739 0.12215087 0.41959839 0.42107719
##  [61] 0.43424455 0.17958293 0.29973146 0.45001751 0.40176863 0.36224273
##  [67] 0.38606462 0.55365940 0.06525293 0.08536250 0.06263378 0.19904312
##  [73] 0.29843855 0.31992817 0.13222397 0.25845303 0.21001090 0.51298249
##  [79] 0.21052614 0.45226727 0.29384499 0.39705597 0.31175077 0.27981094
##  [85] 0.49830580 0.25001053 0.25431068 0.47720238 0.51069573 0.24152024
##  [91] 0.17933615 0.36246213 0.55819933 0.50031907 0.58242507 0.27215634
##  [97] 0.15809045 0.49646500 0.46498158 0.57513913 0.12021481 0.26285500
## [103] 0.26184816 0.33617577 0.14993953 0.17807090 0.07421892 0.30898874
## [109] 0.31735051 0.54329600 0.24986394 0.31125044 0.50716727 0.51580536
## [115] 0.36757592 0.33324880
## 
## [[2]]
##   [1] 0.42437191 0.42166087 0.11034915 0.53628817 0.26664338 0.56019193
##   [7] 0.47742424 0.57393773 0.14084728 0.14362862 0.31236129 0.31721526
##  [13] 0.38596440 0.18880493 0.04812138 0.45638473 0.41078058 0.05256078
##  [19] 0.46492843 0.52171893 0.46108704 0.21290513 0.22310043 0.45409643
##  [25] 0.45930327 0.18577457 0.36158588 0.24971898 0.58879898 0.41575047
##  [31] 0.18554941 0.39445373 0.28522318 0.58346389 0.32451036 0.18008878
##  [37] 0.04132313 0.59830758 0.41320793 0.14514696 0.29721112 0.22637895
##  [43] 0.04849278 0.40651673 0.40851120 0.66014540 0.13879684 0.06363653
##  [49] 0.52363313 0.13815280 0.51922242 0.37840873 0.54789402 0.57943600
##  [55] 0.56132071 0.38889393 0.52519133 0.58779907 0.28938997 0.61631587
##  [61] 0.13184617 0.34006403 0.57948064 0.14434030 0.25303662 0.45252013
##  [67] 0.57257067 0.29465880 0.43297707 0.36033907 0.37258560 0.46463744
##  [73] 0.50987457 0.37172193 0.18361503 0.42113694 0.27689800 0.14365438
##  [79] 0.55354278 0.54706793 0.42226267 0.47061693 0.36409860 0.48147536
##  [85] 0.33758893 0.41467864 0.55986707 0.38533223 0.13734775 0.44497587
##  [91] 0.39932320 0.62210273 0.29616191 0.20060165 0.27928239 0.33310813
##  [97] 0.32835193 0.33672433 0.25222402 0.24816602 0.44472647 0.47488773
## [103] 0.22254282 0.31701187 0.29165748 0.40074613 0.58059227 0.30186144
## [109] 0.15521216 0.25418054 0.20895696 0.05475050 0.28761327 0.24282338
## [115] 0.44208013 0.43015740
## 
## [[3]]
##   [1] 0.51886213 0.42269027 0.44311863 0.42097641 0.08775118 0.68640580
##   [7] 0.38567227 0.35998449 0.42384362 0.53274313 0.06131857 0.34763320
##  [13] 0.13748328 0.54042713 0.33666000 0.25974166 0.08628498 0.41911114
##  [19] 0.33101043 0.54331782 0.05772015 0.21888174 0.30547037 0.26009764
##  [25] 0.18179839 0.38878107 0.23727769 0.53649911 0.38548718 0.21766214
##  [31] 0.60574720 0.46111682 0.44998460 0.33639220 0.33745086 0.19814248
##  [37] 0.58312175 0.56342873 0.41387230 0.51587540 0.18108673 0.50461171
##  [43] 0.05742022 0.54832385 0.39993243 0.26222228 0.46996698 0.39748883
##  [49] 0.05128998 0.32957914 0.28134991 0.38633118 0.17907480 0.48894244
##  [55] 0.27214778 0.38215857 0.16875248 0.57504793 0.43138046 0.40680677
##  [61] 0.38195078 0.62946244 0.48569321 0.45541271 0.08121655 0.29280507
##  [67] 0.48125267 0.50030933 0.11051915 0.13026557 0.59390120 0.49857053
##  [73] 0.05569837 0.33661873 0.06947750 0.27254522 0.08370582 0.47734205
##  [79] 0.25854425 0.52549333 0.36830842 0.31619340 0.56383251 0.43537553
##  [85] 0.54850904 0.09762280 0.39710900 0.54611167 0.19688535 0.38312842
##  [91] 0.26104218 0.26134510 0.19782715 0.12904413 0.43233847 0.54829153
##  [97] 0.25843992 0.46684123 0.46931578 0.44456338 0.09883917 0.41934371
## [103] 0.48932050 0.25744409 0.36440272 0.51494833 0.34001164 0.30144678
## [109] 0.10116837 0.16362342 0.04455765 0.21164136 0.15217418 0.20862540
## [115] 0.48968624
## 
## [[4]]
##   [1] 0.24318525 0.27168888 0.06791553 0.41096761 0.41596643 0.58930087
##   [7] 0.41769771 0.46090447 0.49960951 0.32958534 0.28412914 0.33479565
##  [13] 0.36729217 0.35418922 0.35689552 0.10278960 0.08794669 0.22925602
##  [19] 0.14529536 0.34744209 0.24985377 0.41863983 0.58322111 0.62031029
##  [25] 0.57121680 0.25373312 0.21557363 0.32174822 0.39187201 0.53394426
##  [31] 0.51542553 0.26847875 0.26356160 0.18582352 0.45294571 0.19105650
##  [37] 0.43787822 0.62515087 0.31439221 0.22131296 0.08579988 0.31440747
##  [43] 0.32640030 0.22369113 0.31179650 0.43604764 0.11352461 0.07149760
##  [49] 0.49464436 0.27355690 0.31410435 0.34157473 0.39964022 0.30217861
##  [55] 0.39890980 0.63995049 0.50031623 0.14610220 0.14853607 0.26198229
##  [61] 0.56244909 0.37178915 0.28645553 0.06631427 0.39800367 0.36144240
##  [67] 0.45469685 0.52593675 0.34153796 0.33798933 0.43995311 0.41494533
##  [73] 0.09274013 0.47213131 0.38566597 0.07528619 0.19508148 0.04278647
##  [79] 0.48228020 0.13411580 0.25901665 0.34908591 0.37152620 0.56274387
##  [85] 0.18740821 0.52141060 0.44128652 0.46919588 0.27361018 0.42742451
##  [91] 0.38143163 0.29662040 0.48132690 0.33626072 0.51412002 0.45460133
##  [97] 0.54358228 0.55396576 0.53727964 0.57786145 0.52649933 0.58069430
## [103] 0.61365660 0.64033193 0.40337173 0.55142553 0.29193827 0.25789120
## [109] 0.13084704 0.07202955 0.04114760 0.06072500 0.09584507 0.36074252
## [115] 0.08962574

I can obtain estimates for the remaining individual features in the same way (again using only using a small library for illustration):

# estimate variable importance
tpa_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 6, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
alc_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 8, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
sbp_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 1, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
tob_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 2, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
ldl_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 3, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
adi_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 4, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
obe_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 7, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
age_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = 9, SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.

Now that I have estimates of each of individual feature’s variable importance, I can view them all simultaneously by plotting:

library("dplyr")
library("tibble")
library("ggplot2")
library("cowplot")
theme_set(theme_cowplot())
# combine the objects together
ests <- merge_vim(sbp_vim, tob_vim, ldl_vim, adi_vim,
                fam_vim, tpa_vim, obe_vim, alc_vim, age_vim)
all_vars <- c("Sys. blood press.", "Tobacco consump.", "LDL cholesterol",
              "Adiposity", "Family history", "Type A behavior", "Obesity",
              "Alcohol consump.", "Age")

est_plot_tib <- ests$mat %>%
  mutate(
    var_fct = rev(factor(s, levels = ests$mat$s,
                     labels = all_vars[as.numeric(ests$mat$s)],
                     ordered = TRUE))
  )

# plot
est_plot_tib %>%
  ggplot(aes(x = est, y = var_fct)) +
  geom_point() +
  geom_errorbarh(aes(xmin = cil, xmax = ciu)) +
  xlab(expression(paste("Variable importance estimates: ", R^2, sep = ""))) +
  ylab("") +
  ggtitle("Estimated individual feature importance") +
  labs(subtitle = "in the South African heart disease study data")

Estimating variable importance for a group of variables

Now that I have estimated variable importance for each of the individual features, I can estimate variable importance for each of the groups that I mentioned above: biological and behavioral features.

The only difference between estimating variable importance for a group of features rather than an individual feature is that now I specify a vector for s; I can use any of the options listed in the previous section to compute these estimates.

# get the estimates
behav_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = c(2, 6, 8), SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)
## Warning in cv_vim(type = "r_squared", Y = Y, X = X, cross_fitted_f1 =
## cross_fitted_f1, : Original estimate < 0; returning zero.
bios_vim <- vimp_rsquared(Y = heart$chd, X = X, indx = c(1, 3, 4, 5, 7, 9), SL.library = learners.2, na.rm = TRUE, env = environment(), V = V, cvControl = sl_cvcontrol)

# combine and plot
groups <- merge_vim(behav_vim, bios_vim)
all_grp_nms <- c("Behavioral features", "Biological features")

grp_plot_tib <- groups$mat %>%
  mutate(
    grp_fct = factor(case_when(
      s == "2,6,8" ~ "1",
      s == "1,3,4,5,7,9" ~ "2"
    ), levels = c("1", "2"),  labels = all_grp_nms, ordered = TRUE)
  )
grp_plot_tib %>%
  ggplot(aes(x = est, y = grp_fct)) +
  geom_point() +
  geom_errorbarh(aes(xmin = cil, xmax = ciu)) +
  xlab(expression(paste("Variable importance estimates: ", R^2, sep = ""))) +
  ylab("") +
  ggtitle("Estimated feature group importance") +
  labs(subtitle = "in the South African heart disease study data")

Types of population variable importance

In this document, I have focused on one particular definition of population variable importance that I call conditional variable importance. For a further discussion of what I call marginal variable importance and Shapley population variable importance, please see “Types of VIMs”.

References

Breiman, L. 2001. “Random Forests” 45.

Friedman, JH. 2001. “Greedy Function Approximation: A Gradient Boosting Machine.”

Hastie, TJ, and RJ Tibshirani. 1990. Generalized Additive Models. Vol. 43. CRC Press.

Hastie, T, R Tibshirani, and J Friedman. 2009. The Elements of Statistical Learning.

van der Laan, MJ, EC Polley, and AE Hubbard. 2007. “Super Learner” 6.

Williamson, Brian D, and Jean Feng. 2020. “Efficient Nonparametric Statistical Inference on Population Feature Importance Using Shapley Values.” https://arxiv.org/abs/2006.09481.

Williamson, Brian D, Peter B Gilbert, Marco Carone, and Noah Simon. 2020. “Nonparametric Variable Importance Assessment Using Machine Learning Techniques.”

Williamson, Brian D, Peter B Gilbert, Noah Simon, and Marco Carone. 2020. “A Unified Approach for Inference on Algorithm-Agnostic Variable Importance.” https://arxiv.org/abs/2004.03683.

Zou, H, and TJ Hastie. 2005. “Regularization and Variable Selection via the Elastic Net.”