In the following, we explain the counterfactuals
workflow for both a classification and a regression task using concrete use cases.
library("counterfactuals")
library("iml")
library("randomForest")
library("mlr3")
library("mlr3learners")
library("data.table")
To illustrate the counterfactuals
workflow for classification tasks, we search for counterfactuals for diabetes-tested patients with MOC
(Dandl et al. 2020).
As training data, we use the Pima Indians Diabetes Database from the mlbench
package. The data set contains 768 observations with 8 features and the binary target variable diabetes
.
Variable | Description |
---|---|
pregnant | Number of times pregnant |
glucose | Plasma glucose concentration (glucose tolerance test) |
pressure | Diastolic blood pressure (mm Hg) |
triceps | Triceps skin fold thickness (mm) |
insulin | 2-Hour serum insulin (mu U/ml) |
mass | Body mass index |
pedigree | Diabetes pedigree function |
age | Age (years) |
diabetes | Class variable (test for diabetes) |
We convert integerish
features to the integer
data type to ensure that the counterfactuals will only contain integer
values for these features (for example no 2.76
pregnancies).
First, we train a model to predict diabetes
, omitting one observation from the training data, which is x_interest
.
An iml::Predictor
object serves as a wrapper for different model types. It contains the model and the data for its analysis.
predictor = iml::Predictor$new(rf, type = "prob")
x_interest = PimaIndiansDiabetes[499L, ]
predictor$predict(x_interest)
#> neg pos
#> 1 0.366 0.634
For x_interest
, the model predicts a diabetes probability of 0.63.
Now, we examine which risk factors need to be changed to reduce the predicted diabetes probability to a maximum of 40%.
Since we want to apply MOC to a classification model, we initialize a MOCClassif
object. Individuals whose prediction is farther away from the desired prediction than epsilon
can be penalized. Here, we set epsilon = 0
, penalizing all individuals whose prediction is outside the desired interval. With the fixed_features
argument, we can fix non-actionable features, here pregnant
and age
.
moc_classif = MOCClassif$new(
predictor, epsilon = 0, fixed_features = c("pregnant", "age"), init_strategy = "icecurve"
)
Then, we use the find_counterfactuals()
method to find counterfactuals for x_interest
. As we aim to find counterfactuals with a predicted diabetes probability of at most 40%, we set the desired_class
to "pos"
and the desired_prob
to c(0, 0.4)
; in the binary classification case, this is equivalent to setting desired_class
to "neg"
and desired_prob
to c(0.6, 1)
.
The resulting Counterfactuals
object holds the counterfactuals in the data
field and possesses several methods for their evaluation and visualization.
Printing a Counterfactuals
object, gives an overview of the results.
print(cfactuals)
#> 422 Counterfactual(s)
#>
#> Desired class: pos
#> Desired predicted probability range: [0, 0.4]
#>
#> Head:
#> pregnant glucose pressure triceps insulin mass pedigree age
#> 1: 7 138 70 31.31761 145 25.1 0.2332576 55
#> 2: 7 138 70 33.00000 145 25.1 0.1630000 55
#> 3: 7 138 70 31.31761 145 25.1 0.1630000 55
The predict()
method returns the predictions for the counterfactuals.
head(cbind(cfactuals$data, cfactuals$predict()), 5L)
#> pregnant glucose pressure triceps insulin mass pedigree age neg pos
#> 1: 7 138 70 31.31761 145 25.1 0.2332576 55 0.700 0.300
#> 2: 7 138 70 33.00000 145 25.1 0.1630000 55 0.728 0.272
#> 3: 7 138 70 31.31761 145 25.1 0.1630000 55 0.720 0.280
#> 4: 7 124 70 33.00000 145 25.1 0.1630000 55 0.792 0.208
#> 5: 7 138 70 33.00000 145 25.1 0.2798642 55 0.694 0.306
The evaluate()
method returns the counterfactuals along with the evaluation measures dist_x_interest
, dist_target
, no_changed
, and dist_train
.
Setting the show_diff
argument to TRUE
displays the counterfactuals as their difference to x_interest
: for a numeric feature, positive values indicate an increase compared to the feature value in x_interest
and negative values indicate a decrease; for factors, the counterfactual feature value is displayed if it differs from x_interest.
; NA
means “no difference” in both cases.
head(cfactuals$evaluate(show_diff = TRUE, measures = c("dist_x_interest", "dist_target", "no_changed", "dist_train")), 5L)
#> pregnant glucose pressure triceps insulin mass pedigree age dist_x_interest no_changed dist_train dist_target
#> 1: NA -57 NA -1.682387 NA NA 0.07025758 NA 0.04167812 3 0.04563903 0
#> 2: NA -57 NA NA NA NA NA NA 0.03580402 1 0.05151312 0
#> 3: NA -57 NA -1.682387 NA NA NA NA 0.03792825 2 0.04938890 0
#> 4: NA -71 NA NA NA NA NA NA 0.04459799 1 0.04869469 0
#> 5: NA -57 NA NA NA NA 0.11686417 NA 0.04204143 2 0.04527571 0
The plot_freq_of_feature_changes()
method plots the frequency of feature changes across all counterfactuals.
Setting subset_zero = TRUE
removes all unchanged features from the plot.
The parallel plot connects the (scaled) feature values of each counterfactual and highlights x_interest
in blue.
The white dot in the prediction surface plot represents x_interest
. All counterfactuals that differ from x_interest
only in the selected features are displayed as black dots. The tick marks next to the axes indicate the marginal distribution of the counterfactuals.
Additional diagnostic tools for MOC are available as part of the MOCClassif and MOCRegr class. For example, the hypervolume indicator (Zitzler and Thiele 1998) given a reference point (that represents the maximal values of the objectives) could be computed. The evolution of the hypervolume indicator can be plotted together with the evolution of mean and minimum objective values using the plot_statistics()
method.
[[1]] [[2]]
[[3]]
Ideally, one would like the mean value of each objective to decrease over the generations, leading to an increase of the hypervolume. We could visualize the objective values of the emerging candidates throughout the generations via the plot_search
method for pairs of objectives.
Finding counterfactuals for regression models is analogous to classification models. In this example, we use NICE
(Brughmans et al. (2022)) to search for counterfactuals for housing prices. Brughmans et al. introduced NICE
only for the classification setting but for this package the method was extended to also work for regression tasks by allowing prediction functions to return real-valued outcomes instead of classification scores.
As training data, we use the Boston Housing dataset from the mlbench
package. The dataset contains 506 observations with 13 features and the (continuous) target variable medv
.
First, we train a model to predict medv
, again omitting x_interest
from the training data. This time we use a support vector machine trained with the mlr3
package.
Then, we initialize an iml::Predictor
object.
predictor = iml::Predictor$new(model, data = BostonHousing, y = "medv")
x_interest = BostonHousing[1L, ]
predictor$predict(x_interest)
#> predict.model..newdata...newdata.
#> 1 27.49074
For x_interest
, the model predicts a median housing value of 27.49.
Since we want to apply NICE
to a regression model, we initialize a NICERegr
object. For regression models, we define a correctly predicted datapoint when its prediction is less than a user-specified value away. Here we allow for a deviation of margin_correct = 0.5
. In this example, we aim for plausible counterfactuals in additional to sparse ones, such that we set optimization = "plausiblity"
.
nice_regr = NICERegr$new(predictor, optimization = "plausibility",
margin_correct = 0.5, return_multiple = FALSE)
Then, we use the find_counterfactuals()
method to find counterfactuals for x_interest
with a predicted housing value in the interval [30, 32].
As a result, we obtain a Counterfactuals
object, just like for the classification task.
At the beginning, NICE
calculates the distance of x_interest
to each of the training samples. By default, Gower’s distance measures this but users could also specify their own distance functions in the distance_function
argument. For example, the Gower distance can be replaces by the L_0 norm.
l0_norm = function(x, y, data) {
res = matrix(NA, nrow = nrow(x), ncol = nrow(y))
for (i in seq_len(nrow(x))) {
for (j in seq_len(nrow(y))) {
res[i, j] = sum(x[i,] != y[j,])
}
}
res
}
A short example illustrates the functionality of l0_norm()
.
xt = data.table::data.table(a = c(0.5), b = c("a"))
yt = data.table::data.table(a = c(0.5, 3.2, 0.1), b = c("a", "b", "a"))
l0_norm(xt, yt, data = NULL)
#> [,1] [,2] [,3]
#> [1,] 0 2 1
Replacing the distance function is fairly easy:
nice_regr = NICERegr$new(predictor, optimization = "plausibility",
margin_correct = 0.5, return_multiple = FALSE,
distance_function = l0_norm)
cfactuals = nice_regr$find_counterfactuals(x_interest, desired_outcome = c(30, 40))
cfactuals
#> 1 Counterfactual(s)
#>
#> Desired outcome range: [30, 40]
#>
#> Head:
#> crim zn indus chas nox rm age dis rad tax ptratio b lstat
#> 1: 0.04011 80 1.52 0 0.404 7.287 34.1 7.309 2 329 12.6 396.9 4.08
Dandl, Susanne, Christoph Molnar, Martin Binder, and Bernd Bischl. 2020. “Multi-Objective Counterfactual Explanations.” In Parallel Problem Solving from Nature – PPSN XVI, edited by Thomas Bäck, Mike Preuss, André Deutz, Hao Wang, Carola Doerr, Michael Emmerich, and Heike Trautmann, 448–469. Cham: Springer International Publishing. .
Brughmans D, Martens D (2022). “NICE: An Algorithm for Nearest Instance Counterfactual Explanations.” Technical report, <arXiv:2104.07411> v2.
Zitzler, Eckart, and Lothar Thiele. 1998. “Multiobjective Optimization Using Evolutionary Algorithms—a Comparative Case Study.” In International Conference on Parallel Problem Solving from Nature, 292–301. Springer.