Last updated on 2025-10-09 13:49:41 CEST.
| Flavor | Version | Tinstall | Tcheck | Ttotal | Status | Flags |
|---|---|---|---|---|---|---|
| r-devel-linux-x86_64-debian-clang | 1.2.1 | 9.13 | 202.04 | 211.17 | ERROR | |
| r-devel-linux-x86_64-debian-gcc | 1.2.1 | 6.57 | 141.76 | 148.33 | ERROR | |
| r-devel-linux-x86_64-fedora-clang | 1.2.1 | 331.75 | ERROR | |||
| r-devel-linux-x86_64-fedora-gcc | 1.2.1 | 316.83 | ERROR | |||
| r-devel-windows-x86_64 | 1.2.1 | 11.00 | 188.00 | 199.00 | ERROR | |
| r-patched-linux-x86_64 | 1.2.1 | 9.44 | 194.85 | 204.29 | ERROR | |
| r-release-linux-x86_64 | 1.2.1 | 8.10 | 193.55 | 201.65 | ERROR | |
| r-release-macos-arm64 | 1.2.1 | 111.00 | NOTE | |||
| r-release-macos-x86_64 | 1.2.1 | 213.00 | NOTE | |||
| r-release-windows-x86_64 | 1.2.1 | 11.00 | 190.00 | 201.00 | ERROR | |
| r-oldrel-macos-arm64 | 1.2.1 | 99.00 | NOTE | |||
| r-oldrel-macos-x86_64 | 1.2.1 | 192.00 | NOTE | |||
| r-oldrel-windows-x86_64 | 1.2.1 | 16.00 | 243.00 | 259.00 | ERROR |
Version: 1.2.1
Check: CRAN incoming feasibility
Result: NOTE
Maintainer: ‘Jakub Wiśniewski <jakwisn@gmail.com>’
The Description field contains
<arXiv:2104.00507>.
Please refer to arXiv e-prints via their arXiv DOI <doi:10.48550/arXiv.YYMM.NNNNN>.
Flavors: r-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc
Version: 1.2.1
Check: Rd files
Result: NOTE
checkRd: (-1) choose_metric.Rd:35: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) choose_metric.Rd:36: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) choose_metric.Rd:37: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) confusion_matrx.Rd:20: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) confusion_matrx.Rd:21: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) confusion_matrx.Rd:22: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) confusion_matrx.Rd:23: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) disparate_impact_remover.Rd:28: Lost braces
28 | pigeonholing. The number of pigeonholes is fixed and equal to min{101, unique(a)}, where a is vector with values for subgroup. So if some subgroup is not numerous and
| ^
checkRd: (-1) fairness_check.Rd:47: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:48: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:49: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:50: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:51: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:52: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:53: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:54: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:55: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:56: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:57: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:58: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:61: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:62: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:63: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:64: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:65: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_check.Rd:66: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_heatmap.Rd:12: Lost braces
12 | \item{scale}{logical, if code{TRUE} metrics will be scaled to mean 0 and sd 1. Default \code{FALSE}}
| ^
checkRd: (-1) fairness_heatmap.Rd:19: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_heatmap.Rd:20: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_heatmap.Rd:21: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_heatmap.Rd:22: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_pca.Rd:18: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_pca.Rd:19: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_pca.Rd:20: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_pca.Rd:21: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_pca.Rd:22: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_radar.Rd:18: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) fairness_radar.Rd:19: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_matrices.Rd:25: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_matrices.Rd:26: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_matrices.Rd:27: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_matrices.Rd:28: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_metric.Rd:30: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_metric.Rd:31: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_metric.Rd:32: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) group_metric.Rd:33: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) metric_scores.Rd:18: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) metric_scores.Rd:19: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) performance_and_fairness.Rd:20: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) performance_and_fairness.Rd:21: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) performance_and_fairness.Rd:22: Lost braces in \itemize; \value handles \item{}{} directly
checkRd: (-1) performance_and_fairness.Rd:23: Lost braces in \itemize; \value handles \item{}{} directly
Flavors: r-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc, r-devel-linux-x86_64-fedora-clang, r-devel-linux-x86_64-fedora-gcc, r-devel-windows-x86_64, r-patched-linux-x86_64, r-release-linux-x86_64, r-release-macos-arm64, r-release-macos-x86_64, r-release-windows-x86_64, r-oldrel-macos-arm64, r-oldrel-macos-x86_64, r-oldrel-windows-x86_64
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in ‘fairmodels-Ex.R’ failed
The error most likely occurred in:
> base::assign(".ptime", proc.time(), pos = "CheckExEnv")
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 4.352454e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.07287302 , mean = 0.6989152 , max = 0.9974848
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7219256 , mean = 0.001084826 , max = 0.6142332
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavors: r-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc
Version: 1.2.1
Check: tests
Result: ERROR
Running ‘testthat.R’ [36s/40s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1575654 , mean = 0.5448133 , max = 0.8645892
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8479717 , mean = 6.679325e-05 , max = 0.7820503
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -174.6472 , mean = 742.9023 , max = 1690.151
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -307.5534 , mean = -6.003371e-13 , max = 302.2034
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -174.6472 , mean = 742.9023 , max = 1690.151
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -307.5534 , mean = -6.003371e-13 , max = 302.2034
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6343983
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at tests/testthat/helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-devel-linux-x86_64-debian-clang
Version: 1.2.1
Check: re-building of vignette outputs
Result: ERROR
Error(s) in re-building vignettes:
...
--- re-building ‘Advanced_tutorial.Rmd’ using rmarkdown
--- finished re-building ‘Advanced_tutorial.Rmd’
--- re-building ‘Basic_tutorial.Rmd’ using rmarkdown
Quitting from Basic_tutorial.Rmd:254-257 [unnamed-chunk-19]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<error/rlang_error>
Error in `rep()`:
! attempt to replicate an object of type 'object'
---
Backtrace:
▆
1. ├─base::plot(fheatmap, text_size = 3)
2. └─fairmodels:::plot.fairness_heatmap(fheatmap, text_size = 3)
3. └─base::ifelse(...)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error: processing vignette 'Basic_tutorial.Rmd' failed with diagnostics:
attempt to replicate an object of type 'object'
--- failed re-building ‘Basic_tutorial.Rmd’
SUMMARY: processing the following file failed:
‘Basic_tutorial.Rmd’
Error: Vignette re-building failed.
Execution halted
Flavors: r-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc, r-patched-linux-x86_64, r-release-linux-x86_64
Version: 1.2.1
Check: tests
Result: ERROR
Running ‘testthat.R’ [22s/25s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1661123 , mean = 0.5447374 , max = 0.8744903
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.852705 , mean = 0.0001427116 , max = 0.7740508
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -95.73285 , mean = 750.892 , max = 1547.596
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -314.4113 , mean = -9.066226e-14 , max = 269.3932
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -95.73285 , mean = 750.892 , max = 1547.596
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -314.4113 , mean = -9.066226e-14 , max = 269.3932
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 6/13 metrics calculated for all models ( <1b>[33m7 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6335761
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at tests/testthat/helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-devel-linux-x86_64-debian-gcc
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in ‘fairmodels-Ex.R’ failed
The error most likely occurred in:
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 1.280546e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.06718651 , mean = 0.6975836 , max = 0.9967857
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7225752 , mean = 0.002416363 , max = 0.634748
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavor: r-devel-linux-x86_64-fedora-clang
Version: 1.2.1
Check: tests
Result: ERROR
Running ‘testthat.R’ [61s/91s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1642965 , mean = 0.5447758 , max = 0.8658543
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8494785 , mean = 0.0001043454 , max = 0.7798716
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -165.8516 , mean = 756.1278 , max = 1708.849
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -379.0468 , mean = 4.644133e-13 , max = 268.9282
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -165.8516 , mean = 756.1278 , max = 1708.849
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -379.0468 , mean = 4.644133e-13 , max = 268.9282
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6343983
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 327.8173 , mean = 756.3617 , max = 1159.246
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -635.2394 , mean = 0.1289064 , max = 608.1891
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at tests/testthat/helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-devel-linux-x86_64-fedora-clang
Version: 1.2.1
Check: re-building of vignette outputs
Result: ERROR
Error(s) in re-building vignettes:
--- re-building ‘Advanced_tutorial.Rmd’ using rmarkdown
--- finished re-building ‘Advanced_tutorial.Rmd’
--- re-building ‘Basic_tutorial.Rmd’ using rmarkdown
Quitting from Basic_tutorial.Rmd:254-257 [unnamed-chunk-19]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<error/rlang_error>
Error in `rep()`:
! attempt to replicate an object of type 'object'
---
Backtrace:
▆
1. ├─base::plot(fheatmap, text_size = 3)
2. └─fairmodels:::plot.fairness_heatmap(fheatmap, text_size = 3)
3. └─base::ifelse(...)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error: processing vignette 'Basic_tutorial.Rmd' failed with diagnostics:
attempt to replicate an object of type 'object'
--- failed re-building ‘Basic_tutorial.Rmd’
SUMMARY: processing the following file failed:
‘Basic_tutorial.Rmd’
Error: Vignette re-building failed.
Execution halted
Flavors: r-devel-linux-x86_64-fedora-clang, r-devel-linux-x86_64-fedora-gcc, r-devel-windows-x86_64, r-release-windows-x86_64, r-oldrel-windows-x86_64
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in ‘fairmodels-Ex.R’ failed
The error most likely occurred in:
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 1.280546e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.07287302 , mean = 0.6989152 , max = 0.9974848
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7219256 , mean = 0.001084826 , max = 0.6142332
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavor: r-devel-linux-x86_64-fedora-gcc
Version: 1.2.1
Check: tests
Result: ERROR
Running ‘testthat.R’ [55s/65s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.167971 , mean = 0.5447203 , max = 0.8624487
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8480142 , mean = 0.0001598289 , max = 0.7760066
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -109.083 , mean = 754.6519 , max = 1726.226
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -349.5516 , mean = 2.612e-14 , max = 310.2391
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -109.083 , mean = 754.6519 , max = 1726.226
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -349.5516 , mean = 2.612e-14 , max = 310.2391
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6333333
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at tests/testthat/helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-devel-linux-x86_64-fedora-gcc
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in 'fairmodels-Ex.R' failed
The error most likely occurred in:
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 6.648002e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.07287302 , mean = 0.6989152 , max = 0.9974848
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7219256 , mean = 0.001084826 , max = 0.6142332
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavor: r-devel-windows-x86_64
Version: 1.2.1
Check: tests
Result: ERROR
Running 'testthat.R' [28s]
Running the tests in 'tests/testthat.R' failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1575998 , mean = 0.5446366 , max = 0.8628671
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8453535 , mean = 0.0002435264 , max = 0.7813384
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -54.02757 , mean = 744.8297 , max = 1647.915
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -382.3527 , mean = -1.179752e-13 , max = 283.2315
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -54.02757 , mean = 744.8297 , max = 1647.915
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -382.3527 , mean = -1.179752e-13 , max = 283.2315
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6449945
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.6.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at D:\RCompile\CRANpkg\local\4.6\fairmodels.Rcheck\tests\testthat\helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-devel-windows-x86_64
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in ‘fairmodels-Ex.R’ failed
The error most likely occurred in:
> base::assign(".ptime", proc.time(), pos = "CheckExEnv")
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 4.352454e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.07287302 , mean = 0.6989152 , max = 0.9974848
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7219256 , mean = 0.001084826 , max = 0.6142332
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavors: r-patched-linux-x86_64, r-release-linux-x86_64
Version: 1.2.1
Check: tests
Result: ERROR
Running ‘testthat.R’ [35s/38s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1567624 , mean = 0.5446174 , max = 0.8662601
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8478945 , mean = 0.0002626889 , max = 0.7727163
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -282.7001 , mean = 754.0042 , max = 1623.082
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -369.9672 , mean = 7.268287e-14 , max = 342.6273
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -282.7001 , mean = 754.0042 , max = 1623.082
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -369.9672 , mean = 7.268287e-14 , max = 342.6273
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6451136
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at tests/testthat/helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 1 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-patched-linux-x86_64
Version: 1.2.1
Check: tests
Result: ERROR
Running ‘testthat.R’ [35s/43s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1569291 , mean = 0.5449904 , max = 0.868545
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8542404 , mean = -0.0001103223 , max = 0.7871844
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -363.7743 , mean = 750.5251 , max = 1616.219
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -323.1543 , mean = -5.81169e-14 , max = 358.6111
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -363.7743 , mean = 750.5251 , max = 1616.219
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -323.1543 , mean = -5.81169e-14 , max = 358.6111
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 6/13 metrics calculated for all models ( <1b>[33m7 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6338001
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at tests/testthat/helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-release-linux-x86_64
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in 'fairmodels-Ex.R' failed
The error most likely occurred in:
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 6.648002e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.07287302 , mean = 0.6989152 , max = 0.9974848
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7219256 , mean = 0.001084826 , max = 0.6142332
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavor: r-release-windows-x86_64
Version: 1.2.1
Check: tests
Result: ERROR
Running 'testthat.R' [29s]
Running the tests in 'tests/testthat.R' failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1550912 , mean = 0.5446626 , max = 0.8681592
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8485889 , mean = 0.0002174809 , max = 0.7761809
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -165.7584 , mean = 748.8188 , max = 1712.873
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -287.1747 , mean = -1.477078e-12 , max = 292.0534
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -165.7584 , mean = 748.8188 , max = 1712.873
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -287.1747 , mean = -1.477078e-12 , max = 292.0534
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 6/13 metrics calculated for all models ( <1b>[33m7 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6342211
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.5.1 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at D:\RCompile\CRANpkg\local\4.5\fairmodels.Rcheck\tests\testthat\helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-release-windows-x86_64
Version: 1.2.1
Check: examples
Result: ERROR
Running examples in 'fairmodels-Ex.R' failed
The error most likely occurred in:
> ### Name: fairness_heatmap
> ### Title: Fairness heatmap
> ### Aliases: fairness_heatmap
>
> ### ** Examples
>
>
> data("german")
>
> y_numeric <- as.numeric(german$Risk) - 1
>
> lm_model <- glm(Risk ~ .,
+ data = german,
+ family = binomial(link = "logit")
+ )
>
> rf_model <- ranger::ranger(Risk ~ .,
+ data = german,
+ probability = TRUE,
+ num.trees = 200,
+ num.threads = 1
+ )
>
> explainer_lm <- DALEX::explain(lm_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1369187 , mean = 0.7 , max = 0.9832426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9572803 , mean = 6.648002e-17 , max = 0.8283475
<1b>[32m A new explainer has been created! <1b>[39m
> explainer_rf <- DALEX::explain(rf_model, data = german[, -1], y = y_numeric)
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 9 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.07287302 , mean = 0.6989152 , max = 0.9974848
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.7219256 , mean = 0.001084826 , max = 0.6142332
<1b>[32m A new explainer has been created! <1b>[39m
>
> fobject <- fairness_check(explainer_lm, explainer_rf,
+ protected = german$Sex,
+ privileged = "male"
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
> # same explainers with different cutoffs for female
> fobject <- fairness_check(explainer_lm, explainer_rf, fobject,
+ protected = german$Sex,
+ privileged = "male",
+ cutoff = list(female = 0.4),
+ label = c("lm_2", "rf_2")
+ )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : female: 0.4, male: 0.5
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 10/13 metrics calculated for all models ( <1b>[33m3 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
>
>
> fh <- fairness_heatmap(fobject)
>
> plot(fh)
Error in rep(yes, length.out = len) :
attempt to replicate an object of type 'object'
Calls: plot -> plot.fairness_heatmap -> ifelse
Execution halted
Flavor: r-oldrel-windows-x86_64
Version: 1.2.1
Check: tests
Result: ERROR
Running 'testthat.R' [44s]
Running the tests in 'tests/testthat.R' failed.
Complete output:
> library(testthat)
> library(fairmodels)
>
>
> test_check("fairmodels")
Welcome to DALEX (version: 2.5.2).
Find examples and detailed introduction at: http://ema.drwhy.ai/
Additional features will be available after installation of: ggpubr.
Use 'install_dependencies()' to get all suggested dependencies
Loaded gbm 2.2.2
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1587974 , mean = 0.5448854 , max = 0.8728624
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8513548 , mean = -5.30918e-06 , max = 0.781337
<1b>[32m A new explainer has been created! <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.004522979 , mean = 0.5448801 , max = 0.8855426
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.8822826 , mean = -5.053611e-13 , max = 0.9767658
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
Fairness object created succesfully
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 4 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 3 in total ( <1b>[31m model type not supported <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -59.53575 , mean = 756.8124 , max = 1513.47
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -367.1213 , mean = 1.359858e-13 , max = 323.5572
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
Fairness regression object created succesfully
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from numeric <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[31m model type not supported <1b>[39m )
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -59.53575 , mean = 756.8124 , max = 1513.47
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -367.1213 , mean = 1.359858e-13 , max = 323.5572
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[31m not compatible <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[31m y not equal <1b>[39m )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 11/13 metrics calculated for all models ( <1b>[33m2 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 2 objects ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Performace metric not given, setting deafult ( accuracy )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Fairness Metric not given, setting deafult ( TPR )
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 6/13 metrics calculated for all models ( <1b>[33m7 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Fairness Metric not given, setting deafult ( TPR )
Performace metric not given, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Creating object with:
Fairness metric: FPR
Performance metric: f1
Fairness data top rows for FPR
group score model
1 African_American 0.35204756 lm
2 Asian 0.04347826 lm
3 Caucasian 0.16393443 lm
4 Hispanic 0.11562500 lm
5 Native_American 0.16666667 lm
6 Other 0.07762557 lm
Performance data for f1 :
1 lm 0.6039853
2 ranger 0.6443375
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Performace metric is NULL, setting deafult ( accuracy )
Creating object with:
Fairness metric: non_existing
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: non_existing
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: auc
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: accuracy
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: precision
Fairness Metric is NULL, setting deafult parity loss metric ( TPR )
Creating object with:
Fairness metric: TPR
Performance metric: recall
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 6172 rows 7 cols
-> target variable : 6172 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 0.1144574 , mean = 0.4551199 , max = 0.995477
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -0.9767658 , mean = 5.053909e-13 , max = 0.8822826
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 8/13 metrics calculated for all models ( <1b>[33m5 NA created<1b>[39m )
<1b>[32m Fairness object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.lm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = -119.546 , mean = 756.4906 , max = 1594.562
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -302.6659 , mean = 3.478115e-13 , max = 332.7938
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[33m changed from character <1b>[39m )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Preparation of a new explainer is initiated
-> model label : ranger ( <1b>[33m default <1b>[39m )
-> data : 1000 rows 3 cols
-> target variable : 1000 values
-> predict function : yhat.ranger will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package ranger , ver. 0.17.0 , task regression ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 361.6527 , mean = 756.1869 , max = 1136.792
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -669.0748 , mean = 0.3037205 , max = 630.6428
<1b>[32m A new explainer has been created! <1b>[39m
Creating fairness regression object
-> Privileged subgroup : character (<1b>[33m from first fairness object <1b>[39m )
-> Protected variable : factor (<1b>[33m from first fairness object <1b>[39m )
-> Fairness objects : 1 object ( <1b>[32m compatible <1b>[39m )
-> Checking explainers : 2 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 3/3 metrics calculated for all models
<1b>[32m Fairness regression object created succesfully <1b>[39m
Creating fairness classification object
-> Privileged subgroup : character (<1b>[32m Ok <1b>[39m )
-> Protected variable : factor (<1b>[32m Ok <1b>[39m )
-> Cutoff values for explainers : 0.5 ( for all subgroups )
-> Fairness objects : 0 objects
-> Checking explainers : 1 in total ( <1b>[32m compatible <1b>[39m )
-> Metric calculation : 13/13 metrics calculated for all models
<1b>[32m Fairness object created succesfully <1b>[39m
changing protected to factor
Preparation of a new explainer is initiated
-> model label : lm ( <1b>[33m default <1b>[39m )
-> data : 15 rows 2 cols
-> target variable : 15 values
-> predict function : yhat.glm will be used ( <1b>[33m default <1b>[39m )
-> predicted values : No value for predict function target column. ( <1b>[33m default <1b>[39m )
-> model_info : package stats , ver. 4.4.3 , task classification ( <1b>[33m default <1b>[39m )
-> predicted values : numerical, min = 7.884924e-12 , mean = 0.4666667 , max = 1
-> residual function : difference between y and yhat ( <1b>[33m default <1b>[39m )
-> residuals : numerical, min = -7.884924e-12 , mean = -5.256659e-13 , max = 7.884915e-12
<1b>[32m A new explainer has been created! <1b>[39m
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test_heatmap.R:2:3'): Test heatmap ──────────────────────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::plot(fairness_heatmap(fobject)) at test_heatmap.R:2:3
2. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(fobject))
3. └─base::ifelse(...)
── Failure ('test_plot_density.R:14:3'): Test plot_density ─────────────────────
plt$labels$x not equal to "probability".
target is NULL, current is character
── Error ('test_plot_fairmodels.R:8:3'): Test plot_fairmodels ──────────────────
Error in `rep(yes, length.out = len)`: attempt to replicate an object of type 'object'
Backtrace:
▆
1. ├─base::suppressWarnings(...) at test_plot_fairmodels.R:8:3
2. │ └─base::withCallingHandlers(...)
3. ├─fairmodels:::expect_s3_class(...)
4. │ ├─testthat::expect(...) at D:\RCompile\CRANpkg\local\4.4\fairmodels.Rcheck\tests\testthat\helper_objects.R:70:20
5. │ └─base::class(object) %in% class
6. ├─fairmodels::plot_fairmodels(fc, type = "fairness_heatmap")
7. └─fairmodels:::plot_fairmodels.fairness_object(fc, type = "fairness_heatmap")
8. └─fairmodels:::plot_fairmodels.default(x, type, ...)
9. └─fairmodels:::plot.fairness_heatmap(fairness_heatmap(x, ...))
10. └─base::ifelse(...)
[ FAIL 3 | WARN 2 | SKIP 0 | PASS 299 ]
Error: Test failures
Execution halted
Flavor: r-oldrel-windows-x86_64
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.