The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
run_workflow() chains the data, preprocessing,
selection, modelling, evaluation, and result layers into one object. The
held-out validation and test sets are transformed with parameters
learned from the training set; they are never used to fit preprocessing,
selection, calibration, or thresholds.
The vignette uses the bundled cardiovascular example cohort. It is deliberately trimmed here so the vignette builds quickly; the full cohort has 2000 rows per data set.
library(uniml)
development <- read.csv(
system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)
set.seed(123)
development <- development[1:800, ]
new_admissions <- new_admissions[1:400, ]
spec <- uniml_spec(
task_type = "binary",
outcome = "died",
positive = "yes",
id = "id"
)
spec <- set_preprocess(
spec,
impute = "median_mode",
scale = "zscore",
encode = "onehot"
)
spec <- set_feature_selection(
spec,
methods = c("univariate", "lasso"),
combine = "intersect"
)
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_evaluation(
spec,
ci = "bootstrap",
threshold_method = "youden",
calibration = "platt",
select_on = "validation"
)
# Kernel SHAP is the most expensive step of a workflow, so its two size knobs are
# part of the specification. They are small here to keep the vignette quick; a
# paper-quality explanation wants a few hundred rows on both sides.
spec <- set_explain(spec, n_explain = 20, bg_n = 20)
spec
#> <uniml_spec>
#> task_type: binary
#> outcome: died
#> positive: yes
#> leakage_policy: custom
#> in_cv: selection, balance, tuning
#> selection: univariate, lasso
#> verbose: 1, n_workers: 1The built-in learner keys are lr, rf,
xgb, nnet, knn,
cart, and nb, plus the optional
log_reg, svm, lda, and
qda. Any installed mlr3 classification learner can also be
passed directly, such as classif.log_reg,
classif.svm, or classif.lda. Learners from
packages that are not on CRAN (for example classif.lightgbm
from mlr3extralearners) work the same way once that package
is installed and loaded; external learners use their mlr3 defaults
unless a custom search space is registered.
make_split() returns a uniml_split whose
elements are uniml_data objects. Pass the
$data component to run_workflow(). The
independent new-admission cohort stays outside the development
split.
split <- make_split(
as_uniml_data(development, spec),
prop = c(0.8, 0.2),
stratify = TRUE,
seed = 123
)
data.frame(
set = c("train", "validation"),
n = c(nrow(split$train$data), nrow(split$validation$data)),
events = c(
sum(split$train$data$died == "yes"),
sum(split$validation$data$died == "yes")
)
)
#> set n events
#> 1 train 641 45
#> 2 validation 159 11The first 200 new admissions play the role of an independent test set, the next 200 the role of an external cohort. Both are reporting-only.
result <- run_workflow(
spec,
train = split$train$data,
validation = split$validation$data,
test = new_admissions[1:200, ],
external = new_admissions[201:400, ],
ci_R = 25
)
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: f1, f2,
#> calib_slope, calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: f1, f2,
#> calib_slope, calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_large.
result
#> <uniml_result>
#> data_info (list, 7 elements)
#> preprocess (list, 4 elements)
#> selection (list, 8 elements)
#> tuning (list, 2 elements)
#> cv (list, 6 elements)
#> metrics (16 x 24)
#> ci (336 x 10)
#> calibration (list, 4 elements)
#> tests (list, 3 elements)
#> dca (1188 x 6)
#> ranking (2 x 6)
#> shap (list, 12 elements)
#> plots (list, 4 elements)
#> models (list, 2 elements)
#> meta (list, 13 elements)result$metrics is a wide table with one row per model,
set, and probability track (raw or calibrated):
metrics <- result$metrics
metrics[, c("model", "set", "probability", "auc", "brier", "accuracy")]
#> model set probability auc brier accuracy
#> 1 lr train raw 0.9035421 0.05103773 0.7722309
#> 2 lr train calibrated 0.9035421 0.04995033 0.7722309
#> 3 lr validation raw 0.9066339 0.05000148 0.6918239
#> 4 lr validation calibrated 0.9066339 0.04755764 0.6918239
#> 5 lr test raw 0.9075521 0.03376129 0.7650000
#> 6 lr test calibrated 0.9075521 0.03610441 0.7650000
#> 7 lr external raw 0.8864839 0.04105926 0.7700000
#> 8 lr external calibrated 0.8864839 0.04096392 0.7700000
#> 9 cart train raw 0.6836689 0.04752045 0.9438378
#> 10 cart train calibrated 0.6836689 0.05060635 0.9438378
#> 11 cart validation raw 0.5816953 0.05935709 0.9245283
#> 12 cart validation calibrated 0.5816953 0.05848193 0.9245283
#> 13 cart test raw 0.5530599 0.04209931 0.9450000
#> 14 cart test calibrated 0.5530599 0.04046960 0.9450000
#> 15 cart external raw 0.6269841 0.04804633 0.9400000
#> 16 cart external calibrated 0.6269841 0.04827002 0.9400000Bootstrap confidence intervals are stored separately so that they can be exported on their own:
head(result$ci[result$ci$set == "validation", ], 4)
#> model set probability metric mean se lower
#> 43 lr validation raw auc 0.9170925 0.03262267 0.85815355
#> 44 lr validation raw pr_auc 0.5428291 0.13539597 0.24476019
#> 45 lr validation raw brier 0.0480432 0.01246494 0.03152822
#> 46 lr validation raw accuracy 0.6948428 0.03358161 0.62515723
#> upper n_effective formatted
#> 43 0.97748263 25 0.917 (0.858-0.977)
#> 44 0.73160017 25 0.543 (0.245-0.732)
#> 45 0.07209281 25 0.048 (0.032-0.072)
#> 46 0.75094340 25 0.695 (0.625-0.751)The ranking selects on the comparison set (validation by default) and never on the test or external cohorts:
result$ranking[, c("model", "score", "rank", "metric", "set")]
#> model score rank metric set
#> 1 lr 0.7264577 1 composite validation
#> 2 cart 0.4482745 2 composite validationCross-validation output is available per model and metric:
All plotting helpers accept palette, theme,
title, subtitle, caption, and
set, so figures can be styled for a manuscript without
touching the result.
plot_calibration(result, set = "validation")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?The three held-out sets answer different questions, so they are reported separately and never pooled. Model selection uses the validation set only; the test set and the external cohort are reporting-only.
role_table <- subset(
result$metrics,
set %in% c("validation", "test", "external"),
select = c("model", "set", "probability", "auc", "brier", "sensitivity",
"specificity")
)
role_table[role_table$probability == "calibrated", ]
#> model set probability auc brier sensitivity specificity
#> 4 lr validation calibrated 0.9066339 0.04755764 0.9090909 0.6756757
#> 6 lr test calibrated 0.9075521 0.03610441 1.0000000 0.7552083
#> 8 lr external calibrated 0.8864839 0.04096392 0.8181818 0.7671958
#> 12 cart validation calibrated 0.5816953 0.05848193 0.1818182 0.9797297
#> 14 cart test calibrated 0.5530599 0.04046960 0.1250000 0.9791667
#> 16 cart external calibrated 0.6269841 0.04827002 0.2727273 0.9788360Discrimination on the external cohort, which took no part in training, feature selection, or calibration:
Comparing the calibration curves of the validation and external cohorts shows whether the probabilities travel; a model that only looks good on the development split is usually overfitted to it.
plot_calibration(result, set = "external")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?The classification threshold is derived from out-of-fold training predictions, so it is not tuned on the validation data:
as.data.frame(result$meta$thresholds)
#> lr.raw lr.calibrated cart.raw cart.calibrated
#> 1 0.06938979 0.04994531 0.05263158 0.07681389When two or more models are compared, NRI and IDI are computed against the reference model (the best training AUC by default):
SHAP explanations use kernelshap; the number of
explained rows and the background sample are controlled by
n_explain and bg_n:
explained <- explain_shap(result, model = "best", n_explain = 30, bg_n = 20)
plot_shap_importance(explained)predict_uniml() reuses the stored preprocessing,
selected feature mapping, calibration, threshold, and fitted learner. It
does not refit anything.
predicted <- predict_uniml(
result,
newdata = new_admissions[1:5, ],
model = "best"
)
predicted[, c("model", "prob", "calibrated_prob", "probability", "predicted")]
#> model prob calibrated_prob probability predicted
#> 1 lr 0.01132240 0.002474690 calibrated 0
#> 2 lr 0.02432465 0.008738865 calibrated 0
#> 3 lr 0.03504350 0.016014444 calibrated 0
#> 4 lr 0.04703144 0.026130130 calibrated 0
#> 5 lr 0.16267780 0.200254276 calibrated 1The newdata argument may omit the outcome column. In
that case the returned data frame contains predicted probabilities and
classes without a truth column.
tripod_ai_check() audits the result against the
TRIPOD+AI domains and states where each piece of information lives. It
never claims compliance: items such as study design, subgroup fairness,
and code availability must be written by the author.
checklist <- tripod_ai_check(result)
table(checklist$status)
#>
#> reported partial not_reported not_supported
#> 12 1 1 4
subset(checklist, status != "reported", select = c("domain", "status"))
#> domain status
#> 1 Data source and setting not_supported
#> 2 Participants not_supported
#> 12 Calibration not_reported
#> 16 Fairness and subgroups not_supported
#> 17 Reproducibility partial
#> 18 Code and data availability not_supportedexport_results() writes the tables, figures, and fitted
models, plus the reporting checklist, to a directory that can be used
outside R:
out <- tempfile("uniml-vignette-")
exported <- export_results(result, dir = out, include_plots = FALSE)
basename(exported$files)
#> [1] "export_manifest.csv" "result.rds"
#> [3] "var_info.csv" "scaling_params.csv"
#> [5] "encoder_map.csv" "selected_features.csv"
#> [7] "by_method_detail.csv" "best_hyperparameters.csv"
#> [9] "cv_summary.csv" "cv_detailed.csv"
#> [11] "metrics_train.csv" "metrics_validation.csv"
#> [13] "metrics_test.csv" "metrics_external.csv"
#> [15] "metrics_ci.csv" "before_after_calibration.csv"
#> [17] "thresholds.csv" "predictions.csv"
#> [19] "leaderboard.csv" "overall_rank.csv"
#> [21] "params.csv" "ece_brier.csv"
#> [23] "delong.csv" "nri_idi.csv"
#> [25] "dca_curve.csv" "failures.csv"
#> [27] "tripod_ai.csv" "importance.csv"
#> [29] "values_matrix_std.csv" "values_matrix_raw.csv"
#> [31] "feature_values_std.csv" "feature_values_raw.csv"
#> [33] "final_lr.rds" "final_cart.rds"The default is a transparent custom policy. It can be changed explicitly:
spec_in <- set_leakage_policy(
spec,
mode = "strict_in",
in_cv = c("selection", "balance", "tuning")
)
spec_in$leakage_policy
#> [1] "strict_in"
spec_in$in_cv
#> [1] "selection" "balance" "tuning"strict_in repeats configured steps inside each
cross-validation fold. strict_out keeps those steps outside
the fold loop. The validation and test sets remain held out in both
cases.
See system.file("REFERENCES.md", package = "uniml") for
the methods and reporting guidelines that this workflow implements.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.