The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Running a complete workflow

Overview

run_workflow() chains the data, preprocessing, selection, modelling, evaluation, and result layers into one object. The held-out validation and test sets are transformed with parameters learned from the training set; they are never used to fit preprocessing, selection, calibration, or thresholds.

The vignette uses the bundled cardiovascular example cohort. It is deliberately trimmed here so the vignette builds quickly; the full cohort has 2000 rows per data set.

Build and configure the specification

library(uniml)

development <- read.csv(
  system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
  system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)

set.seed(123)
development <- development[1:800, ]
new_admissions <- new_admissions[1:400, ]

spec <- uniml_spec(
  task_type = "binary",
  outcome = "died",
  positive = "yes",
  id = "id"
)
spec <- set_preprocess(
  spec,
  impute = "median_mode",
  scale = "zscore",
  encode = "onehot"
)
spec <- set_feature_selection(
  spec,
  methods = c("univariate", "lasso"),
  combine = "intersect"
)
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_evaluation(
  spec,
  ci = "bootstrap",
  threshold_method = "youden",
  calibration = "platt",
  select_on = "validation"
)
# Kernel SHAP is the most expensive step of a workflow, so its two size knobs are
# part of the specification. They are small here to keep the vignette quick; a
# paper-quality explanation wants a few hundred rows on both sides.
spec <- set_explain(spec, n_explain = 20, bg_n = 20)
spec
#> <uniml_spec>
#> task_type: binary
#> outcome: died
#> positive: yes
#> leakage_policy: custom
#> in_cv: selection, balance, tuning
#> selection: univariate, lasso
#> verbose: 1, n_workers: 1

The built-in learner keys are lr, rf, xgb, nnet, knn, cart, and nb, plus the optional log_reg, svm, lda, and qda. Any installed mlr3 classification learner can also be passed directly, such as classif.log_reg, classif.svm, or classif.lda. Learners from packages that are not on CRAN (for example classif.lightgbm from mlr3extralearners) work the same way once that package is installed and loaded; external learners use their mlr3 defaults unless a custom search space is registered.

Create leakage-safe splits

make_split() returns a uniml_split whose elements are uniml_data objects. Pass the $data component to run_workflow(). The independent new-admission cohort stays outside the development split.

split <- make_split(
  as_uniml_data(development, spec),
  prop = c(0.8, 0.2),
  stratify = TRUE,
  seed = 123
)

data.frame(
  set = c("train", "validation"),
  n = c(nrow(split$train$data), nrow(split$validation$data)),
  events = c(
    sum(split$train$data$died == "yes"),
    sum(split$validation$data$died == "yes")
  )
)
#>          set   n events
#> 1      train 641     45
#> 2 validation 159     11

Run the workflow

The first 200 new admissions play the role of an independent test set, the next 200 the role of an external cohort. Both are reporting-only.

result <- run_workflow(
  spec,
  train = split$train$data,
  validation = split$validation$data,
  test = new_admissions[1:200, ],
  external = new_admissions[201:400, ],
  ci_R = 25
)
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: f1, f2,
#> calib_slope, calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: f1, f2,
#> calib_slope, calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_large.

result
#> <uniml_result>
#> data_info (list, 7 elements)
#> preprocess (list, 4 elements)
#> selection (list, 8 elements)
#> tuning (list, 2 elements)
#> cv (list, 6 elements)
#> metrics (16 x 24)
#> ci (336 x 10)
#> calibration (list, 4 elements)
#> tests (list, 3 elements)
#> dca (1188 x 6)
#> ranking (2 x 6)
#> shap (list, 12 elements)
#> plots (list, 4 elements)
#> models (list, 2 elements)
#> meta (list, 13 elements)

Inspect the results

result$metrics is a wide table with one row per model, set, and probability track (raw or calibrated):

metrics <- result$metrics
metrics[, c("model", "set", "probability", "auc", "brier", "accuracy")]
#>    model        set probability       auc      brier  accuracy
#> 1     lr      train         raw 0.9035421 0.05103773 0.7722309
#> 2     lr      train  calibrated 0.9035421 0.04995033 0.7722309
#> 3     lr validation         raw 0.9066339 0.05000148 0.6918239
#> 4     lr validation  calibrated 0.9066339 0.04755764 0.6918239
#> 5     lr       test         raw 0.9075521 0.03376129 0.7650000
#> 6     lr       test  calibrated 0.9075521 0.03610441 0.7650000
#> 7     lr   external         raw 0.8864839 0.04105926 0.7700000
#> 8     lr   external  calibrated 0.8864839 0.04096392 0.7700000
#> 9   cart      train         raw 0.6836689 0.04752045 0.9438378
#> 10  cart      train  calibrated 0.6836689 0.05060635 0.9438378
#> 11  cart validation         raw 0.5816953 0.05935709 0.9245283
#> 12  cart validation  calibrated 0.5816953 0.05848193 0.9245283
#> 13  cart       test         raw 0.5530599 0.04209931 0.9450000
#> 14  cart       test  calibrated 0.5530599 0.04046960 0.9450000
#> 15  cart   external         raw 0.6269841 0.04804633 0.9400000
#> 16  cart   external  calibrated 0.6269841 0.04827002 0.9400000

Bootstrap confidence intervals are stored separately so that they can be exported on their own:

head(result$ci[result$ci$set == "validation", ], 4)
#>    model        set probability   metric      mean         se      lower
#> 43    lr validation         raw      auc 0.9170925 0.03262267 0.85815355
#> 44    lr validation         raw   pr_auc 0.5428291 0.13539597 0.24476019
#> 45    lr validation         raw    brier 0.0480432 0.01246494 0.03152822
#> 46    lr validation         raw accuracy 0.6948428 0.03358161 0.62515723
#>         upper n_effective           formatted
#> 43 0.97748263          25 0.917 (0.858-0.977)
#> 44 0.73160017          25 0.543 (0.245-0.732)
#> 45 0.07209281          25 0.048 (0.032-0.072)
#> 46 0.75094340          25 0.695 (0.625-0.751)

The ranking selects on the comparison set (validation by default) and never on the test or external cohorts:

result$ranking[, c("model", "score", "rank", "metric", "set")]
#>   model     score rank    metric        set
#> 1    lr 0.7264577    1 composite validation
#> 2  cart 0.4482745    2 composite validation

Cross-validation output is available per model and metric:

cv_auc <- subset(result$cv$summary, metric == "auc")
cv_auc[, c("model", "mean", "sd", "n")]
#>    model      mean         sd n
#> 1     lr 0.8871087 0.03078246 5
#> 22  cart 0.6744501 0.10515624 5

Performance figures

All plotting helpers accept palette, theme, title, subtitle, caption, and set, so figures can be styled for a manuscript without touching the result.

plot_roc(result, set = "validation")
ROC curves on the held-out validation set.
ROC curves on the held-out validation set.
plot_calibration(result, set = "validation")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
Calibration on the validation set, raw and calibrated probabilities.
Calibration on the validation set, raw and calibrated probabilities.
plot_dca(result)
Decision curve analysis.
Decision curve analysis.
plot_confusion(result, models = "best")
Confusion matrix at the Youden threshold.
Confusion matrix at the Youden threshold.

Validation, test, and external performance

The three held-out sets answer different questions, so they are reported separately and never pooled. Model selection uses the validation set only; the test set and the external cohort are reporting-only.

role_table <- subset(
  result$metrics,
  set %in% c("validation", "test", "external"),
  select = c("model", "set", "probability", "auc", "brier", "sensitivity",
             "specificity")
)
role_table[role_table$probability == "calibrated", ]
#>    model        set probability       auc      brier sensitivity specificity
#> 4     lr validation  calibrated 0.9066339 0.04755764   0.9090909   0.6756757
#> 6     lr       test  calibrated 0.9075521 0.03610441   1.0000000   0.7552083
#> 8     lr   external  calibrated 0.8864839 0.04096392   0.8181818   0.7671958
#> 12  cart validation  calibrated 0.5816953 0.05848193   0.1818182   0.9797297
#> 14  cart       test  calibrated 0.5530599 0.04046960   0.1250000   0.9791667
#> 16  cart   external  calibrated 0.6269841 0.04827002   0.2727273   0.9788360

Discrimination on the external cohort, which took no part in training, feature selection, or calibration:

plot_roc(result, set = "external")
ROC curves on the external cohort.
ROC curves on the external cohort.

Comparing the calibration curves of the validation and external cohorts shows whether the probabilities travel; a model that only looks good on the development split is usually overfitted to it.

plot_calibration(result, set = "external")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
Calibration on the external cohort.
Calibration on the external cohort.

Thresholds, reclassification, and explainability

The classification threshold is derived from out-of-fold training predictions, so it is not tuned on the validation data:

as.data.frame(result$meta$thresholds)
#>       lr.raw lr.calibrated   cart.raw cart.calibrated
#> 1 0.06938979    0.04994531 0.05263158      0.07681389

When two or more models are compared, NRI and IDI are computed against the reference model (the best training AUC by default):

if (!is.null(result$tests$nri_idi)) {
  result$tests$nri_idi$summary
}
#> NULL

SHAP explanations use kernelshap; the number of explained rows and the background sample are controlled by n_explain and bg_n:

explained <- explain_shap(result, model = "best", n_explain = 30, bg_n = 20)
plot_shap_importance(explained)
Global SHAP importance of the best model.
Global SHAP importance of the best model.

Predict new data

predict_uniml() reuses the stored preprocessing, selected feature mapping, calibration, threshold, and fitted learner. It does not refit anything.

predicted <- predict_uniml(
  result,
  newdata = new_admissions[1:5, ],
  model = "best"
)
predicted[, c("model", "prob", "calibrated_prob", "probability", "predicted")]
#>   model       prob calibrated_prob probability predicted
#> 1    lr 0.01132240     0.002474690  calibrated         0
#> 2    lr 0.02432465     0.008738865  calibrated         0
#> 3    lr 0.03504350     0.016014444  calibrated         0
#> 4    lr 0.04703144     0.026130130  calibrated         0
#> 5    lr 0.16267780     0.200254276  calibrated         1

The newdata argument may omit the outcome column. In that case the returned data frame contains predicted probabilities and classes without a truth column.

Reporting checklist

tripod_ai_check() audits the result against the TRIPOD+AI domains and states where each piece of information lives. It never claims compliance: items such as study design, subgroup fairness, and code availability must be written by the author.

checklist <- tripod_ai_check(result)
table(checklist$status)
#> 
#>      reported       partial  not_reported not_supported 
#>            12             1             1             4
subset(checklist, status != "reported", select = c("domain", "status"))
#>                        domain        status
#> 1     Data source and setting not_supported
#> 2                Participants not_supported
#> 12                Calibration  not_reported
#> 16     Fairness and subgroups not_supported
#> 17            Reproducibility       partial
#> 18 Code and data availability not_supported

Export results

export_results() writes the tables, figures, and fitted models, plus the reporting checklist, to a directory that can be used outside R:

out <- tempfile("uniml-vignette-")
exported <- export_results(result, dir = out, include_plots = FALSE)
basename(exported$files)
#>  [1] "export_manifest.csv"          "result.rds"                  
#>  [3] "var_info.csv"                 "scaling_params.csv"          
#>  [5] "encoder_map.csv"              "selected_features.csv"       
#>  [7] "by_method_detail.csv"         "best_hyperparameters.csv"    
#>  [9] "cv_summary.csv"               "cv_detailed.csv"             
#> [11] "metrics_train.csv"            "metrics_validation.csv"      
#> [13] "metrics_test.csv"             "metrics_external.csv"        
#> [15] "metrics_ci.csv"               "before_after_calibration.csv"
#> [17] "thresholds.csv"               "predictions.csv"             
#> [19] "leaderboard.csv"              "overall_rank.csv"            
#> [21] "params.csv"                   "ece_brier.csv"               
#> [23] "delong.csv"                   "nri_idi.csv"                 
#> [25] "dca_curve.csv"                "failures.csv"                
#> [27] "tripod_ai.csv"                "importance.csv"              
#> [29] "values_matrix_std.csv"        "values_matrix_raw.csv"       
#> [31] "feature_values_std.csv"       "feature_values_raw.csv"      
#> [33] "final_lr.rds"                 "final_cart.rds"

Leakage policies

The default is a transparent custom policy. It can be changed explicitly:

spec_in <- set_leakage_policy(
  spec,
  mode = "strict_in",
  in_cv = c("selection", "balance", "tuning")
)

spec_in$leakage_policy
#> [1] "strict_in"
spec_in$in_cv
#> [1] "selection" "balance"   "tuning"

strict_in repeats configured steps inside each cross-validation fold. strict_out keeps those steps outside the fold loop. The validation and test sets remain held out in both cases.

References

See system.file("REFERENCES.md", package = "uniml") for the methods and reporting guidelines that this workflow implements.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.