The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

uniml

English | 简体中文

R-CMD-check

uniml is a unified predictive modeling toolkit for binary and multiclass classification. It chains preprocessing, feature selection, model training and tuning, evaluation, visualization, and SHAP interpretability behind one small set of objects, while keeping a strict separation between the data that fits a model and the data that judges it.

The package is organized as a stack of layers. Every layer can be used on its own, and run_workflow() wires them together.

Layer Entry points Purpose
L0 data uniml_spec(), as_uniml_data(), make_split() Declare the task, roles, and splits
L1 task type list_task_types(), task_type_spec() Extensible per-task behaviour
L2 preprocessing fit_preprocessor(), impute_data(), scale_numeric(), encode_categorical(), balance_classes() Imputation, scaling, encoding, balancing
L3 selection select_features(), select_lasso(), select_boruta(), select_rf_importance(), select_rfe(), select_univariate(), select_multivariate(), combine_selections(), remove_collinear() Feature selection and collinearity handling
L4 modeling make_task(), resolve_learners(), train_model(), tune_model() mlr3 learners, tuning, cross validation
L5 statistics calc_metrics(), calc_ci(), calibration_metrics(), multiclass_calibration(), calibrate_probabilities(), delong_test(), hl_test(), calc_nri_idi(), dca(), find_threshold() Metrics with confidence intervals, calibration numbers, tests, decision curves
L6 results assemble_result(), rank_models(), export_results(), save_result(), read_result() Ranking, on-disk export, and a result you can hand over
L7 plots plot_roc(), plot_pr_curve(), plot_calibration(), plot_dca(), plot_confusion(), theme_uniml() Performance figures
L8 SHAP explain_shap(), plot_shap_importance(), plot_shap_beeswarm(), plot_shap_dependence(), plot_shap_waterfall(), extract_thresholds() Interpretability
L9 workflow run_workflow(), predict_uniml() End-to-end run and reuse of fitted state

Why uniml

uniml does not replace mlr3 or tidymodels. It is opinionated orchestration for one job: producing a binary risk model together with a report you can defend.

Compared with using mlr3 directly. mlr3 gives you learners, tasks, resampling, and tuning, and it lets you assemble them in any order. Order is the part that goes wrong: fit a scaler or a feature selector on the full data frame and every cross-validated number becomes optimistic. uniml fixes the order once — fit on training data, apply to held-out data, never look back — turns the remaining choices into an explicit leakage_policy object instead of an accident of script layout, and adds the reporting layer mlr3 leaves to you: bootstrap intervals, calibration fitted on out-of-fold predictions, DeLong and Hosmer-Lemeshow tests, NRI/IDI, decision curves, Rubin pooling across imputations, and a TRIPOD+AI self-audit.

Compared with tidymodels. recipes, workflows, and tune cover the modelling mechanics well. uniml targets the clinical-prediction reporting path — the statistics and the audit trail — and is built on the mlr3 learner ecosystem, so everything in it is an mlr3 learner you can also use directly.

What neither gives you out of the box. Strict Rubin’s rules applied across the whole pipeline rather than to a single metric, an export layout that rebuilds every figure outside R, and failures recorded in result$meta$failures instead of silently dropped.

When not to use it. If you need a model and an AUC and nothing else, plain mlr3 is simpler. If you need multiclass, survival, or deep learning, this package is not it yet.

Scope and limitations

Installation

The package is not on CRAN yet. Install it from GitHub:

# install.packages("remotes")
remotes::install_github("zhengyangszy/uniml")

Or from a local checkout, after R CMD build uniml:

install.packages("uniml_0.1.0.tar.gz", repos = NULL, type = "source")

Quick start

library(uniml)

development <- read.csv(
  system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
  system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)

spec <- uniml_spec(
  task_type = "binary",
  outcome = "died",
  positive = "yes",
  id = "id"
)
spec <- set_preprocess(spec, impute = "median_mode", scale = "zscore", encode = "onehot")
spec <- set_feature_selection(spec, methods = c("univariate", "lasso"), combine = "intersect")
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_tuning(spec, folds = 5, evals = 10)
spec <- set_evaluation(
  spec,
  ci = "bootstrap",
  calibration = "platt",
  select_on = "validation"
)

split <- make_split(
  as_uniml_data(development, spec),
  prop = c(0.8, 0.2),
  stratify = TRUE
)
result <- run_workflow(
  spec,
  train = split$train$data,
  validation = split$validation$data,
  test = new_admissions
)
result

Predictions for new data reuse the fitted preprocessing, selection, and models; they never refit anything:

predict_uniml(result, newdata = new_admissions, model = "best")

Example cohorts

inst/extdata contains two synthetic cardiovascular cohorts:

File Rows Purpose
uniml-cvd-mortality.csv 2000 Development cohort; split into training and validation
uniml-cvd-mortality-new.csv 2000 Independent new admissions; external test cohort

Both cohorts target an in-hospital mortality probability of 6.5%. Their observed event rates differ slightly because the binary outcomes are sampled. The new-admission cohort is generated independently from the development cohort and has non-overlapping identifiers.

Available models

Every model in uniml is an mlr3 Learner, every data set becomes an mlr3 Task, and tuning and resampling use mlr3 objects directly. The package adds a thin layer on top instead of hiding mlr3:

The registry provides seven stable default binary-classification learners:

Key mlr3 learner
lr classif.cv_glmnet
rf classif.ranger
xgb classif.xgboost
nnet classif.nnet
knn classif.kknn
cart classif.rpart
nb classif.naive_bayes

It also provides an optional set of common learners that can be requested by key but are not installed or tuned by default. These entries use their mlr3 defaults unless an explicit search space is supplied:

Key mlr3 learner Optional package
log_reg classif.log_reg stats
svm classif.svm e1071
lda classif.lda MASS
qda classif.qda MASS

The package is not limited to those keys. Any installed mlr3 classification learner can be passed to set_learners(). Learners outside the registry use their mlr3 defaults and are not tuned unless a search space is supplied.

spec <- set_learners(
  spec,
  c("lr", "classif.log_reg", "classif.svm", "classif.lda"),
  tune = TRUE
)

This is also how you use learners from packages that are not on CRAN, such as LightGBM. Install and load the providing package first, then pass the mlr3 learner key:

# remotes::install_github("mlr-org/mlr3extralearners")
library(mlr3extralearners)
spec <- set_learners(spec, "classif.lightgbm", tune = FALSE)

The retired lgb, randomForest, and gbm shortcuts now fail with a message that points at the mlr3 learner key to use instead; nothing else changes.

For a custom search space, register the learner explicitly:

register_learner(
  "svm_custom",
  mlr3::lrn("classif.svm", predict_type = "prob", probability = TRUE),
  paradox::ps(
    cost = paradox::p_dbl(0.1, 10),
    gamma = paradox::p_dbl(0.001, 1)
  )
)
spec <- set_learners(spec, "svm_custom", tune = TRUE)

Step-level arguments

Every stage accepts an explicit argument list, and unknown names are rejected instead of being silently stored:

spec <- set_preprocess(
  spec,
  impute = "mice",
  impute_args = list(m = 5, max_iter = 10),
  scale_args = list(vars = c("age", "bmi")),
  encode_args = list(max_levels = 20)
)
spec <- set_feature_selection(
  spec,
  methods = c("lasso", "boruta"),
  args = list(
    lasso = list(nfolds = 5, alpha = 0.8),
    boruta = list(tentative = TRUE)
  )
)
spec <- set_tuning(
  spec,
  folds = 5,
  repeats = 5,
  terminator = "seconds",
  seconds = 600
)
spec <- set_evaluation(
  spec,
  calibration = "platt",
  calibration_args = list(folds = 5, bins = 10, fit = "out_of_fold")
)

Leakage policy

Validation and test data never take part in imputation, scaling, encoding, selection, balancing, tuning, threshold selection, or calibration. Fitting happens on the training data only, and the resulting parameters are applied to the held-out data. set_leakage_policy() controls whether selection, balancing, and tuning are re-run inside each cross-validation fold (strict_in), left outside the folds (strict_out), or configured step by step (custom).

Mode Steps re-run inside each fold Use it when
strict_in imputation, scaling, encoding, selection, balance, tuning you want the least optimistic cross-validation estimate and can pay for the run time
strict_out none the steps are expensive (Boruta, RFE, tuning) and the data set is large
custom selection, balance, tuning you want to name the trade-off yourself
spec <- set_leakage_policy(
  spec,
  mode = "custom",
  in_cv = c("selection", "balance")
)

Two cautions:

Data sets and selection

run_workflow() takes up to four data sets. test and external are both optional:

run_workflow(spec, train, validation,
             test = internal_holdout,      # optional
             external = new_admissions)    # optional

validation is used for model comparison and selection; test and external are reporting-only and never take part in fitting, tuning, calibration, threshold selection, or ranking. Requesting select_on = "test" is allowed but warns that the test performance becomes optimistic.

Cross-validation and calibration

run_workflow() reports per-fold cross-validation rows and a summary; repeats > 1 instantiates repeated cross-validation, and the summary averages folds within a repeat before summarising across repeats. Calibration is off by default (calibration = "none"); when requested, the calibrator is fitted on out-of-fold training predictions and both raw and calibrated probabilities are kept. Classification thresholds come from the same out-of-fold predictions. Failed or downgraded steps are never silent: they are listed in result$meta$failures.

With multiple imputation (impute = "mice", m > 1) the workflow fits each completed data set, pools metric estimates and confidence intervals with strict Rubin’s rules (pool_mi(), Barnard-Rubin degrees of freedom), and runs cross-validation once per imputation before pooling. Prediction probabilities are averaged across imputations (“pooled predictions”) and are labelled separately from the Rubin-pooled metric table.

Methodological conventions

The choices that are not obvious are deliberate:

Reporting

NRI/IDI are computed against reference_model (default: the model with the best training AUC) with bootstrap intervals, and written to tests/nri_idi.csv. SHAP zero-crossing thresholds are opt-in:

spec <- set_explain(
  spec,
  model = "best",
  plots = c("importance", "beeswarm", "dependence", "waterfall"),
  thresholds = list(targets = c("age", "bmi"))
)

Plots accept theme, title, subtitle, and caption, and palette can be a ggsci name, a colour, or a colour vector. Export size and format are controlled with export_results(..., plot_args = list(dpi = 600, device = "png")).

Reusing exported results

Everything needed to redraw or re-tabulate is exported as plain CSV plus one RDS: predictions/predictions.csv (model, set, raw and calibrated probabilities, truth) is enough to rebuild ROC/calibration/DCA/confusion figures with plot_roc()/plot_calibration()/plot_dca()/plot_confusion(), performance/thresholds.csv stores the raw and calibrated thresholds per model, and shap/values_matrix_*.csv plus shap/feature_values_*.csv allow custom SHAP figures. result.rds contains the complete fitted state (models, preprocessor, calibrators) for predict_uniml() on new data.

Vignettes

vignette("uniml-atomic", package = "uniml")
vignette("uniml-workflow", package = "uniml")
vignette("uniml-shap", package = "uniml")
vignette("uniml-realdata", package = "uniml")
vignette("uniml-multiclass", package = "uniml")

uniml-realdata runs the whole workflow on pROC::aSAH, a published cohort of 113 patients with subarachnoid haemorrhage, and shows what the reporting layer does when the data is small: the outcome is a collapse of gos6, so that column has to be excluded; ratios whose bootstrap distribution is degenerate come back as NA with a reason; and a Hosmer-Lemeshow statistic that cannot be defined says so instead of returning NaN.

uniml-multiclass covers the three-class path: the metric conventions, the k-by-k confusion table, one-vs-rest ROC curves, and the methods that are deliberately not implemented for more than two classes.

Development status

uniml is at version 0.2.0. The binary and multiclass task types are implemented; survival is registered as a placeholder only. GitHub Actions checks the package on Windows, macOS, and Linux, and tests/testthat/test-consistency.R pins the metric layer to independent implementations (mlr3measures, pROC, PRROC, stats::glm(), stats::isoreg(), ResourceSelection). See NEWS.md for changes.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.