The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
English | 简体中文
uniml is a unified predictive modeling toolkit for
binary and multiclass classification. It chains preprocessing, feature
selection, model training and tuning, evaluation, visualization, and
SHAP interpretability behind one small set of objects, while keeping a
strict separation between the data that fits a model and the data that
judges it.
The package is organized as a stack of layers. Every layer can be
used on its own, and run_workflow() wires them
together.
| Layer | Entry points | Purpose |
|---|---|---|
| L0 data | uniml_spec(), as_uniml_data(),
make_split() |
Declare the task, roles, and splits |
| L1 task type | list_task_types(), task_type_spec() |
Extensible per-task behaviour |
| L2 preprocessing | fit_preprocessor(), impute_data(),
scale_numeric(), encode_categorical(),
balance_classes() |
Imputation, scaling, encoding, balancing |
| L3 selection | select_features(), select_lasso(),
select_boruta(), select_rf_importance(),
select_rfe(), select_univariate(),
select_multivariate(), combine_selections(),
remove_collinear() |
Feature selection and collinearity handling |
| L4 modeling | make_task(), resolve_learners(),
train_model(), tune_model() |
mlr3 learners, tuning, cross validation |
| L5 statistics | calc_metrics(), calc_ci(),
calibration_metrics(),
multiclass_calibration(),
calibrate_probabilities(), delong_test(),
hl_test(), calc_nri_idi(), dca(),
find_threshold() |
Metrics with confidence intervals, calibration numbers, tests, decision curves |
| L6 results | assemble_result(), rank_models(),
export_results(), save_result(),
read_result() |
Ranking, on-disk export, and a result you can hand over |
| L7 plots | plot_roc(), plot_pr_curve(),
plot_calibration(), plot_dca(),
plot_confusion(), theme_uniml() |
Performance figures |
| L8 SHAP | explain_shap(), plot_shap_importance(),
plot_shap_beeswarm(), plot_shap_dependence(),
plot_shap_waterfall(),
extract_thresholds() |
Interpretability |
| L9 workflow | run_workflow(), predict_uniml() |
End-to-end run and reuse of fitted state |
uniml does not replace mlr3 or tidymodels. It is
opinionated orchestration for one job: producing a binary risk model
together with a report you can defend.
Compared with using mlr3 directly. mlr3 gives you
learners, tasks, resampling, and tuning, and it lets you assemble them
in any order. Order is the part that goes wrong: fit a scaler or a
feature selector on the full data frame and every cross-validated number
becomes optimistic. uniml fixes the order once — fit on
training data, apply to held-out data, never look back — turns the
remaining choices into an explicit leakage_policy object
instead of an accident of script layout, and adds the reporting layer
mlr3 leaves to you: bootstrap intervals, calibration fitted on
out-of-fold predictions, DeLong and Hosmer-Lemeshow tests, NRI/IDI,
decision curves, Rubin pooling across imputations, and a TRIPOD+AI
self-audit.
Compared with tidymodels. recipes,
workflows, and tune cover the modelling
mechanics well. uniml targets the clinical-prediction
reporting path — the statistics and the audit trail — and is built on
the mlr3 learner ecosystem, so everything in it is an mlr3 learner you
can also use directly.
What neither gives you out of the box. Strict
Rubin’s rules applied across the whole pipeline rather than to a single
metric, an export layout that rebuilds every figure outside R, and
failures recorded in result$meta$failures instead of
silently dropped.
When not to use it. If you need a model and an AUC and nothing else, plain mlr3 is simpler. If you need multiclass, survival, or deep learning, this package is not it yet.
run_workflow() is binary-only: its prediction table,
thresholds, calibrators, and post-model tests assume a single
probability column, so a multiclass specification stops with a message
pointing at vignette("uniml-multiclass"). Multiclass is
supported at the layer level — specification, data, task, learners,
tuning, metrics, calibration, and plots.multiclass_calibration()). Recalibration is
available too, as temperature scaling
(calibrate_probabilities()): one fitted scalar, which
cannot change the predicted class. The stronger Dirichlet calibration is
not, because its K^2 + K parameters are not identifiable on
the sample size this package targets.inst/extdata come from
data-raw/simulate-cvd-mortality.R, and the 6.5% target
event rate is a simulation parameter. They make the examples runnable
and reproducible. They are not a patient cohort, and numbers computed on
them say nothing about clinical performance or about how the methods
behave on real data.median_mode, mice,
missForest, or missRanger; missingness that
depends on the unobserved value is not modelled.tripod_ai_check() reports those items as missing; writing
them is your job.predict_uniml() scores a data frame in R. There is no REST
endpoint, no model registry, and no drift monitoring.The package is not on CRAN yet. Install it from GitHub:
# install.packages("remotes")
remotes::install_github("zhengyangszy/uniml")Or from a local checkout, after R CMD build uniml:
install.packages("uniml_0.1.0.tar.gz", repos = NULL, type = "source")library(uniml)
development <- read.csv(
system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)
spec <- uniml_spec(
task_type = "binary",
outcome = "died",
positive = "yes",
id = "id"
)
spec <- set_preprocess(spec, impute = "median_mode", scale = "zscore", encode = "onehot")
spec <- set_feature_selection(spec, methods = c("univariate", "lasso"), combine = "intersect")
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_tuning(spec, folds = 5, evals = 10)
spec <- set_evaluation(
spec,
ci = "bootstrap",
calibration = "platt",
select_on = "validation"
)
split <- make_split(
as_uniml_data(development, spec),
prop = c(0.8, 0.2),
stratify = TRUE
)
result <- run_workflow(
spec,
train = split$train$data,
validation = split$validation$data,
test = new_admissions
)
resultPredictions for new data reuse the fitted preprocessing, selection, and models; they never refit anything:
predict_uniml(result, newdata = new_admissions, model = "best")inst/extdata contains two synthetic cardiovascular
cohorts:
| File | Rows | Purpose |
|---|---|---|
uniml-cvd-mortality.csv |
2000 | Development cohort; split into training and validation |
uniml-cvd-mortality-new.csv |
2000 | Independent new admissions; external test cohort |
Both cohorts target an in-hospital mortality probability of 6.5%. Their observed event rates differ slightly because the binary outcomes are sampled. The new-admission cohort is generated independently from the development cohort and has non-overlapping identifiers.
Every model in uniml is an mlr3 Learner,
every data set becomes an mlr3 Task, and tuning and
resampling use mlr3 objects directly. The package adds a thin layer on
top instead of hiding mlr3:
make_task() builds a TaskClassif from your
data frame and specification.resolve_learners() returns mlr3 learners plus their
search spaces, which you can also tune yourself with
mlr3tuning.run_workflow() stores the fitted learners in
result$models[[key]]$learner, so anything that accepts an
mlr3 learner keeps working.as_uniml_binary() converts an mlr3
PredictionClassif into the object used by the evaluation
and plotting helpers, and plot_roc(),
plot_calibration(), and plot_confusion()
accept an mlr3 prediction directly.The registry provides seven stable default binary-classification learners:
| Key | mlr3 learner |
|---|---|
lr |
classif.cv_glmnet |
rf |
classif.ranger |
xgb |
classif.xgboost |
nnet |
classif.nnet |
knn |
classif.kknn |
cart |
classif.rpart |
nb |
classif.naive_bayes |
It also provides an optional set of common learners that can be requested by key but are not installed or tuned by default. These entries use their mlr3 defaults unless an explicit search space is supplied:
| Key | mlr3 learner | Optional package |
|---|---|---|
log_reg |
classif.log_reg |
stats |
svm |
classif.svm |
e1071 |
lda |
classif.lda |
MASS |
qda |
classif.qda |
MASS |
The package is not limited to those keys. Any installed mlr3
classification learner can be passed to set_learners().
Learners outside the registry use their mlr3 defaults and are not tuned
unless a search space is supplied.
spec <- set_learners(
spec,
c("lr", "classif.log_reg", "classif.svm", "classif.lda"),
tune = TRUE
)This is also how you use learners from packages that are not on CRAN, such as LightGBM. Install and load the providing package first, then pass the mlr3 learner key:
# remotes::install_github("mlr-org/mlr3extralearners")
library(mlr3extralearners)
spec <- set_learners(spec, "classif.lightgbm", tune = FALSE)The retired lgb, randomForest, and
gbm shortcuts now fail with a message that points at the
mlr3 learner key to use instead; nothing else changes.
For a custom search space, register the learner explicitly:
register_learner(
"svm_custom",
mlr3::lrn("classif.svm", predict_type = "prob", probability = TRUE),
paradox::ps(
cost = paradox::p_dbl(0.1, 10),
gamma = paradox::p_dbl(0.001, 1)
)
)
spec <- set_learners(spec, "svm_custom", tune = TRUE)Every stage accepts an explicit argument list, and unknown names are rejected instead of being silently stored:
spec <- set_preprocess(
spec,
impute = "mice",
impute_args = list(m = 5, max_iter = 10),
scale_args = list(vars = c("age", "bmi")),
encode_args = list(max_levels = 20)
)
spec <- set_feature_selection(
spec,
methods = c("lasso", "boruta"),
args = list(
lasso = list(nfolds = 5, alpha = 0.8),
boruta = list(tentative = TRUE)
)
)
spec <- set_tuning(
spec,
folds = 5,
repeats = 5,
terminator = "seconds",
seconds = 600
)
spec <- set_evaluation(
spec,
calibration = "platt",
calibration_args = list(folds = 5, bins = 10, fit = "out_of_fold")
)Validation and test data never take part in imputation, scaling,
encoding, selection, balancing, tuning, threshold selection, or
calibration. Fitting happens on the training data only, and the
resulting parameters are applied to the held-out data.
set_leakage_policy() controls whether selection, balancing,
and tuning are re-run inside each cross-validation fold
(strict_in), left outside the folds
(strict_out), or configured step by step
(custom).
| Mode | Steps re-run inside each fold | Use it when |
|---|---|---|
strict_in |
imputation, scaling, encoding, selection, balance, tuning | you want the least optimistic cross-validation estimate and can pay for the run time |
strict_out |
none | the steps are expensive (Boruta, RFE, tuning) and the data set is large |
custom |
selection, balance, tuning | you want to name the trade-off yourself |
spec <- set_leakage_policy(
spec,
mode = "custom",
in_cv = c("selection", "balance")
)Two cautions:
strict_out fits selection, balancing, and tuning
once on the whole training set. Those steps have then
seen every training row, so per-fold scores are optimistic for them.
Cross-validation stays useful for comparing models; do not report it as
an unbiased absolute estimate.validation, test, and
external are never used for fitting, tuning, calibration,
or threshold selection. strict_out relaxes what happens
inside the training set, not outside it.run_workflow() takes up to four data sets.
test and external are both optional:
run_workflow(spec, train, validation,
test = internal_holdout, # optional
external = new_admissions) # optionalvalidation is used for model comparison and selection;
test and external are reporting-only and never
take part in fitting, tuning, calibration, threshold selection, or
ranking. Requesting select_on = "test" is allowed but warns
that the test performance becomes optimistic.
run_workflow() reports per-fold cross-validation rows
and a summary; repeats > 1 instantiates repeated
cross-validation, and the summary averages folds within a repeat before
summarising across repeats. Calibration is off by default
(calibration = "none"); when requested, the calibrator is
fitted on out-of-fold training predictions and both raw and calibrated
probabilities are kept. Classification thresholds come from the same
out-of-fold predictions. Failed or downgraded steps are never silent:
they are listed in result$meta$failures.
With multiple imputation (impute = "mice", m > 1) the
workflow fits each completed data set, pools metric estimates and
confidence intervals with strict Rubin’s rules (pool_mi(),
Barnard-Rubin degrees of freedom), and runs cross-validation once per
imputation before pooling. Prediction probabilities are averaged across
imputations (“pooled predictions”) and are labelled separately from the
Rubin-pooled metric table.
The choices that are not obvious are deliberate:
balance = "none"). Resampling a risk model changes what
the predicted probability means (van den Goorbergh et al., 2022), so it
is opt-in.T = U + (1 + 1/m)B, Barnard-Rubin degrees
of freedom, a t interval, and the interval clamped to the metric’s
domain.result$meta$failures and raises a
warning when the workflow returns.NRI/IDI are computed against reference_model (default:
the model with the best training AUC) with bootstrap intervals, and
written to tests/nri_idi.csv. SHAP zero-crossing thresholds
are opt-in:
spec <- set_explain(
spec,
model = "best",
plots = c("importance", "beeswarm", "dependence", "waterfall"),
thresholds = list(targets = c("age", "bmi"))
)Plots accept theme, title,
subtitle, and caption, and
palette can be a ggsci name, a colour, or a
colour vector. Export size and format are controlled with
export_results(..., plot_args = list(dpi = 600, device = "png")).
Everything needed to redraw or re-tabulate is exported as plain CSV
plus one RDS: predictions/predictions.csv (model, set, raw
and calibrated probabilities, truth) is enough to rebuild
ROC/calibration/DCA/confusion figures with
plot_roc()/plot_calibration()/plot_dca()/plot_confusion(),
performance/thresholds.csv stores the raw and calibrated
thresholds per model, and shap/values_matrix_*.csv plus
shap/feature_values_*.csv allow custom SHAP figures.
result.rds contains the complete fitted state (models,
preprocessor, calibrators) for predict_uniml() on new
data.
vignette("uniml-atomic", package = "uniml")
vignette("uniml-workflow", package = "uniml")
vignette("uniml-shap", package = "uniml")
vignette("uniml-realdata", package = "uniml")
vignette("uniml-multiclass", package = "uniml")uniml-realdata runs the whole workflow on
pROC::aSAH, a published cohort of 113 patients with
subarachnoid haemorrhage, and shows what the reporting layer does when
the data is small: the outcome is a collapse of gos6, so
that column has to be excluded; ratios whose bootstrap distribution is
degenerate come back as NA with a reason; and a
Hosmer-Lemeshow statistic that cannot be defined says so instead of
returning NaN.
uniml-multiclass covers the three-class path: the metric
conventions, the k-by-k confusion table, one-vs-rest ROC curves, and the
methods that are deliberately not implemented for more than two
classes.
uniml is at version 0.2.0. The binary and
multiclass task types are implemented; survival is
registered as a placeholder only. GitHub Actions checks the package on
Windows, macOS, and Linux, and
tests/testthat/test-consistency.R pins the metric layer to
independent implementations (mlr3measures,
pROC, PRROC, stats::glm(),
stats::isoreg(), ResourceSelection). See
NEWS.md for changes.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.