---
title: "Running a complete workflow"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Running a complete workflow}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 6,
  fig.height = 4,
  dpi = 110
)
```

## Overview

`run_workflow()` chains the data, preprocessing, selection, modelling,
evaluation, and result layers into one object. The held-out validation and test
sets are transformed with parameters learned from the training set; they are
never used to fit preprocessing, selection, calibration, or thresholds.

The vignette uses the bundled cardiovascular example cohort. It is deliberately
trimmed here so the vignette builds quickly; the full cohort has 2000 rows per
data set.

## Build and configure the specification

```{r specification}
library(uniml)

development <- read.csv(
  system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
  system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)

set.seed(123)
development <- development[1:800, ]
new_admissions <- new_admissions[1:400, ]

spec <- uniml_spec(
  task_type = "binary",
  outcome = "died",
  positive = "yes",
  id = "id"
)
spec <- set_preprocess(
  spec,
  impute = "median_mode",
  scale = "zscore",
  encode = "onehot"
)
spec <- set_feature_selection(
  spec,
  methods = c("univariate", "lasso"),
  combine = "intersect"
)
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_evaluation(
  spec,
  ci = "bootstrap",
  threshold_method = "youden",
  calibration = "platt",
  select_on = "validation"
)
# Kernel SHAP is the most expensive step of a workflow, so its two size knobs are
# part of the specification. They are small here to keep the vignette quick; a
# paper-quality explanation wants a few hundred rows on both sides.
spec <- set_explain(spec, n_explain = 20, bg_n = 20)
spec
```

The built-in learner keys are `lr`, `rf`, `xgb`, `nnet`, `knn`, `cart`, and
`nb`, plus the optional `log_reg`, `svm`, `lda`, and `qda`. Any installed mlr3
classification learner can also be passed directly, such as
`classif.log_reg`, `classif.svm`, or `classif.lda`. Learners from packages that
are not on CRAN (for example `classif.lightgbm` from `mlr3extralearners`) work
the same way once that package is installed and loaded; external learners use
their mlr3 defaults unless a custom search space is registered.

## Create leakage-safe splits

`make_split()` returns a `uniml_split` whose elements are `uniml_data`
objects. Pass the `$data` component to `run_workflow()`. The independent
new-admission cohort stays outside the development split.

```{r split}
split <- make_split(
  as_uniml_data(development, spec),
  prop = c(0.8, 0.2),
  stratify = TRUE,
  seed = 123
)

data.frame(
  set = c("train", "validation"),
  n = c(nrow(split$train$data), nrow(split$validation$data)),
  events = c(
    sum(split$train$data$died == "yes"),
    sum(split$validation$data$died == "yes")
  )
)
```

## Run the workflow

The first 200 new admissions play the role of an independent test set, the next
200 the role of an external cohort. Both are reporting-only.

```{r workflow}
result <- run_workflow(
  spec,
  train = split$train$data,
  validation = split$validation$data,
  test = new_admissions[1:200, ],
  external = new_admissions[201:400, ],
  ci_R = 25
)

result
```

## Inspect the results

`result$metrics` is a wide table with one row per model, set, and probability
track (raw or calibrated):

```{r metrics}
metrics <- result$metrics
metrics[, c("model", "set", "probability", "auc", "brier", "accuracy")]
```

Bootstrap confidence intervals are stored separately so that they can be
exported on their own:

```{r ci}
head(result$ci[result$ci$set == "validation", ], 4)
```

The ranking selects on the comparison set (validation by default) and never on
the test or external cohorts:

```{r ranking}
result$ranking[, c("model", "score", "rank", "metric", "set")]
```

Cross-validation output is available per model and metric:

```{r cv}
cv_auc <- subset(result$cv$summary, metric == "auc")
cv_auc[, c("model", "mean", "sd", "n")]
```

## Performance figures

All plotting helpers accept `palette`, `theme`, `title`, `subtitle`, `caption`,
and `set`, so figures can be styled for a manuscript without touching the
result.

```{r roc, fig.cap = "ROC curves on the held-out validation set."}
plot_roc(result, set = "validation")
```

```{r calibration, fig.cap = "Calibration on the validation set, raw and calibrated probabilities."}
plot_calibration(result, set = "validation")
```

```{r dca, fig.cap = "Decision curve analysis."}
plot_dca(result)
```

```{r confusion, fig.cap = "Confusion matrix at the Youden threshold."}
plot_confusion(result, models = "best")
```

## Validation, test, and external performance

The three held-out sets answer different questions, so they are reported
separately and never pooled. Model selection uses the validation set only; the
test set and the external cohort are reporting-only.

```{r role-table}
role_table <- subset(
  result$metrics,
  set %in% c("validation", "test", "external"),
  select = c("model", "set", "probability", "auc", "brier", "sensitivity",
             "specificity")
)
role_table[role_table$probability == "calibrated", ]
```

Discrimination on the external cohort, which took no part in training, feature
selection, or calibration:

```{r external-roc, fig.cap = "ROC curves on the external cohort."}
plot_roc(result, set = "external")
```

Comparing the calibration curves of the validation and external cohorts shows
whether the probabilities travel; a model that only looks good on the
development split is usually overfitted to it.

```{r external-calibration, fig.cap = "Calibration on the external cohort."}
plot_calibration(result, set = "external")
```

## Thresholds, reclassification, and explainability

The classification threshold is derived from out-of-fold training predictions,
so it is not tuned on the validation data:

```{r thresholds}
as.data.frame(result$meta$thresholds)
```

When two or more models are compared, NRI and IDI are computed against the
reference model (the best training AUC by default):

```{r nri}
if (!is.null(result$tests$nri_idi)) {
  result$tests$nri_idi$summary
}
```

SHAP explanations use `kernelshap`; the number of explained rows and the
background sample are controlled by `n_explain` and `bg_n`:

```{r shap, fig.cap = "Global SHAP importance of the best model."}
explained <- explain_shap(result, model = "best", n_explain = 30, bg_n = 20)
plot_shap_importance(explained)
```

## Predict new data

`predict_uniml()` reuses the stored preprocessing, selected feature mapping,
calibration, threshold, and fitted learner. It does not refit anything.

```{r predict}
predicted <- predict_uniml(
  result,
  newdata = new_admissions[1:5, ],
  model = "best"
)
predicted[, c("model", "prob", "calibrated_prob", "probability", "predicted")]
```

The `newdata` argument may omit the outcome column. In that case the returned
data frame contains predicted probabilities and classes without a truth
column.

## Reporting checklist

`tripod_ai_check()` audits the result against the TRIPOD+AI domains and states
where each piece of information lives. It never claims compliance: items such as
study design, subgroup fairness, and code availability must be written by the
author.

```{r tripod}
checklist <- tripod_ai_check(result)
table(checklist$status)
subset(checklist, status != "reported", select = c("domain", "status"))
```

## Export results

`export_results()` writes the tables, figures, and fitted models, plus the
reporting checklist, to a directory that can be used outside R:

```{r export}
out <- tempfile("uniml-vignette-")
exported <- export_results(result, dir = out, include_plots = FALSE)
basename(exported$files)
```

## Leakage policies

The default is a transparent custom policy. It can be changed explicitly:

```{r leakage}
spec_in <- set_leakage_policy(
  spec,
  mode = "strict_in",
  in_cv = c("selection", "balance", "tuning")
)

spec_in$leakage_policy
spec_in$in_cv
```

`strict_in` repeats configured steps inside each cross-validation fold.
`strict_out` keeps those steps outside the fold loop. The validation and test
sets remain held out in both cases.

## References

See `system.file("REFERENCES.md", package = "uniml")` for the methods and
reporting guidelines that this workflow implements.
