The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

A real cohort: subarachnoid haemorrhage

Why this vignette

Every other example in this package uses a cohort that data-raw/ simulates. That makes the code reproducible, but it says nothing about how the workflow behaves on real data, where probabilities are lumpy, groups can be empty, and a hundred and thirteen patients is all you have.

This vignette runs the same workflow on a published clinical data set that ships with pROC, so the numbers below are real and you can check them against the source paper.

The data

pROC::aSAH contains 113 patients with aneurysmal subarachnoid haemorrhage. The outcome is the dichotomised Glasgow Outcome Scale at six months, and the candidate predictors are two blood markers (s100b, ndka) plus age, sex, and the World Federation of Neurosurgical Societies grade.

asah <- pROC::aSAH
str(asah[, c("outcome", "gos6", "gender", "age", "wfns", "s100b", "ndka")])
#> 'data.frame':    113 obs. of  7 variables:
#>  $ outcome: Factor w/ 2 levels "Good","Poor": 1 1 1 1 2 2 1 2 1 1 ...
#>  $ gos6   : Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 5 5 5 5 1 1 4 1 5 4 ...
#>  $ gender : Factor w/ 2 levels "Male","Female": 2 2 2 2 2 1 1 1 2 2 ...
#>  $ age    : int  42 37 42 27 42 48 57 41 49 75 ...
#>  $ wfns   : Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 1 1 1 1 3 2 5 4 1 2 ...
#>  $ s100b  : num  0.13 0.14 0.1 0.04 0.13 0.1 0.47 0.16 0.18 0.1 ...
#>  $ ndka   : num  3.01 8.54 8.09 10.42 17.4 ...
table(asah$outcome)
#> 
#> Good Poor 
#>   72   41

Note gos6: it is the five-level version of the outcome, and outcome is exactly gos6 collapsed to two groups.

table(asah$gos6, asah$outcome)
#>    
#>     Good Poor
#>   1    0   28
#>   2    0    0
#>   3    0   13
#>   4    6    0
#>   5   66    0

Leaving gos6 in the predictor set would let the model read the answer off the answer sheet. It goes in exclude, and the specification refuses to use it:

library(uniml)

spec <- uniml_spec(
  task_type = "binary",
  outcome = "outcome",
  positive = "Poor",
  exclude = "gos6"
)
spec <- set_preprocess(
  spec,
  impute = "median_mode",
  scale = "zscore",
  encode = "onehot"
)
spec <- set_feature_selection(
  spec,
  methods = "univariate",
  p_threshold = 0.2
)
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_tuning(spec, folds = 5, repeats = 5, evals = 4, metric = "auc", seed = 123)
spec <- set_evaluation(
  spec,
  ci = "bootstrap",
  calibration = "platt",
  threshold_method = "youden",
  select_on = "validation"
)
spec <- set_leakage_policy(spec, mode = "strict_in")
spec <- set_explain(spec, model = "none", plots = character())
spec
#> <uniml_spec>
#> task_type: binary
#> outcome: outcome
#> positive: Poor
#> leakage_policy: strict_in
#> in_cv: imputation, scaling, encoding, selection, balance, tuning
#> selection: univariate
#> verbose: 1, n_workers: 1

strict_in re-runs the univariate selection inside every fold, so the cross-validated numbers do not profit from having seen the whole training set. With five predictors that costs almost nothing.

Split and run

Seventy per cent of the cohort develops the models, thirty per cent is held out for the whole workflow.

split <- make_split(
  as_uniml_data(asah, spec),
  prop = c(0.7, 0.3),
  stratify = TRUE,
  seed = 123
)

data.frame(
  set = c("train", "validation"),
  n = c(nrow(split$train$data), nrow(split$validation$data)),
  poor_outcome = c(
    sum(split$train$data$outcome == "Poor"),
    sum(split$validation$data$outcome == "Poor")
  )
)
#>          set  n poor_outcome
#> 1      train 80           29
#> 2 validation 33           12
result <- suppressWarnings(run_workflow(
  spec,
  train = split$train$data,
  validation = split$validation$data,
  ci_R = 40
))
result
#> <uniml_result>
#> data_info (list, 7 elements)
#> preprocess (list, 4 elements)
#> selection (list, 8 elements)
#> tuning (list, 2 elements)
#> cv (list, 6 elements)
#> metrics (8 x 24)
#> ci (168 x 10)
#> calibration (list, 4 elements)
#> tests (list, 3 elements)
#> dca (1188 x 6)
#> ranking (2 x 6)
#> models (list, 2 elements)
#> meta (list, 13 elements)

Nothing was dropped, downgraded, or silently skipped:

result$meta$failures
#> [1] model   stage   message
#> <0 rows> (or 0-length row.names)

The chunks above suppress warnings. The ones that fired are expected on data this small: bootstrap resamples in which the positive likelihood ratio and the diagnostic odds ratio are infinite, and a Platt fit on tree probabilities that are exactly 0 or 1. Neither is an error, and both are visible in the output below — the affected intervals come back as NA rather than as a number invented by the package.

What the cross-validation says

Repeated cross-validation on the 80 development patients is the most stable estimate available here.

cv_auc <- subset(result$cv$summary, metric == "auc")
cv_auc[, c("model", "mean", "sd", "lower", "upper", "n")]
#>    model      mean         sd     lower     upper n
#> 1     lr 0.7857858 0.09543635 0.7513352 0.8202363 5
#> 22  cart 0.7287635 0.14001758 0.7073763 0.7501507 5

n = 5 is the number of repeats: each repeat averages five folds, and the interval summarises spread across those five repeat means. It is narrower than the sampling uncertainty of the AUC itself, so do not read it as a confidence interval for new patients — compare it with the bootstrap interval on the validation patients further down, which is much wider.

Validation performance

The held-out set has 33 patients and 12 events.

validation <- subset(
  result$metrics,
  set == "validation",
  select = c("model", "probability", "auc", "brier", "sensitivity",
             "specificity", "ppv", "npv")
)
validation
#>   model probability       auc     brier sensitivity specificity    ppv
#> 3    lr         raw 0.8571429 0.1689515   0.4166667   1.0000000 1.0000
#> 4    lr  calibrated 0.8571429 0.1502564   0.4166667   1.0000000 1.0000
#> 7  cart         raw 0.8115079 0.1776383   0.7500000   0.6666667 0.5625
#> 8  cart  calibrated 0.8115079 0.1639931   0.7500000   0.6666667 0.5625
#>         npv
#> 3 0.7500000
#> 4 0.7500000
#> 7 0.8235294
#> 8 0.8235294

Bootstrap intervals around the same numbers:

ci <- subset(
  result$ci,
  set == "validation" & metric %in% c("auc", "brier"),
  select = c("model", "probability", "metric", "mean", "lower", "upper")
)
ci
#>     model probability metric      mean      lower     upper
#> 43     lr         raw    auc 0.8430825 0.67541322 0.9635862
#> 45     lr         raw  brier 0.1718083 0.11203441 0.2270106
#> 64     lr  calibrated    auc 0.8430825 0.67541322 0.9635862
#> 66     lr  calibrated  brier 0.1552310 0.09028117 0.2167864
#> 127  cart         raw    auc 0.8062441 0.61152597 0.9251188
#> 129  cart         raw  brier 0.1753687 0.08648452 0.2435695
#> 148  cart  calibrated    auc 0.8062441 0.61152597 0.9251188
#> 150  cart  calibrated  brier 0.1614905 0.08523515 0.2240351

The logistic model calibrates better than the tree: Platt scaling cuts its validation Brier score by about 0.019, and cart’s by about 0.014. Reads of a single 33-patient split should be treated as illustrative; the cross-validated table above is what to quote.

Two of the ratios cannot be given an interval on this data, and the package says so rather than inventing bounds:

subset(
  result$ci,
  set == "validation" & metric %in% c("plr", "dor"),
  select = c("model", "metric", "mean", "lower", "upper", "n_effective")
)
#>     model metric      mean    lower    upper n_effective
#> 54     lr    plr        NA       NA       NA           0
#> 56     lr    dor        NA       NA       NA           0
#> 75     lr    plr        NA       NA       NA           0
#> 77     lr    dor        NA       NA       NA           0
#> 138  cart    plr  2.808466 1.232143  5.13000          40
#> 140  cart    dor 11.380938 1.437500 36.45833          36
#> 159  cart    plr  2.808466 1.232143  5.13000          40
#> 161  cart    dor 11.380938 1.437500 36.45833          36

lr calls every validation patient at low risk, so its specificity is exactly 1 and the positive likelihood ratio is infinite in all 200 resamples. The interval comes back as NA with n_effective = 0. cart keeps a finite ratio and therefore a finite interval.

Which model does the package rank first?

rank_models() scores each model on the comparison set (validation by default) using the calibrated probabilities, averaging AUC, sensitivity, specificity, F2, balanced accuracy, and Youden’s index.

result$ranking[, c("model", "score", "rank", "metric", "set")]
#>   model     score rank    metric        set
#> 1  cart 0.6760499    1 composite validation
#> 2    lr 0.6450846    2 composite validation

cart comes first. It has the lower AUC but the much higher sensitivity at the Youden threshold, and the composite weights the two equally. That is a choice, not a law: pass best_model_metric = "auc" to rank_models() if discrimination is all you care about, and pick deliberately rather than by default.

Figures

plot_roc(result, set = "validation")
ROC curves on the held-out validation patients.
ROC curves on the held-out validation patients.
plot_calibration(result, set = "validation")
Calibration on the validation set, before and after Platt scaling.
Calibration on the validation set, before and after Platt scaling.
plot_dca(result)
Decision curve analysis.
Decision curve analysis.
plot_confusion(result, models = "best")
Confusion matrix at the out-of-fold Youden threshold.
Confusion matrix at the out-of-fold Youden threshold.

Thresholds

The classification threshold comes from out-of-fold training predictions, never from the validation patients.

as.data.frame(result$meta$thresholds)
#>      lr.raw lr.calibrated cart.raw cart.calibrated
#> 1 0.4294879     0.4733797     0.24       0.2864583

Do the two models differ?

result$tests$delong$summary
#>   model       auc
#> 1    lr 0.8571429
#> 2  cart 0.8115079
result$tests$delong$formatted
#>      lr       cart    
#> lr   "-"      "0.3616"
#> cart "0.3616" "-"

The DeLong test compares AUCs on the same patients and accounts for the correlation between the two prediction vectors. With 12 events it has very little power, so a non-significant p value means “not shown to differ”, not “equivalent”.

Calibration, for the record

for (track in c("raw", "calibrated")) {
  for (model in c("lr", "cart")) {
    item <- result$tests$hl[[track]][[model]]
    cat(sprintf(
      "%-10s %-5s statistic = %s, df = %s, p = %s (%s)\n",
      track, model,
      format(round(item$statistic, 3)), format(item$df),
      format(round(item$p_value, 3)), item$method
    ))
  }
}
#> raw        lr    statistic = 4.907, df = 3, p = 0.179 (ResourceSelection)
#> raw        cart  statistic = NA, df = 2, p = NA (not_estimable)
#> calibrated lr    statistic = 2.937, df = 3, p = 0.401 (ResourceSelection)
#> calibrated cart  statistic = 0.924, df = 2, p = 0.63 (ResourceSelection)

cart reports not_estimable on the raw track: the tree puts a group of patients at a probability of exactly 1, which leaves the expected non-event count at zero and the Hosmer-Lemeshow statistic undefined. The package says so with a message instead of returning NaN; the fix in practice is fewer groups or more patients. After Platt scaling the same model has spread-out probabilities and the statistic becomes estimable, which is a fair summary of what calibration does.

Reclassification

subset(
  result$tests$nri_idi,
  set == "validation",
  select = c("model", "reference", "nri", "nri_events", "nri_nonevents", "idi")
)
#>   model reference        nri nri_events nri_nonevents        idi
#> 1    lr      cart -0.7142857          0    -0.7142857 -0.1877569
#> 2  cart      cart  0.0000000          0     0.0000000  0.0000000

NRI and IDI are computed against the reference model (the best training AUC, cart here). The result is worth reading carefully: lr reclassifies none of the events and 71% of the non-events downward, so its net reclassification is negative even though its AUC on the same patients is higher. With twelve events, neither number is evidence of anything — but the disagreement is exactly why AUC alone is a poor basis for choosing a risk model.

Reporting checklist

checklist <- tripod_ai_check(result)
table(checklist$status)
#> 
#>      reported       partial  not_reported not_supported 
#>            10             1             3             4
subset(checklist, status != "reported", select = c("domain", "status"))
#>                        domain        status
#> 1     Data source and setting not_supported
#> 2                Participants not_supported
#> 10        External validation  not_reported
#> 12                Calibration  not_reported
#> 15             Explainability  not_reported
#> 16     Fairness and subgroups not_supported
#> 17            Reproducibility       partial
#> 18 Code and data availability not_supported

Export

out <- tempfile("uniml-asah-")
exported <- export_results(result, dir = out, include_plots = FALSE)
basename(exported$files)
#>  [1] "export_manifest.csv"          "result.rds"                  
#>  [3] "var_info.csv"                 "scaling_params.csv"          
#>  [5] "encoder_map.csv"              "selected_features.csv"       
#>  [7] "by_method_detail.csv"         "best_hyperparameters.csv"    
#>  [9] "cv_summary.csv"               "cv_detailed.csv"             
#> [11] "metrics_train.csv"            "metrics_validation.csv"      
#> [13] "metrics_test.csv"             "metrics_ci.csv"              
#> [15] "before_after_calibration.csv" "thresholds.csv"              
#> [17] "predictions.csv"              "leaderboard.csv"             
#> [19] "overall_rank.csv"             "params.csv"                  
#> [21] "ece_brier.csv"                "delong.csv"                  
#> [23] "nri_idi.csv"                  "dca_curve.csv"               
#> [25] "failures.csv"                 "tripod_ai.csv"               
#> [27] "final_lr.rds"                 "final_cart.rds"

What this vignette does and does not show

It shows that the workflow runs end to end on real, complete clinical data, that the leakage guard is not hypothetical (gos6 would have made every model look brilliant), and that the reporting layer degrades honestly when the data is too small for a statistic to be defined.

It does not show a validated model. A hundred and thirteen patients cannot support one, and the package is not a substitute for a prospective validation study. If you report from a cohort this size, report the cross-validated estimates with their intervals, say how wide they are, and do not select between models on thirty-three patients.

References

Sabbagh C, et al. (2005). Protein S100B and neuron-specific enolase in subarachnoid haemorrhage. The aSAH data set is distributed with the pROC package and used here under its licence.

See system.file("REFERENCES.md", package = "uniml") for the methods that each layer implements.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.