The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Multiclass outcomes

What “multiclass” means here

A multiclass task predicts one of three or more classes that carry no order. uniml treats it as a separate task type rather than as a binary problem with a positive class bolted on, because almost every binary report is built around a single event: a threshold, a positive class, a calibration curve, a decision curve. None of those survives contact with four unordered classes without a statement of convention.

What the package fixes for you is the convention:

Quantity Definition
auc Hand and Till multiclass AUC: the mean over ordered class pairs of the AUC on class i’s own score column (mlr3measures::mauc_au1u)
auc_ovr Macro one-vs-rest AUC: each class against the rest, then the unweighted mean; equals mean(per_class$auc)
micro_auc Micro-average AUC: all (patient, class) pairs pooled into one binary problem
brier Multi-class Brier score: mean over rows of the summed squared class errors
logloss Mean negative log probability of the observed class
sensitivity, specificity, ppv, npv, f1 Macro averages of the one-vs-rest values
micro_specificity Specificity pooled over the one-vs-rest cells
balanced_accuracy Mean per-class recall, which equals the macro sensitivity
kappa Cohen’s kappa on the k-by-k table

There is no micro precision, micro recall, or micro F1. For a single-label multiclass outcome all three are identically the accuracy — the same number three times — so the package reports accuracy once and says why. Micro AUC and micro specificity are the two micro numbers that carry information of their own.

Every number is reproducible from the confusion matrix that comes with it, and tests/testthat/test-multiclass.R pins them to mlr3measures and to a direct recomputation.

A worked example

datasets::iris is used here because it ships with R and has three classes; the mechanics are the same for a clinical outcome with three risk groups.

library(uniml)

data <- iris
data$Species <- factor(data$Species)
table(data$Species)
#> 
#>     setosa versicolor  virginica 
#>         50         50         50

spec <- uniml_spec(
  task_type = "multiclass",
  outcome = "Species"
)
spec <- set_learners(spec, c("cart", "rf"), tune = TRUE)
spec
#> <uniml_spec>
#> task_type: multiclass
#> outcome: Species
#> positive: none
#> leakage_policy: custom
#> in_cv: selection, balance, tuning
#> selection: lasso
#> verbose: 1, n_workers: 1

Note there is no positive: a positive class only exists when there are two.

prepared <- as_uniml_data(data, spec)
levels(prepared$outcome)
#> [1] "setosa"     "versicolor" "virginica"

task <- make_task(data, spec)
task$class_names
#> [1] "setosa"     "versicolor" "virginica"
task$positive
#> [1] NA

Train and score

learner <- train_model(
  task,
  mlr3::lrn("classif.rpart", predict_type = "prob")
)
prediction <- learner$predict(task)
head(prediction$prob, 3)
#>      setosa versicolor virginica
#> [1,]      1          0         0
#> [2,]      1          0         0
#> [3,]      1          0         0

calc_metrics() recognises a prediction with more than two probability columns and switches to the multiclass metric set:

metrics <- calc_metrics(prediction)
metrics[setdiff(names(metrics), c("confusion", "per_class"))]
#> $auc
#> [1] 0.98
#> 
#> $accuracy
#> [1] 0.96
#> 
#> $balanced_accuracy
#> [1] 0.96
#> 
#> $kappa
#> [1] 0.94
#> 
#> $logloss
#> [1] 0.1431763
#> 
#> $brier
#> [1] 0.07353731
#> 
#> $sensitivity
#> [1] 0.96
#> 
#> $specificity
#> [1] 0.98
#> 
#> $ppv
#> [1] 0.9618894
#> 
#> $npv
#> [1] 0.9805021
#> 
#> $f1
#> [1] 0.9599359
#> 
#> $auc_ovr
#> [1] 0.98
#> 
#> $micro_auc
#> [1] 0.9905778
#> 
#> $micro_specificity
#> [1] 0.98
#> 
#> $ece
#> [1] 0
#> 
#> $mce
#> [1] 0
#> 
#> $ece_classwise
#> [1] 0
#> 
#> $n_samples
#> [1] 150
#> 
#> $n_classes
#> [1] 3

The confusion matrix has the reference classes in rows and the predicted classes in columns, and the per-class table turns it into one-vs-rest rates:

metrics$confusion
#>             predicted
#> observed     setosa versicolor virginica
#>   setosa         50          0         0
#>   versicolor      0         49         1
#>   virginica       0          5        45
metrics$per_class
#>        class  n tp fp fn  tn sensitivity specificity       ppv       npv
#> 1     setosa 50 50  0  0 100        1.00        1.00 1.0000000 1.0000000
#> 2 versicolor 50 49  5  1  95        0.98        0.95 0.9074074 0.9895833
#> 3  virginica 50 45  1  5  99        0.90        0.99 0.9782609 0.9519231
#>          f1  auc ece mce
#> 1 1.0000000 1.00   0   0
#> 2 0.9423077 0.97   0   0
#> 3 0.9375000 0.97   0   0

The macro sensitivity and the balanced accuracy are the same quantity computed two ways, which is a useful check rather than a duplication:

data.frame(
  sensitivity = metrics$sensitivity,
  balanced_accuracy = metrics$balanced_accuracy,
  accuracy = metrics$accuracy,
  kappa = metrics$kappa
)
#>   sensitivity balanced_accuracy accuracy kappa
#> 1        0.96              0.96     0.96  0.94

Does a model survive resampling?

Nothing here is special-cased for the workflow, so ordinary mlr3 resampling applies; calc_metrics() scores the pooled out-of-fold predictions.

resampling <- mlr3::rsmp("cv", folds = 5L)
resampling$instantiate(task)
cv <- mlr3::resample(task, learner, resampling, store_models = FALSE)

pooled <- as_uniml_multiclass(cv$prediction(), truth = cv$prediction()$truth)
cv_metrics <- calc_metrics(pooled)
round(cv_metrics$auc, 3)
#> [1] 0.967
round(cv_metrics$accuracy, 3)
#> [1] 0.927

Tuning works the same way as for a binary task, because the search space and the resampling are mlr3 objects. resolve_learners() returns the package’s learner specifications, which carry their search spaces:

learners <- resolve_learners(spec, task = task)
tuned <- tune_model(
  task,
  learners[["cart"]],
  resampling = mlr3::rsmp("cv", folds = 3L),
  tuner = "random_search",
  terminator = mlr3tuning::trm("evals", n_evals = 4L),
  seed = 123
)
tuned$status
#> [1] "tuned"
tuned$best_params
#> $xval
#> [1] 10
#> 
#> $minsplit
#> [1] 26
#> 
#> $minbucket
#> [1] 14
#> 
#> $cp
#> [1] 0.01286033
#> 
#> $maxdepth
#> [1] 7

Comparing models

rank_models() averages a composite score whose metric set is declared by the task type, so a multiclass comparison is not ranked on two-class numbers. The multiclass composite is AUC, accuracy, balanced accuracy, and macro F1.

score_row <- function(model, prediction) {
  values <- calc_metrics(prediction)
  data.frame(
    model = model,
    set = "validation",
    auc = values$auc,
    accuracy = values$accuracy,
    balanced_accuracy = values$balanced_accuracy,
    f1 = values$f1,
    stringsAsFactors = FALSE
  )
}

forest <- train_model(task, mlr3::lrn("classif.ranger", predict_type = "prob"))
comparison <- rbind(
  score_row("cart", prediction),
  score_row("rf", forest$predict(task))
)
comparison
#>   model        set    auc accuracy balanced_accuracy        f1
#> 1  cart validation 0.9800     0.96              0.96 0.9599359
#> 2    rf validation 0.9992     0.98              0.98 0.9799980
rank_models(comparison, task_type = "multiclass")
#>   model probability     score rank    metric        set
#> 2    rf        <NA> 0.9847995    1 composite validation
#> 1  cart        <NA> 0.9649840    2 composite validation

A plain metric data frame carries no task type, so task_type has to be passed; a uniml_result remembers it for you. Ranking a binary result is unchanged.

Figures

Discrimination is shown as one-vs-rest ROC curves, one per class. There is no single ROC curve for a multiclass problem, and averaging them into one line hides exactly the class you are worried about.

binary_prediction <- as_uniml_multiclass(prediction)
plot_roc(binary_prediction)
One-vs-rest ROC curves.
One-vs-rest ROC curves.
plot_confusion(binary_prediction)
Confusion matrix: reference classes in rows, predicted in columns.
Confusion matrix: reference classes in rows, predicted in columns.

Calibration

A multiclass model outputs a whole vector of probabilities for every patient, so “is the number right” has to be asked in two ways. Both are reported, because a model can pass one and fail the other.

calibration <- multiclass_calibration(prediction$prob, prediction$truth)
calibration$summary
#>     n n_classes accuracy mean_confidence level_bias ece mce ece_classwise bins
#> 1 150         3     0.96            0.96          0   0   0             0   10
calibration$per_class
#>        class  n prevalence mean_predicted ece mce
#> 1     setosa 50  0.3333333      0.3333333   0   0
#> 2 versicolor 50  0.3333333      0.3333333   0   0
#> 3  virginica 50  0.3333333      0.3333333   0   0

ECE and MCE are bin-dependent summaries, not proper scoring rules; rank models on the Brier score or the log loss, and read ECE as “how wrong is the stated probability when the model is wrong”.

plot_calibration(binary_prediction, scope = "both")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
Reliability of the multiclass predictions: top-label confidence (left) and one-vs-rest per class (right).
Reliability of the multiclass predictions: top-label confidence (left) and one-vs-rest per class (right).

The same numbers ride along with every other metric, so they can be tabulated and bootstrapped without a second call:

metrics <- calc_metrics(prediction)
round(metrics$ece, 3)
#> [1] 0
round(metrics$ece_classwise, 3)
#> [1] 0
metrics$per_class[, c("class", "ece", "mce")]
#>        class ece mce
#> 1     setosa   0   0
#> 2 versicolor   0   0
#> 3  virginica   0   0

Recalibration

Measuring the miscalibration and fixing it are two different jobs. The method for a multinomial model is temperature scaling: divide every logit by one fitted scalar, softmax(log(p) / T). T > 1 softens an over-confident model, T < 1 sharpens an under-confident one.

recalibrated <- calibrate_probabilities(binary_prediction, folds = 5L)
recalibrated$parameters$fitted
#> $type
#> [1] "temperature"
#> 
#> $temperature
#> [1] 1
#> 
#> $nll_before
#> [1] 0.1431763
#> 
#> $nll_after
#> [1] 0.1431763
#> 
#> $converged
#> [1] TRUE
recalibrated$metrics_before[, c("accuracy", "ece", "logloss")]
#>   accuracy ece   logloss
#> 1     0.96   0 0.1431763
recalibrated$metrics_after[, c("accuracy", "ece", "logloss")]
#>   accuracy         ece   logloss
#> 1     0.96 6.67821e-05 0.1466176

Two properties are worth knowing before reading those numbers.

The division is monotone and is applied to every class in the same way, so temperature scaling cannot change which class the model ranks first. The confusion matrix, the accuracy, and every threshold-based metric come out identical; only the probabilities attached to them move. A method that can only make the model better at reporting and cannot make it worse at deciding is worth having even though it has one parameter.

The fitted scalar is chosen to be no worse than leaving the model alone, and the calibrated matrix is cross-fitted, so the before/after table is measured on probabilities that were not used to fit the scalar. That honesty has a cost: on a model that was already calibrated the cross-fitted log loss can come out a hair worse, because each fold’s scalar is a noisy estimate of 1.

plot_calibration(
  as_uniml_multiclass(recalibrated$calibrated, truth = binary_prediction$truth),
  scope = "top_label"
)
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
The same predictions after temperature scaling.
The same predictions after temperature scaling.

calibrate_probabilities() takes new_probs (and optionally new_labels) so a scalar fitted on the validation set can be applied to the test set. The job never crosses sets by accident: the scalar is a property of the model, and the before/ after table is reported on whichever set you hand it.

What is deliberately missing

These are not gaps to be filled in later without a decision; each needs a convention that a package should not choose silently.

References

Hand DJ, Till RJ (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 45, 171-186. doi:10.1023/A:1010920819831

Sokolova M, Lapalme G (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427-437. doi:10.1016/j.ipm.2009.03.002

Cohen J (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. doi:10.1177/001316446002000104

Brier GW (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3.

Guo C, Pleiss G, Sun Y, Weinberger KQ (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 70, 1321-1330.

Kull M, Perello-Nieto M, Kängsepp M, Silva Filho T, Song H, Flach P (2019). Beyond temperature scaling: obtaining well-calibrated multi-class probabilities with Dirichlet calibration. Advances in Neural Information Processing Systems, 32.

See system.file("REFERENCES.md", package = "uniml") for the full list.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.