The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Usmile

Overview

Universal Smile Layout for Explanation (Usmile)

Usmile is an R package for threshold-free, class-specific evaluation and comparison of probabilistic binary classifiers.

The package implements the U-smile methodology, which evaluates changes in predicted probabilities separately for:

non-events with improved prediction;
non-events with worsened prediction;
events with worsened prediction;
events with improved prediction.

The U-smile plot visualizes these four subclasses in a fixed order. The resulting shape provides an immediate graphical assessment of whether a new model improves or worsens prediction for the non-event and event classes.

The package supports the following U-smile coefficients:

BA: absolute average change in prediction;
RB: relative change in prediction error;
I: proportion of individuals with changed prediction;
rLR: relative likelihood-ratio measure of prediction improvement.

The methodology is intended for probabilistic binary classifiers and does not require selecting a classification threshold.

Main functionality

Usmile provides functions for:

comparing two fitted probabilistic binary classification models;
comparing vectors of predicted probabilities;
calculating class- and subclass-specific U-smile coefficients;
visualizing prediction improvement and worsening;
generating U-smile, PIW, ROC, precision-recall, and calibration plots;
applying externally fitted probability calibration models;
analysing training and external test datasets.

The core functions include:

Function	Purpose
`UScalc_mdl()`	Compare two fitted models
`USprep_mdl()`	Extract outcomes and predicted probabilities from a fitted model
`USbind_out()`	Combine prepared outputs from reference and new models
`UScalc_raw()`	Calculate U-smile coefficients from predicted probabilities
`USplot()`	Draw a U-smile plot
`PIWplot()`	Draw a prediction improvement-worsening plot
`ROCplot()`	Compare ROC curves
`PRCplot()`	Compare precision-recall curves
`CLBplot()`	Assess probability calibration
`USfit_calibrator()`	Fit a probability calibration model
`USapply_calibrator()`	Apply a fitted probability calibration model

Installation

CRAN

After the package is published on CRAN, it can be installed with:

install.packages("Usmile")

Development version

The current development version can be installed from GitHub:

# install.packages("remotes")
remotes::install_github("bbwieckowska/Usmile")

Quick start

The following example compares a reference logistic regression model with a larger model containing additional predictors.

library(Usmile)

data("heart_disease_train")
data("heart_disease_test")

heart_disease_train$disease <- as.factor(
  heart_disease_train$disease
)

heart_disease_test$disease <- as.factor(
  heart_disease_test$disease
)

reference_model <- stats::glm(
  disease ~ 1,
  data = heart_disease_train,
  family = stats::binomial()
)

new_model <- stats::glm(
  disease ~ ill_high_asym + age + cp,
  data = heart_disease_train,
  family = stats::binomial()
)

Evaluation on the training dataset

results_train <- UScalc_mdl(
  ref_model = reference_model,
  new_model = new_model,
  y_coef = "rLR",
  testing = FALSE
)

results_train$results

USplot(
  plot_data = results_train$plot_data,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)

Evaluation on an external test dataset

results_test <- UScalc_mdl(
  ref_model = reference_model,
  new_model = new_model,
  y_coef = "rLR",
  dataset = heart_disease_test,
  testing = TRUE
)

results_test$results

USplot(
  plot_data = results_test$plot_data,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)

Comparison from predicted probabilities

The U-smile coefficients can also be calculated without passing fitted model objects.

The input data frame must contain:

y: observed binary outcome;
p_ref: probability predicted by the reference model;
p: probability predicted by the new model.

prediction_data <- data.frame(
  y = c(0, 0, 1, 1),
  p_ref = c(0.10, 0.35, 0.60, 0.85),
  p = c(0.05, 0.40, 0.55, 0.92)
)

results_raw <- UScalc_raw(
  raw_data = prediction_data,
  y_coef = "rLR",
  n_vars_diff = 1
)

USplot(
  plot_data = results_raw,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)

This workflow is model-agnostic because it requires only observed outcomes and predicted probabilities.

Probability calibration

Calibration must be fitted independently of the dataset used for final model evaluation.

calibration_outcomes <- c(0, 0, 0, 1, 1, 1)

calibration_predictions <- c(
  0.10,
  0.25,
  0.40,
  0.55,
  0.75,
  0.90
)

# Fit the calibrator on an independent calibration dataset
calibrator <- USfit_calibrator(
  y = calibration_outcomes,
  p = calibration_predictions,
  method = "logistic"
)

# Example predictions from a separate test dataset
test_predictions <- c(
  0.15,
  0.35,
  0.65,
  0.85
)

calibrated_probabilities <- USapply_calibrator(
  object = calibrator,
  p = test_predictions
)

calibrated_probabilities

Available calibration methods are:

"logistic";
"intercept";
"isotonic".

The calibration model should not be estimated using the final test dataset.

Shiny application

A point-and-click implementation of the U-smile workflow is available as a companion Shiny application:

https://barbarawieckowska.shinyapps.io/ShinyApp/

The Shiny application is intended for interactive analyses, demonstrations, and educational use.

Documentation

Function-level documentation is available directly in R:

help(package = "Usmile")
?UScalc_mdl
?UScalc_raw
?USplot

The package citation can be displayed with:

citation("Usmile")

A PDF reference manual can be generated from the package source with:

devtools::build_manual()

Methodological background

The original U-smile framework, including the BA, RB, and I coefficients and the prediction improvement-worsening matrix, was introduced in:

Kubiak KB, Więckowska B, Jodłowska-Siewert E, Guzik P (2024). Visualising and quantifying the usefulness of new predictors stratified by outcome class: The U-smile method. PLOS ONE, 19(5), e0303276.
https://doi.org/10.1371/journal.pone.0303276

The three-level U-smile approach and its evaluation under class imbalance were described in:

Więckowska B, Kubiak KB, Guzik P (2025). Evaluating the three-level approach of the U-smile method for imbalanced binary classification. PLOS ONE, 20(4), e0321661.
https://doi.org/10.1371/journal.pone.0321661

The likelihood-based extension of the methodology and the rLR coefficient were described in:

Więckowska B, Guzik P (2026). Usmile likelihood evaluation provides robust threshold free assessment of binary classification models for balanced and imbalanced datasets. Scientific Reports, 16, 10000.
https://doi.org/10.1038/s41598-026-40545-z

Citation

When using the package, cite both the software version and the publication describing the methodological component used in the analysis.

The recommended citations can be obtained with:

citation("Usmile")

Reproducibility

For reproducible analyses, report at least:

the Usmile package version;
the R version;
the coefficient used: BA, RB, I, or rLR;
whether results were calculated on training or test data;
the definitions of the reference and new models;
any probability calibration procedure;
the random seed for stochastic modelling algorithms.

Package and R versions can be recorded with:

packageVersion("Usmile")
sessionInfo()

Contributing

Bug reports and feature requests can be submitted through the GitHub issue tracker:

https://github.com/bbwieckowska/Usmile/issues

Contributions should preserve compatibility with the documented statistical definitions of the U-smile coefficients. Changes affecting numerical results should include appropriate tests and documentation.

License

Usmile is released under the MIT License. See the LICENSE file for details.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.