The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
skce() computes the squared kernel calibration error of
Widmann et al. (2019) with three estimators: the unbiased U-statistic
("uq"), the unbiased linear-time estimator
("ul"), and the biased V-statistic ("biased").
It supports the binary/confidence reduction, the strong (canonical)
multiclass form via a matrix-valued kernel, and the classwise
one-vs-rest reduction. In the binary and confidence cases
mmce() equals
sqrt(skce(..., estimator = "biased")).cal_test() performs a kernel calibration hypothesis
test (H0: the model is calibrated). The default
method = "bootstrap" uses the more powerful quadratic
estimator with a wild bootstrap (Widmann et al. 2019, Theorem G.2);
method = "asymptotic" uses the faster linear-estimator
normal test (Lemma 3). It returns an object of class
c("cal_test", "htest"). The test targets are binary,
"confidence", and the strong "canonical"
multiclass form; the classwise average is available only as a point
estimate from skce().cal_ci() returns a percentile bootstrap confidence
interval for ece(), skce(),
mmce(), mce(), or ace(), as a
classed cal_ci object with a print()
method.skce() and cal_test() accept
bandwidth = "median" for the median-heuristic kernel scale
(Widmann et al. 2019), recommended for the canonical multiclass form;
the fixed 0.2 remains the default.ece() gains debiased (the debiased
squared-ECE estimator of Kumar, Liang & Ma 2019, Definition 5.2),
strategy (equal-width or equal-mass bins, Roelofs et
al. 2022), and norm ("l1" or
"l2"). The norm and debiased
choices are independent; debiasing is defined only for
norm = "l2". Defaults (norm = "l1",
debiased = FALSE, strategy = "width")
reproduce the previous numeric output exactly.stratified_folds() (used by cal_cv()) now
scopes the optional fold-assignment seed with
withr::local_seed() instead of touching
.Random.seed directly; withr was added to
Imports. The fold assignments for a given seed are
unchanged.inst/CITATION now reads the version from the package
metadata, and the package author is also declared as copyright holder in
Authors@R.skce(), cal_test(), cal_ci(),
and the debiased, strategy, and
norm arguments of ece()) with a worked
example.cal_temperature() and
cal_cv() accept a logit or probability matrix, and new
constructors cal_vector_scaling(),
cal_dirichlet(), and cal_ovr() cover vector
scaling, Dirichlet calibration, and one-vs-rest calibration.ece(), mce(), ace(), and
reliability_diagram() accept a probability matrix with a
type argument for classwise or top-label confidence
evaluation.mmce(), a binning-free Maximum Mean Calibration
Error metric for binary and multiclass predictions.inst/CITATION so users can cite the package with
citation("probcal").print() and summary() respect
options(probcal.emoji = FALSE) to suppress the decorative
glyph in console output.reliability_diagram() now reports ECE in the subtitle
by default and can use either count-scaled or fixed-size points.netcal and R betacal.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.