evaluatellm: Statistical Inference for Language Model Evaluations

Treats language model evaluations as statistical experiments and supplies the inference they require. Provides central limit theorem and cluster-robust standard errors for evaluation scores, paired and unpaired model comparisons, variance decomposition when several responses are drawn per question, control-variate variance reduction, multiplicity adjustment across benchmark suites, and power and minimum detectable effect calculations for planning evaluations, following Miller (2024) <doi:10.48550/arXiv.2411.00640>. For evaluations scored by a model judge, implements agreement statistics against a human gold standard and prediction-powered inference (Angelopoulos et al. 2023) <doi:10.1126/science.adi6000> with the power-tuned estimator of Angelopoulos, Bates and Jordan (2023) <doi:10.48550/arXiv.2311.01453>, so a small set of human labels debiases a large set of judge scores. Leaderboards are supported through bootstrap rank intervals and Bradley-Terry ratings (Bradley and Terry 1952) <doi:10.2307/2334029>. Accepts scores from any evaluation harness.

Version:	0.1.0
Depends:	R (≥ 4.1.0)
Imports:	cli (≥ 3.6.0), graphics, grDevices, stats, utils
Suggests:	testthat (≥ 3.0.0), knitr, rmarkdown, sandwich
Published:	2026-09-15
DOI:	10.32614/CRAN.package.evaluatellm (may not be active yet)
Author:	Charles Coverdale [aut, cre, cph]
Maintainer:	Charles Coverdale <charlesfcoverdale at gmail.com>
BugReports:	https://github.com/charlescoverdale/evaluatellm/issues
License:	MIT + file LICENSE
URL:	https://charlescoverdale.github.io/evaluatellm/, https://github.com/charlescoverdale/evaluatellm
NeedsCompilation:	no
Language:	en-GB
Materials:	README, NEWS
CRAN checks:	evaluatellm results

Documentation:

Reference manual:	evaluatellm.html , evaluatellm.pdf
Vignettes:	Getting started with evaluatellm (source, R code)

Downloads:

Package source:	evaluatellm_0.1.0.tar.gz
Windows binaries:	r-devel: not available, r-release: not available, r-oldrel: not available
macOS binaries:	r-release (arm64): not available, r-oldrel (arm64): evaluatellm_0.1.0.tgz, r-release (x86_64): evaluatellm_0.1.0.tgz, r-oldrel (x86_64): evaluatellm_0.1.0.tgz

Linking:

Please use the canonical form https://CRAN.R-project.org/package=evaluatellm to link to this page.