The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

aiEvalR

Statistical and Psychometric Evaluation of AI Systems

aiEvalR evaluates AI systems the way psychometricians evaluate tests — as measurement instruments with reliability, fairness, calibration, and error structure — rather than as black boxes scored on accuracy alone.

Modules

Module	What it does
Psychometric quality	The methodological core. Multi-facet generalizability theory (case × prompt × model × run) via `lme4`, with a G-study/D-study workflow. Prompts become “items,” AI outputs become “item responses.”
Reliability	Test-retest (ICC), internal consistency (alpha), bootstrap intervals.
Robustness	Stress testing, prompt sensitivity, adversarial stability.
Fairness	Group disparity + equalized odds; integrates with (does not duplicate) the `aiDIF` package for item-level IRT-based AI-scoring bias.
Calibration	ECE, Brier score, calibration curves, input-validated.
Hallucination	Aggregation of adjudicated verdicts; lexical-overlap baseline.
Dashboard	Descriptive multi-module profile (not a validated composite score).

Quick examples

Generalizability (the core)

library(aiEvalR)

# long-format: one row per case x prompt x model x run
g <- ai_generalizability(data, score = "score", case = "case",
                          facets = c("prompt", "model", "run"))
print(g)  # G-study variance components

# D-study: project dependability for a specific measurement procedure
ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 1)

Fairness (Module 2) — integrates with `aiDIF`

# outcome-level group disparity (this package)
ai_group_disparity(outcome, group, predicted_class = decisions)

# item-level IRT-based AI-scoring bias -> dispatches to aiDIF if installed
ai_fairness(outcome, group, human_mle = h, ai_mle = a)

aiEvalR’s fairness functions and the causalfragility-lab/aiDIF package answer different questions: aiEvalR checks whether overall outcomes/decisions differ by group; aiDIF checks whether an AI scoring engine shifts specific item difficulties differently by group (via paired human/AI IRT calibration and a Differential AI Scoring Bias test). Use both, not one instead of the other. aiDIF is on CRAN (install.packages("aiDIF")).

Naming notes

Some functions carry deliberately precise names to avoid overclaiming what they measure:

Function	Rationale
`lexical_overlap()`	Measures token overlap (Jaccard); named to make clear it does NOT detect negation/contradiction or factual consistency.
`empirical_output_interval()`	Describes the observed spread of repeated outputs; named to avoid implying calibrated prediction-interval coverage guarantees.
`ai_group_disparity()`	A between-group difference is a disparity statistic, not automatically “bias” without further context.
`ai_dif_mh()`	A basic Mantel-Haenszel screening helper; named to distinguish it from the fuller IRT-based `aiDIF` package.

Installation

devtools::install_github("causalfragility-lab/aiEvalR")

Citation

Hait, S. (2026). aiEvalR: Statistical and Psychometric Evaluation of AI Systems. R package version 0.1.0.

Generalizability-theory foundation:

Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The Dependability of Behavioral Measurements. Wiley.

Brennan, R. L. (2001). Generalizability Theory. Springer.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.