The Generalizability-Theory Core

From one reliability number to a variance decomposition

Classical reliability gives you a single coefficient. But when you evaluate an AI system, the “error” in a score has structure: some variation comes from which prompt you used, some from which model version, some from stochastic run-to-run variation. Generalizability theory (Cronbach et al., 1972; Brennan, 2001) decomposes score variance into these separate sources.

aiEvalR treats the object of measurement (a “case” – the underlying task) as what you want a generalizable score about, and treats prompt formulation, model, and run as facets whose variation is error.

A G-study

We simulate a fully crossed design: 20 cases, each evaluated under 3 prompt formulations, 2 models, and 2 stochastic runs. In real use, these would be actual AI outputs; here we plant known variance components so you can see them recovered.

design <- expand.grid(case = 1:20, prompt = 1:3, model = 1:2, run = 1:2)

# plant variance: cases differ most, then prompts, then model, then run
case_eff   <- rnorm(20, sd = 1.0)
prompt_eff <- rnorm(3,  sd = 0.5)
model_eff  <- rnorm(2,  sd = 0.3)
run_eff    <- rnorm(2,  sd = 0.2)

design$score <- 5 +
  case_eff[design$case] + prompt_eff[design$prompt] +
  model_eff[design$model] + run_eff[design$run] +
  rnorm(nrow(design), sd = 0.5)

g <- ai_generalizability(
  design,
  score  = "score",
  case   = "case",
  facets = c("prompt", "model", "run")
)
#> boundary (singular) fit: see help('isSingular')
g
#> Generalizability (G-)Study Variance Components
#> ================================================
#> case:prompt                   0.0256  (  2.1%)
#> case:run                      0.0002  (  0.0%)
#> case:model                    0.0210  (  1.8%)
#> case                          0.7856  ( 66.0%)
#> prompt                        0.0917  (  7.7%)
#> run                           0.0000  (  0.0%)
#> model                         0.0687  (  5.8%)
#> residual                      0.1973  ( 16.6%)
#> 
#> Run ai_dstudy() on this object to get G-coefficient / Phi for a
#> specific number of conditions per facet.

The variance components tell you where the noise lives. If prompt variance is large, your evaluation is sensitive to prompt wording; if run variance dominates, the system is stochastically unstable.

A D-study: how many conditions do you need?

The G-study estimates variance components once. A D-study then projects reliability for any hypothetical measurement design – without refitting – so you can ask “how many prompts and runs do I need for a dependable score?”

# dependability using 3 prompts, 2 models, 2 runs (as observed)
ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 2)
#> D-Study Projection
#> ==================
#> Conditions per facet:  prompt=3, model=2, run=2 
#> 
#> Universe score variance:  0.786
#> Relative error variance:  0.019
#> Absolute error variance:  0.100
#> G-coefficient (relative): 0.976
#> Phi (absolute/dependability): 0.887

Two coefficients are reported. The generalizability coefficient (relative error) suits rank-ordering systems; Phi (absolute error, “dependability”) suits criterion-referenced decisions like “is this system’s score above a fixed bar.” Reducing conditions lowers both:

full    <- ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 2)
minimal <- ai_dstudy(g, n_prompt = 1, n_model = 1, n_run = 1)

c(full_phi = full$phi, minimal_phi = minimal$phi)
#>    full_phi minimal_phi 
#>   0.8865966   0.6601418

Fewer conditions per facet means a less dependable score – the D-study quantifies exactly how much you lose.

Conditional error and decision consistency

Measurement error is often not uniform: a system may be steady on typical inputs and erratic near decision boundaries. ai_conditional_sem() estimates error as a function of the score level. When outputs drive a categorical decision via a threshold, ai_decision_consistency() estimates how often repeated administrations would agree.

# rows = prompts, cols = repeated occasions; some prompts sit near a cut
responses <- rbind(
  c(9, 9, 10),   # clearly above a cut of 5 every time
  c(1, 0, 1),    # clearly below
  c(4, 6, 5)     # right on the boundary -> inconsistent
)
ai_decision_consistency(responses, cutpoint = 5)
#> $consistency
#> [1] 0.6666667
#> 
#> $kappa
#> [1] 0.6

A prompt whose repeated outputs straddle the cutpoint yields inconsistent decisions even when the underlying score is only slightly uncertain – decision consistency captures that risk directly, which a single reliability coefficient does not.

References

Brennan, R. L. (2001). Generalizability Theory. Springer.

Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The Dependability of Behavioral Measurements. Wiley.