---
title: "Getting started with evaluatellm"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with evaluatellm}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
set.seed(3)
```

```{r setup}
library(evaluatellm)
```

An evaluation is an experiment. It draws a sample of questions from the much
larger set of questions someone could have written, runs a model on them, and
reports a mean. Like any experiment it has a standard error, and like any
experiment it can be too small to answer the question it was built to answer.

This vignette walks through the workflow on a simulated reading comprehension
evaluation: 50 passages, 8 questions each, two models.

## Building the evaluation object

Passages differ in difficulty, and every question about a passage inherits that
difficulty. This is what makes the questions dependent.

```{r}
passage_difficulty <- rnorm(50, 0, 1.3)
item_difficulty <- rep(passage_difficulty, each = 8) + rnorm(400, 0, 0.4)

skill <- c(new = 0.95, old = 0.72)
results <- do.call(rbind, lapply(names(skill), function(m) {
  data.frame(
    q       = 1:400,
    passage = rep(1:50, each = 8),
    model   = m,
    correct = rbinom(400, 1, plogis(skill[[m]] - item_difficulty))
  )
}))

e <- as_eval(results, score = correct, item = q, model = model,
             cluster = passage)
e
```

Declaring `cluster` is the single most consequential argument in the package.
Everything downstream uses it.

## Scoring one model

```{r}
ev_score(e, "new")
```

The design effect says the cluster-robust standard error is three times the
variance of the naive one. Turning the cluster correction off shows what would
otherwise have been reported:

```{r}
ev_score(e, "new", cluster = FALSE)
```

Same estimate, an interval roughly 40 per cent too narrow. `ev_cluster()` breaks
this down further, and `ev_icc()` returns the intra-cluster correlation on its
own, which is what you need to plan the next evaluation.

```{r}
ev_icc(e, "new")
```

## Comparing two models

Both models answered the same questions, so the comparison should be paired.

```{r}
ev_paired(e, model_a = "new", model_b = "old")
```

Two things are worth reading here. The correlation between the models' item
scores drives the variance reduction from pairing: the more alike the models,
the more pairing buys. And the difference is far less affected by clustering
than the level was, because taking the difference has already removed the
passage difficulty both models faced.

For models run on different question sets, `ev_unpaired()` gives a Welch
interval, at a real cost in power.

## Sizing an evaluation before running it

Run this first, not last.

```{r}
ev_mde(n_items = 400, p_a = 0.72, p_b = 0.70, correlation = 0.7,
       icc = 0.25, cluster_size = 8)
```

Four hundred clustered questions cannot resolve a two point gap. A null result
from this evaluation would say nothing about the models. To find the size that
would work:

```{r}
ev_power(delta = 0.02, p_a = 0.72, p_b = 0.70, correlation = 0.7,
         icc = 0.25, cluster_size = 8)
```

The best source for `sd_diff` is a pilot run passed back through `pilot =`.

## Several responses per question

Sampling `k` responses per question splits the observed spread into genuine
variation between questions and noise in the model's own sampling.

```{r}
p_i <- plogis(rnorm(200, 0.7, 1.1))
rep_d <- data.frame(
  q    = rep(1:200, each = 6),
  draw = rep(1:6, times = 200),
  s    = rbinom(1200, 1, rep(p_i, each = 6))
)
ev_resample(as_eval(rep_d, score = s, item = q, sample = draw))
```

The projection table is the useful part: more responses per question buys
precision only against the within-question term, and there is a floor no amount
of resampling can beat. Getting below it means writing more questions.

## Evaluations graded by a model judge

A judge is cheap and biased. Bias does not shrink as you grade more items, so
measure it first.

```{r}
truth <- rbinom(500, 1, 0.55)
judge_pilot <- truth
judge_pilot[truth == 0] <- rbinom(sum(truth == 0), 1, 0.35)
ev_judge_agreement(judge_pilot, truth)
```

Read the kappa rather than the raw agreement, and take the McNemar test
seriously: a judge whose errors run one way is putting that error into every
score it grades.

The fix is not a better prompt. Grade everything with the judge, label a random
subset by hand, and combine them.

```{r}
n_total <- 5000
truth_all <- rbinom(n_total, 1, 0.62)
judge <- ifelse(runif(n_total) < 0.87, truth_all, 1 - truth_all)
judge[truth_all == 0 & runif(n_total) < 0.15] <- 1

gold <- rep(NA_real_, n_total)
labelled <- sample(n_total, 250)
gold[labelled] <- truth_all[labelled]

ev_judge_debias(judge, gold)
```

The estimator is unbiased for any tuning constant, so no assumption about judge
quality is being smuggled in, and the default tuning guarantees it is never less
precise than ignoring the judge entirely. To size the labelling budget in
advance:

```{r}
ev_judge_power(n_total = 20000, correlation = 0.8, target_se = 0.01)
```

## Suites and leaderboards

Reporting the tasks a model won, out of the tasks it was tried on, needs a
multiplicity adjustment. `ev_multi()` applies one, pools the tasks by inverse
variance, and tests whether they are telling the same story.

```{r}
suite <- lapply(1:6, function(i) {
  a <- rbinom(250, 1, 0.72)
  b <- rbinom(250, 1, 0.68)
  d <- data.frame(q = rep(1:250, 2), m = rep(c("new", "old"), each = 250),
                  s = c(a, b))
  ev_paired(as_eval(d, score = s, item = q, model = m), "new", "old")
})
names(suite) <- paste0("task_", 1:6)
ev_multi(suite)
```

For leaderboards, `ev_rank()` bootstraps the ordering so you can see which
positions the data actually support, and `ev_elo()` fits Bradley-Terry ratings
with standard errors on pairwise preference data.

## Reporting

`ev_table()` flattens results into a data frame and `ev_plot()` draws them.

```{r, fig.width = 7, fig.height = 3.5}
ev_plot(suite, reference = 0, xlab = "accuracy gain")
```