Information and Error-Rate Metrics

In plain terms

Before running a lineup, you are uncertain whether the suspect is guilty. After the witness responds, you are — usually — less uncertain. EIG measures how much of that uncertainty the procedure removes, on average.

The unit is the bit, which has a concrete meaning: one bit is the uncertainty in a fair coin toss. If you begin thinking guilt is a coin toss and end up certain, you have gained exactly 1 bit. An EIG of 0 means the responses told you nothing; an EIG of 1 (with a 50% prior) would mean every response resolved the case completely.

The virtue of this is that it grades every response, not just identifications. A confident rejection is informative — it makes guilt less likely — and EIG counts that, where a measure built only from suspect identifications cannot.

The arithmetic

Uncertainty is Shannon entropy. For a probability \(p\) of guilt,

\[ H(p) = -p \log_2 p - (1-p) \log_2 (1-p) \]

which is 1 bit at \(p = 0.5\) and 0 bits at \(p = 0\) or \(p = 1\).

For each response category \(x\) — a decision crossed with a confidence bin — Bayes’ theorem gives a posterior \(P(G \mid x)\), and the information that response carried is the drop in entropy:

\[ \mathrm{IG}(x) = H\bigl(P(G)\bigr) - H\bigl(P(G \mid x)\bigr) \]

EIG is the average of those, weighted by how often each response occurs:

\[ \mathrm{EIG} = \sum_x P(x)\,\Bigl[H\bigl(P(G)\bigr) - H\bigl(P(G \mid x)\bigr)\Bigr] \]

Two consequences follow. Individual responses can have negative information gain — a response that moves belief towards 0.5 increases uncertainty — but the expectation over all responses cannot be negative. And EIG depends on the prior \(P(G)\), which is set by prior_guilt and is a property of the investigation, not of the procedure.

eig_obj <- make_eig(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100),
  show_plot = FALSE
)

eig_obj$eig
#> [1] 0.2011801

Against a prior entropy of 1 bit, that is 20% of the uncertainty removed — informative, but a long way from resolving the question.

Response-level information gain

knitr::kable(
  head(as.data.frame(eig_obj$response_data), 8),
  digits = 4, row.names = FALSE,
  caption = "Per-response information gain and posterior probability of guilt, lineup_example."
)

Plots

plot_eig(eig_obj)

plot_eig_posteriors(eig_obj)

When the three coincide, and why

lineup_example has a designated innocent suspect: its target-absent lineups record whether the identification fell on the suspect or a foil. When that is known, nothing needs estimating, and the corrections have no work to do:

ppv_designated <- make_ppv_range(
  lineup_example,
  lineup_size = 6,
  show_plots = FALSE
)

knitr::kable(
  data.frame(
    Correction = c("Nominal size", "Effective size", "None"),
    `Overall PPV` = round(c(ppv_designated$ppv_nominal$overall_ppv,
                            ppv_designated$ppv_effective$overall_ppv,
                            ppv_designated$ppv_none$overall_ppv), 4),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "PPV under three corrections, lineup_example -- which designates an innocent suspect, so all three coincide."
)

All three are identical, and that is correct rather than a fault: the count being corrected for is observed directly. make_ppv_range() says so when it happens.

When they diverge

The corrections are important when no innocent suspect is designated — the usual situation with field data. We can create that case by relabelling the target-absent suspect identifications as foil identifications and removing the suspect_pos column, which is exactly the information loss that occurs when the lineup did not nominate a suspect in advance. Both steps are needed: a non-missing suspect_pos tells make_ppv_range() that a suspect was designated, and the design would then stay designated with zero innocent-suspect identifications, giving PPV = 1 under every correction.

no_designation <- lineup_example
ta_suspect <- !no_designation$target_present &
               no_designation$identification == "suspect"
no_designation$identification[ta_suspect] <- "filler"
no_designation$suspect_pos <- NULL

ppv_estimated <- make_ppv_range(
  no_designation,
  lineup_size = 6,
  innocent_suspect = "estimated",  # state the design rather than rely on "auto"
  show_plots = FALSE
)

knitr::kable(
  data.frame(
    Correction = c("Nominal size (best case)",
                   "Effective size",
                   "None (worst case)"),
    `Overall PPV` = round(c(ppv_estimated$ppv_nominal$overall_ppv,
                            ppv_estimated$ppv_effective$overall_ppv,
                            ppv_estimated$ppv_none$overall_ppv), 4),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "PPV under three corrections, same data with the innocent-suspect designation removed. This is the range the method exists to report."
)

The spread is large, from about 0.45 to about 0.83. An analysis reporting only the nominal-size figure would be asserting that the lineup was perfectly fair, which is precisely what is unknown.

The effective-size correction sits between the two because the lineup’s members do not attract choices equally:

knitr::kable(
  data.frame(
    `Confidence bin` = ppv_estimated$ppv_effective$ppv_data$confidence,
    `Effective size` = round(ppv_estimated$ppv_effective$ppv_data$effective_size, 2),
    `Error rate`     = round(ppv_estimated$ppv_effective$ppv_data$error_rate, 3),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "Effective size estimated within each confidence bin, against a nominal size of 6."
)

Plots

plot_ppv_range(ppv_estimated)

plot_effective_size_conf(ppv_estimated)

EIG

Computed over response categories formed by crossing decision with confidence bin. For each category the posterior comes from Bayes’ theorem using the observed likelihoods \(P(x \mid G)\) and \(P(x \mid \neg G)\), estimated as response proportions within the target-present and target-absent arms respectively. Entropies are in bits (base-2 logarithms). The prior is set by prior_guilt, defaulting to 0.5, at which the prior entropy is exactly 1 bit.

EIG inherits the binning issue discussed in vignette("calibration_decision_analysis"): it is computed over discrete categories, so a category with no observations in one arm gives an unbounded likelihood ratio, and the number of bins is a researcher decision that affects the result.

PPV range

With a designated innocent suspect, innocent-suspect identifications are counted directly and no correction is applied. Otherwise, writing \(e\) for the target-absent error rate within a confidence bin:

\[ \hat{r}_{\text{nominal}} = e/k, \qquad \hat{r}_{\text{effective}} = e/\hat{E}, \qquad \hat{r}_{\text{none}} = e \]

where \(\hat{E}\) is effective size estimated within the bin. PPV then follows as hits / (hits + estimated innocent-suspect identifications).

Assumptions

1. Target-present and target-absent lineups are independent samples.
2. The nominal-size correction assumes a perfectly fair lineup. It is a bound, not an estimate.
3. Effective size is estimated per confidence bin, so bins with few observations give unstable corrections — check the counts before relying on a bin.
4. Effective size is itself a plug-in estimate with small-sample bias; see the “Small-sample bias” section of ?esize_T. In near-fair lineups the bias is downward, which makes the correction more conservative, but its direction is not guaranteed: with strongly unequal choices the plug-in estimate can be biased upward. Bins with few target-absent choices deserve particular care.