--- title: "Information and Error-Rate Metrics" subtitle: "EIG and PPV-Range for Lineup Decisions" author: - "Colin G. Tredoux" - "Tamsyn M. Naylor" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Information and Error-Rate Metrics} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- > **Acknowledgement.** Large language models supplied by Anthropic and OpenAI > were used in preparing this vignette, to assist with checking code, testing > the veracity of the approaches described, and on occasion editing the text. > Responsibility for the content rests with the authors. ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 5, warning = FALSE, message = FALSE ) ``` ## Introduction This vignette demonstrates two complementary approaches for evaluating lineup evidence: 1. **Expected Information Gain (EIG)** following Starns et al., which quantifies how much a response reduces uncertainty about guilt. 2. **PPV-range estimation** following Fitzgerald, Tredoux, and Juncu, which reports how positive predictive value changes under different corrections for lineup size. Both methods use the same data format as ROC/CAC analyses: a data frame with `target_present`, `identification`, and `confidence` columns. ```{r} library(r4lineups) data(lineup_example) head(lineup_example) ``` ## 1. Expected Information Gain (EIG) ### In plain terms Before running a lineup, you are uncertain whether the suspect is guilty. After the witness responds, you are — usually — less uncertain. EIG measures **how much of that uncertainty the procedure removes, on average**. The unit is the **bit**, which has a concrete meaning: one bit is the uncertainty in a fair coin toss. If you begin thinking guilt is a coin toss and end up certain, you have gained exactly 1 bit. An EIG of 0 means the responses told you nothing; an EIG of 1 (with a 50% prior) would mean every response resolved the case completely. The virtue of this is that it grades *every* response, not just identifications. A confident rejection is informative — it makes guilt less likely — and EIG counts that, where a measure built only from suspect identifications cannot. ### The arithmetic Uncertainty is Shannon entropy. For a probability \(p\) of guilt, \[ H(p) = -p \log_2 p - (1-p) \log_2 (1-p) \] which is 1 bit at \(p = 0.5\) and 0 bits at \(p = 0\) or \(p = 1\). For each response category \(x\) — a decision crossed with a confidence bin — Bayes' theorem gives a posterior \(P(G \mid x)\), and the information that response carried is the drop in entropy: \[ \mathrm{IG}(x) = H\bigl(P(G)\bigr) - H\bigl(P(G \mid x)\bigr) \] EIG is the average of those, weighted by how often each response occurs: \[ \mathrm{EIG} = \sum_x P(x)\,\Bigl[H\bigl(P(G)\bigr) - H\bigl(P(G \mid x)\bigr)\Bigr] \] Two consequences follow. Individual responses can have **negative** information gain — a response that moves belief towards 0.5 increases uncertainty — but the expectation over all responses cannot be negative. And EIG depends on the prior \(P(G)\), which is set by `prior_guilt` and is a property of the investigation, not of the procedure. ```{r eig_compute} eig_obj <- make_eig( lineup_example, confidence_bins = c(0, 60, 80, 100), show_plot = FALSE ) eig_obj$eig ``` Against a prior entropy of `r round(eig_obj$prior_entropy, 2)` bit, that is `r round(100 * eig_obj$eig / eig_obj$prior_entropy)`% of the uncertainty removed — informative, but a long way from resolving the question. ### Response-level information gain ```{r eig_responses} knitr::kable( head(as.data.frame(eig_obj$response_data), 8), digits = 4, row.names = FALSE, caption = "Per-response information gain and posterior probability of guilt, lineup_example." ) ``` ### Plots ```{r plot_eig} plot_eig(eig_obj) ``` ```{r plot_eig_posteriors} plot_eig_posteriors(eig_obj) ``` ## 2. PPV-range estimation Positive predictive value is the proportion of suspect identifications that are correct: hits / (hits + false alarms). The difficulty is the denominator. In field data you observe that *someone* was identified, but not whether a target-absent identification landed on the suspect or on a foil — and only the former is a wrongful identification. The response is to *estimate* the innocent-suspect identifications from the observed errors, and the estimate depends on an assumption about lineup size (Fitzgerald, Tredoux & Juncu, 2023): - **Nominal size.** Errors are spread evenly over all \(k\) members, so \(1/k\) of them fall on the innocent suspect. The most optimistic assumption, and it requires the lineup to be perfectly fair. - **Effective size.** Errors are spread over the members that actually attract choices, estimated from the data. Fewer functioning alternatives means more errors land on the suspect. - **No correction.** Every mistaken identification is treated as an innocent-suspect identification. The worst case, and a genuine lower bound. Reporting all three gives a *range* rather than a single figure resting on an assumption that cannot be checked. ### When the three coincide, and why `lineup_example` has a **designated innocent suspect**: its target-absent lineups record whether the identification fell on the suspect or a foil. When that is known, nothing needs estimating, and the corrections have no work to do: ```{r ppv_designated} ppv_designated <- make_ppv_range( lineup_example, lineup_size = 6, show_plots = FALSE ) knitr::kable( data.frame( Correction = c("Nominal size", "Effective size", "None"), `Overall PPV` = round(c(ppv_designated$ppv_nominal$overall_ppv, ppv_designated$ppv_effective$overall_ppv, ppv_designated$ppv_none$overall_ppv), 4), check.names = FALSE ), row.names = FALSE, caption = "PPV under three corrections, lineup_example -- which designates an innocent suspect, so all three coincide." ) ``` All three are identical, and that is correct rather than a fault: the count being corrected for is observed directly. `make_ppv_range()` says so when it happens. ### When they diverge The corrections are important when no innocent suspect is designated — the usual situation with field data. We can create that case by relabelling the target-absent suspect identifications as foil identifications **and** removing the `suspect_pos` column, which is exactly the information loss that occurs when the lineup did not nominate a suspect in advance. Both steps are needed: a non-missing `suspect_pos` tells `make_ppv_range()` that a suspect was designated, and the design would then stay designated with zero innocent-suspect identifications, giving PPV = 1 under every correction. ```{r ppv_estimated} no_designation <- lineup_example ta_suspect <- !no_designation$target_present & no_designation$identification == "suspect" no_designation$identification[ta_suspect] <- "filler" no_designation$suspect_pos <- NULL ppv_estimated <- make_ppv_range( no_designation, lineup_size = 6, innocent_suspect = "estimated", # state the design rather than rely on "auto" show_plots = FALSE ) knitr::kable( data.frame( Correction = c("Nominal size (best case)", "Effective size", "None (worst case)"), `Overall PPV` = round(c(ppv_estimated$ppv_nominal$overall_ppv, ppv_estimated$ppv_effective$overall_ppv, ppv_estimated$ppv_none$overall_ppv), 4), check.names = FALSE ), row.names = FALSE, caption = "PPV under three corrections, same data with the innocent-suspect designation removed. This is the range the method exists to report." ) ``` The spread is large, from about `r round(ppv_estimated$ppv_none$overall_ppv, 2)` to about `r round(ppv_estimated$ppv_nominal$overall_ppv, 2)`. An analysis reporting only the nominal-size figure would be asserting that the lineup was perfectly fair, which is precisely what is unknown. The effective-size correction sits between the two because the lineup's members do not attract choices equally: ```{r effective_sizes} knitr::kable( data.frame( `Confidence bin` = ppv_estimated$ppv_effective$ppv_data$confidence, `Effective size` = round(ppv_estimated$ppv_effective$ppv_data$effective_size, 2), `Error rate` = round(ppv_estimated$ppv_effective$ppv_data$error_rate, 3), check.names = FALSE ), row.names = FALSE, caption = "Effective size estimated within each confidence bin, against a nominal size of 6." ) ``` ### Plots ```{r plot_ppv_range} plot_ppv_range(ppv_estimated) ``` ```{r plot_effective_size} plot_effective_size_conf(ppv_estimated) ``` ## Interpretation guidance - **EIG** grades every response category, including rejections, and reports how much uncertainty the procedure removes overall. Use it to compare procedures, or to identify which responses carry the evidence. - **PPV range** answers the question a court asks — how often is an identification of this kind correct — and makes visible how much the answer depends on an untestable assumption about lineup size. The two are complementary: EIG is base-rate-dependent through `prior_guilt` and covers all responses; PPV concerns identifications only, and is the quantity with direct forensic meaning. ## Technical Details ### EIG Computed over response categories formed by crossing decision with confidence bin. For each category the posterior comes from Bayes' theorem using the observed likelihoods \(P(x \mid G)\) and \(P(x \mid \neg G)\), estimated as response proportions within the target-present and target-absent arms respectively. Entropies are in bits (base-2 logarithms). The prior is set by `prior_guilt`, defaulting to 0.5, at which the prior entropy is exactly 1 bit. EIG inherits the binning issue discussed in `vignette("calibration_decision_analysis")`: it is computed over discrete categories, so a category with no observations in one arm gives an unbounded likelihood ratio, and the number of bins is a researcher decision that affects the result. ### PPV range With a designated innocent suspect, innocent-suspect identifications are counted directly and no correction is applied. Otherwise, writing \(e\) for the target-absent error rate within a confidence bin: \[ \hat{r}_{\text{nominal}} = e/k, \qquad \hat{r}_{\text{effective}} = e/\hat{E}, \qquad \hat{r}_{\text{none}} = e \] where \(\hat{E}\) is effective size estimated within the bin. PPV then follows as hits / (hits + estimated innocent-suspect identifications). ### Assumptions 1. Target-present and target-absent lineups are independent samples. 2. The nominal-size correction assumes a perfectly fair lineup. It is a bound, not an estimate. 3. Effective size is estimated per confidence bin, so bins with few observations give unstable corrections — check the counts before relying on a bin. 4. Effective size is itself a plug-in estimate with small-sample bias; see the "Small-sample bias" section of `?esize_T`. In near-fair lineups the bias is downward, which makes the correction more conservative, but its direction is not guaranteed: with strongly unequal choices the plug-in estimate can be biased upward. Bins with few target-absent choices deserve particular care. ## References ::: {.hanging-indent} Fitzgerald, R. J., Tredoux, C. G., & Juncu, S. (2023). Estimation of eyewitness error rates in fair and biased lineups. *Law and Human Behavior, 47*(4), 463-483. Starns, J. J., Cohen, A. L., & Rotello, C. M. (2023). A complete method for assessing the effectiveness of eyewitness identification procedures: Expected information gain. *Psychological Review, 130*(3), 677-719. ::: ## Further Reading - `?make_eig`, `?make_ppv_range` - `vignette("calibration_decision_analysis")` for the Bayesian information-gain curves, which report the same evidence on a per-response rather than expected basis