Calibration and Decision Analysis

Why there is a lineup_member column

lineup_example records two further columns, for a reason that applies to any lineup data you collect or simulate yourself.

identification collapses a witness’s decision into three categories. That is what the analysis functions consume, but it is not what should be recorded. The raw datum is which lineup member the witness chose — lineup_member, with 0 for a rejection — together with the suspect’s position, suspect_pos. The label is then derived:

head(classify_lineup_choices(
  lineup_example[, c("lineup_member", "suspect_pos")],
  k = 6
))
  lineup_member suspect_pos identification
1             2           1         filler
2             3           1         filler
3             2           1         filler
4             1           1        suspect
5             1           1        suspect
6             0           1         reject

This is important because foils are not interchangeable. They differ in how plausible they are, and a foil that nobody ever chooses is not a functioning alternative to the suspect at all — it is precisely what makes a lineup unfair. Recording only “filler” throws that information away. In lineup_example the five foils attract very different numbers of choices:

lineup_table(
  lineup_example$lineup_member[lineup_example$lineup_member > 0],
  k = 6, suspect_pos = 1
)
Lineup choices (k = 6, N = 1132)

 pos:  1*    2    3    4   5   6
 cnt: 503  248  203  115  53  10

 * = nominated suspect

lineup_example is simulated, and it is simulated at the level of individual foils: each is given its own mean memory strength rather than being drawn from a common distribution. See ?simulate_lineup_data and its foil_bias argument if you want to generate data of your own on the same basis.

Basic Calibration

# Compute calibration with binned confidence
cal_result <- make_calibration(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100),
  choosers_only = TRUE  # Analyze only suspect identifications
)

print(cal_result)

=== Lineup Calibration Analysis ===

Analysis type: Choosers only (suspect IDs)
Total N: 503 

Calibration Statistics:
  C (Calibration):         0.0727 
  O/U (Over/Under):        -0.2290 
  NRI (Resolution):        0.0453 

Overall Performance:
  Mean Accuracy:      0.825 
  Mean Confidence:    0.596 

Calibration Data by Bin:
# A tibble: 3 × 7
  bin          n mean_confidence accuracy n_correct n_incorrect
  <chr>    <dbl>           <dbl>    <dbl>     <dbl>       <dbl>
1 [0,60]     305            42.6    0.764       233          72
2 (60,80]     94            74.0    0.872        82          12
3 (80,100]   104            96.5    0.962       100           4
# ℹ 1 more variable: mean_confidence_prop <dbl>

Plot: save this object first, then extract it with x$plot

The output provides three key statistics:

• C statistic: Overall calibration quality (0 = perfect, higher = worse)
• O/U (Over/Under confidence): Positive = overconfident, Negative = underconfident
• NRI (Normalized Resolution Index): Ability to discriminate accuracy levels with confidence (higher = better)

Interpreting the Calibration Plot

The calibration plot shows:

• Diagonal line: Perfect calibration
• Points above diagonal: Underconfident (accuracy exceeds confidence)
• Points below diagonal: Overconfident (confidence exceeds accuracy)
• Point size: Sample size in each bin

cal_result$plot

Accuracy rises steadily across the confidence bins, from about 76% in the lowest to about 96% in the highest, so confidence here is genuinely informative. The points sit above the diagonal, however, which means these witnesses are underconfident: they are more accurate than their stated confidence implies. That is the direction the O/U statistic reports.

The plot is an ordinary ggplot2 object, so it can be modified and saved in the usual way. ggsave() picks its output format from the file extension alone, so the same call writes PNG, PDF, SVG or TIFF depending on what you ask for:

outfile <- file.path(tempdir(), "calibration_plot.png")
ggplot2::ggsave(outfile, cal_result$plot, width = 8, height = 6, dpi = 300)

# Change only the extension to change the format:
#   ggsave("calibration_plot.pdf", cal_result$plot)   # vector, for print
#   ggsave("calibration_plot.svg", cal_result$plot)   # vector, for the web

Calibration by Condition

The question here is whether calibration differs between two identification procedures. To answer it we need data from two procedures, and lineup_example records only one, so we simulate them.

The comparison is only meaningful if the two conditions genuinely differ. We give Procedure A the stronger memory evidence (d_prime = 1.8 against 1.1), as might arise from better viewing conditions or a better-constructed lineup, and give the two lineups different foil quality as well. Note that both are seeded, so the results below are reproducible rather than changing on every rebuild.

# Procedure A: stronger evidence, and foils that are reasonable alternatives
data_proc_a <- as.data.frame(simulate_lineup_data(
  n_tp = 400, n_ta = 400, d_prime = 1.8, c_criterion = 0.9,
  lineup_size = 6, conf_levels = 9,
  foil_bias = c(0.4, 0.2, 0.0, -0.3, -0.8), seed = 101
))

# Procedure B: weaker evidence, and one near-useless foil
data_proc_b <- as.data.frame(simulate_lineup_data(
  n_tp = 400, n_ta = 400, d_prime = 1.1, c_criterion = 0.9,
  lineup_size = 6, conf_levels = 9,
  foil_bias = c(0.5, 0.1, -0.2, -0.9, -2.0), seed = 102
))

# Put confidence on the same 20-100 scale used by lineup_example
conf_scale <- round(seq(20, 100, length.out = 9))
data_proc_a$confidence <- conf_scale[data_proc_a$confidence]
data_proc_b$confidence <- conf_scale[data_proc_b$confidence]

# Combine into one frame with a condition variable
procedures <- rbind(
  transform(data_proc_a, procedure = "Procedure A"),
  transform(data_proc_b, procedure = "Procedure B")
)

cal_by_cond <- make_calibration_by_condition(
  procedures,
  condition_vars = "procedure",
  confidence_bins = c(0, 60, 80, 100)
)

knitr::kable(
  cal_by_cond$condition_summary, digits = 3,
  caption = "Calibration statistics by procedure, simulated data (choosers only)."
)

The per-bin detail for each procedure is in cal_by_cond$by_condition. The object does not carry a plot; draw one with make_calibration_by_condition_gg():

make_calibration_by_condition_gg(cal_by_cond)

The arithmetic, briefly

Let \(G\) denote the event that the suspect is guilty and \(x\) a particular identification response — say, a suspect identification made with confidence above 80. Bayes’ theorem gives the posterior probability of guilt as

\[ P(G \mid x) \;=\; \frac{P(x \mid G)\,P(G)}{P(x \mid G)\,P(G) + P(x \mid \neg G)\,\bigl(1 - P(G)\bigr)} \]

The two conditional probabilities are estimated directly from the data: \(P(x \mid G)\) is the proportion of target-present witnesses who gave response \(x\), and \(P(x \mid \neg G)\) the proportion of target-absent witnesses who did. Their ratio is the likelihood ratio,

\[ \mathrm{LR}(x) \;=\; \frac{P(x \mid G)}{P(x \mid \neg G)} \]

and it is the whole of what the identification contributes. An \(\mathrm{LR}\) of 1 means the response is uninformative; larger values point towards guilt, smaller towards innocence. make_bayes_curves() returns these quantities in $likelihoods:

lik <- make_bayes_curves(
  lineup_example,
  response_categories = "confidence",
  confidence_bins = c(0, 60, 80, 100)
)$likelihoods

knitr::kable(
  transform(
    as.data.frame(lik)[, c("response", "p_x_given_guilty",
                           "p_x_given_innocent", "n_guilty", "n_innocent")],
    LR = round(lik$p_x_given_guilty / lik$p_x_given_innocent, 2)
  ),
  digits = 3, row.names = FALSE,
  caption = "Response likelihoods and likelihood ratios, lineup_example."
)

A high-confidence suspect identification carries a likelihood ratio of roughly 25 here, while a rejection carries one of about 0.3 — that is, it is evidence against guilt. Note also that the likelihood ratio, not the posterior, is the portable quantity: it belongs to the identification procedure, whereas the posterior additionally depends on the prior, which belongs to the investigation.

One practical warning. If no target-absent witness ever produces a given response, the estimated \(P(x \mid \neg G)\) is zero, the likelihood ratio is infinite, and the posterior curve is pinned at 1.0 for every prior. That is almost always a sign of too few observations or too narrow a bin rather than of perfectly diagnostic evidence, and make_bayes_curves() warns when it happens.

Prior-Posterior Curves

# Compute Bayesian curves for simple responses
bayes_result <- make_bayes_curves(
  lineup_example,
  response_categories = "simple",  # "suspect", "filler", "reject"
  prior_grid = seq(0.01, 0.99, 0.01)
)

print(bayes_result)
$curves
# A tibble: 297 × 4
   response prior posterior information_gain
   <chr>    <dbl>     <dbl>            <dbl>
 1 filler    0.01   0.00533           0.0329
 2 filler    0.02   0.0107            0.0560
 3 filler    0.03   0.0161            0.0752
 4 filler    0.04   0.0216            0.0918
 5 filler    0.05   0.0272            0.106 
 6 filler    0.06   0.0327            0.119 
 7 filler    0.07   0.0384            0.131 
 8 filler    0.08   0.0441            0.141 
 9 filler    0.09   0.0498            0.151 
10 filler    0.1    0.0557            0.159 
# ℹ 287 more rows

$likelihoods
# A tibble: 3 × 5
  response p_x_given_guilty p_x_given_innocent n_guilty n_innocent
  <chr>               <dbl>              <dbl>    <int>      <dbl>
1 filler             0.311               0.587      218        411
2 reject             0.0957              0.287       67        201
3 suspect            0.593               0.126      415         88

$response_counts
# A tibble: 3 × 3
  response n_guilty n_innocent
  <chr>       <int>      <dbl>
1 filler        218        411
2 reject         67        201
3 suspect       415         88

$n_guilty
[1] 700

$n_innocent
[1] 700

$response_categories
[1] "simple"

$lineup_size
[1] 6

$innocent_suspect_method
[1] "designated"

# Plot prior-posterior relationships
plot_bayes_prior_posterior(bayes_result)

The plot shows how each response type updates prior belief:

• Lines above diagonal: Evidence increases guilt belief (diagnostic of guilt)
• Lines below diagonal: Evidence decreases guilt belief (diagnostic of innocence)
• Distance from the diagonal: at a given prior, the further a curve lies from the diagonal, the larger the likelihood ratio of that response. Compare curves at the same prior, not by their local slopes.

The prior-posterior curve is due to Wells, Yang and Smalarz (2015), who introduced it along with the base-rate effect-equivalency curve used later in this vignette. The implementation here is written directly from that paper.

On the shape of these curves. They are curved, and that curvature is not an artefact of any smoothing: no loess or spline is fitted anywhere in these plots. For a response with likelihood ratio \(L\), the posterior as a function of the prior \(p\) is \[ q(p) = \frac{Lp}{1 + (L - 1)p}, \qquad q''(p) = \frac{-2L(L - 1)}{[1 + (L - 1)p]^3}. \] The curvature has one sign throughout: the curve is concave (bowed above the diagonal) when \(L > 1\), convex (bowed below it) when \(L < 1\), and the diagonal itself when \(L = 1\). It is not S-shaped. What the display shows is that the same response moves belief most, in absolute terms, at middling priors, and very little when the prior is already near 0 or near 1.

Information Gain

Information gain quantifies uncertainty reduction, measured in bits: the entropy of the prior minus the entropy of the posterior. Where the posterior curve says where belief ends up, information gain says how much uncertainty was removed:

plot_bayes_information_gain(bayes_result)

Interpretation:

• Positive values: the response moves belief towards certainty (towards 0 or 1), so uncertainty falls.
• Negative values: the response moves belief towards 50/50, so uncertainty rises. That is not a sign of misleading evidence: a suspect identification given a low prior raises the probability of guilt towards .5, which is the correct update and also the most uncertain state.
• Where each curve peaks depends on the direction of the evidence. Responses that point to guilt (suspect identifications) gain most at priors above .5, where they push belief towards 1; responses that point to innocence (rejections, filler identifications) gain most at priors below .5.

The curves bend sharply because binary entropy changes fastest near 0 and 1. That shape is a property of the measure, not of these data or of any smoothing.

Confidence-Based Bayesian Analysis

Analyze by confidence level:

bayes_conf <- make_bayes_curves(
  lineup_example,
  response_categories = "confidence",
  confidence_bins = c(0, 60, 80, 100)
)

# Get suspect responses only
suspect_responses <- grep("^suspect_", unique(bayes_conf$curves$response), value = TRUE)
plot_bayes_prior_posterior(bayes_conf, selected_responses = suspect_responses)

Splitting suspect identifications by confidence separates three quite different kinds of evidence. The highest-confidence band sits well above the others across the whole range of priors, which is the Bayesian restatement of the calibration result found earlier: high-confidence identifications from this procedure are worth considerably more than low-confidence ones.

The three curves stay clearly apart and none is pinned to the top of the plot. A curve that is pinned flat at 1.0 signals the zero-cell problem described above, not evidence of unusual strength.

Base-Rate Equivalency of Evidence (BREE)

The base-rate effect-equivalency curve, also from Wells, Yang and Smalarz (2015), answers a question that comes up whenever two procedures are compared and neither dominates: how much better is one than the other, expressed in units anyone can interpret?

The idea is to express the difference as an exchange rate between base rates. Suppose Procedure A, applied to a case with a 20% prior probability of guilt, yields a certain posterior after a suspect identification. We can then ask what prior Procedure B would need in order to reach that same posterior. If B needs a 40% prior to get where A got from 20%, then A is worth as much as a doubling of the base rate — and a doubling of the base rate is something an investigator has an intuition about, in a way that a difference in pAUC is not.

Plotting that correspondence for every prior gives the BREE curve. plot_bree() draws the base-rate shift \(\Delta\) = prior needed by B − prior used by A, against A’s prior. The dashed horizontal line at \(\Delta = 0\) is the null result of no difference; positive \(\Delta\) means B needs the larger prior, so A’s suspect identification is the more diagnostic, and the further the curve lies above zero, the larger A’s advantage. (Plotting B’s prior directly against A’s would put the reference on the diagonal instead, with A’s advantage above it.)

The two procedures are the ones simulated in the Calibration by Condition section: data_proc_a and data_proc_b are data frames of target-present and target-absent identifications with confidence, one row per witness (not lineup vectors). Procedure A has the stronger memory evidence (d′ = 1.8 against 1.1).

bree_result <- make_bree_curve(
  data_proc_a,
  data_proc_b,
  reference_response = "suspect"
)

plot_bree(bree_result)

The shift is positive throughout, from 0.003 to 0.079, so Procedure B always needs a higher prior probability of guilt to reach the posterior that A reaches — which is to say that a suspect identification from Procedure A is worth more. At a prior of 0.2 for A, B needs 0.255. The shift is largest, 0.079, near a prior of 0.46, and shrinks towards both ends of the scale, where neither procedure’s evidence can move belief much.

Define Utility Matrix

The costs and benefits must be stated explicitly. That is often felt to be the uncomfortable part of the exercise, but the values are being assumed either way; utility analysis only makes them visible and open to argument.

Four outcomes need a value, on any common scale:

utility_matrix <- c(
  tp =  1.0,   # Correct identification of a guilty suspect: a benefit
  fn = -0.5,   # Failing to identify a guilty suspect: a cost
  fp = -2.0,   # Identifying an innocent suspect: the most serious cost
  tn =  0.5    # Correctly declining to identify an innocent suspect: a benefit
)

The relative magnitudes carry the argument. Here a wrongful identification is set at four times the cost of a missed one (-2.0 against -0.5), and twice the absolute value of a correct identification’s benefit. That is a moral and legal judgement, not a statistical one, and it is the natural place for disagreement. Someone who set fp = -10 would be expressing Blackstone’s ratio; the analysis below would then recommend a considerably stricter confidence threshold. Because the choice is this important, report the sensitivity of the conclusion to it, which is what the base-rate section does for the other key assumption.

Compute Utility Curves

# Compute expected utility across confidence criteria
util_result <- make_utility_curves(
  lineup_example,
  base_rate = 0.5,  # Assume 50% guilty suspects
  utility_matrix = utility_matrix,
  lineup_size = 6
)

print(util_result)
$utility_data
# A tibble: 10 × 7
   criterion strategy   hit_rate false_alarm_rate expected_utility n_ids_tp
       <dbl> <chr>         <dbl>            <dbl>            <dbl>    <dbl>
 1       100 threshold    0.0971          0                 0.0729       68
 2        90 threshold    0.143           0.00571           0.1         100
 3        80 threshold    0.189           0.0143            0.124       132
 4        70 threshold    0.26            0.0229            0.166       182
 5        60 threshold    0.351           0.03              0.226       246
 6        50 threshold    0.436           0.0543            0.259       305
 7        40 threshold    0.507           0.0771            0.284       355
 8        30 threshold    0.556           0.103             0.288       389
 9        20 threshold    0.593           0.126             0.288       415
10       101 reject_all   0               0                 0             0
# ℹ 1 more variable: n_ids_ta <dbl>

$max_utility
$max_utility$expected_utility
       tp 
0.2882143 

$max_utility$criterion
[1] 30

$max_utility$strategy
[1] "threshold"

$max_utility$hit_rate
[1] 0.5557143

$max_utility$false_alarm_rate
[1] 0.1028571


$avg_utility
[1] 0.2008333

$utility_all_ids
    tp 
0.2875 

$base_rate
[1] 0.5

$utility_matrix
  tp   fn   fp   tn 
 1.0 -0.5 -2.0  0.5 

$n_target_present
[1] 700

$n_target_absent
[1] 700

$innocent_suspect_method
[1] "designated"

$criteria
[1] "confidence"

# Plot utility curves
plot_utility_curves(util_result)

The plot shows:

• Expected utility at each confidence threshold
• Optimal threshold: Maximum utility point
• Comparison with “all identifications” strategy

Reading it from right to left: a very strict threshold acts on almost nothing, so little is gained and little is risked. Relaxing the threshold admits identifications that are mostly correct, and utility climbs. Past the optimum, each further relaxation admits more wrongful identifications than correct ones, and because those carry four times the weight, utility falls away again. The maximum is the criterion at which the marginal identification is exactly worth acting on.

This is the shape Lampinen et al. (2019) describe. Note that the curve is usually rather flat near its peak, which is reassuring for practice: thresholds somewhat either side of the optimum lose very little, so the recommendation does not depend on locating the maximum precisely.

Compare Procedures

max_utility is a list rather than a single number: it reports the best expected utility together with the criterion that achieves it, and the hit and false-alarm rates at that criterion. Ask for the element you want.

The optimum can be a confidence criterion or the decision to reject every identification. max_utility$strategy says which, so a small helper turns the result into a rule that can be stated:

utility_rule <- function(m) {
  if (m$strategy == "reject_all") "Act on no identification"
  else paste0("Act on confidence >= ", m$criterion)
}

# Compare the two procedures simulated in the Calibration by Condition section
util_proc_a <- make_utility_curves(data_proc_a, base_rate = 0.5,
                                   utility_matrix = utility_matrix)
util_proc_b <- make_utility_curves(data_proc_b, base_rate = 0.5,
                                   utility_matrix = utility_matrix)

knitr::kable(
  data.frame(
    Procedure = c("A (d' = 1.8)", "B (d' = 1.1)"),
    `Max expected utility` = round(c(util_proc_a$max_utility$expected_utility,
                                     util_proc_b$max_utility$expected_utility), 3),
    `Optimal rule` = c(utility_rule(util_proc_a$max_utility),
                       utility_rule(util_proc_b$max_utility)),
    check.names = FALSE
  ),
  caption = "Expected utility at the optimal confidence criterion, simulated data (200 target-present and 200 target-absent lineups per procedure)."
)

The procedure with the stronger evidence supports both a higher expected utility and a stricter criterion: it can afford to wait for high-confidence identifications because it gets more of them.

Sensitivity to Base Rate

Examine how utility depends on base rate:

# Test multiple base rates
base_rates <- c(0.2, 0.5, 0.8)

sensitivity <- do.call(rbind, lapply(base_rates, function(br) {
  util <- make_utility_curves(lineup_example, base_rate = br,
                              utility_matrix = utility_matrix,
                              lineup_size = 6)
  data.frame(
    `Base rate` = br,
    `Max expected utility` = round(util$max_utility$expected_utility, 3),
    `Optimal rule` = utility_rule(util$max_utility),
    check.names = FALSE
  )
}))

knitr::kable(
  sensitivity,
  row.names = FALSE,
  caption = "Sensitivity of expected utility to the assumed base rate, lineup_example data."
)

Base rate is very important. When guilty suspects are rare, any given identification is more likely to be a false one, so the optimal criterion becomes stricter: here “act on confidence >= 60” at a base rate of 0.2, against “act on confidence >= 20” at 0.8. The attainable utility is not ordered by base rate in any simple way, because it also depends on how the cost matrix values the outcomes that become more common; here it is 0.345, 0.288 and 0.349 at the three base rates. Compare utilities across procedures at the same base rate, not across base rates.

Basic DPP

dpp_result <- make_dpp(
  lineup_example,
  lineup_size = 6
)

dpp_result$dpp
[1] 0.7001587
dpp_result$dpp_by_point
# A tibble: 9 × 4
  confidence culprit_rate innocent_rate   dpp
       <dbl>        <dbl>         <dbl> <dbl>
1        100       0.0971       0       0.903
2         90       0.143        0.00571 0.863
3         80       0.189        0.0143  0.826
4         70       0.26         0.0229  0.763
5         60       0.351        0.03    0.679
6         50       0.436        0.0543  0.619
7         40       0.507        0.0771  0.57 
8         30       0.556        0.103   0.547
9         20       0.593        0.126   0.533

plot_dpp(dpp_result)

The table lists each operating point, from the strictest threshold to the most lenient, which counts every identification. The plot shows DPP(\(c\)) at each of them, the average as a dashed line, and the chance reference at 1. The circled point has the lowest DPP and so the highest utility; it is the threshold at which acting on identifications would do the most good under equal costs and a base rate of .5.

Different costs and base rates

Smith et al.’s Equation 3 weights the two errors:

\[DPP_{\alpha\beta}(c) = \beta[1 - \text{Culprit}(c)] + \alpha(1 - \beta)\text{Innocent}(c),\]

where \(\alpha\) is the cost of identifying an innocent suspect relative to missing the culprit, and \(\beta\) is the base rate of culprit presence. With \(\alpha = 1\) and \(\beta = .5\) this is half of Equation 2. Weighted values no longer range from 0 to 2, so they are comparable only with other values computed with the same weights.

# An innocent-suspect identification three times as costly as a miss,
# with culprits present in 30% of lineups
make_dpp(lineup_example, alpha = 3, beta = 0.3)$dpp
[1] 0.296619

Compare DPP Between Groups

The comparison is the usual use. Here the two simulated procedures from earlier differ in the strength of the evidence they generate, so Procedure A should show the smaller deviation. As Smith et al. did, compare_dpp() tests the difference with a nonparametric bootstrap, resampling target-present and target-absent trials separately within each procedure:

# Stored with the vignette; set R4LINEUPS_PRECOMPUTE=true to recompute.
dpp_comparison <- compare_dpp(
  data_proc_a,
  data_proc_b,
  lineup_size = 6,
  n_bootstrap = 500,
  seed = 2026
)

dpp_comparison

=== DPP Comparison ===

Average DPP, procedure A:  0.6022 
Average DPP, procedure B:  0.8247 
Difference (A - B):        -0.2225  (negative = A better)
Bootstrap SE:              0.0268
95% percentile interval:  [-0.2797, -0.1726]  (500 resamples)

plot_dpp_comparison(dpp_comparison)

A negative difference favours Procedure A. The interval is a percentile bootstrap interval for the difference in average DPP.

When to Use DPP

Use DPP when:

• You have confidence-based ROC data
• ROC curves are truncated (witnesses may not use full scale)
• You want a single performance metric comparable across studies
• You need to compare procedures with different confidence distributions

Basic ANRI

# Compute ANRI
anri_result <- compute_anri(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100)
)

print(anri_result)
$anri
[1] 0.04145948

$nri
[1] 0.04527077

$n_total
[1] 503

$n_bins
[1] 3

$calibration_data
# A tibble: 3 × 7
  bin          n mean_confidence accuracy n_correct n_incorrect
  <chr>    <dbl>           <dbl>    <dbl>     <dbl>       <dbl>
1 [0,60]     305            42.6    0.764       233          72
2 (60,80]     94            74.0    0.872        82          12
3 (80,100]   104            96.5    0.962       100           4
# ℹ 1 more variable: mean_confidence_prop <dbl>

$overall_accuracy
[1] 0.8250497

$overall_confidence
[1] 0.5960239

$choosers_only
[1] TRUE

ANRI is always ≤ NRI because it removes positive bias. The bias correction is larger when:

• Sample size N is small
• Number of bins J is large
• The ratio J/N is high

Bootstrap Confidence Intervals

A point estimate is not much use on its own, so the interval comes from a bootstrap. n_bootstrap = 100 keeps this vignette quick; use 2000 or more for anything you intend to report.

anri_boot <- bootstrap_anri(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100),
  n_bootstrap = 100,
  conf_level = 0.95,
  seed = 123
)

knitr::kable(
  data.frame(
    Statistic = c("NRI (uncorrected)", "ANRI (bias-corrected)"),
    Estimate  = round(c(anri_boot$nri, anri_boot$anri), 3),
    `95% CI`  = c("—", sprintf("[%.3f, %.3f]",
                               anri_boot$ci_lower, anri_boot$ci_upper)),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "Resolution with bootstrap interval, lineup_example, 3 confidence bins."
)

No interval is given for NRI because it would add nothing. ANRI is a linear rescaling of NRI, \(ANRI = (N \cdot NRI - J + 1)/(N - J + 1)\), so with \(N =\) 503 suspect identifications and \(J =\) 3 bins the two differ only by the bias correction (0.045 against 0.041 here). An interval for NRI would be the ANRI interval shifted upward by that bias, so it is ANRI and its interval that should be reported and compared.

Compare Groups

The comparison uses the two simulated procedures, which differ in the strength of the evidence they generate. Because they were simulated with genuinely different parameters, any difference found here reflects that design rather than chance group assignment.

anri_comparison <- compare_anri(
  procedures,
  group_var = "procedure",
  confidence_bins = c(0, 60, 80, 100),
  n_bootstrap = 100,
  seed = 456
)

print(anri_comparison)
$anri_group1
[1] 0.06778469

$anri_group2
[1] 0.02682759

$nri_group1
[1] 0.07322035

$nri_group2
[1] 0.03571501

$difference
[1] 0.04095711

$ci_lower
[1] -0.0162656

$ci_upper
[1] 0.09288628

$conf_level
[1] 0.95

$significant
 2.5% 
FALSE 

$group_names
[1] "Procedure A" "Procedure B"

$n_bootstrap
[1] 100

$n_bootstrap_requested
[1] 100

$n_failed
[1] 0

$bootstrap_results_group1
$bootstrap_results_group1$anri
[1] 0.06778469

$bootstrap_results_group1$nri
[1] 0.07322035

$bootstrap_results_group1$ci_lower
[1] 0.02912137

$bootstrap_results_group1$ci_upper
[1] 0.1149089

$bootstrap_results_group1$conf_level
[1] 0.95

$bootstrap_results_group1$n_bootstrap
[1] 100

$bootstrap_results_group1$n_bootstrap_requested
[1] 100

$bootstrap_results_group1$n_failed
[1] 0

$bootstrap_results_group1$n_total
[1] 343

$bootstrap_results_group1$n_bins
[1] 3

$bootstrap_results_group1$bootstrap_distribution
  [1] 0.06708598 0.06570604 0.08047661 0.04987933 0.07221240 0.06148205
  [7] 0.07212585 0.08571190 0.06816675 0.06999423 0.05916390 0.09510135
 [13] 0.03142914 0.04505713 0.05639861 0.09915104 0.03885619 0.07716533
 [19] 0.05518367 0.06160329 0.08011800 0.03582855 0.05369086 0.08262844
 [25] 0.11270127 0.09078520 0.11118346 0.06107025 0.11095935 0.03686786
 [31] 0.07824433 0.10365266 0.06466421 0.07758830 0.06246753 0.04141894
 [37] 0.08328425 0.08469806 0.08008407 0.06726736 0.02703340 0.06715568
 [43] 0.06081143 0.04384714 0.06404480 0.08060125 0.09375399 0.04073706
 [49] 0.05561147 0.06537103 0.08259529 0.07587945 0.06234151 0.11957211
 [55] 0.06314098 0.02570420 0.04332888 0.09238761 0.10012490 0.11869072
 [61] 0.05276179 0.09538591 0.07677354 0.04634813 0.08990006 0.05857359
 [67] 0.10507931 0.07623093 0.11105741 0.07602279 0.08272019 0.04113382
 [73] 0.08082205 0.05360715 0.09284835 0.01839824 0.03796410 0.07069711
 [79] 0.07777091 0.08588510 0.11690619 0.06030880 0.10590118 0.10782745
 [85] 0.06639759 0.06499843 0.07933454 0.06463416 0.09112998 0.06640864
 [91] 0.06755599 0.07182240 0.09123954 0.07012033 0.09326807 0.08488978
 [97] 0.08600358 0.07038152 0.06785717 0.03480324


$bootstrap_results_group2
$bootstrap_results_group2$anri
[1] 0.02682759

$bootstrap_results_group2$nri
[1] 0.03571501

$bootstrap_results_group2$ci_lower
[1] 0.009110349

$bootstrap_results_group2$ci_upper
[1] 0.06445527

$bootstrap_results_group2$conf_level
[1] 0.95

$bootstrap_results_group2$n_bootstrap
[1] 100

$bootstrap_results_group2$n_bootstrap_requested
[1] 100

$bootstrap_results_group2$n_failed
[1] 0

$bootstrap_results_group2$n_total
[1] 219

$bootstrap_results_group2$n_bins
[1] 3

$bootstrap_results_group2$bootstrap_distribution
  [1] 0.054817973 0.022421218 0.026550014 0.047942448 0.026436048 0.051313022
  [7] 0.026715655 0.035606414 0.049266532 0.033202453 0.065100176 0.019272279
 [13] 0.017972351 0.024914903 0.031759105 0.023415879 0.039935663 0.016485797
 [19] 0.028106056 0.026755049 0.042393733 0.014044901 0.047430438 0.025694351
 [25] 0.009603482 0.031974042 0.025750449 0.023649520 0.016634482 0.034929618
 [31] 0.044207564 0.042362668 0.025991477 0.033185154 0.037640835 0.026160620
 [37] 0.011972828 0.007036663 0.041360143 0.024607043 0.027630872 0.030317349
 [43] 0.032542761 0.019641566 0.022733987 0.010714669 0.031485628 0.021524720
 [49] 0.021008661 0.029182388 0.023620195 0.032096170 0.032127258 0.039164861
 [55] 0.022085988 0.045748433 0.022176174 0.023266895 0.034157548 0.023283074
 [61] 0.064850996 0.008664181 0.014826744 0.024504903 0.010977051 0.019082504
 [67] 0.013783056 0.023553399 0.032484769 0.026147752 0.022248487 0.030642495
 [73] 0.007978659 0.057524655 0.027271420 0.064017894 0.080392203 0.014551924
 [79] 0.051559618 0.026730444 0.026491746 0.023184755 0.035857128 0.028955733
 [85] 0.019554188 0.048800873 0.022485638 0.034746265 0.046738180 0.030777372
 [91] 0.026480496 0.025514253 0.037440071 0.038654235 0.034623890 0.033019727
 [97] 0.053811990 0.030239350 0.028706437 0.010009702


$difference_distribution
  [1]  0.0122680065  0.0432848222  0.0539265958  0.0019368818  0.0457763559
  [6]  0.0101690294  0.0454101932  0.0501054840  0.0189002169  0.0367917783
 [11] -0.0059362774  0.0758290756  0.0134567857  0.0201422261  0.0246395094
 [16]  0.0757351573 -0.0010794705  0.0606795334  0.0270776135  0.0348482420
 [21]  0.0377242712  0.0217836462  0.0062604171  0.0569340862  0.1030977881
 [26]  0.0588111547  0.0854330116  0.0374207301  0.0943248658  0.0019382433
 [31]  0.0340367627  0.0612899930  0.0386727321  0.0444031450  0.0248266917
 [36]  0.0152583218  0.0713114246  0.0776613940  0.0387239227  0.0426603164
 [41] -0.0005974759  0.0368383268  0.0282686652  0.0242055779  0.0413108181
 [46]  0.0698865853  0.0622683605  0.0192123438  0.0346028060  0.0361886426
 [51]  0.0589750985  0.0437832779  0.0302142505  0.0804072525  0.0410549946
 [56] -0.0200442308  0.0211527091  0.0691207197  0.0659673547  0.0954076458
 [61] -0.0120892085  0.0867217281  0.0619467959  0.0218432288  0.0789230117
 [66]  0.0394910885  0.0912962539  0.0526775336  0.0785726379  0.0498750399
 [71]  0.0604717015  0.0104913202  0.0728433901 -0.0039175017  0.0655769263
 [76] -0.0456196567 -0.0424281019  0.0561451813  0.0262112892  0.0591546516
 [81]  0.0904144437  0.0371240484  0.0700440502  0.0788717134  0.0468434025
 [86]  0.0161975531  0.0568489007  0.0298878990  0.0443917958  0.0356312668
 [91]  0.0410754920  0.0463081431  0.0537994645  0.0314660929  0.0586441834
 [96]  0.0518700487  0.0321915922  0.0401421675  0.0391507322  0.0247935330

plot_anri_comparison(anri_comparison)

The comparison provides:

• Point estimates for each group
• Difference with CI: Tests whether groups differ
• Significance test: CI excludes 0 = significant difference

The bootstrap difference distribution is available as plot_anri_difference_distribution(anri_comparison) if you want to inspect its shape. Inspect it when an interval looks surprising: a heavily skewed or multimodal distribution is a sign the bins are too thin somewhere.

When to Use ANRI vs NRI

The considerations, rather than rules:

ANRI is the safer default whenever the bias it corrects could be material — modest samples (say a few hundred or fewer), several confidence bins, or any comparison across studies that binned confidence differently. Because the bias grows with the bin-to-sample ratio, it is exactly the studies most likely to be compared informally that are least comparable on raw NRI. Formal tests should use ANRI, since testing a biased estimate against zero tests the wrong thing.

NRI remains reasonable when the correction is negligible — large samples with few bins, where ANRI and NRI will agree closely anyway — or when the point is to compare with older published work that reported NRI. In the analysis above the two differ by very little, which is what a sample of this size should produce.

Reporting both, as the table above does, costs nothing and lets a reader see the size of the correction for themselves.

Example Write-Up

Every number below is computed from the objects created above rather than typed in, so this passage cannot drift out of step with the analysis it describes. The same practice suits a manuscript.

We analysed identification performance using calibration analysis, deviation from perfect performance (DPP), and the adjusted normalized resolution index (ANRI) with bootstrap inference, using 1400 lineups of size 6 (700 target-present, 700 target-absent).

Witnesses were well calibrated overall (C = 0.073), but systematically underconfident (O/U = -0.229): accuracy in the lowest confidence band was substantially higher than stated confidence implied. Average DPP was 0.7 (0 = perfect, 1 = chance), and the operating point of maximum utility was confidence of 20 or above (DPP = 0.533). Resolution, measured by ANRI, was 0.041 (95% CI [0.021, 0.072]), against an uncorrected NRI of 0.045.

Expected utility analysis assumed a base rate of 50% and the cost matrix u(correct suspect ID) = 1, u(missed guilty suspect) = -0.5, u(innocent suspect ID) = -2, u(correct rejection) = 0.5; that is, a wrongful identification was weighted four times as heavily as a missed one. Under those assumptions the rule “act on confidence >= 30” maximised expected utility (EU = 0.288).

Bayesian analysis showed that high-confidence suspect identifications were the most informative response, with a likelihood ratio of 25 (based on 100 guilty-suspect and 4 innocent-suspect identifications in that band).

Note the last parenthesis. The likelihood ratio there rests on very few target-absent observations, and reporting the counts alongside the ratio is what allows a reader to judge how much weight it will bear.