Full ROC Analysis for Eyewitness Lineups

Every response, not just suspect identifications

Conventional ROC analysis of lineup data uses only suspect identifications. A witness who picks a foil, and a witness who rejects the lineup entirely, contribute nothing to either axis.

Spelling out what actually happens when a witness views a lineup makes the omission clear. There are six possible outcomes:

Four of the six outcomes are set aside. Yet they carry information. A witness who picks a foil has demonstrably failed to recognise the culprit, and in a target-absent lineup has done the innocent suspect a service. A confident rejection is evidence against the suspect’s guilt, and an investigator would certainly treat it as such.

The full ROC method uses all six.

Why the partial ROC curve does not span the full range

A concrete example follows, because this is the practical reason full ROC was proposed.

An ROC curve is traced by progressively relaxing the criterion for accepting evidence. In conventional lineup ROC analysis, the only lever is confidence among suspect identifications: start with the highest-confidence suspect IDs and work down. When every suspect identification has been included, the curve stops.

It stops at the point given by the overall suspect identification rates in the two arms — typically well under 0.3 on the x-axis, because most witnesses in a target-absent lineup do not pick the innocent suspect; they pick a foil or reject. There is no way to continue the curve, because there are no further suspect identifications to admit.

data(lineup_example)

partial <- make_roc(lineup_example, lineup_size = 6, show_plot = FALSE)
full    <- make_fullroc(lineup_example, lineup_size = 6, show_plot = FALSE)

knitr::kable(
  data.frame(
    Method = c("Partial ROC (suspect IDs only)", "Full ROC (all responses)"),
    `False-ID axis spans` = c(
      sprintf("0 to %.3f", max(partial$roc_data$false_id_rate)),
      sprintf("0 to %.3f", max(full$roc_data$cumulative_false_alarm_rate))
    ),
    `Area reported` = c(
      sprintf("pAUC = %.4f", partial$pauc),
      sprintf("AUC = %.4f", full$auc)
    ),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "Range covered by the two methods, lineup_example (700 target-present and 700 target-absent lineups of size 6)."
)

The full ROC reaches 1.0 because once foil identifications and rejections are admitted as evidence, every witness eventually contributes, and the cumulative rates must reach 1 in both arms.

The two areas are not comparable. A pAUC of 0.06 and an AUC of 0.78 are not in competition; they are computed over different ranges from different subsets of the data. Never report one as though it were the other.

Data structure

Three columns are required: target_present, identification and confidence. lineup_example has them, along with the member-level columns this package records.

str(lineup_example)
#> 'data.frame':    1400 obs. of  5 variables:
#>  $ target_present: logi  FALSE FALSE TRUE TRUE TRUE FALSE ...
#>  $ lineup_member : int  2 3 2 1 1 0 1 1 1 4 ...
#>  $ suspect_pos   : int  1 1 1 1 1 1 1 1 1 1 ...
#>  $ identification: chr  "filler" "filler" "filler" "suspect" ...
#>  $ confidence    : num  40 50 40 80 60 20 60 70 20 50 ...

As elsewhere in this package, the data are simulated at the level of individual foils — each foil has its own mean memory strength, so the foils are not interchangeable — and identification is derived from lineup_member rather than stored in its place. See ?simulate_lineup_data and its foil_bias argument.

Compute a full ROC

result <- make_fullroc(
  lineup_example,
  lineup_size = 6,
  order       = "apriori",
  show_plot   = FALSE
)

print(result)
#> 
#> === Full Lineup ROC Analysis (Smith & Yang, 2020) ===
#> 
#> Full AUC: 0.76 
#> Target-present trials (N): 700 
#> Target-absent trials (N): 700 
#> Lineup size: 6 
#> Ordering method: apriori 
#> Operating points: 19 
#> 
#> ROC Data (first 10 points):
#>    cumulative_hit_rate cumulative_false_alarm_rate evidence_label
#> 1           0.00000000                 0.000000000         origin
#> 2           0.09714286                 0.000000000    suspect_100
#> 3           0.14285714                 0.005714286     suspect_90
#> 4           0.18857143                 0.014285714     suspect_80
#> 5           0.26000000                 0.022857143     suspect_70
#> 6           0.35142857                 0.030000000     suspect_60
#> 7           0.43571429                 0.054285714     suspect_50
#> 8           0.50714286                 0.077142857     suspect_40
#> 9           0.55571429                 0.102857143     suspect_30
#> 10          0.59285714                 0.125714286     suspect_20
#> 
#> 
#> Diagnosticity Table (ordered by evidence strength):
#>  evidence_label    hit_rate false_alarm_rate diagnosticity_ratio
#>     suspect_100 0.097142857      0.000000000          97.1428571
#>      suspect_90 0.045714286      0.005714286           8.0000000
#>      suspect_80 0.045714286      0.008571429           5.3333333
#>      suspect_70 0.071428571      0.008571429           8.3333333
#>      suspect_60 0.091428571      0.007142857          12.8000000
#>      suspect_50 0.084285714      0.024285714           3.4705882
#>      suspect_40 0.071428571      0.022857143           3.1250000
#>      suspect_30 0.048571429      0.025714286           1.8888889
#>      suspect_20 0.037142857      0.022857143           1.6250000
#>       filler_20 0.047142857      0.108571429           0.4342105
#>       filler_30 0.047142857      0.110000000           0.4285714
#>       filler_40 0.054285714      0.098571429           0.5507246
#>       filler_50 0.054285714      0.090000000           0.6031746
#>       filler_60 0.032857143      0.052857143           0.6216216
#>       filler_70 0.037142857      0.062857143           0.5909091
#>       filler_80 0.020000000      0.021428571           0.9333333
#>       filler_90 0.008571429      0.014285714           0.6000000
#>      filler_100 0.010000000      0.028571429           0.3500000
#>       reject_20 0.095714286      0.287142857           0.3333333
#> 
#> Plot: use plot(x), or save the object and extract x$plot

result$plot + theme_minimal(base_size = 12)

The curve runs corner to corner, and each point is one operating point: a threshold on the ordered evidence categories.

A-priori ordering

order = "apriori" fixes the order in advance on theoretical grounds:

1. Suspect identifications, from high confidence down to low
2. Filler identifications, from low confidence up to high
3. Rejections, from low confidence up to high

The reasoning is that a confident suspect identification is the strongest evidence of guilt, while a confident rejection or a confident filler identification is the strongest evidence of innocence.

Diagnosticity ordering

order = "diagnosticity" instead sorts the categories by their observed diagnosticity ratio, \(DR = HR / FAR\), computed from the data at hand.

That sounds more principled — why assume an order when you can measure it? — but it uses the same data twice: once to choose the ordering that makes the curve bow as far upward as possible, and again to compute the area under it. The result is optimistically biased.

How large is the bias?

Large enough to be important. The following simulates data with no signal whatsoever — the response distribution is identical in target-present and target-absent lineups, so the true AUC is exactly 0.5 — and computes the AUC both ways.

null_auc <- function(n_per_arm, n_reps = 15, seed = 7) {
  set.seed(seed)
  out <- t(vapply(seq_len(n_reps), function(i) {
    d <- data.frame(
      target_present = rep(c(TRUE, FALSE), each = n_per_arm),
      identification = sample(c("suspect", "filler", "reject"),
                              2 * n_per_arm, replace = TRUE,
                              prob = c(0.2, 0.3, 0.5)),
      confidence     = sample(seq(20, 100, 20), 2 * n_per_arm, replace = TRUE)
    )
    c(diagnosticity = make_fullroc(d, lineup_size = 6,
                                   show_plot = FALSE)$auc,
      apriori       = make_fullroc(d, lineup_size = 6, order = "apriori",
                                   show_plot = FALSE)$auc)
  }, numeric(2)))
  colMeans(out)
}

bias_table <- do.call(rbind, lapply(c(100, 200, 500, 1000), function(n) {
  a <- null_auc(n)
  data.frame(`Lineups per arm` = n,
             `Diagnosticity ordering` = round(a[["diagnosticity"]], 3),
             `A-priori ordering` = round(a[["apriori"]], 3),
             check.names = FALSE)
}))

knitr::kable(
  bias_table, row.names = FALSE,
  caption = "Mean AUC from data containing NO signal at all, so the true AUC is 0.500. Simulated, 15 replications per row, 15 decision-by-confidence categories."
)

With 100 lineups per arm — an entirely ordinary study size — pure noise returns an AUC around 0.65 under diagnosticity ordering. On the interpretive scale often quoted for AUC, that is “weak to moderate discriminability”. It is nothing of the kind: there is no signal in these data at all.

A-priori ordering sits at 0.5 throughout, as it should.

The bias shrinks as the sample grows, roughly with the square root, but it is still around 0.05 at 1000 lineups per arm. It also grows with the number of decision-by-confidence categories, so binning confidence reduces it.

Recommendation. Use order = "apriori" for anything inferential: hypothesis tests, comparisons between conditions, published estimates. Diagnosticity ordering is defensible only as an exploratory upper bound, and should be labelled as such. make_fullroc() emits a message when diagnosticity ordering is used on a modest sample, for this reason.

The diagnosticity table

Non-cumulative rates for each decision-by-confidence category:

knitr::kable(
  head(as.data.frame(result$diagnosticity_table)[
    , c("decision", "conf_bin", "hit_rate", "false_alarm_rate",
        "diagnosticity_ratio")], 8),
  digits = 4, row.names = FALSE,
  caption = "Non-cumulative response rates and diagnosticity ratios, lineup_example (first 8 categories under a-priori ordering)."
)

A ratio well above 1 means the response is more common when the suspect is guilty; well below 1, more common when innocent; near 1, uninformative.

The ROC data

knitr::kable(
  head(as.data.frame(result$roc_data), 6),
  digits = 4, row.names = FALSE,
  caption = "Cumulative rates defining the ROC operating points, lineup_example."
)

Interpreting the AUC

The AUC has a direct probabilistic reading: it is the probability that a randomly chosen guilty suspect receives stronger evidence than a randomly chosen innocent one. 0.5 is chance.

Descriptive labels — “0.7 is moderate”, “0.8 is good” — circulate widely, but they should be resisted here for two reasons. The bias demonstrated above means a number in the 0.6s may reflect no signal at all; and the quantity is investigator discriminability, which has no established interpretive convention in this literature. Report the value, the ordering method, the sample sizes, and let the comparison do the work.

How the curve is built

For each decision-by-confidence category \(i\) (three decisions times the number of confidence levels), the non-cumulative rates are

\[HR_i = \frac{n(\text{response}_i \mid \text{target present})}{N_{\text{present}}}, \qquad FAR_i = \frac{n(\text{response}_i \mid \text{target absent})}{N_{\text{absent}}}\]

and the diagnosticity ratio is \(DR_i = HR_i / FAR_i\).

Categories are then placed in order — by \(DR_i\) descending, or by the a-priori scheme — and accumulated:

\[CHR_k = \sum_{i=1}^{k} HR_i, \qquad CFAR_k = \sum_{i=1}^{k} FAR_i\]

Because every response falls in exactly one category, both cumulative sums reach 1 when all categories are included, which is why the curve spans the full range. The area follows by the trapezoidal rule:

\[AUC = \sum_{i=2}^{n} \bigl(CFAR_i - CFAR_{i-1}\bigr)\cdot\frac{CHR_i + CHR_{i-1}}{2}\]

Edge cases

• \(FAR_i = 0\): the ratio is undefined, so a small constant (epsilon, default 0.001) is substituted. The affected category sorts to the top under diagnosticity ordering, which is one route by which that ordering becomes optimistic.
• \(HR_i = FAR_i = 0\): the category is empty and is removed.
• Empty bins after conf_bins are applied are removed.

Assumptions

1. Target-present and target-absent lineups are independent samples.
2. Every witness contributes exactly one response, so the categories partition the sample. If witnesses saw multiple lineups this is violated.
3. Under diagnosticity ordering, the AUC is not an unbiased estimate; see the demonstration above.
4. The quantity estimated is investigator discriminability, not eyewitness discriminability.

No standard errors

make_fullroc() returns a point estimate only. There is no bootstrap interval for the full-ROC AUC in this package, and the optimism of diagnosticity ordering would not be repaired by one — a naive bootstrap that re-selects the ordering within each resample reproduces the bias rather than revealing it. For interval estimates, use a-priori ordering and bootstrap the AUC with the ordering held fixed.