Statistical Comparison of ROC Curves (pAUC Analysis)

The region problem

Partial AUC requires a region, and with two curves of different lengths there is no choice of region that is obviously right.

• Integrate each curve to its own endpoint and the two areas cover different intervals. A larger area may then reflect nothing but a longer curve.
• Integrate both to the shorter curve’s endpoint and information from the longer curve is discarded — including, potentially, the region where the procedures differ most.
• Integrate both to some fixed conventional value (0.10, say) and you may be comparing them somewhere neither actually operated.

This is not hypothetical. compare_pauc() defaults to the largest region both curves attain, which is the second option above, and it says so when the two differ:

weak <- as.data.frame(simulate_lineup_data(
  n_tp = 400, n_ta = 400, d_prime = 1.4, c_criterion = 0.9,
  lineup_size = 6, conf_levels = 5,
  foil_bias = c(0.4, 0.2, 0.0, -0.4, -1.0), seed = 2
))

reach_strong <- max(make_rocdata(strong, lineup_size = 6)$roc_data$false_id_rate)
reach_weak   <- max(make_rocdata(weak,   lineup_size = 6)$roc_data$false_id_rate)

knitr::kable(
  data.frame(
    Condition = c("Strong (d' = 2.0)", "Weak (d' = 1.4)"),
    `Maximum false-ID rate reached` = round(c(reach_strong, reach_weak), 4),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "The two simulated conditions stop at different points, so there is no single region both fully cover."
)

The deeper objection

Smith, Lampinen, Wells, Smalarz and Mackovichova (2019) go further and argue that pAUC does not measure what researchers want it to measure. Their claim is that pAUC fails to track the diagnostic utility of a lineup — the thing that is actually important for deciding which procedure a police force should adopt — and that neither choice of region repairs this. They propose deviation from perfect performance (DPP) instead, which is implemented in this package and covered in vignette("calibration_decision_analysis").

What the curve leaves out

A further criticism concerns what enters the analysis at all. The ROC curve above is built from suspect identifications only. A witness who identifies a foil has made an error that the analysis largely sets aside, and a witness who rejects the lineup contributes nothing to either axis.

Yet foil identifications are informative: they say something about how well the lineup was constructed, and a procedure that converts suspect identifications into foil identifications has genuinely protected an innocent suspect. Lineup rejections are informative too. The full ROC approach, which uses every response, is one answer to this; see vignette("fullroc_analysis").

The point is not that pAUC should be abandoned, but that a single number computed over a contested region from a subset of the responses is carrying more weight in this literature than it can comfortably bear. Reporting it alongside DPP, and alongside the full response distribution, is a more defensible practice than reporting it alone.

Key articles

What the test actually is

compare_pauc() computes

\[Z = \frac{\widehat{pAUC}_1 - \widehat{pAUC}_2}{\widehat{SE}(\widehat{pAUC}_1 - \widehat{pAUC}_2)}\]

and refers \(Z\) to a standard normal distribution.

It is important to make a point of clarification here about what kind of test this is, because it is easy to assume it is something it is not. It is a Wald test: an estimate divided by an estimate of its own standard error. It is not a comparison of two standard normal deviates, and it is not a test on transformed proportions. The normal reference distribution comes from the asymptotic normality of the estimator, not from any assumption that the underlying data are normal.

The standard error in the denominator is obtained by bootstrap: the witnesses in each condition are resampled with replacement, the whole ROC and its partial area are recomputed for each resample, and the standard deviation of the resulting differences is taken as \(\widehat{SE}\).

Why bootstrap the standard error?

Because no usable closed-form expression exists. The partial area is a non-linear functional of a step function estimated from two correlated sets of counts, computed over a region that is itself determined by the data. Deriving its sampling variance analytically is not practical; resampling estimates it directly and makes no distributional assumption in doing so.

Is normality reasonable?

The Wald test assumes the difference in pAUC is approximately normally distributed. That can be checked, rather than assumed:

# Stored with the vignette; set R4LINEUPS_PRECOMPUTE=true to recompute.
comparison <- compare_pauc(
  strong, weak,
  lineup_size = 6,
  label1 = "Strong (d' = 2.0)",
  label2 = "Weak (d' = 1.4)",
  n_bootstrap = 600,
  seed = 123
)

boot_diff <- comparison$bootstrap_results$diff_boot

knitr::kable(
  data.frame(
    Quantity = c("Mean", "SD", "Skewness", "Excess kurtosis",
                 "Shapiro-Wilk p"),
    Value = c(
      round(mean(boot_diff), 5),
      round(sd(boot_diff), 5),
      round(mean((boot_diff - mean(boot_diff))^3) / sd(boot_diff)^3, 3),
      round(mean((boot_diff - mean(boot_diff))^4) / sd(boot_diff)^4 - 3, 3),
      signif(shapiro.test(boot_diff)$p.value, 3)
    )
  ),
  row.names = FALSE,
  caption = "Shape of the bootstrap distribution of the pAUC difference, simulated data (600 bootstrap resamples)."
)

ggplot(data.frame(diff = boot_diff), aes(x = diff)) +
  geom_histogram(aes(y = after_stat(density)), bins = 40,
                 fill = "steelblue", colour = "white", alpha = 0.8) +
  stat_function(fun = dnorm,
                args = list(mean = mean(boot_diff), sd = sd(boot_diff)),
                colour = "firebrick", linewidth = 1) +
  geom_vline(xintercept = 0, linetype = "dotted") +
  labs(
    x = "pAUC difference (strong - weak)",
    y = "Density",
    title = "Bootstrap distribution of the pAUC difference",
    subtitle = "Simulated data. Red curve = normal with the same mean and SD; dotted line = no difference"
  ) +
  theme_minimal(base_size = 12)

Typically the distribution is close to normal but not exactly so — mild skew is common, because pAUC is bounded and the region is data-determined. With several hundred resamples a formal normality test will often reject even when the departure is too small to be important. The practical question is not whether normality holds exactly but whether it holds well enough that the interval is about right, and that is best answered by comparing the two intervals directly:

knitr::kable(
  data.frame(
    Method = c("Wald (normal approximation)", "Bootstrap percentile"),
    Lower = round(c(comparison$ci_diff[["lower"]],
                    unname(quantile(boot_diff, 0.025))), 5),
    Upper = round(c(comparison$ci_diff[["upper"]],
                    unname(quantile(boot_diff, 0.975))), 5)
  ),
  row.names = FALSE,
  caption = "95% confidence intervals for the pAUC difference by two methods, simulated data."
)

When the two agree closely, as here, the normal approximation is doing no harm. When they diverge, prefer the percentile interval: it inherits whatever shape the bootstrap distribution actually has, rather than imposing symmetry on it.

Specifying the region explicitly

The default region is the largest both curves reach. A policy-relevant cutoff can be imposed instead, provided it is one both curves actually attain:

comparison_10 <- compare_pauc(
  strong, weak,
  lineup_size = 6,
  max_false_id_rate = 0.10,
  label1 = "Strong", label2 = "Weak",
  n_bootstrap = 300,
  seed = 456
)

knitr::kable(
  data.frame(
    Region = c(sprintf("Default (to %.3f)", comparison$max_false_id_rate),
               "Policy cutoff (to 0.100)"),
    `pAUC difference` = round(c(comparison$pauc_diff,
                                comparison_10$pauc_diff), 5),
    `p-value` = signif(c(comparison$p_value, comparison_10$p_value), 3),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "The same two conditions compared over two different regions, simulated data."
)

The estimates differ because they are areas over different intervals. This is not an inconsistency to be resolved but the region problem in miniature: the region is part of the estimand, and reporting a pAUC without stating the region it covers is incomplete.

How many bootstrap resamples?

The bootstrap standard error is itself an estimate, and too few resamples make it unstable. For exploratory work a few hundred is adequate; anything intended for publication should use 2000 or more. This vignette uses small numbers so that it builds quickly.

How pAUC is computed

The ROC points are the cumulative correct- and false-identification rates at each confidence threshold \(c\):

\[\text{CID}(c) = \frac{\#\{\text{suspect IDs in target-present lineups with confidence} \ge c\}}{N_{\text{target-present}}}\]

\[\text{FID}(c) = \frac{\#\{\text{suspect IDs in target-absent lineups with confidence} \ge c\}}{N_{\text{target-absent}}}\]

Where target-absent lineups contain no designated innocent suspect, foil identifications divided by lineup size are used to estimate the false identification rate instead.

The partial area over \([0, f_{\max}]\) is computed by the trapezoidal rule over the observed points:

\[\widehat{pAUC} = \sum_{i=2}^{k} \bigl(\text{FID}_i - \text{FID}_{i-1}\bigr) \cdot \frac{\text{CID}_i + \text{CID}_{i-1}}{2}\]

If \(f_{\max}\) falls between two observed points, the curve is linearly interpolated to that point. If \(f_{\max}\) lies beyond the last observed point, no extrapolation is performed: the sum simply stops at the data. Supplying a cutoff beyond a curve’s reach therefore produces a warning and an area truncated at the observed reach, which is smaller than the area up to the requested cutoff would be. Read the warning; it means the two areas being compared may not cover the same false-ID range.

How the standard error is obtained

For each of \(B\) bootstrap resamples, witnesses within each condition are resampled with replacement, preserving the target-present and target-absent sample sizes; the ROC and its partial area are recomputed; and the difference is recorded. Then

\[\widehat{SE}(\widehat{pAUC}_1 - \widehat{pAUC}_2) = \operatorname{sd}\bigl(\{\widehat{pAUC}_1^{(b)} - \widehat{pAUC}_2^{(b)}\}_{b=1}^{B}\bigr)\]

The two conditions are resampled independently, which is correct for a between-subjects design and gives \(\widehat{SE}^2 \approx \widehat{SE}_1^2 + \widehat{SE}_2^2\). It would be wrong for a within-subjects design in which the same witnesses contribute to both conditions, because the two areas would then be correlated and the independent resampling would overstate the standard error. This package provides no within-subjects version.

Assumptions

1. The two conditions are independent samples of witnesses.
2. The difference in pAUC is approximately normally distributed — checkable, as above, and avoidable by using the percentile interval.
3. The region of integration is one both curves attain.
4. Witnesses are independent of one another. If witnesses viewed multiple lineups, or lineups were constructed from a small pool of faces, this is violated and the standard error will be too small.

Effect size

summary() reports a standardised effect size formed by dividing the pAUC difference by a pooled bootstrap standard deviation. It is a useful relative summary, but it should not be read against Cohen’s conventions for \(d\): those were calibrated for standardised mean differences between raw observations, not for a difference between two areas scaled by a resampling standard deviation, and the two are not on comparable footings.