r4lineups MAIN vignette: Statistical Inference on Lineup Fairness

Null foils, and why table() can mislead

One important issue is that a lineup vector records the choice each witness made, but not the number of lineup members. If there were in fact 7 members in our running example, but member 7 was chosen by nobody, that would not be evident from the vector at all. Such an unchosen member is called a null foil.

table() makes this worse, because it tabulates only the positions that were actually chosen and silently omits the rest:

null_foil_vec <- rep(c(5, 3, 2, 1, 8, 6, 7), each = 4)

# Position 4 was never chosen, so this table has 7 entries, not 8
table(null_foil_vec)
null_foil_vec
1 2 3 5 6 7 8 
4 4 4 4 4 4 4 

Null foils are important, because a foil nobody would pick is not a functioning alternative to the suspect. They are one of several things that can make a lineup unfair: poor foils that draw only a few choices weaken it in the same way, and a suspect who stands out does so whatever the foils are like. A zero in a sample is also weaker evidence than it looks. Even in a perfectly fair six-person lineup, a given member receives no choices from ten mock witnesses with probability \((5/6)^{10} \approx .16\). Treat an unchosen member as a prompt to look at that photograph, not as proof that it fails as a foil. Use lineup_table() instead, which tabulates over all k positions, keeps unchosen members as explicit zeroes, and says so:

lineup_table(null_foil_vec, k = 8, suspect_pos = 3)
Lineup choices (k = 8, N = 28)

 pos: 1  2  3*  4  5  6  7  8
 cnt: 4  4   4  0  4  4  4  4

 * = nominated suspect

 Note: 1 null foil (position 4) received no choices.
 Unchosen members reduce effective size; see ?esize_m.

The result inherits from table, so it can be passed to esize_m(), esize_T() and the other table-taking functions in the package exactly as an ordinary table would be. The measures in this package already handle null foils correctly given a complete table; what lineup_table() adds is visibility.

For the remaining functions, which take a vector rather than a table, nominal lineup size must be passed explicitly as an argument for the same reason.

A larger dataset

Let’s look at a slightly larger data set. Only the first three rows are shown here; nortje2012 has 133 rows in total.

data(nortje2012)

• For functions that require data for only one lineup or one lineup pair, you can convert the data from a given lineup to a vector by calling for e.g. lineup_vec <- nortje2012$lineup_1 or alternatively lineup_vec <- nortje2012[[1]]

• For functions that require several lineups or several lineup pairs, you can convert the data to a list, e.g. lineup_list <- as.list(nortje2012)

lineup_vec   <- nortje2012$lineup_1
lineup_list  <- as.list(nortje2012)
lineup_tab   <- lineup_table(lineup_vec, k = 8)

lineup_vec and lineup_tab are used throughout the rest of this vignette.

• Nominal Size

  The notion of ‘Nominal size’ refers to the number of physical members of a lineup. Functions relying on nominal size for accurate calculations require you to manually specify nominal size. If the specified nominal size is not reflected in the dataframe or vector passed to the function, execution of the function is halted with the following error message: “User-declared nominal size does not match observed nominal size. Please check vector of target positions.”

Calculating proportion for a single target/lineup member

lineup_prop_vec(lineup_vec, target_pos = 3, k = 8)
[1] 0.1428571

This function requires three arguments: a vector of lineup data (here lineup_vec, the choices for nortje2012$lineup_1 that we extracted above), the position of the target in the lineup (target_pos), and nominal size (k). The example returns the proportion of mock witnesses choosing the lineup member occupying position 3, which is 0.14 — that is, about 14% of the witnesses. Chance expectation for an 8-person lineup is 1/8 = 0.125, so this member was chosen at close to the chance rate.

bootobject <- boot::boot(lineup_vec, lineup_prop_boot, target_pos = 3, R = 1000)
bootobject

ORDINARY NONPARAMETRIC BOOTSTRAP


Call:
boot::boot(data = lineup_vec, statistic = lineup_prop_boot, R = 1000, 
    target_pos = 3)


Bootstrap Statistics :
     original        bias    std. error
t1* 0.1428571 -0.0004586466   0.0289372

cis <- boot::boot.ci(bootobject, conf = 0.95, type = "bca", target_pos = 3)
cis
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
Based on 1000 bootstrap replicates

CALL : 
boot::boot.ci(boot.out = bootobject, conf = 0.95, type = "bca", 
    target_pos = 3)

Intervals : 
Level       BCa          
95%   ( 0.0902,  0.1955 )  
Calculations and Intervals on Original Scale

n_chose_3 <- sum(lineup_vec == 3)
n_total   <- length(lineup_vec)

# Exact (Clopper-Pearson) interval
binom.test(n_chose_3, n_total)$conf.int
[1] 0.08825399 0.21406762
attr(,"conf.level")
[1] 0.95

# Normal-theory interval
prop.test(n_chose_3, n_total)$conf.int
[1] 0.09037086 0.21662699
attr(,"conf.level")
[1] 0.95

never_chosen <- rep(2:6, 6)   # 30 mock witnesses; member 1 never chosen
b0 <- boot::boot(never_chosen, lineup_prop_boot, target_pos = 1, R = 1000)
range(b0$t)                              # every resample gives 0
[1] 0 0
binom.test(0, length(never_chosen))$conf.int
[1] 0.0000000 0.1157033
attr(,"conf.level")
[1] 0.95

Calculating proportion for each lineup member

Should you want to calculate a proportion for each member of a given lineup, the allprop() function can be called. This might be the case when there is no pre-specified suspect in the lineup, as some researchers do in laboratory experiments. Instead of manually specifying the target position, a vector indexing all target positions is automatically generated from the argument k, within the function itself.

allprop(lineup_vec, k = 8)
        prop
1 0.14285714
2 0.13533835
3 0.14285714
4 0.04511278
5 0.28571429
6 0.04511278
7 0.17293233
8 0.03007519

The function then returns the proportion of mock witnesses selecting each lineup member.

lineuprops_ci <- lineup_boot_allprop(lineup_vec, k = 8, R = 1000)
lineuprops_ci
       ci_low    ci_high
1 0.082706767 0.19548872
2 0.082706767 0.18796992
3 0.081853490 0.20300752
4 0.015037594 0.08270677
5 0.205667634 0.36090226
6 0.015037594 0.08270677
7 0.112781955 0.24060150
8 0.007518797 0.06015038

lineup_boot_allprop(lineup_vec, k = 8, conf = 1 - 0.05 / 8, R = 1000)
       ci_low    ci_high
1 0.065932074 0.24386511
2 0.060150376 0.21804511
3 0.067669173 0.23473395
4 0.007518797 0.10094435
5 0.163254847 0.39308379
6 0.007518797 0.11663662
7 0.090225564 0.26335704
8 0.000000000 0.09022556

Visualising the bootstrap distribution

It is often more informative to look at the whole bootstrap distribution than at its endpoints. lineup_bias_boot_dist() returns the replicates and plot_lineup_bias_distribution() draws them:

bias_dist <- lineup_bias_boot_dist(lineup_vec, target_pos = 3, k = 8, R = 1000)
plot_lineup_bias_distribution(bias_dist, target_pos = 3)

Note that a bootstrapped proportion is a count divided by the number of witnesses, so the replicates can only take values that are multiples of 1/N. The plot bins at exactly that resolution, giving one bar per attainable proportion. If you override this with bins or binwidth, you may see spurious gaps and uneven bar heights, because some bins will contain no attainable value at all.

1. Malpass’s original (1981) and adjusted effective size (see: Tredoux, 1998)

Malpass’ original formula for effective size proposed a denominator computed by entering the number of foils that attracted non-zero choice frequencies. Tredoux (1998) argued against this practice, pointing out that zero frequencies could arise from random guessing with appreciable frequency. He suggested a minor amendment of Malpass’ original formula, as follows.

Malpass’ Effective size B = \(k-\sum_{i=1}^{k}\frac{|o_{i}-e|}{2e}\)

Both Malpass measures can be displayed by calling esize_m() and passing lineup choices (or a table) plus nominal size. The function always returns the Tredoux-adjusted estimate; both = TRUE additionally prints the original Malpass estimate for comparison.

esize_m(lineup_tab, k = 8, both = TRUE)
Effective size (Malpass, 1981) =  5.962406 
Effective size (Malpass, 1981, 
             adj Tredoux, 1998) =  5.962406 
[1] 5.962406

Every member of lineup_1 was chosen at least once, so here the two formulations coincide. They come apart as soon as there is a null foil, which is where the distinction between \(k\) and \(k_a\) bites:

# Position 4 was never chosen
esize_m(lineup_table(null_foil_vec, k = 8), k = 8, both = TRUE)
Effective size (Malpass, 1981) =  7 
Effective size (Malpass, 1981, 
             adj Tredoux, 1998) =  7 
[1] 7

The adjusted estimate deducts the unchosen member. Note that both figures happen to be 7 in this particular example, but for different reasons: the adjusted formula starts from k = 8 and deducts the null foil, while the original formula starts from \(k_a\) = 7 chosen members and finds them perfectly uniform. Using lineup_table() rather than table() is what makes the null foil visible in the first place.

# Create a dataframe of bootstrapped lineup data
bootdata <- gen_boot_samples(lineup_vec, 1000)

# Calculate effective size for each lineup in bootdata.
# Nominal size is again declared by the user.
lineupsizes <- gen_esize_m(bootdata, k = 8)

# Pass the vector of bootstrapped effective sizes to gen_esize_m_ci
# to calculate lower and upper CIs with the desired level of alpha
gen_esize_m_ci(lineupsizes, perc = .025)
    2.5% 
5.345865 
gen_esize_m_ci(lineupsizes, perc = .975)
   97.5% 
6.285714 

gen_boot_propmean_se(lineupsizes)
Boot prop. (mean)   =  5.820346 
Boot prop. (median) =  5.827068 
SD of boot prop     =  0.2395503 
SE of boot prop     =  0.24 
2.5% boot CI lvl    =  5.345865 
97.5% boot CI lvl   =  6.285714 

2. Tredoux’s measure of Effective Size, E’

Tredoux (1998) proposed a measure of effective size that is amenable to statistical inference using distribution theory.

E’ = \(\frac{1}{1-I}\), where I = \(1-\sum_{i=1}^k(\frac{O_i}{N})^2\)

and where k, N, and \(O_i\) are all as defined earlier.

Substituting for I gives \(E' = 1/\sum_{i=1}^k (O_i/N)^2\). This is the reciprocal of Simpson’s (1949) concentration index, known in ecology as the inverse Simpson index and in economics as the inverse Herfindahl index, and it can be read as the effective number of members drawing choices: if all k members are chosen equally often, E’ = k, and if one member draws every choice, E’ = 1. The other vignettes use this \(1/\sum p_i^2\) form; it is the same quantity as Tredoux’s E’.

To compute Tredoux’s measure of effective size, E’, use the function esize_T(). We can compute confidence intervals around the measure of E’ using normal distribution theory, and using bootstrap intervals, with the function esize_T_boot() from r4lineups, as well as boot() and boot.ci() from the boot package. Usage is shown below. Recall that lineup_tab was created near the start of this vignette with lineup_table(lineup_vec, k = 8).

# Compute effective size from a lineup table
esize_T(lineup_tab)
[1] 5.693273

# Compute bootstrapped effective size
esize_boot <- boot::boot(lineup_vec, esize_T_boot, R = 1000)
esize_boot

ORDINARY NONPARAMETRIC BOOTSTRAP


Call:
boot::boot(data = lineup_vec, statistic = esize_T_boot, R = 1000)


Bootstrap Statistics :
    original     bias    std. error
t1* 5.693273 -0.1578132   0.4144991

When requesting confidence intervals, name the types you actually want:

esize_boot_ci <- boot::boot.ci(esize_boot, type = c("perc", "bca"))
esize_boot_ci
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
Based on 1000 bootstrap replicates

CALL : 
boot::boot.ci(boot.out = esize_boot, type = c("perc", "bca"))

Intervals : 
Level     Percentile            BCa          
95%   ( 4.681,  6.333 )   ( 5.029,  6.589 )  
Calculations and Intervals on Original Scale
Some BCa intervals may be unstable

Calling boot.ci() without a type argument asks for all interval types, and produces two warnings that should be understood rather than ignored:

• “bootstrap variances needed for studentized intervals”. The studentized (bootstrap-t) interval needs an estimate of the variance of the statistic at each bootstrap replicate, which esize_T_boot() does not return. That interval simply cannot be computed here, and asking for it is what triggers the warning.
• “extreme order statistics used as endpoints”. The requested percentile falls in the extreme tail of the available replicates, so the endpoint is being read off the very smallest or largest values, where it is unstable. The remedy is more replicates (a larger R), not a different interval type.

These calls demonstrate empirical bootstrap intervals, which have important inferential limits. Do not test exact fairness by checking whether a percentile or BCa interval for plug-in E’ contains nominal size. At N=100 and k=6, the most even possible count vector is (17,17,17,17,16,16), so the largest empirical E’ is 10000/1668 = 5.9952. Every bootstrap value is at most that number: an interval made from its quantiles cannot contain the perfectly fair population value 6. More resamples do not fix this nonregular boundary problem. Normal-theory and corrected-estimator intervals also require separate sampling assessment; none is a universally reliable replacement. Bayesian credible intervals describe the stated posterior and should not be treated as tests of exact equality to k.

You can also look at the whole bootstrap distribution:

esize_dist <- esize_boot_dist(lineup_vec, k = 8, metric = "tredoux", R = 1000)
plot_esize_distribution(esize_dist, metric = "E' (Tredoux, 1998)")

set.seed(4242)
k <- 6L

bias_demo <- do.call(rbind, lapply(c(20, 50, 100, 500), function(N) {
  est <- replicate(2000, {
    counts <- tabulate(sample.int(k, N, replace = TRUE), nbins = k)
    c(plug_in = esize_T(counts), corrected = esize_T(counts, unbiased = TRUE))
  })
  data.frame(
    `N choices`  = N,
    `Plug-in E'` = round(mean(est["plug_in", ]), 3),
    `Predicted`  = round(N * k / (N + k - 1), 3),
    `Corrected`  = round(mean(est["corrected", ]), 3),
    check.names  = FALSE
  )
}))

knitr::kable(
  bias_demo, row.names = FALSE,
  caption = "Mean E' from simulated fair six-person lineups, where the true effective size is exactly 6. Simulated data, 2000 replications per row."
)

c(plug_in = esize_T(lineup_tab),
  corrected = esize_T(lineup_tab, unbiased = TRUE))
  plug_in corrected 
 5.693273  5.903161 

data(lineup_example)

ta <- lineup_example[!lineup_example$target_present, ]
chosen <- ta$lineup_member[ta$lineup_member > 0]

c(plug_in   = esize_T(table(factor(chosen, levels = 1:6))),
  corrected = esize_T(table(factor(chosen, levels = 1:6)), unbiased = TRUE))
  plug_in corrected 
 4.225584  4.253132 

3. Effective Size Per Foils

Malpass (1981) proposed an additional method for computing effective size, based on the number of foils who are chosen at a rate nearly equal to chance expectation. We propose that ‘nearly equal’ could mean that chance expectation falls within a confidence interval constructed around the proportion of mock witnesses choosing the lineup member.

To calculate effective size by counting the number of foils who fall within the CI for chance guessing, we pass a vector of lineup data, a vector of target positions and nominal size.

eff_size_per_foils(lineup_vec, target_pos = seq_len(8), k = 8,
                   conf = 0.95, R = 1000)
[1] 4

Applying this function to our data, we see that effective size for this lineup is 4: four of the eight members were chosen at a rate not distinguishable from chance.

Note that the value returned is a count of members, not a continuous quantity, so it moves only in whole steps as conf is varied, and it stays flat across any range of conf in which no member’s interval crosses 1/k.

The direction of the change with conf can look backwards at first. A member is counted when its interval contains chance, that is, when its choice rate cannot be distinguished from 1/k. A higher conf gives wider intervals, which makes it harder to show that any member departs from chance, so more members are counted and the effective size rises. A higher confidence level is a stricter test of unfairness, and so a more lenient test of each foil. The same logic means the measure rises when there are few mock witnesses, because every interval is then wide: with a small sample, a large count can reflect low power rather than a fair lineup.

for (conf in c(0.5, 0.7, 0.95, 0.99)) {
  cat("conf =", conf, "->",
      eff_size_per_foils(lineup_vec, target_pos = seq_len(8), k = 8,
                         conf = conf, R = 1000), "\n")
}
conf = 0.5 -> 3 
conf = 0.7 -> 3 
conf = 0.95 -> 4 
conf = 0.99 -> 4 

As with lineup_boot_allprop(), on which this function is built, the intervals are not corrected for the fact that k of them are examined at once.

Comparing two effective sizes

To test if the effective sizes of two independent lineups are significantly different from one another, we can take the difference between the effective sizes and calculate confidence intervals. This function, effsize_compare(), requires a dataframe containing lineup data for two independent lineups. Here, we create a new dataframe containing two independent lineups from nortje2012.

# Get data
linedf <- nortje2012[1:2]

# Compare effective size
effsize_compare(linedf)

The two Effective sizes are  5.693273   4.967425
If the interval includes 0, ns at p = .05
Confidence intervals of difference [95%]
Normal Theory -0.302 1.887
Bootstrap: percentile (R = 1000) -0.4 1.737
Bootstrap: bias-corrected (R = 1000) -0.292 1.836

The effective size for lineup_1 is therefore not significantly different from the effective size for lineup_2.

Differences between Independent Diagnosticity Ratios

We can compare diagnosticity ratios to other diagnosticity ratios, as detailed in Tredoux (1998). He showed a method for computing the equivalence of k diagnosticity ratios, in particular.

# Target present data:
TP_lineup1 <- rep(1:6, length.out = 100)
TP_lineup2 <- rep(1:5, length.out = 70)
TP_lineup3 <- rep(1:4, length.out = 20)
lineup_pres_list <- list(TP_lineup1, TP_lineup2, TP_lineup3)

# Target absent data:
TA_lineup1 <- rep(6:1, length.out = 100)
TA_lineup2 <- rep(5:1, length.out = 70)
TA_lineup3 <- rep(4:1, length.out = 20)
lineup_abs_list <- list(TA_lineup1, TA_lineup2, TA_lineup3)

pos_list <- c(3, 2, 1)

k <- c(6, 5, 4)

homog_diag(lineup_pres_list, lineup_abs_list, pos_list, k)
Diagnosticity ratio for each lineup pair:
 pair     d       95% CI
    1 1.000 [0.54, 1.84]
    2 1.000 [0.52, 1.94]
    3 1.000 [0.34, 2.93]

Mean diagnosticity ratio: 1
Chi-square estimate (q): 0
Sig: 1

Note: no evidence of heterogeneity (p >= 0.05), so the ratios are
consistent with a common value and the pooled mean is a reasonable summary.

# A helper: n1 witnesses choose position 1, the rest spread over positions 2-6
choose_pos1 <- function(n1) c(rep(1, n1), rep(2:6, length.out = 100 - n1))

# Suspect ID rate falls from .60 to .32 across the three TP lineups,
# and rises from .10 to .20 across the three TA lineups
het_pres <- list(choose_pos1(60), choose_pos1(45), choose_pos1(32))
het_abs  <- list(choose_pos1(10), choose_pos1(15), choose_pos1(20))

homog_diag(het_pres, het_abs, pos_list = c(1, 1, 1), k = c(6, 6, 6))
Diagnosticity ratio for each lineup pair:
 pair     d        95% CI
    1 6.000 [3.26, 11.04]
    2 3.000  [1.79, 5.02]
    3 1.600  [0.99, 2.60]

Mean diagnosticity ratio: 2.782997
Chi-square estimate (q): 11.18942
Sig: 0.003717485

Note: the ratios are heterogeneous (p < 0.05), so they do not share a
common diagnosticity ratio. Q measures how far the ratios lie from their
pooled value, which is a different question from whether the pooled ratio
differs from 1. The mean above averages ratios that genuinely differ and
should not be reported as a single summary for these lineups; interpret the
pairs separately.

mixed_pres <- list(choose_pos1(60), choose_pos1(30), choose_pos1(20))
mixed_abs  <- list(choose_pos1(10), choose_pos1(30), choose_pos1(45))

homog_diag(mixed_pres, mixed_abs, pos_list = c(1, 1, 1), k = c(6, 6, 6))
Diagnosticity ratio for each lineup pair:
 pair     d        95% CI
    1 6.000 [3.26, 11.04]
    2 1.000  [0.65, 1.53]
    3 0.444  [0.28, 0.70]

Mean diagnosticity ratio: 1.060829
Chi-square estimate (q): 45.62598
Sig: 1.237212e-10

Note: the ratios are heterogeneous (p < 0.05), so they do not share a
common diagnosticity ratio. Q measures how far the ratios lie from their
pooled value, which is a different question from whether the pooled ratio
differs from 1. The mean above averages ratios that genuinely differ and
should not be reported as a single summary for these lineups; interpret the
pairs separately.

homog_diag_boot(lineup_pres_list = het_pres,
                lineup_abs_list = het_abs,
                pos_list = c(1, 1, 1),
                k = c(6, 6, 6),
                R = 200,
                seed = 1)
Diagnosticity ratio for each lineup pair:
 pair     d        95% CI
    1 6.000 [3.26, 11.04]
    2 3.000  [1.79, 5.02]
    3 1.600  [0.99, 2.60]

Mean diagnosticity ratio is 2.783 
Confidence intervals (percentile) 2.059 3.7 
Chi-squared estimate is 11.189 
Confidence intervals (percentile): 2.601 24.833 
Sig (asymptotic, df = 2):  0.003717 

Note: the ratios are heterogeneous (p < 0.05), so they do not share a
common diagnosticity ratio. Q measures how far the ratios lie from their
pooled value, which is a different question from whether the pooled ratio
differs from 1. The mean above averages ratios that genuinely differ and
should not be reported as a single summary for these lineups; interpret the
pairs separately.

homog_diag_boot(lineup_pres_list = mixed_pres,
                lineup_abs_list = mixed_abs,
                pos_list = c(1, 1, 1),
                k = c(6, 6, 6),
                R = 200,
                seed = 1)
Diagnosticity ratio for each lineup pair:
 pair     d        95% CI
    1 6.000 [3.26, 11.04]
    2 1.000  [0.65, 1.53]
    3 0.444  [0.28, 0.70]

Mean diagnosticity ratio is 1.061 
Confidence intervals (percentile) 0.827 1.327 
Chi-squared estimate is 45.626 
Confidence intervals (percentile): 32.879 59.368 
Sig (asymptotic, df = 2):  1.237e-10 

Note: the ratios are heterogeneous (p < 0.05), so they do not share a
common diagnosticity ratio. Q measures how far the ratios lie from their
pooled value, which is a different question from whether the pooled ratio
differs from 1. The mean above averages ratios that genuinely differ and
should not be reported as a single summary for these lineups; interpret the
pairs separately.

Data format for ROC analysis

The ROC functions require a dataframe with three columns: target_present (logical), identification (character: “suspect”, “filler”, or “reject”), and confidence (numeric).

An important point about study design: the raw data you collect should record which lineup member each witness chose, exactly as in the earlier sections of this vignette. The suspect/filler/reject classification is derived from that choice together with the known position of the suspect — it is not something to record instead of the choice. Keeping the original position lets you compute every measure in this vignette from the same dataset, and lets you re-derive the classification if the suspect position is later corrected.

The derivation is straightforward:

# Raw data as it should be collected: the member chosen (0 = rejected the lineup)
raw <- data.frame(
  target_present = c(TRUE, TRUE, TRUE, FALSE, FALSE, FALSE),
  chosen_member  = c(3, 5, 0, 3, 2, 0),
  confidence     = c(90, 70, 40, 80, 60, 50)
)

# The suspect occupies position 3 in both the TP and TA lineups
suspect_pos <- 3

raw$identification <- ifelse(raw$chosen_member == 0, "reject",
                      ifelse(raw$chosen_member == suspect_pos, "suspect",
                             "filler"))
raw
  target_present chosen_member confidence identification
1           TRUE             3         90        suspect
2           TRUE             5         70         filler
3           TRUE             0         40         reject
4          FALSE             3         80        suspect
5          FALSE             2         60         filler
6          FALSE             0         50         reject

The bundled lineup_example dataset is already in the derived form. Only the first 8 rows are shown:

Computing and plotting the ROC

Each ROC and CAC analysis comes in two forms:

• make_rocdata(), make_cacdata() and make_fullroc_data() compute and return the data only.
• make_roc(), make_cac() and make_fullroc() wrap those, additionally build the plot, and return a classed object with the plot stored in $plot.

Either form can be passed to plot():

roc1 <- make_rocdata(lineup_example, lineup_size = 6)
plot(roc1)

ROC Curve for Confidence ~ Accuracy

Printing the wrapper object gives the numerical summary:

roc_result <- make_roc(lineup_example, lineup_size = 6, show_plot = FALSE)
roc_result

=== Lineup ROC Analysis ===

Partial AUC: 0.053 
Target-present trials (N): 700 
Target-absent trials (N): 700 
Lineup size: 6 
Confidence levels: 9 

ROC Data:
# A tibble: 10 × 5
   confidence correct_id_rate false_id_rate n_correct_ids n_false_ids
        <dbl>           <dbl>         <dbl>         <dbl>       <dbl>
 1        101          0            0                   0           0
 2        100          0.0971       0                  68           0
 3         90          0.143        0.00571           100           4
 4         80          0.189        0.0143            132          10
 5         70          0.26         0.0229            182          16
 6         60          0.351        0.03              246          21
 7         50          0.436        0.0543            305          38
 8         40          0.507        0.0771            355          54
 9         30          0.556        0.103             389          72
10         20          0.593        0.126             415          88

Plot: use plot(x), or save the object and extract x$plot

The pAUC is a partial area, so it is not on the familiar 0.5-to-1 scale of the full AUC. It covers false-ID rates from 0 up to the largest observed, 0.126 here, so a perfect procedure would score 0.126 and chance (the diagonal) 0.008. The observed 0.053 is therefore well above chance. pAUC values are comparable only between curves integrated over the same false-ID range, which is what compare_pauc() enforces (vignette("pauc_statistical_comparison")).

The value returned by plot() is an ordinary ggplot2 object, so you can adjust it with the usual + syntax rather than being limited to the arguments of the plotting function:

plot(roc1) +
  ggplot2::theme_minimal(base_size = 12) +
  ggplot2::labs(title = "Partial ROC, Nortje & Tredoux mock-witness data")

Full ROC curves

make_fullroc() implements the Smith, Yang & Wells (2020) full ROC, which uses all responses — suspect IDs, filler IDs and rejections — rather than suspect IDs alone. The ordering method is shown in the plot subtitle, since it materially affects the result:

full_roc1 <- make_fullroc(lineup_example, lineup_size = 6, show_plot = FALSE)
Ordering categories by their sample diagnosticity ratios is optimistically biased: the ordering is estimated from the same data used to compute the AUC. With roughly 700 lineups per arm the bias can be substantial. Use order = "apriori" for inference or between-condition comparisons; see vignette("fullroc_analysis").
plot(full_roc1)

See the Full ROC analysis vignette for the details, including why order = "apriori" is preferable for inference.

The measures, and what each assumes

Nominal, adjusted and effective size

Three different counts appear throughout and are easily confused.

Nominal size \(k\) is how many people stand in the lineup. Adjusted nominal size \(k_a\) is \(k\) less the null foils — members nobody chose in this sample, which, as noted above, may include functioning foils that happened to draw no choices. Effective size is an estimate of how many members function as genuine alternatives, and is generally not a whole number.

The distinction is important because most functions here take \(k\) as an argument and will not detect that it is wrong. lineup_table() exists to make null foils visible before that happens: table() silently omits members with zero counts, so a lineup of eight with one null foil produces a seven-entry table and nothing in the output says a member is missing.

Inference

Three routes are available, and they answer slightly different questions:

• Normal-theory intervals (esize_T_ci_n(), prop.test()) are quick and adequate when counts are large and the estimate is away from a boundary.
• Bootstrap intervals (esize_T_boot() with boot::boot.ci(), lineup_prop_boot()) make no distributional assumption about the transformed statistic, which helps for nonlinear measures such as effective size. They fail at a boundary, though: a member nobody chose gets a \([0, 0]\) interval however many resamples are drawn. For a member proportion near 0 or 1 use the exact binom.test() interval or a posterior.
• Bayesian posteriors (esize_T_bayes(), func_size_bayes()) give credible intervals and direct probability statements such as \(P(E' < 4.5 \mid \text{data})\). A threshold at the nominal size is uninformative, because \(P(E' < k) = 1\) under any continuous posterior. See vignette("bayesian_inference").

Bounded quantities are the recurring difficulty: E’ cannot exceed \(k\) and a proportion cannot leave \([0, 1]\), but a normal-theory interval can and does cross those limits in small samples. That is the main reason to prefer an exact interval, the posterior or, away from a boundary, the bootstrap.

Assumptions common to the fairness measures

1. Each mock witness contributes one choice, so the counts partition the sample.
2. Choices are independent across witnesses.
3. The nominal size passed to the function is the true number of lineup members. Where null foils are present, effective size is the quantity of interest and nominal size overstates the alternatives available.
4. Mock-witness measures describe the lineup relative to a description. A different description can give a different answer for the same lineup; see vignette("face_similarity") on what follows from that.