--- title: "SDT Summary-Level Comparisons" author: - "Colin G. Tredoux" - "Tamsyn M. Naylor" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{SDT Summary-Level Comparisons} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- > **Acknowledgement.** Large language models supplied by Anthropic and OpenAI > were used in preparing this vignette, to assist with checking code, testing > the veracity of the approaches described, and on occasion editing the text. > Responsibility for the content rests with the authors. ```{r setup, include=FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", warning = FALSE, message = FALSE) # The slowest results are computed with the code shown and stored in # precomputed/, together with the random-number state, so that the rest of the # vignette is unchanged. Set R4LINEUPS_PRECOMPUTE=true to recompute them # (data-raw/vignette_precompute.R). precompute <- identical(Sys.getenv("R4LINEUPS_PRECOMPUTE"), "true") stored <- function(name, objects) { path <- file.path("precomputed", paste0(name, ".rds")) env <- knitr::knit_global() if (precompute) { saveRDS(list(objects = mget(objects, envir = env), seed = globalenv()[[".Random.seed"]]), path, compress = "xz") } else { s <- readRDS(path) list2env(s$objects, envir = env) if (!is.null(s$seed)) assign(".Random.seed", s$seed, envir = globalenv()) } invisible(NULL) } ``` # Overview This vignette shows how to compare **conditions using only summary SDT counts** (hits, false alarms, misses, correct rejections). Use these tools when **trial-level data are unavailable** and you have only summary counts per condition — the usual situation when working from published tables, or when synthesising results across studies for a meta-analysis. ## Where the methods come from The variance estimators follow Gourevitch and Galanter (1967), Miller (1996), and Suero, Privado and Botella (2017). The bootstrap option follows the procedure used in the meta-analysis of Tredoux, Sporer, Vredeveldt, Kempen and Nortje (2021), where summary counts extracted from published studies had to be compared without access to the underlying trial data — which is exactly the problem these functions address. ## A caution about what d′ means for a lineup These functions treat the data as a standard **yes/no detection** problem: hits, misses, false alarms and correct rejections, from which d′ follows as \(z(\text{hit rate}) - z(\text{false-alarm rate})\). A lineup is not that. A witness viewing a lineup performs a **compound** task — deciding whether the culprit is present at all, *and* which member it is. Duncan (2006) set this out formally as a compound signal-detection model, in which the lineup is a 1-of-*m* detection task combined with an *m*-alternative forced-choice identification task, with a single set of latent memory parameters driving both. Estimating d′ from collapsed 2 × 2 counts ignores that structure, and in particular it discards which lineup member was chosen. That does not make the summary-level comparison useless — it is often the only thing available, and it is what published tables report — but the d′ it yields is a yes/no analogue and should be labelled as such rather than presented as the lineup's discriminability. Where trial-level data exist, prefer the lineup models in `vignette("lineup_memory_models")` or `vignette("model_comparison")`, which respect the compound structure. # Summary metrics from counts ```{r} library(r4lineups) # Condition A counts hits_a <- 34 fas_a <- 6 misses_a <- 16 cr_a <- 44 summ_a <- sdt_summary_from_counts(hits_a, fas_a, misses_a, cr_a) summ_a ``` # Variance estimation Miller’s exact‑binomial method is recommended for most cases: ```{r} var_a_miller <- sdt_summary_variance(hits_a, fas_a, misses_a, cr_a, method = "miller") var_a_miller ``` Gourevitch & Galanter’s delta method is faster but more approximate: ```{r} var_a_gouv <- sdt_summary_variance(hits_a, fas_a, misses_a, cr_a, method = "gourevitch") var_a_gouv ``` Bootstrap variance can also be used: ```{r} set.seed(1) var_a_boot <- sdt_summary_variance(hits_a, fas_a, misses_a, cr_a, method = "bootstrap", nboot = 1000) var_a_boot ``` # Comparing two conditions ```{r} # Condition B counts hits_b <- 28 fas_b <- 10 misses_b <- 22 cr_b <- 40 compare_sdt_summary(hits_a, fas_a, misses_a, cr_a, hits_b, fas_b, misses_b, cr_b, metric = "dprime", method = "miller") ``` You can also compare **criterion** or **lnβ**: ```{r} compare_sdt_summary(hits_a, fas_a, misses_a, cr_a, hits_b, fas_b, misses_b, cr_b, metric = "c", method = "bootstrap", nboot = 2000) ``` # Notes - Extreme rates (0 or 1) are corrected by default (loglinear). - If only **published d′ or β** are available, you need a reported **SE/variance** to compare conditions. Without counts or SEs, z-tests are not identifiable. # The compound-model alternative `fit_max_sdt()` fits the compound (MAX SDT) model of Duncan (2006) directly to lineup counts, estimating d' and a criterion together, and gives the sampling distribution by parametric bootstrap. It is the lineup-appropriate counterpart to everything above. ## What it needs, and why you may not have it The yes/no functions need four numbers per condition. The compound model needs **five**, because it distinguishes outcomes the 2 x 2 framing merges: | | Yes/no summary | Compound model | |---|---|---| | Target-present | hits, misses | correct IDs, **filler IDs**, rejections | | Target-absent | false alarms, correct rejections | suspect/filler choices, rejections | The extra cell is the filler identification, and this is the practical constraint: **you cannot convert 2 x 2 counts into compound-model input**, because collapsing to "miss" has already merged filler identifications with rejections. If a published table reports only hits and false alarms, the yes/no route above is the only one available. If the suspect/filler/rejection breakdown survives, prefer the compound model. ## A worked example These are the Profile-lineup counts from Reisberg and Pezdek (2024), the data on which the reference implementation of this model was written: two conditions, each with 96 target-present and 96 or 106 target-absent lineups of size 6. ```{r compound_fit, eval = precompute} # Stored with the vignette; set R4LINEUPS_PRECOMPUTE=true to recompute. fit_free <- fit_max_sdt( n_hit = 69, n_tp_choose = 82, n_fa = 64, N_tp = 96, N_ta = 106, n_hit_2 = 68, n_tp_choose_2 = 78, n_fa_2 = 42, N_tp_2 = 96, N_ta_2 = 96, n = 6, nboot = 300, seed = 11 ) ``` ```{r compound_fit_stored, include = FALSE} stored("compound_fit", "fit_free") ``` ```{r compound_table} knitr::kable( data.frame( Condition = c("Condition 1", "Condition 2"), `d'` = round(c(fit_free$dprime_1, fit_free$dprime_2), 3), `95% CI` = c(sprintf("[%.2f, %.2f]", fit_free$boot_ci["lower", "d1"], fit_free$boot_ci["upper", "d1"]), sprintf("[%.2f, %.2f]", fit_free$boot_ci["lower", "d2"], fit_free$boot_ci["upper", "d2"])), `Criterion` = round(c(fit_free$lambda_1, fit_free$lambda_2), 3), check.names = FALSE ), row.names = FALSE, caption = "Compound (MAX SDT) model fitted to the Reisberg & Pezdek Profile-lineup counts, with parametric bootstrap intervals from 300 replicates." ) ``` The bootstrap intervals **are** the sampling distribution: replicates are simulated from the fitted model and refitted, so the spread reflects the estimator's behaviour rather than an asymptotic approximation. ```{r compound_diagnostics} cat("Goodness of fit: chi-square =", round(fit_free$chisq, 3), "on", fit_free$df, "df, p =", round(fit_free$p_value, 4), " ") cat("Bootstrap replicates used:", fit_free$boot_n_used, " failed:", fit_free$boot_n_failed, " ") ``` Unlike the exactly-identified summary estimators, this model has residual degrees of freedom, so it **can** be found to fit badly — which is informative in itself. ## Comparing conditions The comparison is a nested test rather than a Wald test on a difference: fit the model with d' free across conditions, refit with d' constrained equal, and compare the chi-square change. ```{r compound_compare} fit_equal_d <- fit_max_sdt( n_hit = 69, n_tp_choose = 82, n_fa = 64, N_tp = 96, N_ta = 106, n_hit_2 = 68, n_tp_choose_2 = 78, n_fa_2 = 42, N_tp_2 = 96, N_ta_2 = 96, n = 6, constrain_d = TRUE ) compare_max_sdt(fit_free, fit_equal_d) ``` Constraining d' to be equal costs 0.42 in chi-square for one degree of freedom, which is nothing: these two conditions do not differ detectably in discriminability. `constrain_c = TRUE` tests the criterion in the same way. ## Which to use | Situation | Use | |---|---| | Only hits and false alarms survive, e.g. from a published table | The yes/no summary functions above, stating that d' is a yes/no analogue | | Full suspect/filler/rejection breakdown available | `fit_max_sdt()` — it respects the compound structure and gives a goodness-of-fit test | | Trial-level data with confidence | The lineup models in `vignette("model_comparison")` and `vignette("lineup_memory_models")` | A note for anyone reproducing published analyses: `fit_max_sdt()` uses mutually exclusive response cells, whereas some reference scripts for this model fit total target-present choices alongside culprit identifications, which counts the latter twice. Estimates therefore differ — see the "Relation to Smith's scripts" section of `?fit_max_sdt`. # Technical Details ## The estimators From hits \(H\), false alarms \(F\), signal trials \(N_s\) and noise trials \(N_n\): \[ d' = z(p_H) - z(p_F), \qquad c = -\tfrac{1}{2}\bigl[z(p_H) + z(p_F)\bigr], \qquad \ln\beta = d'c \] with \(p_H = H/N_s\) and \(p_F = F/N_n\). The \(\ln\beta = d'c\) identity holds under equal variance only. ## Variance, and why there are three methods Comparing two conditions needs a variance for the difference. The three options differ in what they assume: | Method | Basis | When to prefer it | |---|---|---| | `"gourevitch"` | Delta-method approximation (Gourevitch & Galanter, 1967) | Quick, and adequate with large counts away from the extremes | | `"miller"` | Binomial moment sums evaluated at estimated rates (Miller, 1996) | The package default; its plug-in Wald test is approximate and can be liberal with sparse false alarms | | `"bootstrap"` | Parametric resampling of independent binomial counts at estimated rates | Avoids a normal approximation for the transformed draws, but still assumes the binomial sampling model; boundary behaviour needs assessment | Exact binomial moment summation does not make the resulting plug-in comparison an exact test. In an independent audit scenario with 200 signal and 200 noise trials per condition, true hit rate .15 and false-alarm rate .01, nominal 5% Miller Wald tests rejected equal conditions about 8% of the time for d' and 9% for ln-beta (12,000 datasets). These are limits of the approximation, rather than evidence that one variance option is uniformly preferable. The comparison is a Wald test, with \(\mathrm{Var}(\Delta) = \mathrm{Var}_A + \mathrm{Var}_B\), valid because the two conditions are independent samples. It would be wrong for a within-subjects design, where the two estimates are correlated; the package provides no within-subjects version. ## The extreme-rate correction A hit rate of exactly 1 or a false-alarm rate of exactly 0 gives an infinite \(z\), so a correction is unavoidable. Three are available via `correction`: | Option | Rate computed as | Applied | |---|---|---| | `"loglinear"` (default) | \((x + 0.5)/(n + 1)\) | To every case, extreme or not | | `"half"` | \(0.5/n\) or \((n - 0.5)/n\) | Only when the count is 0 or \(n\) | | `"none"` | \(x/n\) | Never (rates are clamped away from 0 and 1 to keep \(z\) finite) | The default applies unconditionally rather than only to offending cells, which is the form that behaves better in simulation. The choice is recorded in the returned object rather than being applied silently, and it should be stated in any writeup: with small counts it moves d′ appreciably. ## Assumptions 1. Equal-variance signal detection. Unequal variance requires the full ROC, not a single point; see `fit_sdt_roc()`. 2. The two conditions are independent samples. 3. Every trial contributes one binary response, so the four counts partition the sample. 4. The task is treated as yes/no detection. See the caution above on compound decision tasks (Duncan, 2006) for what this costs when the data are lineups. # References ::: {.hanging-indent} Duncan, M. (2006). *A signal detection model of compound decision tasks* (Technical Note DRDC TR 2006-256). Defence R&D Canada, Toronto. Gourevitch, V., & Galanter, E. (1967). A significance test for one parameter isosensitivity functions. *Psychometrika, 32*(1), 25-33. Miller, J. (1996). The sampling distribution of d'. *Perception & Psychophysics, 58*(1), 65-72. Reisberg, D., & Pezdek, K. (2024). Data analysed with the MAX SDT model in scripts supplied by A. M. Smith (personal communication, 4 April 2024). Suero, M., Privado, J., & Botella, J. (2017). Estimating the variances and covariance of the parameters of signal detection theory. *Psicothema, 29*(3), 432-437. Tredoux, C. G., Sporer, S. L., Vredeveldt, A., Kempen, K., & Nortje, A. (2021). Does constructing a facial composite affect eyewitness memory? A research synthesis and meta-analysis. *Journal of Experimental Criminology, 17*(4), 713-741. https://doi.org/10.1007/s11292-020-09432-z ::: # Further Reading - `?sdt_summary_from_counts`, `?sdt_summary_variance`, `?compare_sdt_summary` - `vignette("sdt_glm_analysis")` for trial-level SDT with covariates and random effects - `vignette("model_comparison")` and `vignette("lineup_memory_models")` for models that respect the compound structure of a lineup decision