How the matching thresholds were calibrated

The labeled corpus

599 references in three groups:

• 200 exact-title retracted — records sampled from the Retraction Watch corpus, cited with their exact title (as a well-formatted bibliography would).
• 200 perturbed retracted — the same records with a realistic citation variation (about 15% of title words dropped, lower-cased), to probe how the thresholds behave on imperfect titles.
• 199 clean — non-retracted articles sampled from OpenAlex.

Every reference is matched by title, year, and author only (the DOI is withheld), which is exactly the hard case the thresholds govern.

Result 1: the assertion gate never false-accuses

At the title_exact gate (0.985 + year + first author):

Flag rate by group: exact-title retracted 1.000, perturbed retracted 0.065, clean 0.000.

Reading: no clean reference was ever asserted as retracted (zero false accusations), and every exact-title retracted reference was recovered. The perturbed titles almost never clear the gate (0.065) — by design they fall to “possible” rather than being asserted, which is the conservative behavior we want. The overall recall of 0.532 is dominated by the perturbed group; on citations that reproduce the title faithfully, recall is 1.0.

Result 2: 0.90 is the empirical sweet spot for surfacing

Sweeping the fuzzy threshold and measuring how often a retracted reference is surfaced (as flagged or possible) versus how often a clean reference is wrongly surfaced:

0.90 is the lowest threshold at which no clean reference is surfaced while still recovering 88.5% of retracted references. Below 0.88 the clean-surfacing rate climbs steeply (43.7% at 0.86), which would bury real hits in false positives. Above 0.90 precision stays perfect but recall falls with no benefit.

Conclusion

The defaults are validated by this corpus:

• 0.985 title_exact gate: perfect precision, zero false accusations — the right posture for a tool that could otherwise mislabel a clean paper.
• 0.90 fuzzy threshold: the operating point where clean false-positives reach zero while retaining high recall.

These are heuristics, not calibrated probabilities, and the corpus is modest (599 references); the numbers should be revisited as the corpus grows and across non-English titles. But they show the current thresholds are conservative in the direction that matters: the package prefers “possible, please verify” over a false assertion.

Reproducing

The two scripts below are in the data-raw/ directory of the source repository (they are not part of the installed package). Run them from a repository checkout:

retraction_sync()                       # local corpus
source("data-raw/calibration_corpus.R")   # rebuild the labeled set (online)
source("data-raw/calibration_analysis.R") # recompute the tables above