The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
First stable release. The token core, the DuckDB backend, embedding matching, diagnostics, and calibration are feature-complete and the public API is stable. This release adds the documentation that makes the package usable end to end.
workshop_register,
workshop_listings,
workshop_panel,
match_labels_example: synthetic
woodworking-workshop data with planted difficulty tiers and ground-truth
links, used throughout the articles. Each tier (containment, movers,
phonetic twins, hub tokens) has a minority that measurably benefits from
the feature it exercises.Staged entity resolution, region-free linking across blocks, and an always-on cost guard, plus embedding reuse and faster preparers.
Run strategies in order, carry residuals forward, resolve entities once at the end.
multi_stage_dedup() and
multi_stage_search(): run an ordered list
of strategies as successive passes. multi_stage_dedup()
finds duplicates within one table; multi_stage_search()
links records across tables, or across years of one pooled table with
self = TRUE. Both take a mix of exact, fuzzy, and embedding
strategies; multi_stage_search() supports
collapse-and-continue so a slowly drifting name links one step at a
time. Renames multi_stage_match().exact_strategy(): identical-token-set
matching as a strategy, for a cheap first pass. Runs through
detect_duplicates() and search_candidates()
like any strategy. Optional containment matches a subset rather than an
exact set, with a per-column min_containment_tokens
floor.resolve_entities(): group an edge list
into entities (connected components) and pick a representative per
group.materialize_records(): fetch the
original rows for a set of ids, the complement of
extract_unmatched().plan_strategy(): compare blocking keys
before matching. Reports each candidate’s block sizes, comparison cost,
and how many true twins stay co-blocked, without computing any
scores.rarity_distribution(): report a
column’s token frequency and rarity before matching, with a suggested
min_rarity.find_stopwords(): list a column’s
high-frequency, low-information tokens for
filter_stopwords().duckdb_control(): one object for
DuckDB execution tuning (batch size, scoring chunk key, per-chunk
failure policy, progress), passed as control =. Replaces
the loose batch arguments.Follow an entity across geographic blocks (movers, name drift, year to year) without giving up block-based cost control.
block_on_tokens(): block on a record’s
own rare name tokens instead of a fixed key, so two records sharing any
rare token are compared wherever they sit. Mix it with plain column
names in block_by.rarity_scope = "global": measure
rarity across the whole corpus, so a distinctive name reads as strong
evidence in any block and a common one stays weak.max_fanout / on_fanout:
an automatic ceiling on comparison cost. When a hot or boilerplate token
would fan a dense block into a near-quadratic join, joinery drops the
offending tokens with a warning ("cap", the default) or
stops ("abort"). On by default. Replaces
max_comparisons.joinery.embedding_cache_dir to persist across sessions, or
joinery.embedding_reuse = FALSE to opt out.clear_embedding_cache(): empty the
cache, optionally on disk too.score_embeddings()
scores all pairs in a block as one matrix product, dropping a few
hundred thousand pairs from seconds to a fraction of a second.drop_short_tokens(): drop tokens below
a length, useful after phonetic encoding.as_cologne(), as_soundex(),
as_metaphone(), and as_nysiis() now encode
token lists as well as raw strings, so you can encode after
tokenizing.normalize_street() gains
drop_house_numbers and drop_stopwords to strip
address noise.word_tokens(),
filter_stopwords(), drop_numeric_tokens(),
token_shapes(), and extract_initials() now run
group-wise over token tables.search_candidates()
rejects overlapping id spaces and prepare_search_data()
rejects duplicate ids, both of which corrupt results silently
otherwise.resolve_entities() no longer drops singletons when ids
mix integer and double forms.summarise_matches() (DuckDB) no longer produces an
out-of-range histogram bin for scores just above 1.0.drop_joinery_temp_tables() is now exported.Internal consolidation after the calibration work, plus fixes surfaced by a full-scale Yellow-Pages panel build. Output schemas unchanged.
[0, sum(weights)].tbl |> filter(...)) are accepted everywhere.summarise_matches(entity_cols =):
count duplicate groups whose listed columns are single-valued,
separating real stopword clusters from cardinality artefacts.cli::cli_abort() with rlang argument checks
across exported verbs.R/ reorganised under an
eight-prefix naming scheme (see CLAUDE.md).An optional post-match filter that learns to drop false positives from a small labelled sample. The same verb works on token and embedding strategies.
match_features(): build a
one-row-per-pair feature table from a match result, with token-overlap
counts, auxiliary-side informativeness (aIP, after Doherr
2023), and string similarities.fit_filter() /
apply_filter(): fit a logistic false-positive
filter and apply it, choosing a threshold by Youden’s J unless you set
one.calibrate_matches(): one verb
composing features, fit, and apply.calibrate(): evaluate a fitted filter
on a labelled set; returns reliability, Brier score, log-loss, confusion
matrix, and a threshold sweep.sample_matches()
stratification, plus export_for_labelling() /
import_labels() for a CSV round-trip.fit_filter() via joinery_recipe(). All
tidymodels packages are optional; the glm path needs none.Verbs to answer four questions about a strategy and its results: will it work, did it work, why this pair, and where to look.
audit_strategy(): grade a strategy
before matching.summarise_matches(): overview of a
dedup or candidate result, unified across backends.explain_match(): per-token attribution
of a single pair’s score.sample_matches(): draw pairs by mode
(high, low, borderline, ambiguous, top-gap, random).compare_stages(): per-stage coverage
for multi-stage workflows.tinyplot functions, one per view.recommendations().Optional semantic matching that complements rather than replaces the
token core. Use embeddings for fields where word-overlap fails
(paraphrases, multilingual variants, fuzzy free-text descriptions) and
combine them with token strategies via
multi_stage_match().
embedding_strategy(): declarative
strategy for embedding-based linkage, mirroring the ergonomics of
search_strategy(). Specify one or more embedding columns,
an optional block_by, an optional threshold,
and an optional weights vector across embedding
columns.normalize flag for users who want to keep raw
magnitudes.detect_duplicates(), search_candidates(), and
extract_unmatched() all accept an
Embedding_Strategy and return the standard joinery output
schemas (duplicate_group / match_id,
score, rank, original columns).multi_stage_match() accepts a sequence of mixed
Search_Strategy and Embedding_Strategy
objects, threading residuals between stages and stopping early when
either side is exhausted. Useful pattern: cheap token stage first, then
embedding stage on the residual.block_by support for embeddings so
cosine search runs within blocks (e.g. country, year bucket) instead of
across the whole table.tidyllm
(optional Suggests dependency): provider-agnostic helpers
for Ollama, OpenAI, and other tidyllm-supported backends, so users can
move from raw text to a matchable embedding column without leaving
R.block_by SQL bug fixed.A maintenance release with no new user-facing features. The goal was to harden the test suite and close coverage gaps before resuming feature work on embeddings and diagnostics.
methods_duckdb.R coverage raised from 34% to 90%; full
behavioural parity with the data.table backend now exercised by
tests.embedding_methods_* coverage raised to 95%+ on both
data.table and DuckDB backends.batch_duckdb brittleness diagnosed and
fixed (see notes/batch_duckdb_brittleness.md). User-facing
impact: small inputs no longer hit pathological batching behaviour.local_tests/.batch_duckdb small-table brittleness.This release implements advanced matching heuristics that significantly improve accuracy and robustness.
smoothing(method = "log"): Log transformationsmoothing(method = "softmax", temperature = 1.0):
Softmax with temperaturesmoothing(method = "offset", alpha = 0.1): Additive
smoothingsmoothing(method = "none"): No smoothing (default)max_candidates parameter limits top-N matchesfeedback_strength parameter (0-1) controls
intensity.score_pairs_sql() helper consolidates scoring
logicdetect_duplicates() and
search_candidates()These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.