The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
dqcheckr checks recurring data deliveries: each time a file arrives, one function call verifies it against a per-dataset configuration, records a snapshot, and renders an HTML report. The workflow is deliberately plain:
This vignette walks the whole loop on a small example: generate the configs, read what was generated, edit a rule, validate, run, list the history, and compare two deliveries.
Our example delivery has a wrinkle that real files often have: the
header repeats a column name (Amount appears twice).
A brand-new deployment needs exactly two calls: one for the shared global options, one per dataset. Both are create-only — they will never overwrite a config that already exists.
library(dqcheckr)
generate_global_config(config_dir = "config")
#> [dqcheckr] Global config written: config/dqcheckr.yml
#> Relative snapshot_db/report_output_dir resolve against the working directory -- run from the deployment root.
generate_dataset_config("orders.csv", config_dir = "config")
#> [dqcheckr] Config written: config/orders.yml
#> Review the commented options, then check with validate_config("orders", config_dir = "config").(The detection itself is also available standalone:
sniff_dataset("orders.csv") returns everything the
generator is about to write — format, delimiter, encoding, column names
and types, key-column candidates — without touching disk, if you want to
inspect a delivery before committing to a config.)
The dataset config is fully-optioned and self-documenting: every key the package understands appears exactly once — detected values live, optional settings commented out with their defaults — each with the same description the validator uses. Note three things as you read it:
Amount arrived renamed
positionally (Amount_2, with a
# was "Amount" note), and csv_skip: 1 drops
the file’s own header so the renamed list replaces it;col_names list sits under a warning: it is
positional, so you must never comment out a single
entry — every entry below it would shift onto the wrong column;key_columns candidates
commented out — columns unique in the sample, for you
to confirm, never enforced silently.writeLines(readLines(file.path("config", "orders.yml")))
#> # dqcheckr dataset config -- generated by generate_dataset_config()
#> # from: orders.csv
#> # Detected values are set live; optional settings are shown commented out
#> # with their defaults. Uncomment and edit only what you want to change.
#> # Check your edits any time with: validate_config("orders")
#>
#> ## Identity of the dataset. Written into every config by convention;
#> ## dqcheckr itself takes the name as a function argument and matches it to
#> ## the config filename.
#> dataset_name: "orders"
#>
#> ## Free-text description of the dataset, for humans reading the config. Not
#> ## read by the checks; the GUI wizard has always written it.
#> # description: "describe this dataset"
#>
#> ## Explicit path to the current delivery. Required unless folder is set;
#> ## takes precedence over folder.
#> current_file: "orders.csv"
#>
#> ## Directory holding the deliveries; the two most recently modified files
#> ## become current and previous. Required unless current_file is set.
#> # folder: "path/to/deliveries/"
#>
#> ## Explicit path to the previous delivery for comparison checks. Only
#> ## honoured alongside current_file.
#> # previous_file: "path/to/previous.csv"
#>
#> ## File format of the delivery. Defaults to CSV; "fwf" switches to the
#> ## fixed-width reader and makes fwf_widths required.
#> format: "csv"
#>
#> ## Text encoding of the delivery. ASCII and its aliases are read as UTF-8
#> ## (lossless superset); a declared UTF-8 is validity-scanned before parsing.
#> encoding: "UTF-8"
#>
#> ## Field separator for CSV files.
#> delimiter: ","
#>
#> ## Quote character for CSV files, passed through to the reader.
#> # quote_char: "\""
#>
#> ## POSITIONAL LIST -- always keep one entry per physical column, in file
#> ## order. NEVER comment out a single entry: every entry below it would
#> ## shift onto the wrong column. To exclude a column from checks, leave it
#> ## here and omit it from expected_columns / rules instead.
#> ## Explicit column names replacing the file's own header, in physical column
#> ## order. Omit to use the file's header. POSITIONAL: always list every
#> ## column; pair with csv_skip: 1 when the file has a header row being
#> ## replaced.
#> col_names:
#> - "Date"
#> - "Amount"
#> - "Currency"
#> - "Amount_2" # was "Amount"
#> - "Status"
#>
#> ## Leading lines to drop from a CSV before reading, e.g. the original header
#> ## row when col_names supplies replacement names.
#> csv_skip: 1
#>
#>
#> ## Column widths for fixed-width files, in physical order. Required when
#> ## format is "fwf". POSITIONAL: always list every column; never comment out
#> ## an entry.
#> # fwf_widths: [5, 10]
#>
#> ## Column names for fixed-width files, matching fwf_widths entry-for-entry.
#> ## POSITIONAL.
#> # fwf_col_names:
#> # - "col_1"
#> # - "col_2"
#>
#> ## Leading lines to drop from a fixed-width file before reading.
#> # fwf_skip: 0
#>
#> ## Columns the delivery is expected to contain; missing or unexpected
#> ## columns are flagged. Name-keyed: commenting an entry out safely drops
#> ## that column from the expectation.
#> expected_columns: ["Date", "Amount", "Currency", "Amount_2", "Status"]
#>
#> ## Columns forming the row identity, checked for uniqueness and missingness.
#> # key_columns: ["Date", "Amount", "Amount_2"] # candidates unique in the sample -- confirm before uncommenting
#>
#> ## Per-column type pins (character, numeric, or date) overriding type
#> ## inference.
#> column_types:
#> "Date": date
#> "Amount": numeric
#> "Currency": character
#> "Amount_2": numeric
#> "Status": character
#>
#> ## Per-column quality rules; see the rule vocabulary for the keys valid
#> ## inside each column's map.
#> # column_rules:
#> # some_column:
#> # max_missing_rate: 0.05
#> # min_value: 0
#> # pattern: "^[A-Z0-9]+$"
#>
#> ## Dataset-level overrides merged over the global default_rules; see the
#> ## rule vocabulary.
#> # rule_overrides:
#> # max_missing_rate: 0.05
#> # min_row_count: 1
#>
#> ## Path to a user-supplied R file of custom check functions run after the
#> ## built-in checks.
#> # custom_checks_file: "custom_checks.R"
#>
#> ## SQLite snapshot database recording every run. Dataset value overrides the
#> ## global one; relative paths resolve against the working directory.
#> # snapshot_db: "data/snapshots.sqlite"
#>
#> ## Directory receiving rendered HTML reports. Dataset value overrides the
#> ## global one; relative paths resolve against the working directory.
#> # report_output_dir: "reports/"Editing a config is editing a text file. Here we do with code what
you would do in an editor: uncomment the rule_overrides
block and require at least one data row.
validate_config() reports all findings
in one pass — vocabulary, types and ranges, positional-list consistency
— and, when the delivery file is reachable, cross-checks the config
against the file’s header (never its body, so this is cheap even for
very large files).
validate_config("orders", config_dir = "config")
#> dqcheckr config validation: orders -- VALID (tier: config+header)
#> No findings.Typos get a suggestion rather than a silent no-op. Watch what happens if we misspell a key:
writeLines(c(lines, 'delimitter: ";"'), cfg_path)
validate_config("orders", config_dir = "config")
#> dqcheckr config validation: orders -- VALID (tier: config+header)
#> [warning] orders.yml: Unknown key 'delimitter' in the dataset config. Did you mean 'delimiter'?
writeLines(lines, cfg_path) # undoFindings follow one rule worth understanding, because it decides what can stop a run:
run_dq_check() before
any snapshot is written. You cannot accidentally run an unrunnable
config.col_names count that no longer matches — these can mean a
config typo or the supplier changed the delivery, and drift
must be recorded, not crash the run. The run proceeds, and each
warning is persisted into the run’s results as a VC-01
check — visible in the report, the WARN counts, and the history — as
well as printed to the console. Unknown keys (your own annotations
round-trip safely) are warnings too.result <- run_dq_check("orders", config_dir = "config", open_report = FALSE)
#> [dqcheckr] orders: PASS - 0 warning(s), 0 failure(s). Report: /private/var/folders/ys/99f86vb54ys1_r7kg9lhfw780000gn/T/RtmpkHmFAn/dq-demo/reports/orders_20260725_211016_1.html
result$status
#> [1] "PASS"Each run appends a snapshot to the SQLite database and renders a
self-contained HTML report into reports/ (when the Quarto
CLI is available; open_report = TRUE, the default, opens it
for you in interactive use). The report is a plain file — open it from
the file manager, mail it, archive it.
The history is one call away, newest first. The id
column is what the drift comparison takes.
runs <- list_runs("orders", config_dir = "config")
runs[, c("id", "run_timestamp", "overall_status",
"check_pass_count", "check_warn_count", "check_fail_count")]
#> id run_timestamp overall_status check_pass_count check_warn_count
#> 1 1 2026-07-25T21:10:16Z PASS 19 0
#> check_fail_count
#> 1 0list_runs() is per-dataset and resolves the right
database from the configs (honouring a per-dataset
snapshot_db override). For the whole deployment at once —
every dataset sharing a database — drop to the primitive and name the
file yourself:
The next delivery arrives — with a suspiciously changed
Amount profile and a missing status. Run the check again,
then compare the two snapshots.
writeLines(c(
"Date,Amount,Currency,Amount,Status",
"2026-08-01,900.00,AUD,15.00,settled",
"2026-08-02,1250.50,AUD,0.00,",
"2026-08-03,880.25,NZD,8.10,pending"
), "orders.csv")
result2 <- run_dq_check("orders", config_dir = "config", open_report = FALSE)
#> [dqcheckr] orders: FAIL - 0 warning(s), 1 failure(s). Report: /private/var/folders/ys/99f86vb54ys1_r7kg9lhfw780000gn/T/RtmpkHmFAn/dq-demo/reports/orders_20260725_211018_2.htmlcompare_snapshots() defaults to the two most recent
runs; pass two ids from list_runs() to compare
any historical pair. With report = TRUE (the default) it
renders a drift report HTML like the run report; here we just
compute.
cmp <- compare_snapshots("orders", config_dir = "config",
report = FALSE, open_report = FALSE)
#> [dqcheckr] drift: orders snapshot #1 vs #2
names(cmp)
#> [1] "dataset_name" "snap_prev" "snap_curr"
#> [4] "table_drift" "schema_changes" "missing_rate_changes"
#> [7] "non_numeric_changes" "mean_shifts" "spread_changes"
#> [10] "distinct_changes" "report_path"The drift result also carries each numeric column’s
spread (min, max, standard deviation). These are always
reported — but never judged until you set a threshold,
so a config that predates the rule gains information without gaining new
failure modes. The Amount column’s spread more than doubled
with the second delivery, yet exceeds stays
FALSE:
cmp$spread_changes[, c("Column", "numeric_sd_prev", "numeric_sd_curr",
"numeric_sd_shift_pct", "numeric_sd_exceeds")]
#> Column numeric_sd_prev numeric_sd_curr numeric_sd_shift_pct
#> 1 Amount 93.117645 208.296813 1.236921
#> 2 Amount_2 7.507996 7.507996 0.000000
#> numeric_sd_exceeds
#> 1 FALSE
#> 2 FALSEActivating the judgement is the usual edit — uncomment the rule in
the global config (one default for every dataset; a per-column value in
column_rules would win for its column):
gcfg_path <- file.path("config", "dqcheckr.yml")
glines <- readLines(gcfg_path)
glines <- sub("^# default_rules:", "default_rules:", glines)
glines <- sub("^# max_numeric_sd_shift_pct:.*",
" max_numeric_sd_shift_pct: 0.2", glines)
writeLines(glines, gcfg_path)
cmp2 <- compare_snapshots("orders", config_dir = "config",
report = FALSE, open_report = FALSE)
#> [dqcheckr] drift: orders snapshot #1 vs #2
cmp2$spread_changes[, c("Column", "numeric_sd_shift_pct", "numeric_sd_exceeds")]
#> Column numeric_sd_shift_pct numeric_sd_exceeds
#> 1 Amount 1.236921 TRUE
#> 2 Amount_2 0.000000 FALSEFixed-width files. The sniffer guesses column
boundaries from blank gutters and always emits a character-position
ruler comment above fwf_widths so you can
correct a mis-split by counting. A packed file — columns
touching, no gutters — gets explicit TODO widths that
validate_config() flags as an error, so a run refuses to
start until you fill them in against the ruler:
writeLines(c("AB12XY0099QQWW2026",
"CD34ZW0100RRTT2026"), "ledger.txt")
generate_dataset_config("ledger.txt", config_dir = "config")
#> [dqcheckr] Config written: config/ledger.yml
#> Review the commented options, then check with validate_config("ledger", config_dir = "config").
validate_config("ledger", config_dir = "config")
#> dqcheckr config validation: ledger -- INVALID (tier: config+header)
#> [error] ledger.yml: 'fwf_widths' still contains the generator's TODO placeholder. Fill in the column widths (see the ruler comment in the config) before running.The never-overwrite rule. Generators create; humans edit. Re-running a generator against an existing config is refused, and the file is untouched — your hand-tuning can never be silently clobbered:
generate_dataset_config("orders.csv", config_dir = "config")
#> Error in `generate_dataset_config()`:
#> ! Config already exists and will not be overwritten: config/orders.yml
#> Generated configs are created once and then owned by hand edits (the never-overwrite rule). To re-sniff, pass a different dataset_name or config_dir and diff the result.To re-sniff a structurally changed delivery, generate under another name (or into a scratch directory) and diff.
deployment-root/
├── config/ # one YAML per dataset + dqcheckr.yml -- the dataset list
├── data/
│ └── snapshots.sqlite
├── reports/ # one self-contained HTML per run
└── orders.csv # deliveries (or point configs at their real location)The config/ folder is the dataset list:
add a dataset by generating a config, remove one by deleting its file,
inspect one by opening it. The whole workflow is these calls:
| Action | Call |
|---|---|
| New deployment | generate_global_config("config") |
| New dataset | generate_dataset_config("file.csv", config_dir = "config") |
| Check an edit | validate_config("name", config_dir = "config") |
| Run | run_dq_check("name", config_dir = "config") |
| History | list_runs("name", config_dir = "config") |
| Drift | compare_snapshots("name", config_dir = "config") |
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.