The DCC cleaning pipeline: Detect, Execute, Report

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

DCC treats data cleaning as an auditable pipeline. Raw data is immutable, every correction is driven by a declarative and versioned rule, and each change is recorded at the cell level so the whole run can be reproduced from a manifest. This vignette walks one small dataset through the full Detect -> Execute -> Report workflow.

A small dataset and a rule set

We start from a tiny response file. S2 has an out-of-range score and S3 is missing an item.

csv <- tempfile(fileext = ".csv")
writeLines(c(
  "sid,score,q1,q2,q3",
  "S1,90,1,2,3",
  "S2,150,2,2,2",
  "S3,70,1,,3"
), csv)

Rules live outside the code as declarative YAML. Here we flag scores outside [0, 100] and respondents missing more than a third of their items.

rules_file <- tempfile(fileext = ".yaml")
writeLines(c(
  "checks:",
  "  - id: R001",
  "    type: range",
  "    variable: score",
  "    min: 0",
  "    max: 100",
  "    severity: fail",
  "  - id: D001",
  "    type: missing_items",
  "    items: [q1, q2, q3]",
  "    max_prop: 0.3"
), rules_file)

Detect

dcc_read() loads the file (with encoding detection and structural diagnostics) and dcc_rules() parses the rule set, recording its hash for the audit trail. dcc_detect() then returns a structured dcc_findings object.

x     <- dcc_read(csv)
rules <- dcc_rules(rules_file)
found <- dcc_detect(x, rules, id_var = "sid")
found

Execute

dcc_execute() applies declarative actions mapped to rule IDs. Input data is never mutated, the whole plan is validated before any change, and findings without an explicit action are returned unhandled rather than silently flagged or dropped. Below we blank the out-of-range score and flag the incomplete respondent.

res <- dcc_execute(
  x, found,
  actions = list(R001 = "set_na", Q_MISSING_ITEMS = "flag"),
  id_var = "sid"
)
#> Warning: Action ID 'Q_MISSING_ITEMS' is deprecated; use 'D001'.
dcc_cleaned(res)
#> <dcc_data>
#>   rows: 3  cols: 5
#>   source: /private/var/folders/f2/vt0pwn11523g8jj9yryz403h0000gn/T/RtmpW6lgf6/file7733490cfb1.csv (csv, latin1)
#>   provenance: read -> execute
#>   L0 findings: 0
#>   data preview:
#>       sid score    q1    q2    q3
#>    <char> <int> <int> <int> <int>
#> 1:     S1    90     1     2     3
#> 2:     S2    NA     2     2     2
#> 3:     S3    70     1    NA     3

Every change is captured in the cell-level audit log, with the old and new value, the triggering rule, and the method.

dcc_audit_log(res)
#>                                                                                              finding_id
#>                                                                                                  <char>
#> 1:    72:detect-1cbd78178e1a99729a1442154703c8cc-284e81edb08f1365d32a8e68d2c18231|4:R001|2:S2|5:score|0
#> 2: 72:detect-1cbd78178e1a99729a1442154703c8cc-284e81edb08f1365d32a8e68d2c18231|4:D001|2:S3|8:<record>|0
#>    record_id variable old_value new_value action check_id
#>       <char>   <char>    <char>    <char> <char>   <char>
#> 1:        S2    score       150      <NA> set_na     R001
#> 2:        S3     <NA>      <NA>      <NA>   flag     D001
#>                                method                    timestamp dcc_version
#>                                <char>                       <char>      <char>
#> 1:                     cell set to NA 2026-09-02T19:30:23.494+0800       1.2.1
#> 2: reviewed and kept (no data change) 2026-09-02T19:30:23.494+0800       1.2.1
#>                        ruleset_hash keyfile_hash
#>                              <char>       <char>
#> 1: 1cbd78178e1a99729a1442154703c8cc         <NA>
#> 2: 1cbd78178e1a99729a1442154703c8cc         <NA>

Report

dcc_report() writes a self-contained HTML report – a management summary plus an audit layer that reconciles findings against changes – with no external rendering dependency.

report_path <- tempfile(fileext = ".html")
dcc_report(res, report_path)
file.exists(report_path)
#> [1] TRUE

Trace and reproduce

dcc_trace() returns the full history of any cell in the cleaned data.

dcc_trace(res, "S2", "score")
#> <dcc_trace> record 'S2', variable 'score'
#>   findings: 1
#>    check_id variable                         evidence severity
#>      <char>   <char>                           <char>   <char>
#> 1:     R001    score value 150 outside range [0, 100]     fail
#>   changes: 1
#>    variable old_value new_value action check_id                    timestamp
#>      <char>    <char>    <char> <char>   <char>                       <char>
#> 1:    score       150      <NA> set_na     R001 2026-09-02T19:30:23.494+0800

Finally, dcc_manifest() captures the input and rule hashes, the actions, and the output hashes; dcc_rerun() re-executes the whole pipeline and verifies the result is byte-identical (timestamps excluded).

check <- dcc_rerun(dcc_manifest(res))
check$reproduced
#> [1] TRUE

Larger-than-memory input

For files that do not fit in memory, dcc_detect_chunked() streams the input with an adaptive backend – data.table for delimited text, arrow for Parquet/Feather – producing findings identical to the in-memory path for record-local checks.

dcc_detect_chunked(csv, rules, chunk_size = 2L, id_var = "sid",
                   encoding = "UTF-8")

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.