README

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

dbcturbo: High-Performance Streaming Reader for DATASUS DBC Files

dbcturbo is an R package to inspect, read, and convert DATASUS .dbc files, with direct streaming CSV/DBF conversion and UTF-8 CSV output.

The C99 engine writes CSV rows directly from the decompression stream with bounded memory. read_dbc() uses a direct native reader for files up to 50 MiB by default and a CSV reader for larger files; it intentionally materializes its final data frame in memory. dbc_to_parquet() uses a temporary CSV before conversion with Arrow.

⚡ Why dbcturbo?

Reads files with millions of records: Process complete national databases (1.6M+ rows) in seconds.
Streaming architecture: True chunk-by-chunk decompress and write cycle — peak RAM never grows with file size during CSV/DBF conversion.
Small-file fast path: Native decoding avoids CSV and DBF temporary files for low-latency reads.
Bounded conversion memory: CSV/DBF conversion keeps only one DBF record and its output buffer in memory.
Clean UTF-8 output: Automatic CP850/Latin1 transcoding with UTF-8 BOM — Portuguese characters (ã, ç, é) render perfectly in Excel.
Parquet conversion: Converts through a temporary CSV using arrow.
Thread-safe conversion engine: No global mutable C state; callers control any parallelism.
Pure C99 & cross-platform: Zero external C library dependencies — compiles out of the box on Linux, macOS, and Windows.

🥊 Feature Comparison

Feature	`dbcturbo`	`read.dbc`
Streaming conversion in batches	✅	❌
Clean UTF-8 output with BOM	✅	❌
Parquet conversion via Arrow	✅	❌
Direct CSV export	✅	❌
Inspect metadata without decompressing	✅	❌
Directory batch conversion	✅	❌
Thread-safe C99 engine	✅	❌
Bounded-memory CSV/DBF conversion	✅	❌

⚙️ How the Streaming Engine Works

Unlike traditional readers that load the entire compressed and uncompressed DBF payload into RAM simultaneously, dbcturbo processes records iteratively:

       ┌──────────────┐
       │   .dbc File  │  Compressed DATASUS database
       └──────┬───────┘
              │
              ▼
   ┌──────────────────────┐
   │ blast() Decompressor │  64 KB chunk decompression
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────┐
   │  C99 Parsing Engine  │  Parse row + transcode encoding to UTF-8
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────┐
   │ Stream Write to Disk │  Write batch to CSV / DBF
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────┐
   │   Discard from RAM   │  Bounded memory for CSV / DBF conversion
   └──────────────────────┘

📊 Performance

Performance depends on the file, encoding, storage device, batch size, and hardware. On a local Ubuntu test, converting DENGBR23.dbc (1,645,956 rows, 121 fields) to CSV completed in 7.78 seconds. Re-run benchmarks on the target machine before relying on a throughput estimate.

⚡ Quick Start

library(dbcturbo)

# Check file metadata instantly (no decompression needed)
meta <- dbc_inspect("DENGBR23.dbc")
cat("Records:", meta$nrecords, "| Columns:", nrow(meta$fields), "\n")
#> Records: 1,645,956 | Columns: 121

# Small files use the native C reader automatically (up to 50 MiB)
df <- read_dbc("small_file.dbc")
head(df)

For large files, install the optional data.table package. read_dbc() will use it automatically above the native threshold, or request it explicitly with engine = "data.table"; use engine = "native" to force direct native decoding and engine = "base" to force R’s built-in reader.

install.packages("data.table")
df <- read_dbc("DENGBR23.dbc", engine = "data.table")

📦 Installation

# From CRAN (stable)
install.packages("dbcturbo")

# From GitHub (development)
remotes::install_github("GPimentel14/dbcturbo")

Requirements: A standard C compiler is needed to build from source — Rtools on Windows, or GCC/Clang on Linux/macOS. CRAN binaries require no compiler.

🔧 Output Formats

Choose the format that best fits your workflow:

1. 📊 In-Memory Data Frame (small to medium files)

library(dbcturbo)
df <- read_dbc("DENGBR23.dbc")
head(df)

2. 📄 CSV — Universal, UTF-8 encoded

Outputs a clean UTF-8 CSV with BOM — Portuguese characters (ã, ç, é) display correctly in Excel without configuration.

library(dbcturbo)
# Select fields in the C writer to reduce output size and downstream work
dbc_to_csv(
  "DENGBR23.dbc", "dengue_2023.csv",
  cols = c("DT_NOTIFIC", "SG_UF_NOT", "NU_IDADE_N"),
  verbose = TRUE
)

3. 🗜️ Parquet — Conversion through a temporary CSV

Parquet is columnar and convenient to query with R (arrow), Python (pandas, polars), Power BI, and DuckDB. The conversion first creates a temporary CSV, so ensure enough temporary disk space is available.

library(dbcturbo)
dbc_to_parquet("DENGBR23.dbc", "dengue_2023.parquet")

⚠️ Always use .parquet as the output extension. Passing a .csv path to dbc_to_parquet() will raise an informative error.

4. 🗂️ DBF — Legacy compatibility

For EpiInfo, QGIS, and other tools that read dBase format.

library(dbcturbo)
dbc2dbf("DENGBR23.dbc", "dengue_2023.dbf")

5. 🔍 Metadata Inspection (Header-only, no decompression)

library(dbcturbo)
meta <- dbc_inspect("DENGBR23.dbc")
print(meta$nrecords)   # total records
print(meta$fields)     # field names, types, widths, decimals

⚠️ Excel Row Limit

Microsoft Excel supports a maximum of 1,048,576 rows.
Many national DATASUS files (e.g., Dengue, SINASC) exceed this limit.

Solution — filter in R before exporting:

library(dbcturbo)

df <- read_dbc("DENGBR23.dbc")

# Filter to one state (two-digit IBGE code)
df_rs <- df[df$SG_UF_NOT == "43", ]   # Rio Grande do Sul
df_sp <- df[df$SG_UF_NOT == "35", ]   # São Paulo

# Now it fits in Excel
write.csv(df_rs, "dengue_2023_RS.csv", row.names = FALSE)

Common state codes: "35" SP · "33" RJ · "43" RS · "41" PR · "29" BA · "13" AM

🔬 Practical Epidemiology Examples

Decode DATASUS age encoding (`NU_IDADE_N`)

decode_age <- function(x) {
  x <- as.integer(x)
  unit  <- x %/% 1000
  value <- x %%  1000
  ifelse(unit == 4, value,
  ifelse(unit == 3, value / 12,
  ifelse(unit == 2, value / 365,
  ifelse(unit == 1, value / 8760, NA_real_))))
}
df$age_years <- decode_age(df$NU_IDADE_N)

Calculate notification delays

# read_dbc() already returns DBF date fields as Date by default
df$delay_days <- as.numeric(df$DT_NOTIFIC - df$DT_SIN_PRI)

Convert a directory of DBC files

result <- dbc_batch_to_csv(
  input_dir  = "dbc/2023/",
  output_dir = "csv/2023/",
  recursive  = TRUE,
  workers    = 4L # macOS/Linux; use 1L on Windows
)

Parallel conversion

parallel::mclapply() uses multiple processes on macOS and Linux. On Windows, use dbc_batch_to_csv() sequentially or a Windows-compatible parallel backend.

library(parallel)
files <- list.files("datasus/", pattern = "\\.dbc$", full.names = TRUE)
mclapply(files, function(f) {
  dbc_to_csv(f, sub("\\.dbc$", ".csv", f))
}, mc.cores = 4L)

DuckDB SQL on Parquet (zero RAM overhead)

library(duckdb)
con <- dbConnect(duckdb())
dbGetQuery(con, "
  SELECT SG_UF_NOT, COUNT(*) AS cases
  FROM 'dengue_2023.parquet'
  GROUP BY SG_UF_NOT ORDER BY cases DESC
")
dbDisconnect(con)

📚 Documentation

Full vignette: vignette("introduction", package = "dbcturbo")
Function reference: ?dbc_to_csv, ?dbc_to_parquet, ?read_dbc, ?dbc_inspect

📖 Citation

If you use dbcturbo in your research or institutional pipelines, please cite it:

@Manual{,
  title  = {dbcturbo: High-Performance Streaming Reader for DATASUS DBC Files},
  author = {Gumercindo {Pimentel Peralta} and Juliana {da Silva}},
  year   = {2026},
  note   = {R package version 0.2.0},
  url    = {https://github.com/GPimentel14/dbcturbo},
}

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.