The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

dbcturbo: High-Performance Streaming Reader for DATASUS DBC Files

dbcturbo dbcturbo hex logo

CRAN status R-CMD-check License: AGPL-3 Streaming conversion Lifecycle: stable

dbcturbo is an R package to inspect, validate, read, and convert DATASUS .dbc files, with a pure C99 streaming engine for direct native reading, chunked streaming, UTF-8 CSV/DBF export, and direct streaming Parquet conversion.

The C99 engine decodes and transcodes records directly from the decompression stream with bounded memory. read_dbc() uses the native C reader by default for every file size (with optional early stop via n_max), read_dbc_chunks() streams memory-bounded chunks to a user callback, and dbc_to_parquet() writes directly to Parquet via Arrow without creating an intermediate CSV file.


⚡ Why dbcturbo?


🥊 Feature Comparison

Feature dbcturbo read.dbc
Direct native C99 reader (with n_max early stop) ✅ ❌
Memory-bounded chunked streaming (read_dbc_chunks) ✅ ❌
Direct streaming Parquet via Arrow (no temp CSV) ✅ ❌
Direct streaming CSV export (256 KB buffer, UTF-8 BOM) ✅ ❌
Header inspection & deep stream validation (dbc_validate) ✅ ❌
Column projection in C (cols) ✅ ❌
Directory batch conversion (dbc_batch_to_csv) ✅ ❌
Thread-safe pure C99 engine (zero required deps) ✅ ❌

⚙️ How the Streaming Engine Works

Unlike traditional readers that load the entire compressed and uncompressed DBF payload into RAM simultaneously, dbcturbo processes records iteratively:

       ┌──────────────┐
       │   .dbc File  │  Compressed DATASUS database
       └──────┬───────┘
              │
              ▼
   ┌──────────────────────┐
   │ blast() Decompressor │  64 KB chunk decompression
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────┐
   │  C99 Parsing Engine  │  Parse row + transcode encoding to UTF-8
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────┐
   │ Stream Write to Disk │  Write batch to CSV / DBF
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────┐
   │   Discard from RAM   │  Bounded memory for CSV / DBF conversion
   └──────────────────────┘

📊 Performance

Performance depends on the file, encoding, storage device, batch size, and hardware. On a local Ubuntu test, converting DENGBR23.dbc (1,645,956 rows, 121 fields) to CSV completed in 7.78 seconds. Re-run benchmarks on the target machine before relying on a throughput estimate.


⚡ Quick Start

library(dbcturbo)

# Check file metadata instantly (no decompression needed)
meta <- dbc_inspect("DENGBR23.dbc")
cat("Records:", meta$nrecords, "| Columns:", nrow(meta$fields), "\n")
#> Records: 1,645,956 | Columns: 121

# Read directly into R with the native C engine (all file sizes by default)
df <- read_dbc("DENGBR23.dbc")
head(df)

# Preview the first 100 rows in ~1 ms (early decompression stop)
preview <- read_dbc("DENGBR23.dbc", n_max = 100)

# Process huge files in memory-bounded chunks
total_rows <- 0L
read_dbc_chunks("DENGBR23.dbc", chunk_size = 100000L, callback = function(chunk) {
  total_rows <<- total_rows + nrow(chunk)
})

# Validate structural integrity without allocating a data frame
dbc_validate("DENGBR23.dbc", deep = TRUE)

If you prefer the temporary-CSV route (e.g., to forward custom data.table::fread() arguments), pass engine = "data.table" or engine = "base".

install.packages("data.table")
df <- read_dbc("DENGBR23.dbc", engine = "data.table")

📦 Installation

# From CRAN (stable)
install.packages("dbcturbo")

# From GitHub (development)
remotes::install_github("GPimentel14/dbcturbo")

Requirements: A standard C compiler is needed to build from source — Rtools on Windows, or GCC/Clang on Linux/macOS. CRAN binaries require no compiler.


🔧 Output Formats

Choose the format that best fits your workflow:

1. 📊 In-Memory Data Frame (small to medium files)

library(dbcturbo)
df <- read_dbc("DENGBR23.dbc")
head(df)

2. 📄 CSV — Universal, UTF-8 encoded

Outputs a clean UTF-8 CSV with BOM — Portuguese characters (ã, ç, é) display correctly in Excel without configuration.

library(dbcturbo)
# Select fields in the C writer to reduce output size and downstream work
dbc_to_csv(
  "DENGBR23.dbc", "dengue_2023.csv",
  cols = c("DT_NOTIFIC", "SG_UF_NOT", "NU_IDADE_N"),
  verbose = TRUE
)

3. 🗜️ Parquet — Direct streaming via Arrow (no temporary CSV)

Parquet is columnar and convenient to query with R (arrow), Python (pandas, polars), Power BI, and DuckDB. dbc_to_parquet() streams typed chunks directly into arrow::ParquetFileWriter without writing an intermediate CSV file.

library(dbcturbo)
dbc_to_parquet("DENGBR23.dbc", "dengue_2023.parquet",
               cols = c("DT_NOTIFIC", "SG_UF_NOT", "NU_IDADE_N"))

⚠️ Always use .parquet as the output extension. Passing a .csv path to dbc_to_parquet() will raise an informative error.

4. 🗂️ DBF — Legacy compatibility

For EpiInfo, QGIS, and other tools that read dBase format.

library(dbcturbo)
dbc2dbf("DENGBR23.dbc", "dengue_2023.dbf")

5. 🔍 Metadata Inspection (Header-only, no decompression)

library(dbcturbo)
meta <- dbc_inspect("DENGBR23.dbc")
print(meta$nrecords)   # total records
print(meta$fields)     # field names, types, widths, decimals

⚠️ Excel Row Limit

Microsoft Excel supports a maximum of 1,048,576 rows.
Many national DATASUS files (e.g., Dengue, SINASC) exceed this limit.

Solution — filter in R before exporting:

library(dbcturbo)

df <- read_dbc("DENGBR23.dbc")

# Filter to one state (two-digit IBGE code)
df_rs <- df[df$SG_UF_NOT == "43", ]   # Rio Grande do Sul
df_sp <- df[df$SG_UF_NOT == "35", ]   # São Paulo

# Now it fits in Excel
write.csv(df_rs, "dengue_2023_RS.csv", row.names = FALSE)

Common state codes: "35" SP · "33" RJ · "43" RS · "41" PR · "29" BA · "13" AM


🔬 Practical Epidemiology Examples

Decode DATASUS age encoding (NU_IDADE_N)

decode_age <- function(x) {
  x <- as.integer(x)
  unit  <- x %/% 1000
  value <- x %%  1000
  ifelse(unit == 4, value,
  ifelse(unit == 3, value / 12,
  ifelse(unit == 2, value / 365,
  ifelse(unit == 1, value / 8760, NA_real_))))
}
df$age_years <- decode_age(df$NU_IDADE_N)

Calculate notification delays

# read_dbc() already returns DBF date fields as Date by default
df$delay_days <- as.numeric(df$DT_NOTIFIC - df$DT_SIN_PRI)

Convert a directory of DBC files

result <- dbc_batch_to_csv(
  input_dir  = "dbc/2023/",
  output_dir = "csv/2023/",
  recursive  = TRUE,
  workers    = 4L # macOS/Linux; use 1L on Windows
)

Parallel conversion

parallel::mclapply() uses multiple processes on macOS and Linux. On Windows, use dbc_batch_to_csv() sequentially or a Windows-compatible parallel backend.

library(parallel)
files <- list.files("datasus/", pattern = "\\.dbc$", full.names = TRUE)
mclapply(files, function(f) {
  dbc_to_csv(f, sub("\\.dbc$", ".csv", f))
}, mc.cores = 4L)

DuckDB SQL on Parquet (zero RAM overhead)

library(duckdb)
con <- dbConnect(duckdb())
dbGetQuery(con, "
  SELECT SG_UF_NOT, COUNT(*) AS cases
  FROM 'dengue_2023.parquet'
  GROUP BY SG_UF_NOT ORDER BY cases DESC
")
dbDisconnect(con)

📚 Documentation


📖 Citation

If you use dbcturbo in your research or institutional pipelines, please cite it:

@Manual{,
  title  = {dbcturbo: High-Performance Streaming Reader for DATASUS DBC Files},
  author = {Gumercindo {Pimentel Peralta} and Juliana {da Silva}},
  year   = {2026},
  note   = {R package version 0.3.0},
  url    = {https://github.com/GPimentel14/dbcturbo},
}

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.