Introduction to datacaged

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Overview

The datacaged package simplifies access to CAGED microdata (Cadastro Geral de Empregados e Desempregados) directly from HuggingFace, loading data into a local DuckDB database for efficient analysis.

It supports three series:

Period	Series	Database table
Jan/2020 – present	Novo CAGED	`caged_mov`, `caged_for`, `caged_exc`
Jan/1992 – Dec/2019	Legacy CAGED	`caged_antigo`
Jan/1992 – Dec/2019	CAGED Adjustments	`caged_ajustes`

Installation

# Via remotes
remotes::install_github("gecomt/datacaged")

Platform compatibility

The datacaged package is compatible with Windows, macOS and Linux with no extra configuration for Novo CAGED (2020+).

Recurso	Windows	macOS	Linux
Download (HTTPS)	OK	OK	OK
Downloads paralelos	OK	OK	OK
Novo CAGED (2020+)	OK	OK	OK
Legacy CAGED (pre-2020, PPMd)	(!) requires 7-Zip	(!) requires 7-Zip	(!) requires 7-Zip

Legacy CAGED uses PPMd compression. If you need this data, install 7-Zip:

Windows: baixe o instalador e instale normalmente
macOS: brew install 7-zip
Linux: sudo apt install 7zip ou sudo dnf install 7zip

Novo CAGED (2020+) uses LZMA, natively supported by the archive package on all platforms.

Parallel downloads

By default, the package downloads 3 files simultaneously (MOV, FOR and EXC for each month), resulting in approximately 3× faster downloads compared to sequential mode.

# Control the number of workers
caged_download(years = 2023, months = 1:3, workers = 3)  # padrão

# Set globally for the entire session
options(datacaged.workers = 4)

# Sequential mode (useful for unstable connections)
caged_download(years = 2023, months = 1, workers = 1)

Basic usage: Full pipeline

The caged_load() function does everything in a single command: downloads .7z files from HuggingFace, extracts, normalises and writes to DuckDB.

library(datacaged)

# Download Novo CAGED Jan–Dec/2023
# Novo CAGED: national file, `states` does not filter
caged_load(
  years    = 2023,
  months   = seq_len(12L),
  db_path = "caged.duckdb"
)

Progress is displayed in the terminal with a progress bar and final summary.

Querying the data

After populating the database, connect and query with dplyr or plain SQL:

library(dplyr)

con <- caged_connect("caged.duckdb")

# Monthly employment balance in 2023
saldo_mensal <- tbl(con, "caged_mov") |>
  group_by(competenciamov) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  arrange(competenciamov) |>
  collect()

saldo_mensal

# Or with direct SQL
DBI::dbGetQuery(con, "
  SELECT
    competenciamov,
    uf,
    SUM(saldomovimentacao) AS saldo,
    AVG(salario)           AS salario_medio,
    COUNT(*)               AS movimentacoes
  FROM caged_mov
  WHERE uf = 35          -- Sao Paulo
  GROUP BY competenciamov, uf
  ORDER BY competenciamov
")

Always close the connection when done:

DBI::dbDisconnect(con, shutdown = TRUE)

Granular functions

For more control, use the functions individually:

1. Download files only

# Download and save to local cache (~/.local/share/R/datacaged por padrão)
manifest <- caged_download(
  years    = 2023,
  months   = c(1L, 2L, 3L),
  destdir = "~/meus_dados/caged_cache"
)

# manifest is a data.frame with the status of each file
dplyr::count(manifest, status)

2. Parse files manually

# One file at a time
df <- caged_parse("~/meus_dados/caged_cache/caged_mov/2023/CAGEDMOV202301.7z")
glimpse(df)

# Several at once
arquivos <- list.files(
  "~/meus_dados/caged_cache/NOVO_CAGED/2023",
  pattern    = "CAGEDMOV",
  full.names = TRUE
)
df_todos <- caged_parse_batch(arquivos)

3. Write to database

caged_to_duckdb(df_todos, db_path = "caged.duckdb")

Inspect the database

caged_info("caged.duckdb")
#> ── caged.duckdb ────────────────────────────────────────
#> Tamanho do arquivo: 142.3 MB
#> ── Tabelas ──────────────────────────────────────────────
#> * "caged_mov"   Registros: 3,665,155
#> * "caged_for"      Registros:    91,098
#> * "caged_exc"       Registros:     7,900
#>   Registros   : 4.823.901
#>   Competências: 202301 – 202312

Example: Historical series with legacy CAGED

# Baixa Legacy CAGED para Nordeste (2015–2019)
nordeste <- c("MA", "PI", "CE", "RN", "PB", "PE", "AL", "SE", "BA")

caged_load(
  years    = 2015:2019,
  db_path = "caged_historico.duckdb"
)

con <- caged_connect("caged_historico.duckdb")

# Evolução anual do saldo formal no Nordeste
tbl(con, "caged_antigo") |>
  mutate(ano = as.integer(substr(as.character(competencia), 1, 4))) |>
  group_by(ano, uf) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  collect() |>
  tidyr::pivot_wider(names_from = uf, values_from = saldo)

DBI::dbDisconnect(con, shutdown = TRUE)

CAGED Adjustments

CAGED Adjustments contain retroactive corrections to legacy CAGED records (up to 2019). Use caged_adjustments_load() to download and write to the caged_ajustes table.

# Baixar ajustes de 2019
caged_adjustments_load(years = 2019, months = seq_len(12L), db_path = "caged.duckdb")

# Listar o que está disponível no HuggingFace
caged_hf_files(type = "ajustes")

# Comparar saldo original vs ajustado
con <- caged_connect("caged.duckdb")

antigo  <- dplyr::tbl(con, "caged_antigo")  |>
  dplyr::group_by(competencia) |>
  dplyr::summarise(saldo_original = sum(saldomovimentacao, na.rm = TRUE))

ajustes <- dplyr::tbl(con, "caged_ajustes") |>
  dplyr::group_by(competencia) |>
  dplyr::summarise(saldo_ajuste = sum(saldomovimentacao, na.rm = TRUE))

dplyr::full_join(antigo, ajustes, by = "competencia") |>
  dplyr::mutate(saldo_final = saldo_original + saldo_ajuste) |>
  dplyr::collect()

DBI::dbDisconnect(con, shutdown = TRUE)

Performance tips

DuckDB is columnar: prefer select() before collect() to bring only the necessary columns.
Re-runs are safe: caged_load() skips competencies already in the database by default. Use overwrite_competencies = TRUE to overwrite.
.7z cache: downloaded files are stored in ~/.local/share/R/datacaged. They can be reused without a new HuggingFace connection.
Memory: the pipeline processes one competency at a time to avoid excessive RAM usage with large periods.

Utilities

# Verificar se o HuggingFace está online antes de baixar
caged_status()

# Listar competências disponíveis no HuggingFace
caged_hf_files()                     # Novo CAGED (últimos 12 meses)
caged_hf_files(type = "antigo")      # Legacy CAGED
caged_hf_files(type = "ajustes")     # CAGED Adjustments

# Atualização incremental — baixa apenas o que ainda não está no banco
caged_update(db_path = "caged.duckdb")
caged_update(db_path = "caged.duckdb", series = c("novo", "antigo"))

# Exportar tabelas para Parquet (nativo DuckDB, muito rápido)
caged_to_parquet("caged.duckdb", output_dir = "~/exports")
caged_to_parquet("caged.duckdb", output_dir = "~/exports",
                 tables = "caged_mov", partition_by = "uf")

Main variables

Column	Description
`competenciamov`	Competency in Novo CAGED, `YYYYMM` format (ex: `202301`)
`competencia`	Competency in legacy CAGED and Adjustments, format `AAAAMM`
`uf`	IBGE state code (ex: `35` = SP)
`municipio`	IBGE municipality code
`saldomovimentacao`	`+1` hire, `-1` dismissal
`salario`	Contracted wage in BRL
`sexo`	`1` male, `3` female
`idade`	Age in years
`escolaridade`	Education level code (1–9)
`racacor`	Race/colour code (1–5)
`tipomovimentacao`	Reason for movement code
`secao`	CNAE 2.0 section (Novo CAGED)
`fonte_tipo`	`MOV`, `FOR`, `EXC`, `ANTIGO` or `AJUSTES`

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.