The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

datacaged 0.2.3

Message language

• All console messages (errors, warnings, progress) are now written in English in the source code and translated to Brazilian Portuguese through the standard R/gettext mechanism. Users with R set to Portuguese see them in Portuguese automatically; Sys.setLanguage("pt_BR") or Sys.setLanguage("en") switches the language within a session. Previously the messages mixed both languages and the shipped translations were never applied, because the cli calls did not go through gettext().
• The translation catalog now contains only R-pt_BR.po; R-en.po was removed because the source language is English.
• The tests run in English regardless of the machine’s locale, and a new test checks that every translation keeps the {...} code blocks of the original message unchanged.

Bug fixes

• Messages such as “20 registro inserido” now use the correct plural (“20 records inserted” / “20 registros inseridos”): the count was formatted as text before reaching cli, which then always used the singular.

datacaged 0.2.2

Bug fixes

• Fixed test failures on CRAN with duckdb >= 1.5, which caches database instances per file and refuses read_only = TRUE when an instance for the same path is already open (read_only can't be applied to the database instance ... which already exists):
  • caged_load() and caged_adjustments_load() now close their write connection before calling caged_info().
  • The internal .disconnect() helper now also calls duckdb::duckdb_shutdown(), since dbDisconnect(shutdown = TRUE) no longer releases the cached instance.
  • caged_info(), caged_to_parquet() and caged_update() reuse an already-open instance (e.g. a user connection) instead of failing.
• Data loss in caged_for and caged_exc: the deduplication key competencia was taken from competenciamov, but FOR/EXC files contain movements from earlier months that repeat across files, so loading month t+1 silently skipped rows whose competenciamov had already appeared (e.g. 46% of CAGEDFOR202303). The key is now competenciadec for caged_for and competenciaexc for caged_exc (one value per file). caged_mov, caged_antigo and caged_ajustes are unchanged. Action required: databases built with earlier versions should drop caged_for and caged_exc and reload them with caged_load(). Use competenciamov for analyses by month of movement.
• Fixed column-name normalization when the package is installed or run in a non-UTF-8 locale (e.g. C/POSIX): accented headers such as competênciamov became compet_nciamov, and caged_to_duckdb() aborted with “Nenhuma coluna de competencia encontrada”. .normalize_names() now uses chartr() and read_delim() uses name_repair = "unique_quiet".

Behaviour changes

• caged_download_layouts() now downloads the layout files from the HuggingFace repository instead of the MTE FTP server, which was slow and often unreachable. Local file names have accents removed; the original name is kept in the arquivo column.
• caged_update() now compares competencies table by table instead of using the maximum competency. It also fills gaps inside the period already in the database (e.g. a FOR file whose download failed) and calls the pipeline once per year, so it no longer downloads unneeded months (requesting 202512 and 202601 used to also download 202501 and 202612).
• Year validation now matches the data available on HuggingFace: legacy CAGED from 2007 (caged_load(), caged_download()) and CAGED Adjustments from 2002 (caged_adjustments_load()). Earlier years now fail immediately with a clear message instead of finding no files.

More bug fixes

• caged_to_duckdb(overwrite_competencies = TRUE) now runs DELETE and INSERT in a single transaction: if the insert fails, the previous data is kept.
• Errors while checking which competencies are already stored are no longer swallowed (they used to be treated as “none stored”, duplicating rows).
• caged_to_parquet(): partition_by accepts several columns, overwrite = FALSE is respected for partitioned output, a missing partition column gives a clear warning, and paths containing ' work. Partition directories for code columns (competencies, uf, regiao, municipio) are now named competencia=202301/uf=35 instead of competencia=202301.0/uf=35.0; the cast to integer happens only in the export, so the database schema is unchanged.
• caged_adjustments_load() for 2002-2009, where the source is one annual file per year: the file is now downloaded and parsed once instead of once per requested month (about 6x faster for a full year), and only the requested months are stored (previously the whole year was stored).
• Downloads are written to a temporary .part file and renamed when complete, so an interrupted write can no longer leave a truncated file that the cache would treat as valid.
• The internal HuggingFace listing no longer fails under a non-UTF-8 locale for file names with accents.
• caged_hf_files(verbose = FALSE) no longer prints the “Querying HuggingFace” message.

Tests

• No test accesses the network on CRAN anymore: the layouts test and the sequential download test are mocked; real-network tests use skip_on_cran() and skip_if_offline().
• Fixed mocks leaking across test files: on.exit() without add = TRUE after local_mocked_bindings() discarded the restore handler.

Documentation

• Corrected data coverage: legacy CAGED is available from 2007 and CAGED Adjustments from 2002 (previously documented as 1992).
• caged_to_duckdb() documentation now describes the actual deduplication mechanism and the key column per table (it never used INSERT OR IGNORE).
• ?datacaged now lists caged_update(), caged_to_parquet() and caged_download_layouts().
• inst/CITATION now reads the version from DESCRIPTION.
• Updated caged_info() sample output in the vignettes.

datacaged 0.2.1

Breaking changes

• caged_ftp_files() removed — use caged_hf_files() instead.

Bug fixes

• Fixed caged_hf_files(type = "ajustes") returning years > 2019 — now correctly filters to years <= 2019.
• Fixed CAGED Adjustments file name pattern (CAGEDEST_AJUSTES_ was incorrectly read as CAGEDAJUSTES_); updated prefixo_len from 13 to 17.
• Added support for CAGED_AJUSTES/2002a2009/ folder (annual files CAGEDEST_AJUSTES_AAAA.7z).
• Fixed .insert_rows() to handle schema evolution when loading data from multiple years with different column sets (ALTER TABLE ... ADD COLUMN).
• Converted \donttest{} examples that require network access or a populated database to \dontrun{} to pass R CMD check --run-donttest.
• Fixed duplicate caged_ftp_files alias in documentation.
• Removed U+FE0F variation selector from source files (LaTeX PDF error).
• Updated CI workflows: added concurrency, CI=true, reduced coverage timeout to 15 min, added dependabot.yml.
• Replaced absolute file paths in examples (~/Downloads/, D:/dados/) with portable alternatives (file.path(tempdir(), ...), file.path("~", ...)).

datacaged 0.2.0

Breaking changes

• Source migrated from FTP do MTE (ftp://ftp.mtps.gov.br/pdet/microdados/) to HuggingFace dataset (https://huggingface.co/datasets/alexsandroprado/caged). Downloads now use HTTPS — no more FTP port 21 issues behind firewalls.

New functions

• caged_hf_files(): replaces caged_ftp_files() — lists available competencies from the HuggingFace repository via its REST API.
• caged_update(): incremental update — downloads only competencies not yet in the local DuckDB bank, querying the bank max competência and HuggingFace availability.
• caged_to_parquet(): exports DuckDB tables to Parquet files using DuckDB’s native COPY TO ... (FORMAT PARQUET) — supports optional column partitioning. — lists available competencies from the HuggingFace repository via its REST API.

Deprecated

• caged_ftp_files(): kept as an alias with a deprecation warning. Will be removed in a future version. Use caged_hf_files() instead.

Internal changes

• download.R: .ftp_url() renamed to .hf_url(); .download_file() now uses method = "libcurl" with HTTPS and --location to follow CDN redirects.

• availability.R: .ftp_list() renamed to .hf_list_dir(); uses httr2 instead of curl FTP listing.

• status.R: caged_status() now checks HuggingFace API endpoint.

• Added httr2 (>= 1.0.0) to Imports. # datacaged 0.1.2

• DESCRIPTION: URL do FTP do MTE adicionada ao campo Description: conforme política do CRAN.

• man/*.Rd: \dontrun{} substituído por \donttest{} em todos os exemplos — os exemplos requerem internet mas podem ser executados pelo usuário.

• man/*.Rd: db_path = "caged.duckdb" substituído por db_path = file.path(tempdir(), "caged.duckdb") em todos os exemplos para evitar escrita no diretório de trabalho do usuário. # datacaged 0.1.1

• DESCRIPTION: 'DuckDB' entre aspas simples em Title e Description conforme política do CRAN. # datacaged 0.1.0

Primeira versão do pacote — submetida ao CRAN.

Funções exportadas

• caged_load() — pipeline completo: download → parse → DuckDB (Novo CAGED e antigo).
• caged_download() — baixa arquivos .7z do FTP do MTE com cache local e retry automático.
• caged_parse() — lê e normaliza microdados de um arquivo .7z.
• caged_parse_batch() — versão vetorizada de caged_parse() com barra de progresso.
• caged_to_duckdb() — grava dados no DuckDB com controle de duplicatas por competência.
• caged_connect() — abre conexão DBI com o banco DuckDB.
• caged_info() — exibe estatísticas das tabelas no banco.
• caged_adjustments_load() — pipeline completo para o CAGED Ajustes.
• caged_status() — verifica disponibilidade do servidor FTP do MTE.
• caged_ftp_files() — lista competências disponíveis no FTP.
• caged_download_layouts() — baixa os layouts oficiais do MTE.
• Dataset uf_codigos — tabela de UFs com códigos IBGE e regiões.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.