The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Large files, raw microdata and provenance

Renato Prado Siqueira

2026-08-24

Overview

Record-level health data can be much larger than TABNET aggregates. The package provides two complementary workflows:

Both workflows support discovery, local caching, selected columns, standardized schemas and provenance.

Traditional DBC/DBF microdata

The raw microdata catalog covers SIM, SINASC and SIH/SUS:

microdados_catalogo()

microdados_arquivos(
  sistema = "sih",
  ano = 2024,
  mes = 1,
  uf = "AC"
)

Select only the columns required by the analysis and start with a small number of records:

admissions <- sih_microdados(
  ano = 2024,
  mes = 1,
  uf = "AC",
  colunas = c(
    "MUNIC_RES", "DT_INTER", "DIAG_PRINC", "VAL_TOT"
  ),
  n_max = 1000,
  normalizar = TRUE
)

deaths <- sim_microdados(
  ano = 2023,
  uf = "RR",
  colunas = c("CODMUNRES", "DTOBITO", "CAUSABAS"),
  n_max = 1000,
  normalizar = TRUE
)

births <- sinasc_microdados(
  ano = 2023,
  uf = "RR",
  colunas = c("CODMUNRES", "DTNASC", "SEXO", "PESO"),
  n_max = 1000,
  normalizar = TRUE
)

DBC files are decoded directly and read through the same public interface as DBF files.

Inspect schemas before reading

head(datasus_dicionario("sih"), 10)
#>   sistema tipo      campo           campo_padronizado
#> 1     sih   RD      n_aih                         aih
#> 2     sih   RD       cnes                        cnes
#> 3     sih   RD  munic_res codigo_municipio_residencia
#> 4     sih   RD   dt_inter             data_internacao
#> 5     sih   RD   dt_saida                  data_saida
#> 6     sih   RD diag_princ diagnostico_principal_cid10
#> 7     sih   RD    val_tot                 valor_total
#> 8     sih   RD       sexo                        sexo
#> 9     sih   RD      idade                  idade_anos
#>                               descricao    classe formato
#> 1                  N<U+00FA>mero da AIH character        
#> 2                    C<U+00F3>digo CNES character        
#> 3 Munic<U+00ED>pio de resid<U+00EA>ncia character        
#> 4      Data de interna<U+00E7><U+00E3>o      date  %Y%m%d
#> 5                  Data de sa<U+00ED>da      date  %Y%m%d
#> 6   Diagn<U+00F3>stico principal CID-10 character        
#> 7                  Valor total aprovado   numeric        
#> 8                                  Sexo character        
#> 9                         Idade em anos   numeric

After a read, validate the critical analysis fields:

datasus_validar_esquema(
  admissions,
  sistema = "sih",
  campos = c(
    "codigo_municipio_residencia",
    "data_internacao",
    "diagnostico_principal_cid10",
    "valor_total"
  ),
  estrito = TRUE
)

Selected OpenDataSUS columns

For ordinary files, opendatasus_ler() can select columns while parsing:

resources <- opendatasus_recursos("arboviroses-dengue")
csv_id <- resources$id[resources$formato == "CSV"][1]

sample <- opendatasus_ler(
  "arboviroses-dengue",
  recurso = csv_id,
  colunas = c("DT_NOTIFIC", "SG_UF", "ID_MUNICIP"),
  n_max = 1000
)

Convenience wrappers should be preferred when available because they encode the dataset’s partition rules and curated schema.

Process CSV files in bounded memory

opendatasus_processar() calls a function for each block instead of retaining the entire dataset:

resources <- opendatasus_recursos(
  "notificacoes-de-sindrome-gripal-leve-2020"
)
ms_id <- resources$id[
  resources$formato == "CSV" & grepl("^Dados MS", resources$nome)
][1]

processed <- opendatasus_processar(
  "notificacoes-de-sindrome-gripal-leve-2020",
  recurso = ms_id,
  ano = NULL,
  colunas = c("municipioIBGE", "resultadoTeste"),
  tamanho_bloco = 50000,
  sistema = "sindrome_gripal",
  FUN = function(dados, posicao, arquivo) {
    data.frame(
      arquivo = arquivo,
      bloco_inicial = posicao,
      registros = nrow(dados),
      positivos = sum(
        dados$resultado_teste == "Positivo",
        na.rm = TRUE
      )
    )
  }
)

processed$linhas
processed$blocos
processed$resultados

The callback receives the standardized block when sistema is supplied. Physical parts are processed in catalog order and are listed in processed$arquivos.

Cache and provenance

Downloads are written atomically. Cached files are reused unless atualizar = TRUE is requested:

first <- ocupacao_hospitalar(
  ano = 2022,
  n_max = 1000,
  cache = TRUE
)

source <- datasus_proveniencia(first)
str(source)

The provenance record identifies the official URL, resource, portal update time, local path, download time and MD5 checksum. Multipart reads retain a record for each physical file.

Practical strategy

For large resources:

  1. inspect resource metadata and partitions;
  2. select the smallest useful year, month and state;
  3. provide colunas before increasing n_max;
  4. use opendatasus_processar() for reductions that do not need all rows in memory;
  5. validate critical fields with datasus_validar_esquema();
  6. retain provenance with the analytical output.

n_max limits parsed rows, but a remote file may still need to be downloaded in full before parsing. Cache reuse prevents that transfer from being repeated.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.