The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

zstd

R-CMD-check Codecov test coverage

Zstandard compression and decompression rfor R.

Installation

Install the stable version from CRAN:

install.packages("zstd")

Install the development version from GitHub:

pak::pak("r-lib/zstd")

Usage

library(zstd)

Compress data in memory

zstd_mem_compress() and zstd_mem_decompress() work on raw vectors:

x <- charToRaw(paste(rep("hello world ", 1000), collapse = ""))
cmp <- zstd_mem_compress(x)
length(x)
#> [1] 12000
length(cmp)
#> [1] 30
identical(zstd_mem_decompress(cmp), x)
#> [1] TRUE

Use the level argument to trade speed for a better compression ratio:

zstd_min_clevel()
#> [1] -131072
zstd_default_clevel()
#> [1] 3
zstd_max_clevel()
#> [1] 22
cmp19 <- zstd_mem_compress(x, level = 19)

Compress files

zstd_compress() and zstd_decompress() work on files:

src <- tempfile()
writeLines(rep("hello world", 1000), src)
dst <- tempfile(fileext = ".zst")
zstd_compress(src, dst)
file.size(src)
#> [1] 12000
file.size(dst)
#> [1] 30

zstd_info() shows information about compressed files:

zstd_info(dst)[, -1]
#>    type compressed_size content_size window_size dict_id checksum
#> 1 frame              30        12000       12000       0    FALSE
out <- tempfile()
zstd_decompress(dst, out)
identical(readLines(src), readLines(out))
#> [1] TRUE

Compressed tar archives

zstd_tar_compress() creates a .tar.zst archive from files and directories, and zstd_tar_decompress() extracts it:

dir <- tempfile()
dir.create(file.path(dir, "subdir"), recursive = TRUE)
writeLines("hello", file.path(dir, "a.txt"))
writeLines("world", file.path(dir, "subdir", "b.txt"))

archive <- tempfile(fileext = ".tar.zst")
zstd_tar_compress(dir, archive)

exdir <- tempfile()
zstd_tar_decompress(archive, exdir)
dir(exdir, recursive = TRUE)
#> [1] "file14133fd31f45/a.txt"        "file14133fd31f45/subdir/b.txt"

Dictionaries

For many small, similar inputs, train a dictionary with zstd_train_dict() and use it for both compression and decompression:

samples <- lapply(1:100, function(i) {
  charToRaw(paste0('{"id":', i, ',"name":"sample"}'))
})
dict <- zstd_train_dict(samples, size = 1000)
length(samples[[1]])
#> [1] 24
length(zstd_mem_compress(samples[[1]]))
#> [1] 33
cmp <- zstd_mem_compress(samples[[1]], dict = dict)
length(cmp)
#> [1] 20
identical(zstd_mem_decompress(cmp, dict = dict), samples[[1]])
#> [1] TRUE

License

MIT © Posit Software, PBC

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.