Package {summarisebig}

summarisebig: Grouped Summaries for Large 'arrow' Datasets

Description

Provides a 'dplyr'-like interface for grouped summaries on large 'arrow' datasets. Complete summaries are attempted in 'arrow' first. When a result can be reconstructed from 'arrow'-computable sufficient statistics, an explicit MapReduce-style reduction and 'R' finalization strategy is available. For arbitrary 'R' functions that require raw group observations, complete groups are materialized in bounded chunks, with optional parallel execution and shared-memory processing. The MapReduce strategy follows the programming model described by Dean and Ghemawat (2008) doi:10.1145/1327452.1327492.

Author(s)

Maintainer: Lorenzo Isella lorenzo.isella@gmail.com

Authors:

• Lorenzo Isella lorenzo.isella@gmail.com

See Also

Useful links:

• https://github.com/larry77/summarisebig

• https://larry77.github.io/summarisebig/

• Report bugs at https://github.com/larry77/summarisebig/issues

Summarise a large Arrow dataset by group

Description

summarise_big() is designed for grouped computations on Arrow datasets when the result may be expressible in Arrow, reconstructible from compact Arrow-computable state, or require arbitrary R code on complete groups.

Usage

summarise_big(
  .data,
  ...,
  .by,
  .order_by = NULL,
  .workers = 2L,
  .strategy = c("parallel_chunks", "shared_chunk", "map_reduce"),
  .chunk_rows = 1e+06,
  .task_rows = 1e+05,
  .oversize = c("error", "warning"),
  .map_reduce = NULL,
  .finalize = NULL,
  .seed = TRUE,
  .try_arrow = TRUE,
  .tmp = tempfile("summarise-big-"),
  .keep_tmp = FALSE
)

Arguments

Details

The ordinary strategies first attempt the complete summary lazily in Arrow. If Arrow cannot execute it, parallel_chunks materializes different group-safe chunks independently, while shared_chunk materializes one chunk at a time and can share it between workers with mori.

With strategy = "map_reduce", Arrow computes the expressions supplied in .map_reduce, only that reduced table is collected, and .finalize runs in ordinary R.

Value

A data frame containing grouped summary results.

Examples

tab <- arrow::Table$create(data.frame(
  grp = c("a", "a", "b", "b"),
  x = c(1, 2, 10, 20)
))

# Arrow fast path.
summarise_big(tab, result = mean(x), .by = grp)

# Arrow reduction followed by an ordinary R finalizer.
summarise_big(
  tab,
  .by = grp,
  .strategy = "map_reduce",
  .map_reduce = list(n = ~ dplyr::n(), sx = ~ sum(x)),
  .finalize = function(d) {
    dplyr::mutate(d, result = sx / n)
  }
)


# Arbitrary R function on complete groups in a Parquet Dataset.
# This example is self-contained, but starts parallel workers and therefore
# may take more than a few seconds on some systems.
custom_slope <- function(x, y) {
  unname(stats::coef(stats::lm(y ~ x))[2])
}

path <- tempfile("summarisebig-example-")
arrow::write_dataset(
  data.frame(
    grp = rep(c("a", "b"), each = 4),
    x = rep(1:4, 2),
    y = c(2, 4, 6, 8, 1, 2, 4, 8)
  ),
  path
)
ds <- arrow::open_dataset(path)

summarise_big(
  ds,
  result = custom_slope(x, y),
  .by = grp,
  .strategy = "parallel_chunks",
  .workers = 2,
  .chunk_rows = 4
)

rm(ds)
gc()
unlink(path, recursive = TRUE, force = TRUE)