The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

dasklyr

R front-end for Dask, aimed at the same user-facing shape as sparklyr (connect / copy_to / dplyr / read-write / collect). R talks to the Python Dask client through reticulate. Workers stay Python.

Table of Contents

• Installation
• Connecting to Dask
• Using dplyr
• Reading and Writing Data
• Table Utilities
• Connection Utilities
• Docker cluster

Installation

install.packages("dasklyr")

Until the package is on CRAN, install from this repository (Rscript on PATH):

install.packages(".", repos = NULL, type = "source")

Install Python Dask at the same major release as the cluster you connect to:

pip install "dask[dataframe,distributed]" pyarrow pandas

Point reticulate at that interpreter with environment variables, not hardcoded paths:

Connecting to Dask

Connect to a local client or a remote scheduler. dask_connect() with no arguments is always local; pass master for a cluster (or dask_connect(Sys.getenv("DASK_SCHEDULER"))).

library(dasklyr)
sc <- dask_connect(master = "local")
# sc <- dask_connect(master = Sys.getenv("DASK_SCHEDULER"))

The returned connection (sc) is a dplyr source. The same master + app_name pair reuses an open connection.

Using dplyr

Copy data frames from R into Dask, then use dplyr verbs. Computation stays lazy until collect().

library(dplyr)

students <- data.frame(
  name = c("Ann", "Bob", "Cara", "Dan"),
  class = c("a", "a", "b", "b"),
  math = c(92L, 78L, 88L, 95L),
  english = c(85L, 90L, 76L, 88L)
)
classes <- data.frame(
  class = c("a", "b"),
  teacher = c("Chen", "Liu")
)

st <- copy_to(sc, students, overwrite = TRUE)
cl <- copy_to(sc, classes, overwrite = TRUE)
src_tbls(sc)

Filter, mutate, aggregate:

st %>%
  filter(math >= 90) %>%
  mutate(total = math + english) %>%
  group_by(class) %>%
  summarise(n = n(), avg_math = mean(math), .groups = "drop") %>%
  collect()

Joins use the same by argument as dplyr, including named c("a" = "b"):

st %>%
  left_join(cl, by = "class") %>%
  select(name, class, teacher, math) %>%
  collect()

Supported verbs: select, rename, filter (comparisons, %in%, is.na), mutate (columns, literals, + - * /), arrange / desc(), group_by / ungroup, summarise (n(), n_distinct, mean / sum / min / max), count, distinct, slice_head, left_join / inner_join / semi_join, collect().

head() is the first rows of the first partition. Use slice_head() when you need a global first-n.

Window functions, across(), and nested case_when are not implemented.

Rscript examples/local_demo.R

Reading and Writing Data

dask_write_parquet(st, "students.parquet", overwrite = TRUE)
pq <- dask_read_parquet(sc, "students_pq", "students.parquet")

dask_write_csv(st, "students.csv")
csv <- dask_read_csv(sc, "students_csv", "students.csv")

path must be visible to workers. A laptop tempfile() is fine for master = "local". On a Docker cluster it is not your Windows temp directory; use a shared volume or object store.

Table Utilities

Counterparts of sparklyr’s sdf_* helpers, named ddf_*:

ddf_npartitions(st)
ddf_schema(st)
ddf_nrow(st)
ddf_repartition(st, npartitions = 4)
ddf_persist(st)

dask_dataframe(st) returns the Python Dask DataFrame. dask_invoke(st, "repartition", npartitions = 2L) calls an unwrapped Python method (sparklyr’s invoke()).

Connection Utilities

dask_web(sc)              # dashboard URL (opens a browser when interactive)
dask_connection_find()
dask_disconnect(sc)
dask_disconnect_all()

Docker cluster

Image tag is pinned to dask 2026.8.0. Scheduler port 8786, dashboard port 8787 on the scheduler host (Docker publishes these on the local machine).

The official image is a conda stack (often Python 3.10 / pandas 2.x). Your client Python should use the same dask release; other libraries may still warn VersionMismatchWarning. Computations can still run. For fewer warnings, match Python/pandas as well, or install lz4 on the client.

docker compose up -d --wait --scale worker=2
Rscript examples/cluster_demo.R
docker compose down

Or examples/run_cluster_demo.ps1 (docker and Rscript on PATH). Default scheduler: set DASK_SCHEDULER to the scheduler tcp:// address (Docker Compose uses host port 8786).

Not in scope

MLlib-style ml_*, running R on workers (spark_apply), Spark streaming, JDBC, and a SQL dbGetQuery surface. Those have no Dask counterpart worth shipping as empty wrappers.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.