The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

staRburst staRburst logo

CRAN status R-CMD-check License

Seamless AWS cloud bursting for parallel R workloads

staRburst lets you run parallel R code on AWS with zero infrastructure management. Scale from your laptop to 100+ cloud workers with a single function call. Supports both EC2 (recommended for performance and cost) and Fargate (serverless) backends.

Features

Installation

From CRAN:

install.packages("starburst")

Development version (from GitHub):

remotes::install_github("scttfrdmn/starburst")

Quick Start

library(starburst)

# One-time setup (~2 min to provision; first run also builds the worker image,
# +5-10 min once)
starburst_setup()

# Run parallel computation on AWS (each task should be real work — seconds+ —
# so it's worth shipping; batch tiny items first). Console output is illustrative:
results <- starburst_map(
  inputs,                       # e.g. a list of scenarios / parameter sets
  function(x) expensive_computation(x),
  workers = 50
)
#> [Starting] Starting starburst cluster with 50 workers
#> [Status] Processing 200 items with 50 workers
#> [Starting] Submitting 200 tasks...
#> [Wait] Progress: 200/200
#> [OK] Completed
#> [Cost] Estimated cost: (printed per run)

Example: Monte Carlo Simulation

library(starburst)

# Define simulation
simulate_portfolio <- function(seed) {
  set.seed(seed)
  returns <- rnorm(252, mean = 0.0003, sd = 0.02)
  prices <- cumprod(1 + returns)

  list(
    final_value = prices[252],
    sharpe_ratio = mean(returns) / sd(returns) * sqrt(252)
  )
}

# Each simulation is tiny (sub-millisecond), so BATCH them: 100 tasks of 100
# simulations each, not 10,000 one-simulation tasks. (Thousands of tiny tasks are
# an anti-pattern — see the "Workload Shapes" guide for measured numbers.)
batches <- split(1:10000, ceiling(seq_along(1:10000) / 100))
results <- starburst_map(
  batches,
  function(seeds) lapply(seeds, simulate_portfolio),
  workers = 50
)
results <- unlist(results, recursive = FALSE)  # flatten to 10,000 results

# Extract results
final_values <- sapply(results, function(x) x$final_value)
sharpe_ratios <- sapply(results, function(x) x$sharpe_ratio)

# Summary
mean(final_values)    # Average portfolio outcome
quantile(final_values, c(0.05, 0.95))  # Risk range

For real, measured performance (when the cloud wins, when it loses, and how to size tasks/workers), see the Workload Shapes and Performance guides.

Advanced Usage

Reuse Cluster for Multiple Operations

# Create cluster once
cluster <- starburst_cluster(workers = 50, cpu = 4, memory = "8GB")

# Run multiple analyses
results1 <- cluster$map(dataset1, analysis_function)
results2 <- cluster$map(dataset2, processing_function)
results3 <- cluster$map(dataset3, modeling_function)

# All use the same Docker image and configuration

Custom Worker Configuration

# For memory-intensive workloads
results <- starburst_map(
  large_datasets,
  memory_intensive_function,
  workers = 20,
  cpu = 8,
  memory = "16GB"
)

# For CPU-intensive workloads
results <- starburst_map(
  cpu_tasks,
  cpu_intensive_function,
  workers = 50,
  cpu = 4,
  memory = "8GB"
)

Detached Sessions

Run long jobs and disconnect - results persist in S3:

# Start a detached session (EC2 + Spot by default)
session <- starburst_session(workers = 50)

# Fan work out by submitting one task per input
task_ids <- lapply(inputs, function(input) {
  session$submit(quote(expensive_function(input)),
                 globals = list(input = input))
})
session_id <- session$session_id

# Disconnect - job continues running.
# Later (hours/days), reconnect from a fresh R session:
session <- starburst_session_attach(session_id)
status  <- session$status()          # Check progress
results <- session$collect(wait = TRUE)  # Collect results (in submission order)

# Cleanup when done
session$cleanup()

How It Works

  1. Environment Snapshot: Captures your R packages using renv
  2. Container Build: Creates a Docker image with your environment, cached in ECR
  3. Task Creation: Creates one task per element of your input (.x) — there is no automatic chunking; batch small items yourself if per-task overhead matters
  4. Worker Launch: Starts EC2 workers by default (Spot-enabled), or Fargate tasks if you pass launch_type = "FARGATE"; falls back to waves if quota-limited
  5. Data Transfer: Serializes each task’s function, inputs, and globals to S3 using the fast qs2 format
  6. Execution: Each worker pulls a task from S3, runs your function, pushes the result back to S3
  7. Result Collection: Downloads and combines results in the original order
  8. Cleanup: Automatically shuts workers down (warm-pool timeout configurable)

Cost Management

# Set an hourly cost ceiling (USD/hour)
starburst_config(
  max_hourly_cost = 10,       # Jobs estimated over $10/hour won't start
  cost_alert_threshold = 5     # Warn at $5/hour
)

# Costs shown transparently
results <- starburst_map(data, fn, workers = 100)
#> [Starting] Starting starburst cluster with 100 workers
#> [OK] Completed in 1380.0 seconds
#> [Cost] Estimated cost: $1.34

Quota Management

When using the Fargate backend, staRburst automatically handles AWS Fargate vCPU quota limits (EC2 uses your standard On-Demand/Spot limits):

results <- starburst_map(data, fn, workers = 100, cpu = 4, launch_type = "FARGATE")
#> [!] Requested 100 workers (400 vCPUs) but quota allows 25 workers (100 vCPUs)
#> [!] Using 25 workers instead

Your work still completes, just with fewer workers. You can request quota increases through AWS Service Quotas.

API Reference

Main Functions

Configuration Options

starburst_config(
  max_hourly_cost = 10,        # USD/hour rate ceiling
  cost_alert_threshold = 5     # USD/hour warning
)

Documentation

Full documentation available at starburst.ing

Comparison

Feature staRburst RStudio Server on EC2 Coiled (Python)
Setup time 2 minutes 30+ minutes 5 minutes
Infrastructure management Zero Manual Zero
Learning curve Minimal Medium Medium
Auto scaling Yes No Yes
Cost optimization Automatic Manual Automatic
R-native Yes Yes No (Python)

Requirements

For detailed setup instructions, see the Getting Started guide.

Roadmap

v0.3.9 (current CRAN release)

v0.3.8

Future Releases

Contributing

Contributions welcome! See the GitHub repository for contribution guidelines.

License

Apache License 2.0 - see LICENSE

Copyright 2026 Scott Friedman

Citation

@software{starburst,
  title = {staRburst: Seamless AWS Cloud Bursting for R},
  author = {Scott Friedman},
  year = {2026},
  version = {0.3.9},
  url = {https://starburst.ing},
  license = {Apache-2.0}
}

Credits

Built using the paws AWS SDK for R.

Container management with renv and rocker.

Inspired by Coiled for Python/Dask.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.