Get started with s3pe

Configuration

Put the endpoint and the key pair in ~/.Renviron and restart R. S3PE_BUCKET is the bucket used when you do not say which one.

S3PE_ENDPOINT=https://example.org
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
S3PE_BUCKET=my-bucket

For one session, use_endpoint() and use_bucket() do the same. s3_config() shows what the package resolved (the secret is never printed). Every verb takes a config argument, so you can also build one explicitly, for another store or another key:

library(s3pe)
s3_config()

cfg <- s3_config(endpoint = "https://example.org", bucket = "archive",
                 access_key = Sys.getenv("ARCHIVE_KEY"),
                 secret_key = Sys.getenv("ARCHIVE_SECRET"))

Keys are either relative to the bucket or full s3://bucket/key URLs; the URL wins.

Reading

read_s3() chooses the reader by the key’s extension:

# JSON -> list
meta <- read_s3("data/manifest.json")
meta$generated_at

# gzipped CSV -> tibble (readr); `...` reaches the reader
sales <- read_s3("series/sales.csv.gz", col_types = "Dncl")

# GeoJSON -> sf, when sf is installed
regions <- read_s3("data/regions.geojson")

# Parquet -> data frame (nanoparquet)
panel <- read_s3("etl/panel.parquet")

# anything else -> raw bytes; or ask for them
png <- read_s3("report/figure1.png")
txt <- read_s3("data/manifest.json", format = "text")

A good publisher uploads a manifest last. If manifest.json says the batch is 2026-08, every file of that batch is already there: read it first and use it as the version of what you cache.

cache <- "batch-cache.rds"
meta <- read_s3("data/manifest.json")
if (!file.exists(cache) || readRDS(cache)$generated_at != meta$generated_at) {
  batch <- list(
    generated_at = meta$generated_at,
    series = read_s3("data/series.json"),
    ranking = read_s3("data/ranking.json")
  )
  saveRDS(batch, cache)
}

Writing

write_s3() is the reverse: the extension decides the format and the content type.

write_s3(panel, "report/panel.parquet")
write_s3(panel, "report/panel.csv.gz")
write_s3(list(generated_at = Sys.time(), rows = nrow(panel)),
         "report/manifest.json", pretty = TRUE)
write_s3(model, "report/model.rds")
write_s3(c("# Report", "", "Text..."), "report/README.md")

Upload the data first, the manifest last: readers that follow the convention above will then never see a half-written batch.

Files

upload_s3() and download_s3() move files without reading them into R. A key ending in / gets the file’s name appended; downloads are written to a .s3pe-part file and renamed when complete and verified.

upload_s3("output/figure1.png", "report/")
download_s3("data/regions.geojson", "data/regions.geojson", overwrite = TRUE)

The rest of the bucket

list_s3("data/")                   # tibble: key, size, last_modified, etag
head_s3("data/manifest.json")      # one row, or NULL
exists_s3("data/manifest.json")    # TRUE/FALSE
delete_s3("report/draft.csv")
buckets_s3()

Big files, sharing and sync

Bodies above 64 MiB are uploaded in parts, with every part’s ETag checked and the whole upload aborted on failure; s3_config(multipart_threshold =, part_size =) tune it. metadata = stores user metadata with the object (head_s3() returns it) and headers = sends any extra header.

upload_s3("exports/full.parquet", "archive/", bucket = "my-bucket",
          metadata = c(run = "2026-10-23"))

A pre-signed URL lets someone without credentials read (or write) one object for a while:

presign_s3("reports/report.pdf", bucket = "my-bucket", expires = 3600)

sync_up_s3() and sync_down_s3() mirror a directory and a prefix, moving only what differs (size and ETag). last uploads a manifest after everything else, so readers who check the manifest first never see a half-published batch; delete = TRUE removes extras on the destination.

sync_up_s3("output/", "reports/2026/", bucket = "my-bucket", last = "manifest.json")
sync_down_s3("reports/2026/", "reports-2026/", bucket = "my-bucket")
copy_s3("reports/2026/manifest.json", "reports/latest.json", bucket = "my-bucket")
delete_s3(list_s3("tmp/", bucket = "my-bucket")$key, bucket = "my-bucket")

When things go wrong

Every failure is an error with a class, so a pipeline stops by default and you can catch exactly what you expect:

meta <- tryCatch(
  read_s3("data/manifest.json"),
  s3pe_error_not_found = function(e) NULL
)

• s3pe_error_config: no endpoint, no credentials, no bucket.
• s3pe_error_not_found (404), s3pe_error_access (403), s3pe_error_http (other statuses, with $status and $code).
• s3pe_error_transport: the server could not be reached, after three attempts with a 1 s and a 2 s pause.
• s3pe_error_integrity: the bytes that arrived are not the bytes that were sent.

Transfers print one line each (↑ s3://my-bucket/report/panel.parquet (12.3 MB, 0.8 s)); silence them with options(s3pe.quiet = TRUE) or S3PE_QUIET=1.

A local store

Inside the machine that hosts the store, or against a MinIO or Garage started for development, point the session at it:

use_endpoint("http://127.0.0.1:9000")

Other editions

The same verbs exist in Python (pip install s3pe), Rust (cargo install s3pe, with the s3pe command line) and Go (go get github.com/StrategicProjects/s3pe). They share one specification and one test vector, so a key written by one is read by the others.