The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Getting started with rcloner

library(rcloner)
has_rclone <- rclone_available()
if (!has_rclone) {
  message("rclone is not installed on this system. ",
          "Code chunks that require rclone are skipped. ",
          "Install with install_rclone().")
}

Overview

rcloner provides an R interface to rclone, a command-line program that supports over 40 cloud storage backends, including:

All file operations (copy, sync, list, move, delete, …) use the same consistent interface regardless of the storage backend.

Installation

Install from CRAN:

install.packages("rcloner")

Or the development version from GitHub:

# install.packages("pak")
pak::pak("boettiger-lab/rcloner")

Installing the rclone binary

rcloner automatically locates a system-installed rclone binary. If rclone is not already on your PATH, install it with:

install_rclone()

This downloads the appropriate pre-built binary from https://downloads.rclone.org for your operating system and architecture and stores it in a user-writable directory — no system privileges required.

Check the installed version:

rclone_version()
#> rclone v1.60.1-DEV
#> - os/version: ubuntu 24.04 (64 bit)
#> - os/kernel: 6.14.0-32-generic (x86_64)
#> - os/type: linux
#> - os/arch: amd64
#> - go/version: go1.22.2
#> - go/linking: dynamic
#> - go/tags: none

Configuring a remote

rcloner manages cloud storage credentials through remotes, which are named configurations stored in rclone’s config file.

Amazon S3

rclone_config_create(
  "aws",
  type     = "s3",
  provider = "AWS",
  access_key_id     = Sys.getenv("AWS_ACCESS_KEY_ID"),
  secret_access_key = Sys.getenv("AWS_SECRET_ACCESS_KEY"),
  region            = "us-east-1"
)

MinIO / S3-compatible

rclone_config_create(
  "minio",
  type     = "s3",
  provider = "Minio",
  access_key_id     = Sys.getenv("MINIO_ACCESS_KEY"),
  secret_access_key = Sys.getenv("MINIO_SECRET_KEY"),
  endpoint          = "https://minio.example.com"
)

Listing configured remotes

rclone_listremotes()

Listing objects

rclone_ls() returns a data frame of objects at a given path.

Local paths (no credentials needed)

# List a local directory
rclone_ls(tempdir(), files_only = TRUE)
#> [1] Path     Name     Size     MimeType ModTime  IsDir   
#> <0 rows> (or 0-length row.names)

Remote paths

# List a bucket on a configured remote
rclone_ls("aws:my-bucket")

# Recursive listing
rclone_ls("aws:my-bucket/data/", recursive = TRUE)

# Directories only
rclone_lsd("aws:my-bucket")

rclone_ls() parses rclone lsjson output and returns a data frame with columns Path, Name, Size, MimeType, ModTime, and IsDir.

Copying and syncing files

Copy

rclone_copy() copies files from source to destination, skipping identical files. It never deletes destination files.

src  <- tempfile()
dest <- tempfile()
dir.create(src)
dir.create(dest)
writeLines("hello from rcloner", file.path(src, "readme.txt"))

rclone_copy(src, dest)
list.files(dest)
#> [1] "readme.txt"

Copy to/from the cloud

# Upload a local directory to S3
rclone_copy("/local/data", "aws:my-bucket/data")

# Download a file from S3
rclone_copy("aws:my-bucket/report.csv", "/local/downloads/")

# Copy a URL directly to cloud storage (no local intermediate)
rclone_copyurl(
  "https://raw.githubusercontent.com/tidyverse/readr/main/inst/extdata/mtcars.csv",
  "aws:my-bucket/mtcars.csv"
)

Sync

rclone_sync() makes the destination identical to the source, deleting destination files that are not in the source. Use with care.

rclone_sync("aws:my-bucket/data", "/local/backup")

Move

rclone_move() copies files and then deletes the source.

rclone_move("aws:staging/file.csv", "aws:archive/2024/file.csv")

Other file operations

# Read a remote file into R
contents <- rclone_cat("aws:my-bucket/config.yaml")

# Get metadata for an object
rclone_stat("aws:my-bucket/data.csv")

# Total size of a path
rclone_size("aws:my-bucket")

# Create a bucket/directory
rclone_mkdir("aws:new-bucket")

# Delete files (keeps directories)
rclone_delete("aws:my-bucket/old-data/")

# Remove a path and all its contents
rclone_purge("aws:my-bucket/scratch")

# Generate a public link (where supported)
rclone_link("aws:my-bucket/report.html")

# Get storage quota info
rclone_about("aws:")

Using the low-level rclone() wrapper

Every rclone subcommand is accessible via the rclone() function, which accepts a character vector of arguments:

# Equivalent to: rclone version
rclone("version")

# Run any rclone command
rclone(c("check", "aws:bucket", "/local/backup", "--one-way"))

Migrating from minioclient

If you are migrating from the minioclient package, the function mapping is:

minioclient rcloner
mc_alias_set() rclone_config_create()
mc_cp() rclone_copy()
mc_mv() rclone_move()
mc_mirror() rclone_sync()
mc_ls() rclone_ls()
mc_cat() rclone_cat()
mc_mb() rclone_mkdir()
mc_rb() rclone_purge()
mc_rm() rclone_delete()
mc_du() rclone_size()
mc_stat() rclone_stat()
mc() rclone()

The main difference is that rcloner uses remotes (e.g. "aws:bucket") rather than aliases (e.g. "alias/bucket"). Remote configuration is done with rclone_config_create() instead of mc_alias_set().

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.