Working with Unity Catalog Volumes

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

In most day-to-day workflows, you will spend most of your time with db_volume_* once a volume already exists.

Which Function to Use

This table is focused on filesystem operations inside an existing volume:

Operation	Scope	Function	Notes
Upload file	Single file	`db_volume_write()`	Upload one local file to a volume path
Download file	Single file	`db_volume_read()`	Download one volume file to local disk
Delete file	Single file	`db_volume_delete()`	Remove one file from a volume
Check file exists	Single file	`db_volume_file_exists()`	Returns `TRUE`/`FALSE`
List contents	Directory	`db_volume_list()`	Lists files/subdirectories for a directory
Create directory	Directory	`db_volume_dir_create()`	Creates target directory path
Check directory exists	Directory	`db_volume_dir_exists()`	Returns `TRUE`/`FALSE`
Delete directory	Directory	`db_volume_dir_delete()`	Use `recursive = TRUE` for non-empty directories
Upload directory	Bulk transfer	`db_volume_upload_dir()`	Parallel upload, `recursive = TRUE` includes subdirectories
Download directory	Bulk transfer	`db_volume_download_dir()`	Parallel download, `recursive = TRUE` includes subdirectories

Example Workflows

Single File Round-Trip

If you just need to move one file in and out of a volume. This example goes beyond the minimum to showcase more than just upload/download.

library(brickster)

volume_root <- "/Volumes/zacdav/default/data"
incoming_dir <- file.path(volume_root, "incoming")
incoming_file <- file.path(incoming_dir, "example.csv")

# create local file
local_file <- tempfile(fileext = ".csv")
write.csv(mtcars, local_file, row.names = FALSE)

# ensure target directory exists
db_volume_dir_create(incoming_dir)

# upload file
db_volume_write(
  path = incoming_file,
  file = local_file,
  overwrite = TRUE
)

# verify + inspect
db_volume_file_exists(incoming_file)
db_volume_list(incoming_dir)

# download file back to local path
downloaded_file <- tempfile(fileext = ".csv")
db_volume_read(
  path = incoming_file,
  destination = downloaded_file
)

# verify that file can be read as csv
read.csv(downloaded_file)

# clean up (optional)
db_volume_delete(incoming_file)
db_volume_dir_delete(incoming_dir)

Bulk Directory Transfer (Upload + Download)

This is a compact pattern for a larger transfer: sample a local dataset to 100 million rows, write a 2-level partitioned Arrow dataset, upload it, then download it back.

library(brickster)
library(arrow)
library(dplyr)

volume_root <- "/Volumes/zacdav/default/data"
landing_dir <- file.path(volume_root, "sample_10m")
local_dir <- tempfile("arrow_sample_")

# sample to 10M rows
# write partitioned Arrow dataset (2 levels deep: cyl/gear)
mtcars |>
  sample_n(size = 1e+07, replace = TRUE) |>
  write_dataset(
    path = local_dir,
    format = "parquet",
    partitioning = c("cyl", "gear")
  )

# bulk upload
db_volume_upload_dir(
  local_dir = local_dir,
  volume_dir = landing_dir,
  overwrite = TRUE,
  recursive = TRUE
)

# bulk download
local_download <- tempfile("arrow_download_")
db_volume_download_dir(
  volume_dir = landing_dir,
  local_dir = local_download,
  overwrite = TRUE,
  recursive = TRUE
)
list.files(local_download, recursive = TRUE)

# cleanup example directory recursively (optional)
db_volume_dir_delete(
  path = landing_dir,
  recursive = TRUE
)

Set recursive = FALSE for non-recursive transfer: only files directly under the source directory are transferred, and nested subdirectories are skipped.

Managing Volume Objects (Optional)

Use db_uc_volumes_* when you need to create or manage the volume object itself (not files inside it).

Operation	Function	Notes
List volumes in schema	`db_uc_volumes_list()`	Returns volumes under `<catalog>.<schema>`
Get one volume	`db_uc_volumes_get()`	Returns metadata for one volume
Create volume	`db_uc_volumes_create()`	Supports `MANAGED` and `EXTERNAL`
Update volume metadata	`db_uc_volumes_update()`	Rename/comment/owner updates
Delete volume	`db_uc_volumes_delete()`	Removes the Unity Catalog volume object

# list volumes in a schema
db_uc_volumes_list(catalog = "<catalog>", schema = "<schema>")

# create a managed volume
db_uc_volumes_create(
  catalog = "<catalog>",
  schema = "<schema>",
  volume = "my_volume",
  volume_type = "MANAGED"
)

# inspect one volume
db_uc_volumes_get(
  catalog = "<catalog>",
  schema = "<schema>",
  volume = "my_volume"
)

After a volume exists, use /Volumes/<catalog>/<schema>/<volume>/... paths with db_volume_* for file operations.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.