Catalog configuration

Connection recipes for the catalogs icebergr supports, how credentials are handled, and what to do when a connection does not work. Code in this vignette is shown but not run.

Credentials are never arguments

icebergr reads credentials from environment variables and provides no argument to pass one. This is deliberate. A token passed as an argument ends up in the script that called it, in .Rhistory, in any knitr cache of the chunk that ran it, and in the traceback() of any error raised nearby.

Variable	Iceberg property	Used for
`ICEBERGR_REST_TOKEN`	`token`	Bearer token for a REST catalog
`ICEBERGR_REST_CREDENTIAL`	`credential`	OAuth2 client credential
`ICEBERGR_REST_OAUTH2_SERVER_URI`	`oauth2-server-uri`	OAuth2 token endpoint
`ICEBERGR_REST_SCOPE`	`scope`	OAuth2 scope
`ICEBERGR_S3_ACCESS_KEY_ID`	`s3.access-key-id`	Object storage access key
`ICEBERGR_S3_SECRET_ACCESS_KEY`	`s3.secret-access-key`	Object storage secret
`ICEBERGR_S3_SESSION_TOKEN`	`s3.session-token`	Temporary session token
`ICEBERGR_S3_REGION`	`s3.region`	Object storage region
`ICEBERGR_S3_ENDPOINT`	`s3.endpoint`	Custom or S3-compatible endpoint

The standard AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_REGION variables are consulted as a fallback, so an environment already configured for AWS works without further setup.

Set them outside your scripts — in ~/.Renviron, in your shell profile, or from your platform’s secret manager:

# ~/.Renviron
ICEBERGR_REST_TOKEN=eyJhbGciOi...
ICEBERGR_S3_REGION=us-east-1

Two guarantees worth relying on:

print() on a catalog never shows its properties.
Errors raised while connecting list property keys only, never values.

You can still pass a credential through ... if you must, but icebergr warns when you do, because it is nearly always a mistake.

Iceberg REST catalog

The common case: Polaris, Lakekeeper, Nessie, Unity Catalog’s Iceberg endpoint, Tabular-style services, or a self-hosted REST catalog.

Sys.setenv(ICEBERGR_REST_TOKEN = "...") # better: set this in ~/.Renviron

catalog <- icebergr_catalog(
  "rest",
  uri = "https://catalog.example.com/api/catalog",
  warehouse = "analytics"
)

icebergr_list_namespaces(catalog)
icebergr_list_tables(catalog, "analytics")

warehouse is whatever the server expects to identify the warehouse — often a name rather than a path.

With OAuth2 client credentials

Sys.setenv(
  ICEBERGR_REST_CREDENTIAL = "client_id:client_secret",
  ICEBERGR_REST_OAUTH2_SERVER_URI = "https://auth.example.com/oauth/token",
  ICEBERGR_REST_SCOPE = "catalog"
)

catalog <- icebergr_catalog("rest", uri = "https://catalog.example.com")

Extra, non-secret properties

Anything iceberg-rust accepts can be passed through ...:

catalog <- icebergr_catalog(
  "rest",
  uri = "https://catalog.example.com",
  `rest.signing-region` = "us-east-1",
  `rest.sigv4-enabled` = "true"
)

Object storage

REST catalogs usually serve tables that live in S3. That needs the optional s3 Cargo feature, which is off by default because opendal is a substantial subtree:

ICEBERGR_CARGO_FEATURES=s3 R CMD INSTALL --preclean .

That build needs network access. The crates vendored inside the CRAN tarball cover the default install only, so cargo fetches the s3 subtree from crates.io.

Check before relying on it:

"s3" %in% icebergr_spec_support()$cargo_features

Sys.setenv(
  ICEBERGR_S3_ACCESS_KEY_ID = "...",
  ICEBERGR_S3_SECRET_ACCESS_KEY = "...",
  ICEBERGR_S3_REGION = "us-east-1"
)

catalog <- icebergr_catalog(
  "rest",
  uri = "https://catalog.example.com",
  storage = "s3"
)

S3-compatible storage (MinIO, R2, Ceph)

Sys.setenv(
  ICEBERGR_S3_ENDPOINT = "https://minio.internal:9000",
  ICEBERGR_S3_ACCESS_KEY_ID = "...",
  ICEBERGR_S3_SECRET_ACCESS_KEY = "..."
)

catalog <- icebergr_catalog(
  "rest",
  uri = "https://catalog.internal/api/catalog",
  storage = "s3",
  `s3.path-style-access` = "true"
)

Most S3-compatible services need path-style access.

AWS Glue

Needs the glue Cargo feature, which implies s3:

ICEBERGR_CARGO_FEATURES=glue R CMD INSTALL --preclean .

catalog <- icebergr_catalog(
  "glue",
  warehouse = "s3://my-bucket/warehouse",
  `glue.region` = "us-east-1"
)

icebergr_list_namespaces(catalog)

Glue uses the AWS SDK’s own credential chain, so instance roles, IRSA, SSO profiles and AWS_PROFILE all work as they do elsewhere.

If the feature is not compiled in, the error says so and how to get it, rather than failing at link time.

Local warehouses

For local files, testing, or a warehouse on a mounted filesystem, use the in-process memory catalog. This needs no server and no credentials, and is what this package’s own tests run against.

warehouse <- "/data/warehouse"
catalog <- icebergr_catalog("memory", warehouse = warehouse)

icebergr_create_namespace(catalog, "db")
tbl <- icebergr_create_table(catalog, "db.events", data.frame(id = integer()))

There is deliberately no "hadoop" catalog type: iceberg-rust does not implement a Hadoop or filesystem catalog. memory is the local equivalent.

Its one real limitation is that the table registry lives in memory, so it does not persist between sessions. Tables already on disk are re-attached with icebergr_register_table():

catalog <- icebergr_catalog("memory", warehouse = "/data/warehouse")
icebergr_create_namespace(catalog, "db")

tbl <- icebergr_register_table(
  catalog, "db.events",
  "/data/warehouse/db/events/metadata/00003-....metadata.json"
)

Point it at the newest metadata file: Iceberg writes a new one per commit, and the newest is the current state of the table.

Performance notes

Push filters down. filter and select are the difference between reading a table and reading the part of it you want. Confirm with icebergr_scan_plan().
limit is not pushdown. iceberg-rust has no row limit in its scan API, so limit bounds decoding, not planning. To read less, filter.
Batch size trades memory for call overhead. batch_size controls rows per Arrow batch; the default is usually right.
Worker threads. The tokio runtime uses 2 worker threads, which keeps parallelism within CRAN’s limits for checks. For large scans over object storage, raise it before loading the package:
```
Sys.setenv(ICEBERGR_WORKER_THREADS = "8")
library(icebergr)
```
The runtime starts on first use, so this has no effect once you have connected.
Parallelise with a PSOCK cluster, not with fork. icebergr drives an async runtime whose worker threads do not survive fork(), and R’s parallel::mclapply() forks. Once you have made any icebergr call in the parent, a forked child that calls into icebergr hangs rather than erroring. Use parallel::makeCluster(), which starts fresh R processes, and open the catalog inside each worker:
```
cl <- parallel::makeCluster(4)
parallel::clusterEvalQ(cl, library(icebergr))
parallel::parLapply(cl, shards, function(shard) {
  catalog <- icebergr_catalog("memory", warehouse = "/data/warehouse")
  tbl <- icebergr_table(catalog, "db.events")
  icebergr_collect(icebergr_scan(tbl, filter = day == shard))
})
```
Note that the handle is opened in the worker. A handle cannot be sent to one: it holds a pointer into the Rust side, and serialising it produces the “no longer usable” error rather than a wrong answer.
Reads stream. Batches are pulled one at a time rather than collected up front, so memory stays bounded by batch size, not table size — until icebergr_collect() materialises the result into a tibble, which is necessarily all of it. To process a table larger than memory, scan it in filtered pieces.

Troubleshooting

could not connect to the REST catalog, with a list of property keys. The keys tell you what the catalog was given. A missing token usually means the environment variable is not visible to R — check Sys.getenv("ICEBERGR_REST_TOKEN"), and remember that ~/.Renviron is read only at startup.

TLS failures behind a corporate proxy. icebergr uses rustls with the platform trust store, so a CA installed in the system store is honoured. A CA present only in R’s own bundle is not.

is not available in this build of icebergr. An optional Cargo feature is missing. The message names the feature and gives the install command.

The compiled Rust component of icebergr is not available. The package loaded but its native routines did not. This is a half-finished source install; reinstall and read the build log for cargo errors.

Reads succeed, writes fail with a permissions error. Iceberg commits write both data files and new metadata. Write access to the table’s metadata/ prefix is required, not just to data/.

A table shows fewer rows than expected. Reading a merge-on-read table is supported: iceberg-rust applies positional and equality deletes during the scan, so rows another engine deleted will correctly be absent. If the count still looks wrong, check whether you are reading an older snapshot — print() on a table shows which one — and whether a filter is pruning more than you meant, with icebergr_scan_plan().

Writes to a table another engine performs deletes on. icebergr only appends, which is always safe. It cannot delete or update rows, so a workflow needing that must do it elsewhere; see icebergr_spec_support().

mclapply() never returns. The async runtime’s worker threads do not survive a fork(), so a forked child that calls into icebergr waits forever on threads that did not come across. There is no error to catch, which is what makes it worth knowing about. Forking before any icebergr call is fine — each child then starts its own runtime — but once the parent has connected, use parallel::makeCluster() instead and open the catalog inside each worker. See Performance notes above.

cannot append to "db.events": it is partitioned by …. Reading a partitioned table works, and icebergr_partitions() reports its spec, but appending to one would have to compute a partition value for every row, which this version does not do. The append is refused before any data file is written, so nothing is left behind in the warehouse. Write to such a table with an engine that supports partitioned writes.