The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

CoinGecko Pro backfill: one-shot survivorship-bias archive

Scope

The free-tier cg_* functions are the right primary entry point for almost all crypto2 users — they require no key and accept the same arguments as their CMC counterparts.

For one specific scenario — bootstrapping a survivorship-bias-corrected archive from scratch in a single batch run — the Pro tier (pro-api.coingecko.com) is the cheapest path: a one-shot subscription gets you per-coin OHLC and listing snapshots for every coin CoinGecko has ever tracked, in a few hours rather than the months of accumulated snapshots that the free tier requires.

This vignette holds the recipes. The functions are written inline rather than exported by crypto2 — they are deliberately kept out of the package namespace so that:

there is no encouragement of paid-API patterns inside a key-free package, and
the recipes can be adapted to any change in the Pro endpoints without bumping the package version.

To use the recipes: copy the function definitions below into a script, provide your Pro API key, and run.

Setup

library(crypto2)         # only for the column conventions we mirror
library(dplyr)
library(tibble)
library(purrr)
library(jsonlite)
library(httr)
library(arrow)

# Your Pro key. Store in .Renviron as COINGECKO_PRO_KEY and read here.
CG_PRO_KEY <- Sys.getenv("COINGECKO_PRO_KEY", unset = NA)
stopifnot(!is.na(CG_PRO_KEY))

A polite Pro client

# Pro tier nominal cap: 500 req / min. Stay below ~ 6 req / s.
pro_sleep <- 0.2

pro_get <- function(path, query = NULL) {
  url <- paste0("https://pro-api.coingecko.com/api/v3/", sub("^/", "", path))
  resp <- httr::GET(
    url,
    query = query,
    httr::add_headers(`x-cg-pro-api-key` = CG_PRO_KEY),
    httr::timeout(60)
  )
  sc <- httr::status_code(resp)
  if (sc == 429) {
    ra <- suppressWarnings(as.numeric(
      httr::headers(resp)[["retry-after"]]))
    if (is.na(ra)) ra <- 30
    Sys.sleep(ra)
    return(pro_get(path, query))  # one retry on 429
  }
  if (sc < 200 || sc >= 300) return(NULL)
  jsonlite::fromJSON(httr::content(resp, as = "text", encoding = "UTF-8"))
}

Recipe 1: full historic id/slug mapping

The Pro endpoint /coins/list?include_platform=false&status=active,inactive returns every coin CoinGecko has ever tracked, active or not. This is the input to the survivorship-bias-corrected universe.

pro_id_mapping <- function() {
  raw <- pro_get("coins/list",
                 query = list(include_platform = "false",
                              status = "active,inactive"))
  if (is.null(raw) || !length(raw)) return(tibble::tibble())
  tibble::tibble(
    slug         = raw$id,
    symbol       = raw$symbol,
    name         = raw$name,
    harvested_at = Sys.Date()
  )
}

mapping <- pro_id_mapping()
nrow(mapping)
#> [1] ~ 17 000 (active) + 5 000-10 000 inactive

To enrich with the numeric CoinGecko IDs, page through /coins/markets:

pro_numeric_ids <- function() {
  per_page <- 250L
  pages <- vector("list", 200L)
  for (i in seq_along(pages)) {
    Sys.sleep(pro_sleep)
    page <- pro_get("coins/markets",
                    query = list(vs_currency = "usd",
                                 per_page = per_page, page = i))
    if (is.null(page) || !nrow(page)) break
    pages[[i]] <- tibble::tibble(
      slug   = page$id,
      id     = as.integer(sub("^.*/coins/images/([0-9]+).*$", "\\1",
                              page$image)),
      rank   = page$market_cap_rank
    )
    if (nrow(pages[[i]]) < per_page) break
  }
  dplyr::bind_rows(pages)
}

ids <- pro_numeric_ids()
mapping_full <- dplyr::left_join(mapping, ids, by = "slug")

Recipe 2: full historic OHLC per coin

Pro /coins/{slug}/ohlc?vs_currency=...&days=max returns daily OHLC for the entire history of the coin in a single call.

pro_ohlc_one <- function(slug, vs = "usd") {
  Sys.sleep(pro_sleep)
  raw <- pro_get(sprintf("coins/%s/ohlc", slug),
                 query = list(vs_currency = vs, days = "max"))
  if (is.null(raw) || !length(raw)) return(NULL)
  tibble::tibble(
    slug      = slug,
    timestamp = as.POSIXct(raw[, 1] / 1000, origin = "1970-01-01", tz = "UTC"),
    open      = raw[, 2],
    high      = raw[, 3],
    low       = raw[, 4],
    close     = raw[, 5]
  )
}

# Run for the entire universe
hist <- purrr::map_dfr(mapping_full$slug, pro_ohlc_one)

Recipe 3: persist as a parquet dataset

The accumulated parquet is the survivorship-bias-corrected archive. Combined with the id mapping it lets cg_history() and cg_list(only_active = FALSE) work correctly on the free tier forever after.

arrow::write_parquet(mapping_full, "cg_id_mapping_pro.parquet")
arrow::write_dataset(
  hist,
  path        = "data/cg_history_pro",
  partitioning = "slug"
)

Where to host the mapping for other users

If you intend the mapping to be reused by cg_id_mapping() (in this package or by other consumers), upload the parquet to a stable, anonymous public URL. The default download path baked into cg_id_mapping() is the Hugging Face dataset sstoeckl/opencryptoassetpricing at data/_static.parquet. Drop your parquet there (after stripping anything but the four columns id, slug, symbol, name, harvested_at) and the free-tier package will pick it up automatically.

Rate-limit budgeting

A one-shot historical bootstrap of ~ 17 000 coins at 0.2 s per call is about 57 minutes for the OHLC sweep alone, plus a few minutes for the mapping and listing snapshot. Plan for ~ 2 hours total wall-clock with generous safety margins. The Pro 30-day trial period is sufficient for exactly one bootstrap.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.