The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Monitoring and Retrieving Extraction Jobs

Overview

When extract_batch() submits a table-exporter job, it runs asynchronously on the RAP cloud. The job_* functions let you monitor progress, inspect job history, and load results once the job completes.

Typical Workflow

library(ukbflow)

# 1. Submit extraction job
job_id <- extract_batch(c(31, 53, 21022, 22189), file = "ukb_demographics")

# 2. Wait for completion
job_wait(job_id)

# 3. Load result (RAP only)
df <- job_result(job_id)

Monitoring a Job

Check status

job_status() returns the current state of a job:

job_status(job_id)
#> job-XXXXXXXXXXXX
#>            done

Possible states:

State Meaning
idle Queued, waiting to be scheduled
runnable Resources being allocated
running Actively executing
done Completed successfully
failed Failed — see failure message
terminated Manually terminated

For failed jobs, the error message is accessible via:

s <- job_status(job_id)
if (s == "failed") cli::cli_inform(attr(s, "failure_message"))

Wait for completion

job_wait() polls at regular intervals until the job reaches a terminal state:

job_wait(job_id)                    # wait indefinitely (default)
job_wait(job_id, interval = 60)     # poll every 60 seconds
job_wait(job_id, timeout = 7200)    # give up after 2 hours

job_wait() stops with an error if the job fails or is terminated, so you can safely chain it with job_result():

job_wait(job_id)
df <- job_result(job_id)

Retrieving Results

Get the file path

job_path() returns the /mnt/project/ path of the output CSV on RAP:

path <- job_path(job_id)
#> "/mnt/project/results/ukb_demographics.csv"

Use this to read the file directly or pass it to other tools:

df <- data.table::fread(job_path(job_id))

Load into R

job_result() combines job_path() and fread() in one step. Must be run inside the RAP environment:

df <- job_result(job_id)
# returns a data.table, e.g. 502353 rows x 5 cols (incl. eid)

Browsing Job History

job_ls() returns a summary of recent jobs:

job_ls()          # last 20 jobs
job_ls(n = 5)     # last 5 jobs

# Filter by state
job_ls(state = "failed")
job_ls(state = c("done", "failed"))

The result is a data.frame with columns:

Column Description
job_id Job ID, e.g. job-XXXXXXXXXXXX
name Job name (typically Table exporter)
state Current state
created Job creation time (POSIXct)
runtime Runtime string, e.g. 0:04:36 (NA if still running)

Getting Help

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.