Introduction to jobqueue

Create a jobqueue

library(jobqueue)
jq <- jobqueue()

When you create a jobqueue, several worker processes are created in the background. You can then evaluate R code on those background processes while your main R process is free to do other work.

Important

The background processes are persisent R sessions. Avoid modifying their .GlobalEnv, otherwise your R code will produce different results based on which worker evaluates it.

Create a job

Main article: vignette('eval')

job <- jq$run({ paste('Hello', 'World!') })

This job will begin evaluating immediately, assuming no other jobs are ahead of it in the jobqueue.

Access the Result

Main article: vignette('results')

job$result
#> [1] "Hello World!"

Running <job>$result will block until the job finishes and the result is ready.

Add a Callback Hook

Main article: vignette('hooks')

job$on('done', function (job) message(job$result))
#> [1] "Hello World!"

Adding a callback hook to trigger when the Job is done allows the result to be handled without blocking the main R process.

Interrupt Evaluation

Main article: vignette('stops')

job <- jq$run({ Sys.sleep(10); 'Zzzzz' })
job$stop()
job$result
#> <interrupt: job stopped by user>

If the job’s result is no longer needed and you want to free up compute resources, calling <job>$stop() will terminate the background process. Terminated background process are automatically replaced by new ones.

Passing Data

# Variables to permanently store on the Worker.
jq <- jobqueue(globals = list(MY_DATA = mtcars))

# Variables to temporary add to the Worker.
vars <- list(n = 2, fields = c('mpg', 'cyl', 'disp'))

# The expression to evaluate on the Worker.
expr <- quote(head(MY_DATA, n)[,fields])

job <- jq$run(expr = expr, vars = vars)

job$result
#>               mpg cyl disp
#> Mazda RX4      21   6  160
#> Mazda RX4 Wag  21   6  160