Package {Rhpc}

Rhpc: MPI-based apply-style parallel computing for R

Description

Provides MPI-aware apply-style functions and worker utilities for high-performance computing in R.

Details

The Rhpc package offers MPI-aware versions of common apply-style functions and helper utilities for distributed computation. It supports both static worker configuration and dynamic process generation.

Main functions:

Rhpc_initialize

Initialize MPI environment

Rhpc_getHandle

Get communicator handle to workers

Rhpc_lapply, Rhpc_lapplyLB

Parallel lapply variants

Rhpc_sapply, Rhpc_sapplyLB

Parallel sapply variants

Rhpc_apply

Parallel apply for arrays/matrices

Rhpc_worker_call

Call functions on workers

Rhpc_worker_noback

Call functions without waiting for results on workers

Rhpc_Export

Export variables to workers

Rhpc_setupRNG

Setup random number generators

Rhpc_finalize

Clean up MPI environment

MPI-related options are exposed through getOption(): Rhpc.mpi.c.comm, Rhpc.mpi.f.comm, Rhpc.mpi.rank, and Rhpc.mpi.procs.

See https://github.com/e-nakama/Rhpc for more information and examples.

Evaluate an expression on worker processes

Description

Evaluates an R expression on all worker processes.

Usage

Rhpc_EvalQ(cl, expr,
           usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                             getOption("Rhpc.usequote"), TRUE),
           envir = .GlobalEnv)

Arguments

Details

This function evaluates the given expression on all worker processes and returns the results. It is useful for initialization tasks or querying worker state.

Value

Results of evaluating the expression on all workers.

See Also

Rhpc_worker_call, Rhpc_Export

Export variables to worker processes

Description

Sends R objects from the master process to all worker processes.

Usage

Rhpc_Export(cl, variableNames,
            usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                              getOption("Rhpc.usequote"), TRUE),
            pos = 1,
            envir = as.environment(pos))

Arguments

Details

This function retrieves objects from the master's environment and sends them to all worker processes, making them available in the workers' global environment.

Objects are sent via collective communication.

Value

Returns invisibly. Objects are assigned in worker environments.

See Also

Rhpc_worker_call, Rhpc_EvalQ

Parallel apply using MPI

Description

MPI-based version of apply for arrays and matrices.

Usage

Rhpc_apply(cl = NULL, X, MARGIN, FUN, ...,
           usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                             getOption("Rhpc.usequote"), TRUE))

Arguments

Details

This function applies FUN to array margins specified by MARGIN, distributing the work across MPI workers.

If X has dimension subscripts not in MARGIN, those are used as the margin dimensions for applying the function.

Value

Array or matrix of results from applying FUN to the specified margins.

See Also

Rhpc_lapply, Rhpc_sapply

Quote arguments for remote evaluation

Description

Quotes arguments for use in remote function calls on worker processes.

Usage

Rhpc_enquote(...)

Arguments

Details

This function quotes its arguments to prevent premature evaluation in the master process. The quoted expressions are then evaluated on the worker processes where they have access to the worker's data and environment.

This is primarily an internal utility; users typically do not need to call it directly.

Value

A quoted representation of the arguments suitable for remote evaluation.

See Also

Rhpc_worker_call, Rhpc_EvalQ

Finalize MPI environment

Description

Finalizes the MPI environment and terminates all worker processes.

Usage

Rhpc_finalize()

Details

This function should be called at the end of an Rhpc session to properly shut down the MPI environment and clean up all resources.

Value

Returns invisibly. Resources are cleaned up.

See Also

Rhpc_initialize, Rhpc_getHandle

Get MPI communicator handle

Description

Obtains a handle to the MPI communicator for worker processes.

Usage

Rhpc_getHandle(procs = NA)

Arguments

Details

This function returns an external pointer to the MPI communicator, which is used in all subsequent Rhpc function calls. It also initializes the random number generation stream for workers.

If procs is specified, dynamic process generation is used (MPI_Comm_spawn). If procs is NA, static worker configuration is assumed.

Value

An external pointer to the MPI communicator handle, which can be passed to other Rhpc functions.

See Also

Rhpc_initialize, Rhpc_numberOfWorker, Rhpc_finalize

Initialize MPI environment

Description

Initializes the MPI environment for Rhpc.

Usage

Rhpc_initialize()

Details

This function must be called before any other Rhpc functions to initialize the MPI communicator. It sets up the MPI environment and prepares the system for distributed computation.

Value

Returns invisibly. MPI initialization results are stored internally.

See Also

Rhpc_finalize, Rhpc_getHandle

Parallel lapply using MPI

Description

MPI-based version of lapply that distributes work across worker processes sequentially.

Usage

Rhpc_lapply(cl, X, FUN, ...,
            usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                              getOption("Rhpc.usequote"), TRUE))

Arguments

Details

This function distributes elements of X sequentially to worker processes. The input X is divided among workers and each worker processes its portion in order.

Results are collected and returned in a list maintaining the original order.

Value

A list with results from applying FUN to each element of X.

See Also

Rhpc_lapplyLB, Rhpc_sapply, Rhpc_apply

Parallel lapply with load balancing

Description

MPI-based version of lapply with load balancing for better performance on heterogeneous systems.

Usage

Rhpc_lapplyLB(cl, X, FUN, ...,
              usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                                getOption("Rhpc.usequote"), TRUE))

Arguments

Details

This function distributes work to workers using a load-balancing approach. Workers receive tasks dynamically as they become available, which can provide better performance when task execution times vary significantly.

Results are collected and returned in a list maintaining the original order.

Value

A list with results from applying FUN to each element of X.

See Also

Rhpc_lapply, Rhpc_sapplyLB, Rhpc_apply

Get number of worker processes

Description

Returns the number of active worker processes.

Usage

Rhpc_numberOfWorker(cl)

Arguments

Value

An integer giving the number of worker processes.

See Also

Rhpc_getHandle

Parallel sapply using MPI

Description

MPI-based version of sapply that distributes work across worker processes.

Usage

Rhpc_sapply(cl = NULL, X, FUN, ...,
            usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                              getOption("Rhpc.usequote"), TRUE),
            simplify = TRUE,
            USE.NAMES = TRUE)

Arguments

Details

This function combines the functionality of Rhpc_lapply with automatic simplification of results. When possible, results are simplified to a vector or matrix form rather than remaining as a list.

Value

A simplified version of the result (vector, matrix, or array) or list depending on simplify parameter.

See Also

Rhpc_sapplyLB, Rhpc_lapply, Rhpc_apply

Parallel sapply with load balancing

Description

MPI-based version of sapply with load balancing for better performance.

Usage

Rhpc_sapplyLB(cl = NULL, X, FUN, ...,
              usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                                getOption("Rhpc.usequote"), TRUE),
              simplify = TRUE,
              USE.NAMES = TRUE)

Arguments

Details

This function combines load-balanced distribution with result simplification. Workers receive tasks dynamically as they become available, and results are simplified when possible.

Value

A simplified version of the result (vector, matrix, array, or list) depending on simplify.

See Also

Rhpc_sapply, Rhpc_lapplyLB, Rhpc_apply

Serialize and unserialize R objects

Description

Serialize R objects for transmission or storage, and unserialize them back.

Usage

Rhpc_serialize(obj)
Rhpc_serialize_onlysize(obj)
Rhpc_serialize_norealloc(obj)
Rhpc_unserialize(obj)

Arguments

Details

These functions provide low-level serialization utilities for handling R objects:

Rhpc_serialize

Fully serializes an object to raw bytes, allocating necessary memory

Rhpc_serialize_onlysize

Returns only the size needed for serialization without actual serialization

Rhpc_serialize_norealloc

Serializes without reallocating memory (for efficiency)

Rhpc_unserialize

Reconstructs an R object from its serialized form

These functions are primarily intended for internal Rhpc use but may be useful for debugging or advanced users.

Value

For Rhpc_serialize: raw vector containing serialized data

For Rhpc_serialize_onlysize: integer giving the size in bytes

For Rhpc_serialize_norealloc: raw vector containing serialized data

For Rhpc_unserialize: original R object

Examples

## Serialization example (executable without MPI)
data_to_serialize <- list(a = 1:5, b = c("x", "y", "z"))
serialized <- Rhpc_serialize(data_to_serialize)
deserialized <- Rhpc_unserialize(serialized)
stopifnot(identical(data_to_serialize, deserialized))

Setup random number generators for worker processes

Description

Initializes independent random number streams for each worker process using L'Ecuyer-CMRG algorithm.

Usage

Rhpc_setupRNG(cl, iseed = NULL)

Arguments

Details

This function sets up independent random number streams for each worker to ensure they produce different but reproducible random sequences. It uses the L'Ecuyer-CMRG (Combined Multiple Recursive Generator) algorithm.

The master process's original .Random.seed is saved to options("Rhpc.oldseed") but is NOT automatically restored due to CRAN policy. Users should manually restore if needed: assign(".Random.seed", getOption("Rhpc.oldseed"), envir = globalenv())

Value

Returns invisibly the original .Random.seed that was saved.

See Also

Rhpc_getHandle, Rhpc_lapply

Split a list into chunks

Description

Divides a list or vector into approximately equal-sized chunks for distribution to workers.

Usage

Rhpc_splitList(var, num)

Arguments

Details

This function divides var into num roughly equal-sized chunks. It is useful for manually distributing work to workers when fine-grained control is needed.

This is primarily an internal utility; the apply-style functions handle splitting automatically.

Value

A list of length num, each element containing a portion of var.

See Also

Rhpc_lapply, Rhpc_numberOfWorker

Call a function on worker processes

Description

Executes a function on all worker processes and collects the results.

Usage

Rhpc_worker_call(cl, FUN, ...,
                 usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                                   getOption("Rhpc.usequote"), TRUE))

Arguments

Details

This function broadcasts a function call to all worker processes and waits for results. Results from all workers are collected and returned.

The function or its name is sent to workers via collective communication only on the first call; subsequent calls with the same function reuse the cached version.

Value

Results collected from all worker processes.

See Also

Rhpc_worker_noback, Rhpc_lapply, Rhpc_EvalQ

Call a function on worker processes without collecting results

Description

Executes a function on all worker processes without waiting for or collecting results.

Usage

Rhpc_worker_noback(cl, FUN, ...,
                   usequote = ifelse(is.logical(getOption("Rhpc.usequote")),
                                     getOption("Rhpc.usequote"), TRUE))

Arguments

Details

This function sends a function call to all worker processes but does not wait for results. It is useful for operations like setting options or initializing worker state where return values are not needed.

Value

Returns invisibly. No results are collected from workers.

See Also

Rhpc_worker_call, Rhpc_Export