The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

TRexSelector R package guide

Working without data copies

The rule

The backend works on the caller’s data wherever possible. An R matrix is bound by its memory address, a memory-mapped file by its mapping, and the preprocessing a method needs is applied in place and undone afterwards. A design of several gigabytes is therefore held once, by R or by the file, and not a second time by the backend. This page states the contract that makes it safe. The bindings hand the memory of an R matrix to the C++ core through Rcpp [1] and RcppEigen [2], as an Eigen map [3] of the same buffer, and a memory-mapped file through Boost.Interprocess [4].

Pass double matrices

R stores a matrix of whole numbers as integers. The backend binds double storage, so an integer matrix, a data frame or a logical matrix is converted first, and the conversion is a copy by R’s own rules. The conversion happens visibly in R before the call, once, and a double matrix passes through untouched. To keep a large design copy-free, create it as double.

Xi <- matrix(sample(0:2, 60 * 20, replace = TRUE), 60, 20)
storage.mode(Xi)
#> [1] "integer"
storage.mode(Xi) <- "double"     # one explicit conversion, then zero-copy from here on
storage.mode(Xi)
#> [1] "double"

Selectors: in place at construction, back at the end

A selector centers and scales X in place when it is constructed and writes the original values back when select() returns. The values come back up to rounding, because centering, scaling and their inverse are floating-point operations. In between, X holds the working design. restore() writes back early for a selector that will not run, and a selector that is garbage collected without running writes back in its finalizer. A select() that ends with an error writes back at once and retires the selector. A selector runs once: after select() the design is back at its original scale, so a second select() on the same object is an error, and a new selector is constructed for a new run.

set.seed(1)
n <- 80
p <- 40
X <- matrix(rnorm(n * p), n, p)
y <- as.numeric(X[, 1] * 2 + rnorm(n))
X_before <- X + 0                      # a copy, a plain assignment would share the buffer with X
sel <- TRexSelector$new(X, y, tFDR = 0.1, seed = 2, verbose = FALSE, control = trex_control(K = 10))
round(sqrt(colSums(X^2))[1:3], 6)     # the working design: unit columns
#> [1] 1 1 1
sel$select()
max(abs(X - X_before))                 # restored up to rounding
#> [1] 4.440892e-16

T-solvers: writers and readers

A T-solver with normalize = TRUE or intercept = TRUE is a writer. It transforms X and D in place, keeps the statistics, and holds a claim on both matrices until restore() or garbage collection writes them back. A second writer on a claimed matrix is refused, because a second transformation would move the data by rounding.

D <- matrix(rnorm(n * p), n, p)
w1 <- TLARS_Solver$new(X, D, y)
tryCatch(TLASSO_Solver$new(X, D, y), error = function(e) cat("error:", conditionMessage(e), "\n"))
#> error: TLASSO: the input matrix X is already held by another live T-solver or T-Rex selector that preprocessed it in place. A second in-place preprocessing is not idempotent (it moves the data by rounding) and would corrupt both. Pre-scale once in front and construct the solvers with normalize = false and intercept = false (readers never claim a buffer), or restore() the previous solver, or let it go out of scope first.
w1$restore()

A solver with normalize = FALSE and intercept = FALSE is a reader. It runs on the data as it finds them and never writes, so any number of readers can share one design. A pipeline scales the design once with a scaler and constructs readers.

sx <- MLLpNormScaler$new(norm_type = 2, center = TRUE)
invisible(sx$fit_transform_inplace(X))
sd <- MLLpNormScaler$new(norm_type = 2, center = TRUE)
invisible(sd$fit_transform_inplace(D))
y_c <- y - mean(y)
opts <- tsolver_options(normalize = FALSE, intercept = FALSE, precentered = TRUE)
r1 <- TLARS_Solver$new(X, D, y_c, options = opts)
r2 <- TLASSO_Solver$new(X, D, y_c, options = opts)
r1$execute_step(T_stop = 2)
r2$execute_step(T_stop = 2)
invisible(sx$inverse_transform_inplace(X))
invisible(sd$inverse_transform_inplace(D))
max(abs(X - X_before))
#> [1] 8.881784e-16

y is never written. The solver centers an internal copy of the response.

Loading a saved solver

load() restores a solver into one constructed on the same data in the same state. Load into a reader on pre-scaled data, or into a writer constructed with the same options on the same raw data, which then keeps the duty to write back.

Mapped files are never read into memory

A memory-mapped matrix is bound through the same interface as an R matrix. The backend touches the pages it needs and the file keeps the data, so the working set of a run is the backend’s own buffers, never a copy of the design. The R-level accessors ([ and as.matrix()) do copy the requested block into R memory, by design.

A file opened in "readonly" mode cannot be written, so it serves exactly the code that only reads. That is a selector under scaling_mode = "NONE", which takes the design as supplied, a T-solver that is a reader, and the fits of the regressions. A selector under "L2" or "ZSCORE", a writer among the T-solvers, the in-place scalers and TRexSPCASelector write into the design, so they refuse a read-only file with an error and need "readwrite" mode.

The low-memory engines

The ridge and elastic-net classes have a low_memory engine that avoids every allocation of the size of X.

Class low_memory = FALSE low_memory = TRUE
MLRidgeCV each fold gathers and standardizes its training rows and takes a thin singular value decomposition (SVD), about three to four copies of X in flight the dual (kernel) form, \(n \times n\) kernels accumulated from column blocks read in place, memory \(O(n^2)\)
MLElasticNet coordinate descent on a centered and standardized working copy, one \(n \times p\) matrix coordinate descent on X itself, no allocation of the size of X
MLElasticNetCV one working copy per fit every fold on X itself with a training mask

The two engines compute the same estimator on the same grid and agree to rounding. The selectors that tune \(\lambda_2\) by cross-validation (GVS, SPCA, Tikhonov) use these engines, and the elastic-net tuner of the sparse principal component selector keeps its peak memory under 0.3 times the size of X.

cv_a <- MLRidgeCV$new()
cv_a$fit(X, y, n_folds = 5, n_lambda = 30)
cv_b <- MLRidgeCV$new(low_memory = TRUE)
cv_b$fit(X, y, n_folds = 5, n_lambda = 30)
all.equal(cv_a$get_cv_errors(), cv_b$get_cv_errors())
#> [1] TRUE

Measuring the footprint

get_memory_bytes() reports what a T-solver holds on the heap. X and D are borrowed and not counted.

r1$get_memory_bytes()
#> [1] 14140

Bibliography

[1]
D. Eddelbuettel and R. François, “Rcpp: Seamless R and C++ integration,” J. Stat. Softw., vol. 40, no. 8, pp. 1–18, 2011, doi: 10.18637/jss.v040.i08.
[2]
D. Bates and D. Eddelbuettel, “Fast and elegant numerical linear algebra using the RcppEigen package,” J. Stat. Softw., vol. 52, no. 5, pp. 1–24, 2013, doi: 10.18637/jss.v052.i05.
[3]
G. Guennebaud, B. Jacob, et al., “Eigen v3.” 2010. Available: https://libeigen.gitlab.io/
[4]
I. Gaztañaga, “Boost.Interprocess.” 2008. Available: https://www.boost.org/libs/interprocess

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.