The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
The backend works on the caller’s data wherever possible. An R matrix is bound by its memory address, a memory-mapped file by its mapping, and the preprocessing a method needs is applied in place and undone afterwards. A design of several gigabytes is therefore held once, by R or by the file, and not a second time by the backend. This page states the contract that makes it safe. The bindings hand the memory of an R matrix to the C++ core through Rcpp [1] and RcppEigen [2], as an Eigen map [3] of the same buffer, and a memory-mapped file through Boost.Interprocess [4].
R stores a matrix of whole numbers as integers. The backend binds double storage, so an integer matrix, a data frame or a logical matrix is converted first, and the conversion is a copy by R’s own rules. The conversion happens visibly in R before the call, once, and a double matrix passes through untouched. To keep a large design copy-free, create it as double.
A selector centers and scales X in place when it is
constructed and writes the original values back when
select() returns. The values come back up to rounding,
because centering, scaling and their inverse are floating-point
operations. In between, X holds the working design.
restore() writes back early for a selector that will not
run, and a selector that is garbage collected without running writes
back in its finalizer. A select() that ends with an error
writes back at once and retires the selector. A selector runs once:
after select() the design is back at its original scale, so
a second select() on the same object is an error, and a new
selector is constructed for a new run.
set.seed(1)
n <- 80
p <- 40
X <- matrix(rnorm(n * p), n, p)
y <- as.numeric(X[, 1] * 2 + rnorm(n))
X_before <- X + 0 # a copy, a plain assignment would share the buffer with X
sel <- TRexSelector$new(X, y, tFDR = 0.1, seed = 2, verbose = FALSE, control = trex_control(K = 10))
round(sqrt(colSums(X^2))[1:3], 6) # the working design: unit columns
#> [1] 1 1 1
sel$select()
max(abs(X - X_before)) # restored up to rounding
#> [1] 4.440892e-16A T-solver with normalize = TRUE or
intercept = TRUE is a writer. It transforms X
and D in place, keeps the statistics, and holds a claim on
both matrices until restore() or garbage collection writes
them back. A second writer on a claimed matrix is refused, because a
second transformation would move the data by rounding.
D <- matrix(rnorm(n * p), n, p)
w1 <- TLARS_Solver$new(X, D, y)
tryCatch(TLASSO_Solver$new(X, D, y), error = function(e) cat("error:", conditionMessage(e), "\n"))
#> error: TLASSO: the input matrix X is already held by another live T-solver or T-Rex selector that preprocessed it in place. A second in-place preprocessing is not idempotent (it moves the data by rounding) and would corrupt both. Pre-scale once in front and construct the solvers with normalize = false and intercept = false (readers never claim a buffer), or restore() the previous solver, or let it go out of scope first.
w1$restore()A solver with normalize = FALSE and
intercept = FALSE is a reader. It runs on the data as it
finds them and never writes, so any number of readers can share one
design. A pipeline scales the design once with a scaler and constructs
readers.
sx <- MLLpNormScaler$new(norm_type = 2, center = TRUE)
invisible(sx$fit_transform_inplace(X))
sd <- MLLpNormScaler$new(norm_type = 2, center = TRUE)
invisible(sd$fit_transform_inplace(D))
y_c <- y - mean(y)
opts <- tsolver_options(normalize = FALSE, intercept = FALSE, precentered = TRUE)
r1 <- TLARS_Solver$new(X, D, y_c, options = opts)
r2 <- TLASSO_Solver$new(X, D, y_c, options = opts)
r1$execute_step(T_stop = 2)
r2$execute_step(T_stop = 2)
invisible(sx$inverse_transform_inplace(X))
invisible(sd$inverse_transform_inplace(D))
max(abs(X - X_before))
#> [1] 8.881784e-16y is never written. The solver centers an internal copy
of the response.
load() restores a solver into one constructed on the
same data in the same state. Load into a reader on pre-scaled data, or
into a writer constructed with the same options on the same raw data,
which then keeps the duty to write back.
A memory-mapped matrix is bound through the same interface as an R
matrix. The backend touches the pages it needs and the file keeps the
data, so the working set of a run is the backend’s own buffers, never a
copy of the design. The R-level accessors ([ and
as.matrix()) do copy the requested block into R memory, by
design.
A file opened in "readonly" mode cannot be written, so
it serves exactly the code that only reads. That is a selector under
scaling_mode = "NONE", which takes the design as supplied,
a T-solver that is a reader, and the fits of the regressions. A selector
under "L2" or "ZSCORE", a writer among the
T-solvers, the in-place scalers and TRexSPCASelector write
into the design, so they refuse a read-only file with an error and need
"readwrite" mode.
The ridge and elastic-net classes have a low_memory
engine that avoids every allocation of the size of X.
| Class | low_memory = FALSE |
low_memory = TRUE |
|---|---|---|
MLRidgeCV |
each fold gathers and standardizes its training rows and takes a
thin singular value decomposition (SVD), about three to four copies of
X in flight |
the dual (kernel) form, \(n \times n\) kernels accumulated from column blocks read in place, memory \(O(n^2)\) |
MLElasticNet |
coordinate descent on a centered and standardized working copy, one \(n \times p\) matrix | coordinate descent on X itself, no allocation of the
size of X |
MLElasticNetCV |
one working copy per fit | every fold on X itself with a training mask |
The two engines compute the same estimator on the same grid and agree
to rounding. The selectors that tune \(\lambda_2\) by cross-validation (GVS, SPCA,
Tikhonov) use these engines, and the elastic-net tuner of the sparse
principal component selector keeps its peak memory under 0.3 times the
size of X.
get_memory_bytes() reports what a T-solver holds on the
heap. X and D are borrowed and not
counted.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.