| Type: | Package |
| Title: | 'GGML' Tensor Operations for Machine Learning |
| Version: | 0.9.0 |
| Description: | Provides 'R' bindings to the 'GGML' tensor library for machine learning, optimized for 'Vulkan' GPU acceleration with a transparent CPU fallback. The package features a 'Keras'-like sequential API and a 'PyTorch'-style 'autograd' engine for building, training, and deploying neural networks. Key capabilities include high-performance 5D tensor operations, 'f16' precision, and efficient quantization. It supports native 'ONNX' model import (90+ operators) and 'GGUF' weight loading from the 'llama.cpp' and 'Hugging Face' ecosystems. Designed for zero-overhead inference via dedicated weight buffering, it integrates seamlessly as a 'parsnip' engine for 'tidymodels' and provides first-class learners for the 'mlr3' framework. See https://github.com/ggml-org/ggml for more information about the underlying library. |
| Depends: | R (≥ 4.1.0) |
| Imports: | generics, R6, methods, stats, utils |
| License: | MIT + file LICENSE |
| URL: | https://github.com/Zabis13/ggmlR |
| BugReports: | https://github.com/Zabis13/ggmlR/issues |
| Encoding: | UTF-8 |
| SystemRequirements: | C++17, GNU make, libvulkan-dev, glslc (optional, for GPU on Linux), 'Vulkan' 'SDK' (optional, for GPU on Windows) |
| Suggests: | testthat (≥ 3.0.0), mlr3 (≥ 0.21.0), paradox, digest, parsnip, tibble, rlang, dials, lgr, knitr, rmarkdown, Matrix, SeuratObject, RSpectra, irlba, FNN, SingleCellExperiment, SummarizedExperiment, S4Vectors, withr, hardhat |
| VignetteBuilder: | knitr, rmarkdown |
| RoxygenNote: | 7.3.3 |
| Config/testthat/edition: | 3 |
| Config/build/clean-inst-doc: | false |
| NeedsCompilation: | yes |
| Packaged: | 2026-11-21 18:10:29 UTC; yuri |
| Author: | Yuri Baramykov |
| Maintainer: | Yuri Baramykov <lbsbmsu@mail.ru> |
| Repository: | CRAN |
| Date/Publication: | 2026-11-22 00:20:09 UTC |
ggmlR: 'GGML' Tensor Operations for Machine Learning
Description
Provides 'R' bindings to the 'GGML' tensor library for machine learning, optimized for 'Vulkan' GPU acceleration with a transparent CPU fallback. The package features a 'Keras'-like sequential API and a 'PyTorch'-style 'autograd' engine for building, training, and deploying neural networks. Key capabilities include high-performance 5D tensor operations, 'f16' precision, and efficient quantization. It supports native 'ONNX' model import (90+ operators) and 'GGUF' weight loading from the 'llama.cpp' and 'Hugging Face' ecosystems. Designed for zero-overhead inference via dedicated weight buffering, it integrates seamlessly as a 'parsnip' engine for 'tidymodels' and provides first-class learners for the 'mlr3' framework. See https://github.com/ggml-org/ggml for more information about the underlying library.
Author(s)
Maintainer: Yuri Baramykov lbsbmsu@mail.ru (ORCID)
Other contributors:
Georgi Gerganov (Author of the GGML library) [contributor, copyright holder]
Jeffrey Quesnelle (Contributor to ops.cpp) [contributor, copyright holder]
Bowen Peng (Contributor to ops.cpp) [contributor, copyright holder]
Mozilla Foundation (Author of llamafile/sgemm.cpp) [contributor, copyright holder]
See Also
Useful links:
GLU Operation Types
Description
Constants for GLU (Gated Linear Unit) operation types. Used with ggml_glu() and ggml_glu_split().
Usage
GGML_GLU_OP_REGLU
GGML_GLU_OP_GEGLU
GGML_GLU_OP_SWIGLU
GGML_GLU_OP_SWIGLU_OAI
GGML_GLU_OP_GEGLU_ERF
GGML_GLU_OP_GEGLU_QUICK
Format
Integer constants
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
Details
-
GGML_GLU_OP_REGLU(0): ReGLU - ReLU gating -
GGML_GLU_OP_GEGLU(1): GeGLU - GELU gating (used in GPT-NeoX, Falcon) -
GGML_GLU_OP_SWIGLU(2): SwiGLU - SiLU/Swish gating (used in LLaMA, Mistral) -
GGML_GLU_OP_SWIGLU_OAI(3): SwiGLU OpenAI variant -
GGML_GLU_OP_GEGLU_ERF(4): GeGLU with exact erf implementation -
GGML_GLU_OP_GEGLU_QUICK(5): GeGLU with fast approximation
Value
An integer constant representing a GLU operation type
Examples
GGML_GLU_OP_REGLU # 0 - ReLU gating
GGML_GLU_OP_GEGLU # 1 - GELU gating
GGML_GLU_OP_SWIGLU # 2 - SiLU/Swish gating
GGML_GLU_OP_SWIGLU_OAI # 3 - SwiGLU OpenAI
GGML_GLU_OP_GEGLU_ERF # 4 - GELU with erf
GGML_GLU_OP_GEGLU_QUICK # 5 - Fast GELU
Maximum Task Count for Custom Ops
Description
Sentinel for the n_tasks argument of ggml_custom and
ggml_custom_inplace, meaning "use as many threads as the
backend allows" (mirrors GGML_N_TASKS_MAX in ggml.h).
Usage
GGML_N_TASKS_MAX
Format
An object of class integer of length 1.
Sort Order Constants
Description
Sort Order Constants
Usage
GGML_SORT_ORDER_ASC
GGML_SORT_ORDER_DESC
Format
Integer constants
An object of class integer of length 1.
Details
Constants for specifying sort order in argsort operations.
-
GGML_SORT_ORDER_ASC(0): Ascending order (smallest first) -
GGML_SORT_ORDER_DESC(1): Descending order (largest first)
Value
An integer constant representing a sort order
Examples
GGML_SORT_ORDER_ASC # 0 - Ascending order
GGML_SORT_ORDER_DESC # 1 - Descending order
# Usage with ggml_argsort
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(3, 1, 4, 1, 5))
# Get ascending sort indices
idx_asc <- ggml_argsort(ctx, a, GGML_SORT_ORDER_ASC)
# Get descending sort indices
idx_desc <- ggml_argsort(ctx, a, GGML_SORT_ORDER_DESC)
ggml_free(ctx)
Split-axis codes for the meta backend
Description
Integer codes of ggml_backend_meta_split_axis, for a raw
split_fn passed to ggml_backend_meta_device.
ggml_meta_backend takes the names instead
("cols", "rows", "mirror", ...).
Usage
GGML_SPLIT_AXIS_0
GGML_SPLIT_AXIS_1
GGML_SPLIT_AXIS_2
GGML_SPLIT_AXIS_3
GGML_SPLIT_AXIS_MIRRORED
GGML_SPLIT_AXIS_PARTIAL
Format
Integer scalars.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
See Also
Other meta backend:
ggml_meta_backend(),
ggml_tp_forward()
GGML Data Types
Description
Constants representing different data types supported by GGML.
Usage
GGML_TYPE_F32
GGML_TYPE_F16
GGML_TYPE_Q4_0
GGML_TYPE_Q4_1
GGML_TYPE_Q8_0
GGML_TYPE_Q2_K
GGML_TYPE_Q3_K
GGML_TYPE_Q4_K
GGML_TYPE_Q5_K
GGML_TYPE_Q6_K
GGML_TYPE_I32
GGML_TYPE_BF16
GGML_TYPE_I64
GGML_TYPE_F64
Format
Integer constants
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
Details
-
GGML_TYPE_F32: 32-bit floating point (default) -
GGML_TYPE_F16: 16-bit floating point (half precision) -
GGML_TYPE_Q4_0: 4-bit quantization type 0 -
GGML_TYPE_Q4_1: 4-bit quantization type 1 -
GGML_TYPE_Q8_0: 8-bit quantization type 0 -
GGML_TYPE_I32: 32-bit integer -
GGML_TYPE_BF16: 16-bit brain float (bfloat16) -
GGML_TYPE_I64: 64-bit integer (held in R as double, exact for magnitudes up to 2^53) -
GGML_TYPE_F64: 64-bit floating point
Value
An integer constant representing a GGML data type
Examples
GGML_TYPE_F32
GGML_TYPE_F16
GGML_TYPE_I32
Run a GGML GPU operation on a Seurat object
Description
High-level, Seurat-style entry point: extracts the expression matrix from the
object, runs the requested operation on the GGML backend (Vulkan GPU with CPU
fallback) and writes the result back into the object. Returns the updated
object, so it slots into a %>% / |> pipeline next to
Seurat::RunPCA(). Methods exist for Seurat and
SingleCellExperiment objects (and a .default for bare matrices).
Usage
## S3 method for class 'SingleCellExperiment'
RunGGML(
object,
op = "embed",
assay = NULL,
layer = NULL,
n_components = 50L,
reduction_name = "ggml",
device = "auto",
genes = NULL,
cells = NULL,
reduction = NULL,
dims = NULL,
...
)
RunGGML(
object,
op = "embed",
assay = NULL,
layer = NULL,
n_components = 50L,
reduction_name = "ggml",
device = "auto",
genes = NULL,
cells = NULL,
reduction = NULL,
dims = NULL,
...
)
## Default S3 method:
RunGGML(
object,
op = "embed",
assay = NULL,
layer = NULL,
n_components = 50L,
reduction_name = "ggml",
device = "auto",
genes = NULL,
cells = NULL,
...
)
## S3 method for class 'Seurat'
RunGGML(
object,
op = "embed",
assay = NULL,
layer = NULL,
n_components = 50L,
reduction_name = "ggml",
device = "auto",
genes = NULL,
cells = NULL,
reduction = NULL,
dims = NULL,
...
)
Arguments
object |
A |
op |
Operation name; see |
assay |
Assay to read (Seurat); defaults to the object's default assay. |
layer |
Layer/slot to read; default |
n_components |
Number of components for |
reduction_name |
Name of the reduction slot to create. Default
|
device |
|
genes, cells |
Optional feature/cell subsets passed to extraction. |
reduction |
Optional name of an existing reduction to use as the input
feature space (e.g. |
dims |
Optional integer vector selecting columns of |
... |
Additional parameters forwarded to the engine. A notable one is
|
Details
Supported operations include "embed" (PCA), "normalize",
"scale", "umap", "neighbors" and "largest_gene"
(per-cell highest-expressed gene QC metric); see
ggml_ops_registry.
Value
For a Seurat object, the updated object with a new reduction. For a
bare matrix, a ggml_result.
Examples
## Not run:
library(Seurat)
pbmc <- RunGGML(pbmc, op = "embed", n_components = 30)
DimPlot(pbmc, reduction = "ggml")
## End(Not run)
Element-wise addition with broadcasting
Description
Computes A + B. If B is [m, 1] and A is
[m, n], B is broadcast across columns (useful for bias
vectors). Either operand may be a plain number; on the GPU that is one
ggml_scale_bias node with nothing uploaded, so it can be recorded by
ag_capture.
Usage
ag_add(A, B)
Arguments
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix (may be |
Value
ag_tensor
Column-wise argmax
Description
Index of the largest entry in every column of x, for action selection
over a [n_actions, batch] matrix of logits or log-probabilities.
Usage
ag_argmax(x, as_tensor = FALSE)
Arguments
x |
ag_tensor or numeric matrix, |
as_tensor |
|
Details
Ties go to the FIRST index on every path (R, CPU kernel, Vulkan shader),
matching which.max. NaN is never selected; a column with nothing
above -Inf (all NaN / all -Inf) gives 0.
Not differentiable. On the GPU the default reads the result back to the
host, so it is a synchronisation point: fine where the environment needs the
action, but keep it out of the training step. With as_tensor = TRUE
the indices stay on the device as a [1, batch] ag_tensor (whole
numbers stored as floats), which is what lets action selection be part of a
recorded graph (ag_capture) and feed ag_gather.
Value
Integer vector of length ncol(x), 0-based; or, with
as_tensor = TRUE, an ag_tensor [1, batch] holding the same.
Examples
ag_argmax(matrix(c(1, 3, 3, 5, 2, 5), 3, 2)) # 1 0
Create a Batch Normalisation layer
Description
Normalises each feature (row) over the batch dimension.
Learnable scale gamma [F,1] and shift beta [F,1].
Usage
ag_batch_norm(num_features, eps = 1e-05, momentum = 0.1)
Arguments
num_features |
Number of features (rows of input) |
eps |
Numerical stability constant (default 1e-5) |
momentum |
Running-stats momentum (default 0.1) |
Details
Training mode: use batch statistics; update running mean/var. Eval mode: use stored running statistics.
Value
An ag_batch_norm environment
Examples
bn <- ag_batch_norm(16L)
x <- ag_tensor(matrix(rnorm(16 * 32), 16, 32))
out <- bn$forward(x)
Device buffer for column-indexed reads
Description
A matrix of rows x capacity kept on the compute device without a
gradient: written in place by column ranges with ag_buffer_write()
and read by column indices with ag_get_rows() inside ag_*
computations, including ag_capture() recordings. Typical use
is a replay buffer: one column per transition.
Usage
ag_buffer(rows, capacity, dtype = "f32")
ag_buffer_write(buf, data, col_offset)
ag_buffer_read(buf, col_offset = 0, k = buf$capacity - col_offset)
ag_buffer_free(buf)
Arguments
rows |
Number of rows (values per column). |
capacity |
Number of columns. |
dtype |
Storage type. Only |
buf |
An |
data |
Numeric matrix with |
col_offset |
0-based index of the first column written or read. |
k |
Number of columns to read. |
Details
On the CPU the buffer is an R matrix and the same calls work on it. On the
GPU the memory is allocated when the buffer is created (or first used on the
device) and zero-filled; creation fails with an error when the buffer needs
more than 85% of the free memory the device reports (driver estimates are
approximate). A device switch (ag_device("cpu")) downloads the value
to the host in 16 MB parts before the backend is released (the host copy is
an R double matrix, about twice the buffer's device size), and the switch
fails with an error if that download does; the next use on the GPU uploads
it again, and recordings that read the buffer are then made anew.
Writes are synchronous: when ag_buffer_write() returns, the data is on
the device, and any graph computed afterwards, including a captured replay,
reads it. Graph nodes queued before the write are computed first, so they
read the old value.
ag_buffer_write() writes columns col_offset to
col_offset + ncol(data) - 1; the range must lie inside the buffer
(a ring buffer wraps with two writes).
ag_buffer_read() returns columns col_offset to
col_offset + k - 1 as a rows x k matrix; reading a large
buffer in parts bounds the host memory used.
ag_buffer_free() releases the memory at once instead of at
garbage collection; the buffer cannot be used afterwards.
Value
An object of class ag_buffer with fields rows,
capacity and dtype.
See Also
Examples
buf <- ag_buffer(3, 10)
ag_buffer_write(buf, matrix(1:6, 3, 2), col_offset = 4)
ag_buffer_read(buf, 4, 2)
as.matrix(ag_get_rows(buf, c(5, 4, 5)))
ag_buffer_free(buf)
Copy a device tensor into buffer columns
Description
Writes a ggml tensor that already lives on the GPU – for example the
observations of an environment computed in its own backend – into columns
of an ag_buffer, device to device: the data does not pass
through the host or R.
Usage
ag_buffer_copy_from(
buf,
tensor,
backend,
col_offset = 0,
via_host = c("error", "allow")
)
Arguments
buf |
An |
tensor |
A ggml tensor pointer, normally in device memory. |
backend |
The backend that owns |
col_offset |
0-based index of the first column written. |
via_host |
|
Details
The contract is strict and nothing is converted:
-
tensoris f32 and contiguous, with one buffer column per item along its trailing axes: 2-d[rows, k]in ggml order, or N-d whose leading axes make up a column – the buffer's row count isne0,ne0 * ne1orne0 * ne1 * ne2, and the remaining axes count the columns (a state[S, 6, n]fillsncolumns of a buffer with6 Srows). A tensor laid out[k, rows]must be transposed by its producer; the buffer does not. -
tensoris on the same device as the buffer, and the ag device is the GPU.
Otherwise the copy would have to go through the host: with ag on the CPU
(the buffer lives on the host), with tensor in host memory, or with
tensor on another device. By default (via_host = "error")
each of these is an error naming the reason, not a silent slow copy.
via_host = "allow" takes the host path instead – the tensor is
downloaded and written with ag_buffer_write(): same values, same
synchronous semantics. A message names the reason once per session for
each reason (silence it with suppressMessages()).
The write is synchronous, like ag_buffer_write(): queued ag nodes
are computed first (they read the old value), and when the call returns
the columns hold the new data. The buffer is not reallocated, so
recordings that read it stay valid.
Value
buf, invisibly.
See Also
Capture an ag_* computation as a reusable device graph
Description
Wraps fn so that, on the GPU, its ag_* operations are recorded once
per input shape and every later call runs the recorded ggml graph: upload the
inputs, one graph compute, download the outputs – no R-level op dispatch.
On the CPU the wrapper simply calls fn, so the same code runs on both.
Usage
ag_capture(fn, params = NULL)
Arguments
fn |
Function of named ag_tensor arguments returning an ag_tensor or a
named list of ag_tensors, or |
params |
Optional list of ag_tensors (typically the model parameters)
and |
Details
fn must depend on the data only through its arguments. Anything it
computes in R from data, or uploads itself (ag_tensor(f(x))), would be
frozen into the graph at the first call; such uploads, any download and any
eager launch during the recording are errors. Weights must be resident on
the device (e.g. trained or used on the GPU) before the first call.
The recording is redone when the input shapes, compute dtype or matmul
precision change, when the persistent device pool is reset (device switch),
and when a tensor in params no longer points at the device memory it
had at recording time.
Value
A function taking the same named arguments as numeric matrices (a
vector is one column) and returning a matrix, a named list of matrices,
or NULL invisibly (writes only: nothing is downloaded).
See Also
Examples
W <- ag_param(matrix(rnorm(6), 2, 3))
f <- ag_capture(function(x) ag_relu(ag_matmul(W, x)), params = list(W))
f(x = matrix(1, 3, 4))
Release the device memory held by a captured function
Description
Frees every recording made by ag_capture()'s function f. The
function stays usable: the next call records again. Recordings are also
released when the persistent device pool is reset (device switch).
Usage
ag_capture_free(f)
Arguments
f |
A function returned by |
Value
f, invisibly.
Run a captured function several times in one go
Description
Calls f, a function from ag_capture() that only writes
(ag_set_rows) and returns NULL, n times with
the same arguments. On the GPU the inputs are uploaded once and the
recorded graph is queued n times without waiting in between; R waits
once, at the end. Each run sees the device state the previous one left (a
counter, game boards): the backend orders the runs by the memory they
share. On the CPU f is simply called n times.
Usage
ag_capture_repeat(f, n, ...)
Arguments
f |
A function returned by |
n |
Number of runs, a positive whole number. |
... |
Named arguments of |
Details
On Vulkan the graph's command buffers are recorded once and submitted
n times in one go, so a run costs no command recording on the CPU;
the recording is made again after a device buffer it uses was freed (its
handle may be reused). Where that is not possible (no push descriptors) each run
is recorded and queued as usual. GGMLR_VK_GRAPH_RECORD=0 turns the
recorded path off.
Value
NULL, invisibly.
See Also
Change a setting of a captured training step
Description
Settings are inputs of the recording, read on every call, so changing one does not record again.
Usage
ag_capture_set(f, max_grad_norm)
Arguments
f |
A function returned by |
max_grad_norm |
A positive number: the new gradient-clipping limit. The
step must have been created with a |
Value
f, invisibly.
See Also
Capture a whole training step as reusable device graphs
Description
Like ag_capture, for a training step: the forward pass in
fn, its backward pass and the Adam update are recorded once, and every
later call is upload the inputs, two graph computes, download the outputs –
no R-level op dispatch. On the CPU the wrapper runs the same step as ordinary
code (zero_grad, tape, backward, step).
Usage
ag_capture_step(fn, params = NULL, optimizer, max_grad_norm = NULL)
Arguments
fn |
Function of named ag_tensor arguments returning the loss (an
ag_tensor), or a named list of ag_tensors with an element |
params |
Optional list of further ag_tensors the recording references
(e.g. frozen weights) and |
optimizer |
An optimizer from |
max_grad_norm |
|
Details
Two graphs, not one: forward with backward, then the update. Adam writes each weight in place, and a weight is also read inside the first graph (by the forward, and by the backward of the layer below) through tensors that share its memory but are not connected to the update in the graph – so a single graph would leave the order of that read and that write to the backend.
fn is called WITHOUT with_grad_tape: the wrapper records the
tape itself. Everything ag_capture requires of fn applies: data
only through the arguments, no reading values back, weights on the device.
The backward must be one that runs as a graph (the ops the graph backward
supports).
Input names and shapes are fixed by the first call; a call with other shapes
is an error (pad a short last minibatch, or run it uncaptured). The step
counter and the learning rate are read from optimizer on every call,
so learning-rate schedules work, and optimizer$state() stays valid.
Outputs are the values of the forward pass, before the update, as in an
ordinary training loop. After a captured step $grad is not set.
Supported: optimizer_adam without gradient accumulation, F32
compute dtype.
Gradient clipping by global norm (max_grad_norm): the norm of all
gradients is computed in the first graph and every gradient is multiplied by
min(1, max_grad_norm / (norm + 1e-6)) before the update. Branch-free,
unlike clip_grad_norm: for a norm within about 1e-6 below
max_grad_norm the factor is slightly under 1 instead of exactly 1.
The pre-clipping norm is returned as an extra output grad_norm. A
non-finite norm is an error raised BEFORE the update: weights, moments and
the step counter are left as they were. The moments accumulate clipped
gradients, so optimizer state saved with and without clipping is not
interchangeable on resume. Change the limit between calls with
ag_capture_set (no re-recording).
Value
A function taking the same named arguments as numeric matrices and returning the loss as a matrix, or a named list of matrices.
See Also
ag_capture, ag_capture_set,
ag_capture_free
Examples
W <- ag_param(matrix(rnorm(6, sd = 0.1), 2, 3))
opt <- optimizer_adam(list(W), lr = 1e-2)
train <- ag_capture_step(function(x, y) ag_mse_loss(ag_matmul(W, x), y),
optimizer = opt)
for (i in 1:5) loss <- train(x = matrix(1, 3, 4), y = matrix(0, 2, 4))
Run part of the forward pass without storing its activations
Description
Executes fn(...) with the gradient tape switched off, so none of the
intermediate activations inside fn are kept. During
backward() the segment is run again – with the same RNG state
– to rebuild what the gradient rules need.
Usage
ag_checkpoint(fn, ...)
Arguments
fn |
A function taking the tensors in One caveat, because it bites in the most natural usage: |
... |
Input |
Details
This trades compute for memory: the segment's forward pass runs twice, and in exchange the tape never holds its activations. Use it on the deep middle of a network, not on cheap layers.
Value
An ag_tensor holding the segment's output. Its gradient flows
back into both ... and any parameters used inside fn.
Examples
block <- ag_linear(8L, 8L, activation = "relu")
x <- ag_tensor(matrix(runif(8 * 4), 8, 4))
y <- ag_tensor(matrix(runif(8 * 4), 8, 4))
with_grad_tape({
h <- ag_checkpoint(function(inp) block$forward(inp), x)
loss <- ag_mse_loss(h, y)
})
backward(loss)
# Checkpointing every second block of a deeper stack. local() gives each
# segment its own binding -- see the note on `fn` for why that matters.
layers <- lapply(1:4, function(i) ag_linear(8L, 8L, activation = "relu"))
with_grad_tape({
h <- x
for (i in seq_along(layers)) {
seg <- local({
lyr <- layers[[i]]
function(inp) lyr$forward(inp)
})
h <- if (i %% 2L == 0L) ag_checkpoint(seg, h) else seg(h)
}
loss <- ag_mse_loss(h, y)
})
backward(loss)
Element-wise clamp
Description
Clamps values to [lo, hi]. Gradient is 1 inside the interval,
0 at the boundary (straight-through estimator).
Usage
ag_clamp(x, lo = -Inf, hi = Inf)
Arguments
x |
ag_tensor |
lo |
Lower bound (default |
hi |
Upper bound (default |
Value
ag_tensor
Concatenate two tensors by rows
Description
Stacks a above b, like rbind(a, b): both must have the
same number of columns (the batch). Typical use is a critic input built from
observations and an action computed by the actor. The gradient of a
is rows 1..nrow(a) of the incoming gradient and that of b the
rest; an operand without gradient gets none.
Usage
ag_concat_rows(a, b)
Arguments
a, b |
ag_tensors (or matrices) with the same number of columns |
Value
ag_tensor of shape (nrow(a) + nrow(b)) x ncol(a)
See Also
Examples
a <- ag_tensor(matrix(1:6, 2, 3)); b <- ag_tensor(matrix(7:9, 1, 3))
as.matrix(ag_concat_rows(a, b))
2-D convolution
Description
Convolves a batch of images stored as columns. x is
[C*H*W, B], each column an image flattened with W fastest, then H,
then C (as.vector(array(img, c(W, H, C)))); the result is
[OC*OH*OW, B] in the same order, ready for ag_linear.
The output size per axis is
\lfloor (n + 2p - d(k - 1) - 1) / s \rfloor + 1.
Usage
ag_conv2d(
x,
weight,
bias = NULL,
in_shape,
kernel_size,
stride = 1L,
padding = 0L,
dilation = 1L
)
Arguments
x |
ag_tensor (or matrix) |
weight |
ag_tensor |
bias |
|
in_shape |
|
kernel_size |
|
stride, padding, dilation |
|
Details
The weight is [OC, C*KH*KW], its columns in the row-major order of
a PyTorch kernel [OC, C, KH, KW]: a torch weight w (an R
array [OC, C, KH, KW]) becomes matrix(aperm(w, c(1, 4, 3, 2)), OC).
On the GPU the convolution is one graph (im2col, a matmul and a permute, all F32; an f16/bf16 operand is cast first). On the CPU it is computed in R by an index gather: a reference path for checking, not for speed. Scaling the input (e.g. dividing pixels by 255) is the caller's.
Value
ag_tensor [OC*OH*OW, B].
See Also
Examples
x <- ag_tensor(matrix(runif(3 * 8 * 8 * 2), 3 * 8 * 8, 2)) # 2 RGB 8x8 images
w <- ag_param(matrix(rnorm(4 * 3 * 3 * 3, sd = 0.1), 4)) # 4 filters 3x3
y <- ag_conv2d(x, w, in_shape = c(3, 8, 8), kernel_size = 3, padding = 1)
dim(as.matrix(y)) # 4*8*8 x 2
Convolution layer
Description
A module around ag_conv2d for a fixed image size: the output
size is computed once and the layer refuses inputs of another size.
Usage
ag_conv2d_layer(
in_channels,
out_channels,
kernel_size,
in_hw,
stride = 1L,
padding = 0L,
dilation = 1L,
bias = TRUE,
init = c("orthogonal", "torch"),
gain = sqrt(2)
)
Arguments
in_channels, out_channels |
Channels in and out. |
kernel_size, stride, padding, dilation |
See |
in_hw |
|
bias |
Use a per-channel bias (default |
init |
|
gain |
Scale of the orthogonal initialisation (ignored by |
Details
The backward runs as a graph on the GPU (im2col is recomputed from
the input instead of being kept from the forward), so a training step with
convolutions can be recorded by ag_capture_step. There, as for
every input, the batch size is fixed by the first call: another size is an
error, not a re-recording (pad a short last minibatch).
init:
"orthogonal"(default)the weight
[OC, C*KH*KW]orthogonal (as CleanRL / SB3 for Atari) timesgain(sqrt(2)for ReLU), bias 0."torch"weight and bias
U(-1/\sqrt{fan_{in}}, 1/\sqrt{fan_{in}}),fan_{in} = C \cdot KH \cdot KW, as PyTorch's default (kaiming_uniform_(a = sqrt(5))).
Value
An ag_conv2d_layer environment with forward(x),
parameters() (weight, bias), in_shape
c(C, H, W), out_shape c(OC, OH, OW) and
out_features (OC*OH*OW, the input size of the next linear
layer).
Examples
# the first layer of the Nature-DQN network on 4 stacked 84x84 frames
conv <- ag_conv2d_layer(4L, 32L, kernel_size = 8L, in_hw = c(84L, 84L), stride = 4L)
conv$out_shape # 32 20 20
2-D transposed convolution
Description
The transposed ("fractionally strided") convolution of PyTorch's
nn.ConvTranspose2d: the input gradient of ag_conv2d,
used to upsample in decoders. Images are columns as in
ag_conv2d: x is [C*H*W, B], each column W
fastest, then H, then C; the result is [OC*OH*OW, B] in the same
order. The output size per axis is PyTorch's
(n - 1) s - 2p + d (k - 1) + output\_padding + 1
so kernel 4, stride 2, padding 1 (or kernel 3, stride 2, padding 1, output_padding 1) doubles H and W.
Usage
ag_conv_transpose2d(
x,
weight,
bias = NULL,
in_shape,
kernel_size,
stride = 1L,
padding = 0L,
output_padding = 0L,
dilation = 1L,
groups = 1L
)
Arguments
x |
ag_tensor (or matrix) |
weight |
ag_tensor |
bias |
|
in_shape |
|
kernel_size |
|
stride, padding, dilation |
|
output_padding |
|
groups |
Must be 1. |
Details
The weight is [C, OC*KH*KW], the layout of a PyTorch
ConvTranspose2d weight [C, OC, KH, KW] with its columns in
row-major order: a torch weight w (an R array [C, OC, KH, KW])
becomes matrix(aperm(w, c(1, 4, 3, 2)), C). The number of output
channels is ncol(weight) / (KH*KW).
Differences from PyTorch, deliberate: groups must be 1 (grouped
transposed convolutions are an error), and output_padding must be
smaller than stride (PyTorch also allows values below
dilation).
On the GPU the forward is one graph (a matmul and im2col_back) and
the backward runs in the graph backward, so the op can be part of an
ag_capture_step recording. On the CPU it is computed in R: a
reference path for checking, not for speed.
Value
ag_tensor [OC*OH*OW, B].
See Also
ag_conv_transpose2d_layer, ag_conv2d
Examples
x <- ag_tensor(matrix(runif(8 * 4 * 4 * 2), 8 * 4 * 4, 2)) # 2 images 8x4x4
w <- ag_param(matrix(rnorm(8 * 3 * 4 * 4, sd = 0.1), 8)) # 8 -> 3 channels, 4x4
y <- ag_conv_transpose2d(x, w, in_shape = c(8, 4, 4), kernel_size = 4,
stride = 2, padding = 1)
dim(as.matrix(y)) # 3*8*8 x 2
Transposed convolution layer
Description
A module around ag_conv_transpose2d for a fixed image size,
the counterpart of ag_conv2d_layer.
Usage
ag_conv_transpose2d_layer(
in_channels,
out_channels,
kernel_size,
in_hw,
stride = 1L,
padding = 0L,
output_padding = 0L,
dilation = 1L,
bias = TRUE,
init = c("orthogonal", "torch"),
gain = 1
)
Arguments
in_channels, out_channels |
Channels in and out. |
kernel_size, stride, padding, output_padding, dilation |
See
|
in_hw |
|
bias |
Use a per-channel bias (default |
init |
|
gain |
Scale of the orthogonal initialisation (ignored by |
Details
init:
"orthogonal"(default)the weight
[C, OC*KH*KW]orthogonal timesgain, bias 0."torch"weight and bias
U(-1/\sqrt{fan_{in}}, 1/\sqrt{fan_{in}})with PyTorch'sfan_{in}forConvTranspose2d, taken from the SECOND axis of its weight[C, OC, KH, KW]:fan_{in} = OC \cdot KH \cdot KW.
Value
An ag_conv_transpose2d_layer environment with
forward(x), parameters() (weight, bias),
in_shape c(C, H, W), out_shape c(OC, OH, OW)
and out_features (OC*OH*OW).
Examples
# a decoder step: 32 x 8 x 8 -> 16 x 16 x 16
up <- ag_conv_transpose2d_layer(32L, 16L, kernel_size = 4L, in_hw = c(8L, 8L),
stride = 2L, padding = 1L)
up$out_shape # 16 16 16
Categorical Cross-Entropy loss
Description
Generic CE: -sum(target * log(pred)) / batch_size.
The gradient w.r.t. pred is -target / pred / n.
Use ag_softmax_cross_entropy_loss() for the numerically stable
combined softmax + CE (fused gradient (p - y) / n).
Usage
ag_cross_entropy_loss(pred, target)
Arguments
pred |
ag_tensor [classes, batch_size] probabilities (any, not just softmax) |
target |
matrix [classes, batch_size] one-hot (or soft) labels |
Value
scalar ag_tensor
Create a mini-batch data loader
Description
Returns an iterator environment. Each call to $next_batch() returns
a named list list(x, y) with ag_tensor objects of shape
[features, batch_size] / [labels, batch_size].
After the last batch, $has_next() returns FALSE; call
$reset() (or start a new epoch via $epoch()) to reshuffle
and restart.
Usage
ag_dataloader(x, y = NULL, batch_size = 32L, shuffle = TRUE, col_major = TRUE)
Arguments
x |
Feature matrix |
y |
Label matrix with the same convention. |
batch_size |
Integer batch size. |
shuffle |
Logical; if |
col_major |
Logical; if |
Value
An ag_dataloader environment
Examples
n <- 128L
x <- matrix(runif(4 * n), 4, n) # [4, 128] col-major
y <- matrix(runif(2 * n), 2, n)
dl <- ag_dataloader(x, y, batch_size = 32L)
dl$reset()
while (dl$has_next()) {
batch <- dl$next_batch()
# batch$x: [4, 32], batch$y: [2, 32]
}
Return the current default compute device
Description
Return the current default compute device
Usage
ag_default_device()
Value
"cpu" or "gpu"
Return the current default dtype for GPU operations
Description
Return the current default dtype for GPU operations
Usage
ag_default_dtype()
Value
"f32", "f16", or "bf16"
Stop gradient
Description
Returns the value of x as a tensor that is not recorded on the tape:
nothing flows back through it. Use it for TD targets, advantages, or the
old-policy term of a KL penalty.
Usage
ag_detach(x)
Arguments
x |
ag_tensor or numeric matrix |
Details
When x is resident on the device the result shares its buffer (no
download, no copy), as detach() does in PyTorch. A later in-place
update of x – an optimizer step on a parameter – is therefore
visible through the result; copy the weights instead when a value must be
frozen across steps (e.g. a target network).
Value
ag_tensor with requires_grad = FALSE
Examples
W <- ag_param(matrix(1:4, 2, 2))
ag_detach(W)$requires_grad
Set the default compute device for ag_* operations
Description
Switches all subsequent ag_tensor / ag_param operations to run
on the specified device. Calling ag_device("gpu") initialises the
best available ggml backend (Vulkan, Metal, CUDA, or CPU fallback) the first
time it is called.
Usage
ag_device(device)
Arguments
device |
|
Details
Switching to the CPU moves every ag_buffer on the GPU
to the host first; plan for host memory of about twice the buffers' device
size (R stores the values as doubles).
Value
Invisibly the previous device string
Element-wise division
Description
A / B. A numeric scalar B becomes ag_scale(A, 1 / B);
otherwise A * B^-1, broadcasting B as ag_mul does.
Usage
ag_div(A, B)
Arguments
A |
ag_tensor or numeric matrix |
B |
ag_tensor, numeric matrix, or numeric scalar |
Value
ag_tensor
Examples
as.matrix(ag_div(ag_tensor(matrix(1:4, 2, 2)), 2))
Create a Dropout layer
Description
In training mode applies inverted dropout (random Bernoulli mask, scale by
1/(1-rate) to preserve expected values). In eval mode is identity.
Usage
ag_dropout(rate)
Arguments
rate |
Drop probability in [0, 1) |
Value
An ag_dropout environment
Examples
drop <- ag_dropout(0.5)
x <- ag_tensor(matrix(runif(8), 4, 2))
out <- drop$forward(x) # training mode by default
ag_eval(drop)
out2 <- drop$forward(x) # identity
Set the default floating-point precision for ag_* GPU operations
Description
Controls the dtype used when uploading tensors to the ggml backend.
"bf16" halves memory usage vs "f32" with minimal accuracy loss.
Gradients are never computed in f16: the GPU graph backward runs in f32 only,
so with "f16" or "bf16" backward() takes the closure path,
which computes them in double – correct, but slower (the reason is counted
by ggmlR:::ag_backward_fallbacks()). Training in f32 keeps the graph path.
Usage
ag_dtype(dtype)
Arguments
dtype |
|
Value
Invisibly the previous dtype string
Create an Embedding layer
Description
Maps 0-based integer indices to dense vectors via table lookup.
Input: integer matrix or vector of 0-based indices.
Output: float tensor [dim, length(idx)].
Usage
ag_embedding(vocab_size, dim)
Arguments
vocab_size |
Vocabulary size |
dim |
Embedding dimension |
Details
Backward: scatter-add — only the looked-up rows accumulate gradient.
Value
An ag_embedding environment
Examples
emb <- ag_embedding(100L, 16L)
idx <- c(0L, 3L, 7L, 2L)
out <- emb$forward(idx) # [16, 4]
Entropy of a categorical distribution given by logits
Description
-sum(p * log p) down every column, computed as
-colSums(exp(logp) * logp) with logp = ag_log_softmax(logits):
masked entries (a large FINITE negative logit such as -1e4) give
0 * finite, never 0 * -Inf. -Inf logits are not
supported (see ag_log_softmax).
Usage
ag_entropy(logits, reduction = c("none", "mean", "sum"))
Arguments
logits |
ag_tensor or numeric matrix |
reduction |
|
Value
ag_tensor [1, batch], or scalar
Examples
as.matrix(ag_entropy(ag_tensor(matrix(0, 4, 2)))) # log(4)
Estimate the memory a training run needs
Description
Adds up the four things that are alive at the same time during training on
the autograd (ag_*) path: the weights, their gradients, the
optimizer's per-parameter state, and the activations held by the tape until
zero_grad() clears it.
Usage
ag_estimate_training_memory(
shapes,
batch_size = 32L,
optimizer = "adam",
activation_frac = 0.01,
quiet = FALSE
)
Arguments
shapes |
List of parameter shapes, each a numeric vector of dimensions
(e.g. |
batch_size |
Batch size the activations are sized for. Activations scale with it; weights and optimizer state do not. |
optimizer |
An optimizer object (counted by the state slots it holds)
or one of |
activation_frac |
Activations as a fraction of the weight bytes, per
unit of batch. The default 0.01 reproduces the measured tape composition
for a dense stack ( |
quiet |
Suppress the printed report and return the figures only. Note
that the double-vs-f32 caveat is part of that report, so a caller passing
|
Details
Everything is counted at 8 bytes per scalar, because that is what the path
actually uses: gradients are accumulated by R closures in double, and
optimizer moments are R matrices. ag_dtype affects only
forward tensors uploaded into a ggml buffer and does not reach either. An
estimate made in f32 would be half the real figure, so the returned list
carries a note saying so.
Value
Invisibly, a list of byte counts: params, weights,
gradients, optimizer, activations, total, plus
bytes_per_scalar and note. Printed as a report.
See Also
ag_tape_memory measures a real tape instead of
estimating one; ggml_estimate_memory sizes a single tensor in
a ggml buffer, where dtype does apply.
Examples
# a 3-layer MLP, 512 wide, trained with Adam at batch 64
ag_estimate_training_memory(
shapes = list(c(512, 512), c(512, 512), c(512, 512)),
batch_size = 64)
Switch a layer or sequential model to eval mode
Description
Switch a layer or sequential model to eval mode
Usage
ag_eval(model)
Arguments
model |
An ag_sequential, ag_batch_norm, or ag_dropout layer |
Value
The model/layer (invisibly)
Multi-head attention in a single fused operation
Description
Computes scaled dot-product attention over all heads at once with
ggml_flash_attn_ext(), instead of the per-head loop
ag_multihead_attention uses. Both the forward pass and its
gradient are one operation each, so the tape holds a single node rather than
the dozens a head loop records.
Usage
ag_flash_attention(
q,
k,
v,
n_heads,
scale = NULL,
mask = NULL,
causal = FALSE,
batch = 1L
)
Arguments
q, k, v |
|
n_heads |
Number of attention heads. Must divide |
scale |
Softmax scale. Defaults to |
mask |
Optional attention mask, |
causal |
|
batch |
Number of independent sequences packed side by side in the
columns (default 1). With |
Details
q, k and v are [d_model, seq] matrices whose rows
are split into n_heads contiguous blocks – the same head layout
ag_multihead_attention() uses. Projections are not included: apply
W_q, W_k, W_v before the call and W_o after it.
Value
An ag_tensor of shape [d_model, seq_q]
([d_model, batch * seq_q] when batch > 1).
Examples
d_model <- 16L; seq_len <- 8L
q <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len))
k <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len))
v <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len))
with_grad_tape({
out <- ag_flash_attention(q, k, v, n_heads = 4L)
loss <- ag_mse_loss(out, ag_tensor(matrix(0, d_model, seq_len)))
})
backward(loss)
# Four sequences of length 8 in one call
qb <- ag_param(matrix(runif(d_model * 4 * seq_len, -1, 1), d_model, 4 * seq_len))
ob <- ag_flash_attention(qb, qb, qb, n_heads = 4L, batch = 4L)
Generalised advantage estimation and discounted returns
Description
ag_gae computes GAE(\lambda) (Schulman et al. 2016) backwards
over a rollout:
\delta_t = r_t + \gamma V_{next} - V(s_t), \quad
A_t = \delta_t + \gamma \lambda (1 - done_t) A_{t+1}
and returns = advantages + values, as in Stable-Baselines3 and
CleanRL. ag_discounted_returns is the bootstrapped Monte-Carlo
return G_t = r_t + \gamma G_{next}; it equals ag_gae with
lambda = 1 plus values.
Usage
ag_gae(
rewards,
values,
terminated = NULL,
truncated = NULL,
next_values = NULL,
last_value = 0,
gamma = 0.99,
lambda = 0.95,
turn_sign = 1,
continues = NULL
)
ag_discounted_returns(
rewards,
terminated,
truncated = NULL,
next_values = NULL,
last_value = 0,
gamma = 0.99,
turn_sign = 1
)
Arguments
rewards |
Numeric |
values |
Numeric |
terminated |
Logical or 0/1 |
truncated |
Logical or 0/1 |
next_values |
Numeric |
last_value |
Value of the state after the rollout, one per environment (default 0). |
gamma |
Discount factor in |
lambda |
GAE |
turn_sign |
1 (default) or -1, one number or one per environment. |
continues |
|
Details
V_{next} after step t is
0 when
terminated[t](true terminal state);-
next_values[t]whentruncated[t]: the value of the FINAL observation of the cut episode – not of the first observation after an automatic reset, which belongs to the next episode (Gymnasium'sfinal_obs/ SB3'sterminal_observation); -
values[t + 1]inside an episode, andlast_valueafter the last step of the rollout (not used when that step ended an episode).
The recursion is cut at every episode end. A step both terminated and truncated counts as terminated.
Without truncated, terminated plays the role of CleanRL's
dones: a time-limit cut is then treated as a terminal state, which
biases the values of states near the limit downwards.
turn_sign is for two-player self-play, where the next state is seen
by the other player: V_{next} (including next_values and
last_value) and the carried A_{t+1} / G_{t+1} are
multiplied by it. Use -1 for alternating zero-sum moves, 1 (default)
otherwise.
Value
ag_gae: list of advantages and returns, both
[T, n_envs]. ag_discounted_returns: the [T, n_envs]
matrix of returns.
See Also
Examples
r <- matrix(1, 4, 2); v <- matrix(0.5, 4, 2)
term <- matrix(c(FALSE, TRUE, FALSE, FALSE), 4, 2)
ag_gae(r, v, term, last_value = c(0.5, 0.5), gamma = 0.99, lambda = 0.95)$advantages
Pick one entry per column
Description
out[1, j] = x[index[j] + 1, j]: the log-probability of the action
taken in sample j, for example. Differentiable in x; the gradient of
column j lands on the picked row only.
Usage
ag_gather(x, index)
Arguments
x |
ag_tensor or numeric matrix, |
index |
Integer-valued vector of length |
Details
Built as colSums(x * onehot(index)), so the graph backward covers it
with no rule of its own. The one-hot is [n, batch] and is uploaded on
every call: cheap for tens of classes (games), wasteful for thousands –
there a dedicated op with an I32 index tensor is the right tool (TODO.md).
index may also be a [1, batch] ag_tensor, as returned by
ag_argmax(as_tensor = TRUE) or ag_sample_categorical(key = ).
With x on the GPU nothing then touches the host: the pick is a
get_rows over the flattened x, so it can be recorded by
ag_capture. That path is for collection and is not
differentiable, and the index is not range-checked (it never leaves the
device); pass a host vector in the training step.
Value
ag_tensor [1, batch].
Examples
ag_gather(matrix(1:6 + 0, 3, 2), c(2L, 0L)) # 3 4
Read buffer columns by index
Description
Selects columns of an ag_buffer by 0-based index, as a device
operation: on the GPU only the indices cross the bus. The name follows
ggml_get_rows, which selects along the second ggml dimension (ne1);
in the R rows x capacity view that is a column.
Usage
ag_get_rows(buf, idx)
Arguments
buf |
An |
idx |
0-based column indices: a numeric vector, or an ag_tensor with
one column (f32; indices are exact below |
Details
The result is a rows x length(idx) ag_tensor without gradient, usable
as an input to further ag_* operations. Inside ag_capture()
pass idx as an argument of the captured function: the recording is
then reused for every index vector of the same length (the batch size is
part of the recording's shape), and the buffer must be listed in
params so that freeing it invalidates the recording.
Indices given as host values are checked against the buffer and an
index out of range is an error. An index tensor already on the device
(inside a recording) cannot be checked; on the device every index is
clamped to 0..capacity-1 before the read, so an index out of range
reads the first or last column instead of memory outside the buffer.
Value
An ag_tensor of shape rows x length(idx).
Gradient of a tensor as an R matrix
Description
After backward(), x$grad holds the gradient – as an R matrix
on the closure path, but as a device handle when the graph backward kept it
on the GPU (the default there). ag_grad() returns the value as a
matrix either way, so code that reads gradients does not depend on where
they live. In graph mode it is a synchronisation point.
Usage
ag_grad(x)
Arguments
x |
An ag_tensor (usually a parameter). |
Value
A numeric matrix, or NULL when x has no gradient.
Examples
w <- ag_param(matrix(1:4 + 0, 2, 2))
with_grad_tape({ loss <- ag_sum(ag_mul(w, w)) })
backward(loss)
ag_grad(w) # 2 * w
Numerical gradient check (like torch.autograd.gradcheck)
Description
Compares analytical gradients (from backward()) with finite-difference
numerical gradients for all input tensors with requires_grad = TRUE.
Usage
ag_gradcheck(
fn,
inputs,
eps = 1e-05,
atol = 1e-04,
verbose = FALSE,
quiet = FALSE
)
Arguments
fn |
A function that takes a list of ag_tensor inputs and returns a
scalar ag_tensor loss (must be used inside |
inputs |
Named list of ag_tensor objects. Only those with
|
eps |
Finite-difference step size (default 1e-5). |
atol |
Absolute tolerance for pass/fail (default 1e-4). |
verbose |
Print per-element comparison (default FALSE). |
quiet |
Suppress per-parameter and overall status lines (default FALSE).
Useful when calling from |
Value
Invisibly TRUE if all gradients match, FALSE otherwise.
When quiet = FALSE (default), prints a summary report.
Examples
W <- ag_param(matrix(runif(6), 2, 3))
x <- ag_tensor(matrix(runif(3), 3, 1))
ag_gradcheck(
fn = function(ins) ag_mse_loss(ag_relu(ag_matmul(ins$W, ins$x)),
matrix(0, 2, 1)),
inputs = list(W = W, x = x)
)
Graph mode for ag_* operations on the GPU
Description
With graph mode on, a GPU ag_* operation that has a device-resident
path does not run at once: it adds its node to a queue and returns a pending
value. The queue runs as ONE ggml graph when a value is read on the host
(as.matrix(), a printed loss, an action index) or when
backward() needs it – the graph backward then joins the same graph.
Each separate launch costs on the order of a millisecond on the device, so a
small network (an RL policy, a few layers) is dominated by launches, and
fusing them is where the speed comes from.
Usage
ag_graph_mode(on, check = NULL)
Arguments
on |
|
check |
|
Details
Results are the same as without it. Operations that have no resident path yet still run one by one; they first run whatever is queued, so the order of computation is always the data order.
When the queue runs (synchronisation points):
reading a value that is still queued:
as.matrix(), printing, a loss value,ag_argmax();an operation without a resident path (it runs the queue first);
writing a parameter: an optimizer step,
ag_load_state_dict();the device optimizer step (it never joins the queue);
turning graph mode off.
backward() does NOT force it: the graph backward appends to the
queue, so forward and backward run as one graph at the next point above.
Off by default; turn it on around the code that wants it, restoring the
previous value on every exit – ag_local_mode does that. The one
thing it can break is code that orders graphs BY HAND rather than through
data – reading a buffer in one graph and overwriting it in a later one with
nothing linking the two. The package's own optimizer step does this and is
protected; code of your own that does it must run with graph mode off.
On the CPU it has no effect.
Errors. A failure inside a queued operation surfaces where the queue
runs, not where the operation was called. The error names the queued
operations; check = TRUE (or GGMLR_AG_GRAPH_CHECK=1) runs each
operation as soon as it is queued, so the error appears at its own call –
for debugging, since it gives up the fusion.
Value
The current setting when on is missing; otherwise the previous
setting, invisibly. Turning it off runs anything still queued.
See Also
ag_local_mode, ag_matmul_precision
Examples
old <- ag_graph_mode(TRUE)
ag_graph_mode()
ag_graph_mode(old)
Huber and smooth-L1 losses
Description
Element-wise, with d = |pred - target| and q = min(d, delta):
huber = 0.5 q^2 + delta (d - q)
i.e. 0.5 d^2 for d <= delta and delta (d - 0.5 delta)
beyond. ag_smooth_l1_loss is huber(delta = beta) / beta (the
Fast R-CNN form, slope 1 in the linear part); beta = 0 is plain L1.
Usage
ag_huber_loss(pred, target, delta = 1, reduction = c("mean", "sum", "none"))
ag_smooth_l1_loss(pred, target, beta = 1, reduction = c("mean", "sum", "none"))
Arguments
pred |
ag_tensor |
target |
ag_tensor or numeric matrix, same shape |
delta |
Positive threshold (Huber) |
reduction |
|
beta |
Non-negative threshold (smooth-L1); 0 gives L1 |
Details
The gradient is clip(pred - target, -delta, delta) for Huber, also at
d = delta (the loss is C1 there) and 0 at d = 0.
Value
ag_tensor: scalar, or [n, batch] for "none"
Examples
as.matrix(ag_huber_loss(ag_tensor(matrix(c(0.5, 3), 1, 2)), matrix(0, 1, 2),
reduction = "none"))
Create a Layer Normalization layer
Description
Normalizes each example independently over its normalized_shape
features, then applies a learnable per-feature scale (gamma) and shift
(beta): y = gamma * (x - mu) / sqrt(var + eps) + beta.
Usage
ag_layer_norm(normalized_shape, eps = 1e-05, elementwise_affine = TRUE)
Arguments
normalized_shape |
Number of features to normalize over (the number of rows of the input). |
eps |
Small constant added to the variance for numerical stability (default 1e-5). |
elementwise_affine |
Logical; when |
Details
Unlike ag_batch_norm, which normalizes each feature across the
batch, LayerNorm normalizes each column (example) across its features. It
therefore has no running statistics and behaves identically in training and
evaluation mode, which makes it the usual choice for transformers and for
small or varying batch sizes.
The backward pass is exact: because mu and sigma are computed
from the same example being normalized, their contribution to the gradient is
included rather than treated as constant. The normalization is a single tape
node (ggml_norm / ggml_norm_back); gamma and
beta are an ordinary ag_mul and ag_add.
On the GPU all three stay on the device and run in the graph backward, so
the layer can be part of an ag_capture_step recording.
Value
An ag_layer_norm environment with $forward() and
$parameters().
See Also
Examples
ln <- ag_layer_norm(4L)
x <- ag_param(matrix(rnorm(8), 4, 2)) # [features, batch]
with_grad_tape({
y <- ln$forward(x)
loss <- ag_mse_loss(y, matrix(0, 4, 2))
})
grads <- backward(loss)
Create a dense layer with learnable parameters
Description
Returns a closure-based layer. Because ag_param uses environment semantics, the optimizer updates W and b in-place, and forward() always uses the latest weights.
Usage
ag_linear(
in_features,
out_features,
activation = NULL,
init = "glorot_uniform",
gain = 1
)
Arguments
in_features |
Input dimension |
out_features |
Output dimension |
activation |
"relu", "sigmoid", "tanh", "softmax", or NULL |
init |
Weight initialisation,
The bias always starts at zero. |
gain |
Multiplier for the string initialisations (e.g. |
Value
List with W, b, forward(x), parameters()
(and params(), the same function, kept for compatibility)
Examples
layer <- ag_linear(4L, 8L, activation = "relu")
x <- ag_tensor(matrix(runif(4 * 16), 4, 16))
out <- layer$forward(x)
head <- ag_linear(8L, 2L, init = "orthogonal", gain = 0.01)
Load an autograd module from a saved state
Description
Reconstructs an ag_* module saved with ag_save_model.
The architecture is rebuilt by calling model_fn (either the one passed
here, or the one stored inside the container at save time), and the saved
parameter and buffer values are copied back by name.
Usage
ag_load_model(path, model_fn = NULL, device = NULL, dtype = NULL)
Arguments
path |
File path written by |
model_fn |
Optional zero-argument rebuild function. Required if no
|
device |
Optional device for the rebuilt module ( |
dtype |
Optional GPU upload precision for the restored parameters
( Note that this is an upload/compute precision, not a storage
precision: parameter values always live in |
Value
The reconstructed module with restored weights, in eval mode.
See Also
Examples
build <- function() ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 3L))
f <- tempfile(fileext = ".rds")
ag_save_model(build(), f, model_fn = build)
model <- ag_load_model(f)
Load weights into an existing model
Description
Writes a state from ag_state_dict (or the container of an
ag_save_model file read with readRDS) into model
in place. Device-resident weights are overwritten in their own buffers, so
an optimizer built on the model stays valid and keeps its residency. Values
are converted to the model's dtype on upload.
Usage
ag_load_state_dict(model, state, strict = TRUE)
Arguments
model |
Target model (see |
state |
A state dict. |
strict |
|
Details
Everything is checked before anything is written: an error leaves the model untouched.
Value
model, invisibly.
Set ag_* GPU modes for the rest of the calling function
Description
Sets graph mode (ag_graph_mode) and/or matrix-multiply
precision (ag_matmul_precision) and restores the previous
values when the calling function exits – normally or by an error. One place
for both, instead of a pair of on.exit calls per setting.
Usage
ag_local_mode(graph = NULL, matmul_precision = NULL, envir = parent.frame())
Arguments
graph |
|
matmul_precision |
|
envir |
The frame whose exit restores the settings (the caller's). |
Value
The previous values, invisibly.
Examples
f <- function() {
ag_local_mode(graph = TRUE, matmul_precision = "f32")
c(ag_graph_mode(), ag_matmul_precision())
}
f() # TRUE "f32"
ag_graph_mode() # back to the previous value
Column-wise log-softmax
Description
log(softmax(x)) down every column, computed without forming
softmax first: masked entries keep finite log-probabilities instead of
log(0) = -Inf, so p * logp stays finite (entropy terms).
Usage
ag_log_softmax(x)
Arguments
x |
ag_tensor or numeric matrix, |
Details
Mask illegal entries by ADDING a large but FINITE negative number (e.g.
-1e4, which also fits f16). -Inf is not supported: the shift
below multiplies x by p, and 0 * -Inf is NaN. A column with
every entry masked is finite but meaningless (uniform over masked actions);
callers must leave at least one entry unmasked.
Stability shift. Any per-column c with max - log(n) <= c <= max
keeps exp(x - c) from overflowing and the largest term from
underflowing. c = colSums(x * softmax(x)) satisfies it
(lse - c is the entropy, at most log(n), and lse >= max),
and on the device it is three ggml ops – no max reduction, which ggml lacks
along this axis. The result does not depend on c, and the gradient treats it
as a constant: dx = g - exp(logp) * colSums(g).
f16 compute dtype: colSums(exp(x - c)) is at most n and is stored in
the compute type, so n must stay below 65504 (the f16 maximum); larger
vocabularies need ag_dtype("f32"). Inputs of magnitude ~1e4 are
themselves quantised to a few units in f16 – a property of the input.
Value
ag_tensor of the same shape.
Examples
lp <- ag_log_softmax(ag_tensor(matrix(c(1, 2, 3, 0, -1e4, 0), 3, 2)))
Column-wise log-sum-exp
Description
log(colSums(exp(x))) for a [n, batch] tensor, the package's
layout (one column = one sample). Computed as x[1, ] - log_softmax(x)[1, ],
so it inherits the stability of ag_log_softmax and needs no max
reduction or host synchronisation. Gradient softmax(x).
Usage
ag_logsumexp(x)
Arguments
x |
ag_tensor or numeric matrix, |
Value
ag_tensor [1, batch]
Examples
as.matrix(ag_logsumexp(ag_tensor(matrix(c(0, 0, 1000, 1000), 2, 2))))
LSTM and GRU over a sequence
Description
Loops an ag_lstm_cell / ag_gru_cell over a
list of time steps; the layer owns the cell, so layer$cell$forward
runs single steps (rollouts) with the same weights the sequence trains.
Each step adds its ops to the tape, so the recording grows with T;
sequences of 16..128 steps, as in recurrent PPO, are the intended use.
Usage
ag_lstm(input_size, hidden_size, init = c("rl", "torch"))
ag_gru(input_size, hidden_size, init = c("rl", "torch"))
Arguments
input_size |
Number of input features. |
|
Size of the state. | |
init |
|
Value
An ag_lstm / ag_gru environment with
forward(xs, state = NULL, starts = NULL): xs a list of
T inputs [in, B], starts NULL, a
[T, B] 0/1 (logical) matrix or a [T, B] ag_tensor of 0/1
(see ag_lstm_cell), row t applied before step
t. Returns list(outputs, state): the T hidden states
h and the final state. Plus parameters() (the cell's) and
cell.
Examples
rnn <- ag_gru(3L, 5L)
xs <- lapply(1:4, function(t) ag_tensor(matrix(rnorm(3 * 2), 3, 2)))
r <- rnn$forward(xs)
length(r$outputs)
LSTM and GRU cells
Description
One time step of an LSTM or a GRU, composed of ag_* ops. Inputs are
[in, B] (features by batch, as everywhere in the autograd engine);
the state is a list: list(h, c) for the LSTM, list(h) for the
GRU, each [hidden, B].
Usage
ag_lstm_cell(input_size, hidden_size, init = c("rl", "torch"))
ag_gru_cell(input_size, hidden_size, init = c("rl", "torch"))
Arguments
input_size |
Number of input features. |
|
Size of the state. | |
init |
|
Details
Weight layout and gate order follow the PyTorch documentation, so a model
moves from torch.nn.LSTM / torch.nn.GRU (or the R torch
package) by copying weight_ih_l0, weight_hh_l0,
bias_ih_l0, bias_hh_l0 into weight_ih, weight_hh,
bias_ih, bias_hh:
LSTM, gates
i, f, g, ostacked by rows ([4 hidden, in]):c' = \sigma(f) c + \sigma(i) \tanh(g),h' = \sigma(o) \tanh(c');GRU, gates
r, z, n([3 hidden, in]):n = \tanh(W_{in} x + b_{in} + r (W_{hn} h + b_{hn})),h' = (1 - z) n + z h. The two biases are not interchangeable here.
init:
"rl"(default)weight_ihGlorot uniform,weight_hhorthogonal per gate block (each[hidden, hidden]map is orthogonal), biases 0 except the LSTM forget gate, 1 (Gers et al. 2000), so the cell starts by keeping its memory. This deliberately differs from PyTorch. For the GRU,zstarts at\sigma(0) = 0.5: half memory, half new; there is no forget-bias analogue on purpose."torch"everything
U(-1/\sqrt{hidden}, 1/\sqrt{hidden}), as PyTorch.
starts marks the copies whose episode begins at this step: their
incoming state is zeroed before the step (SB3 RecurrentPPO's
episode_starts, i.e. the done of the previous step under
automatic resets, when the observation is already the new episode's).
It is a length-B 0/1 (or logical) vector, checked; or an ag_tensor
[1, B] (an input of ag_capture_step, where host data
inside the recording is refused), applied as s - s \cdot starts on the
device. A tensor's values are NOT checked: they must be 0 or 1, a fraction
resets the state partly.
Value
An ag_lstm_cell / ag_gru_cell environment with
forward(x, state = NULL, starts = NULL) returning the new state
(state = NULL: zeros; starts: length-B 0/1 or logical),
parameters() (weight_ih, weight_hh, bias_ih,
bias_hh; biases are [gates * hidden, 1]) and the four
parameters as fields.
See Also
ag_lstm, ag_gru for whole sequences.
Examples
cell <- ag_lstm_cell(4L, 8L)
s <- NULL
for (t in 1:3) s <- cell$forward(ag_tensor(matrix(runif(4 * 2), 4, 2)), s,
starts = c(t == 1, FALSE))
dim(as.matrix(s$h))
Matrix multiplication
Description
Computes A %*% B and records the operation on the gradient tape.
Usage
ag_matmul(A, B)
Arguments
A |
ag_tensor or numeric matrix of shape |
B |
ag_tensor or numeric matrix of shape |
Value
ag_tensor of shape [m, n]
Matrix-multiply precision for ag_* operations on the GPU
Description
On a Vulkan device with fp16 or cooperative-matrix support, an f32 x f32
matrix multiply with more than 8 columns runs a kernel that keeps its tiles in
f16: relative error around 4e-4, while up to 8 columns (and the CPU)
give f32 accuracy (~1e-7). "f32" asks every ag_matmul
(forward, and the input gradient of the graph backward) for the f32-tile
kernel, so the result no longer depends on the batch size or the device.
The same setting asks ag_flash_attention for f32 accumulation
in its forward. That is a request, not a guarantee: on cooperative-matrix
devices (e.g. RDNA4 under RADV) Vulkan multiplies the attention operands as
f16 whatever the accumulator, and the forward stays about 3e-4 to
7e-4 off an f32 reference with or without it. The backward of
ag_flash_attention is f32 regardless.
Usage
ag_matmul_precision(precision)
Arguments
precision |
|
Details
Cost: none measurable on small layers (RL policies, tens to hundreds of
units); 2-4x slower on large ones (1024 x 1024 and up). That is why the default
stays "default": set it around the code that needs it and restore the
previous value, e.g.
old <- ag_matmul_precision("f32"); on.exit(ag_matmul_precision(old)).
Has no effect on the CPU or with ag_dtype("f16") / "bf16".
Value
The current setting when precision is missing; otherwise the
previous setting, invisibly.
Examples
old <- ag_matmul_precision("f32")
ag_matmul_precision()
ag_matmul_precision(old)
Column-wise maximum
Description
ag_gather(x, ag_argmax(x)): the largest entry of every column, as a
[1, batch] ag_tensor. Differentiable; the gradient goes to the
selected entry only. With ties that is the FIRST maximum (the
ag_argmax rule), and it is not split between the tied entries –
a convention, which some frameworks resolve differently.
Usage
ag_max(x)
Arguments
x |
ag_tensor or numeric matrix, |
Details
Calls ag_argmax, so on the GPU it is a synchronisation point.
Value
ag_tensor [1, batch].
Mean of elements (or along a dim)
Description
Mean of elements (or along a dim)
Usage
ag_mean(x, dim = NULL, keepdim = FALSE)
Arguments
x |
ag_tensor |
dim |
NULL (all), 1 (row-wise), or 2 (col-wise) |
keepdim |
Logical |
Value
ag_tensor
Element-wise minimum and maximum of two tensors
Description
minimum(A, B) = A - relu(A - B), maximum(A, B) = A + relu(B - A).
The result has the shape of A; B may broadcast over it as in
ag_sub. On a tie the whole gradient goes to A.
Typical use is the PPO clipped objective
ag_minimum(ag_mul(r, adv), ag_mul(ag_clamp(r, 1 - eps, 1 + eps), adv)).
Usage
ag_minimum(A, B)
ag_maximum(A, B)
Arguments
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix (same shape, |
Value
ag_tensor
Mean Squared Error loss
Description
Mean Squared Error loss
Usage
ag_mse_loss(pred, target)
Arguments
pred |
ag_tensor [units, batch_size] |
target |
ag_tensor or matrix [units, batch_size] |
Value
scalar ag_tensor
Element-wise multiplication
Description
Element-wise multiplication
Usage
ag_mul(A, B)
Arguments
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix |
Value
ag_tensor
Create a Multi-Head Attention layer
Description
Implements scaled dot-product multi-head attention as in "Attention Is All You Need" (Vaswani et al., 2017).
Usage
ag_multihead_attention(d_model, n_heads, dropout = 0, bias = TRUE)
Arguments
d_model |
Model (embedding) dimension |
n_heads |
Number of attention heads. |
dropout |
Attention dropout probability (default 0, applied in training mode only) |
bias |
Logical: add bias to output projection (default TRUE) |
Details
Calling convention (mirrors PyTorch nn.MultiheadAttention):
-
layer$forward(q)— self-attention (k = v = q) -
layer$forward(q, k, v)— cross-attention
Tensor layout: [d_model, seq_len] — columns are tokens,
consistent with the rest of the ag_* API.
Forward pass:
Q = W_q %*% q [d_k * n_heads, seq_len]
K = W_k %*% k [d_k * n_heads, seq_len]
V = W_v %*% v [d_v * n_heads, seq_len]
for each head h:
q_h = Q[h*d_k+1 : (h+1)*d_k, ] [d_k, seq_len]
k_h = K[h*d_k+1 : (h+1)*d_k, ] [d_k, seq_len]
v_h = V[h*d_v+1 : (h+1)*d_v, ] [d_v, seq_len]
A_h = softmax(t(q_h) %*% k_h / sqrt(d_k)) [seq_len, seq_len]
if causal_mask: A_h[i,j] = 0 for j > i
head_h = v_h %*% A_h [d_v, seq_len]
concat = rbind(head_1, ..., head_H) [d_v*n_heads, seq_len]
out = W_o %*% concat + b_o [d_model, seq_len]
Value
An ag_multihead_attention environment with
$forward(q, k, v, causal_mask) and $parameters()
Choosing between this and ag_flash_attention()
This layer loops over heads in R. Since ag_* has no slice operation,
extracting one head is a matmul against a selector matrix, so a 4-head block
records around 52 tape nodes for the attention core – against one for
ag_flash_attention, which fuses all heads into a single
ggml_flash_attn_ext() call and its gradient into one
ggml_flash_attn_back(). Measured on the attention core alone, the
fused op is 15-37x faster depending on shape
(inst/scripts/measure_ag_flash_attn.R).
Prefer ag_flash_attention() for plain and causal multi-head
attention. This layer remains the fallback for what the fused op cannot do:
attention dropout (
dropout > 0);packaged projections – it owns
W_q,W_k,W_v,W_oandb_o, whereasag_flash_attention()takes already-projected Q/K/V and leaves the output projection to the caller;shapes or options
ggml_flash_attn_back()rejects – ALiBi, logit softcap and attention sinks all assert.
Which API this belongs to
This is an ag_* layer: it runs on the dynamic autograd path, where a
tape records each operation as it executes and backward
differentiates through R closures. It is not a layer for
ggml_model_sequential or ggml_model and cannot be
passed to them; the two APIs do not share layer objects. For attention inside
a compiled model, build the block from nn_* layers instead (see the
transformer encoder section of the README).
The practical consequences of being on this path: gradients and optimizer
state are R doubles regardless of ag_dtype, so budget 8 bytes
per scalar (ag_estimate_training_memory); the tape holds every
activation until zero_grad() (ag_tape_memory); and
ag_device("gpu") is usually slower than the CPU for individual ops,
because each one pays a separate upload and download.
See Also
ag_flash_attention for the fused implementation.
Examples
# Self-attention
mha <- ag_multihead_attention(64L, 8L)
x <- ag_tensor(matrix(rnorm(64 * 10), 64, 10)) # [d_model=64, seq_len=10]
out <- mha$forward(x) # [64, 10]
# Cross-attention
q <- ag_tensor(matrix(rnorm(64 * 10), 64, 10))
kv <- ag_tensor(matrix(rnorm(64 * 15), 64, 15))
out <- mha$forward(q, kv, kv)
# Causal (GPT-style)
out <- mha$forward(x, causal_mask = TRUE)
Diagonal normal distribution: log-density and entropy
Description
For x, mu: [n, batch] and log_std: [n, batch] or [n, 1]
(state-independent, broadcast over the batch, as in
ag_sample_normal), per sample:
log p = sum_i [-0.5 ((x_i - mu_i) / sigma_i)^2 - log sigma_i] - n/2 log(2 pi)
H = sum_i log sigma_i + n/2 (1 + log(2 pi))
Usage
ag_normal_log_prob(x, mu, log_std, reduction = c("none", "mean", "sum"))
ag_normal_entropy(log_std, batch = NULL, reduction = c("none", "mean", "sum"))
Arguments
x |
ag_tensor or numeric matrix |
mu |
ag_tensor or numeric matrix |
log_std |
ag_tensor or numeric matrix |
reduction |
|
batch |
Number of samples for |
Details
Both include the constant: callers must not add -n/2 log(2 pi)
themselves. (It cancels in a PPO ratio, but not in a SAC entropy term.)
ag_normal_entropy does not depend on x; with a [n, 1]
log_std it is expanded to [1, batch] so it has the same shape
as ag_normal_log_prob and ag_entropy.
Value
ag_tensor [1, batch], or scalar
Examples
as.matrix(ag_normal_log_prob(matrix(0, 2, 1), matrix(0, 2, 1),
matrix(0, 2, 1))) # -log(2 pi)
n-step bootstrapped targets
Description
For every step t (DQN-style replay targets):
y_t = \sum_{k=0}^{m-1} \gamma^k r_{t+k} + \gamma^m (1 - terminated_{t+m-1})\, next\_values_{t+m-1}
where the window length m \le n stops early at the first episode end
(terminated or truncated) or at the end of the rollout. The bootstrap is
always next_values of the LAST step of the window: 0 after a
terminated step, the value of the final observation after a truncated one,
V(s_{t+m}) otherwise. n = 1 gives the one-step TD target.
Usage
ag_nstep_targets(
rewards,
next_values,
terminated,
truncated = NULL,
gamma = 0.99,
n = 1L
)
Arguments
rewards |
Numeric |
next_values |
Numeric |
terminated |
Logical or 0/1 |
truncated |
Logical or 0/1 |
gamma |
Discount factor in |
n |
Number of steps, a positive integer. |
Details
Unlike ag_gae, next_values is read on every step:
next_values[t] is V(s_{t+1}) (target network on the next
observation, the final observation when the episode ended at t). It may be
NA only on terminated steps.
Value
Numeric [T, n_envs].
Examples
r <- matrix(1, 5, 1); nv <- matrix(10, 5, 1)
ag_nstep_targets(r, nv, terminated = c(FALSE, FALSE, TRUE, FALSE, FALSE),
gamma = 0.9, n = 3)
Create a parameter tensor (gradient tracked)
Description
Create a parameter tensor (gradient tracked)
Usage
ag_param(
data,
device = .ag_device_state$device,
dtype = .ag_device_state$dtype
)
Arguments
data |
Numeric matrix or vector |
device |
|
dtype |
Floating-point precision: |
Value
An ag_tensor with requires_grad = TRUE
Evaluate eager device work in a private, self-freeing pass pool
Description
Device memory for activations and intermediates (the "pass" pool) is
normally freed only when with_grad_tape starts the next step.
Eager ag_* forwards run outside a tape – for example validation
between ag_capture_step replays – never reach that point, so
each one keeps its memory until the process ends. ag_pass_scope()
gives expr a pass pool of its own and frees it when expr
returns or fails.
Usage
ag_pass_scope(expr)
Arguments
expr |
Expression to evaluate. |
Details
Bring results out as host values (for example with as.matrix()).
Returning an ag_tensor whose value exists only in the scope's pool is
an error, since that memory is freed on exit. Weights in the persistent pool
(ag_param) are not affected. Scopes may be nested. On the CPU
device expr is evaluated as is.
Value
The value of expr.
Examples
ag_device("cpu")
W <- ag_param(matrix(rnorm(12), 3, 4))
y <- ag_pass_scope(as.matrix(ag_matmul(W, ag_tensor(matrix(1, 4, 2)))))
dim(y)
Prioritized replay sampler
Description
Proportional prioritized experience replay (Schaul et al. 2016) over slot
indices 0 .. capacity - 1; it stores priorities only, not
transitions, so it pairs with any storage (ag_replay_buffer
uses it with prioritized = TRUE).
Usage
ag_per_sampler(capacity, alpha = 0.6, eps = 1e-06)
Arguments
capacity |
Number of slots. |
alpha |
Priority exponent, |
eps |
Added to |
Details
priority
p_i = |\delta_i| + eps, probabilityP(i) = p_i^\alpha / \sum_k p_k^\alpha;-
$add(slots)gives new (or overwritten) slots the largest priority seen so far (1 at the start), so each is drawn at least once with high probability; -
$sample(batch, beta)draws stratified: the total mass is cut intobatchequal segments, one draw in each. It returns the 0-based slots and the importance weightsw_i = (N P(i))^{-\beta} / \max_j w_j, normalised by the maximum over the WHOLE sampler (smallest priority), not over the batch, as in the paper and OpenAI baselines; N is the number of slots with a priority; -
$update(slots, td_errors)setsp = |\delta| + epsfor the slots a batch returned; -
$set_alpha(a)rebuilds the tree in one pass (alpha schedules);betais an argument of every draw, so annealing it is up to the caller.
alpha = 0 samples uniformly, beta = 0 gives weights 1.
The tree lives in C memory: it does not survive saveRDS; use
$priorities() / $load_priorities() (ag_replay_buffer
saves them with the transitions).
Value
An ag_per_sampler environment with the methods above, plus
$size() (slots with a priority), $alpha, $eps,
$capacity.
Examples
s <- ag_per_sampler(8L, alpha = 0.6)
s$add(0:7)
b <- s$sample(4L, beta = 0.4)
s$update(b$idx, td_errors = c(0.1, 2, 0.5, 1))
Element-wise power
Description
Element-wise power
Usage
ag_pow(x, p)
Arguments
x |
ag_tensor |
p |
Numeric exponent (scalar, not tracked for gradients) |
Value
ag_tensor
Random numbers on the device
Description
A [n, batch] ag_tensor of counter-based random numbers
(ggml_rand, Philox4x32-10) computed where the tensor lives.
Element i depends only on the key, stream and i, so:
a graph recorded by
ag_capturedraws new numbers at every replay when the key is one of its inputs – two numbers go up, no noise matrix;the CPU and GPU devices draw the same uniform numbers bit for bit (Gumbel and normal agree to the last bits of
log/cos).
Usage
ag_rand(
key,
dist = c("uniform", "gumbel", "normal"),
dims,
stream = 0L,
device = NULL
)
Arguments
key |
Two numbers, or an ag_tensor |
dist |
|
dims |
|
stream |
Non-negative integer separating ops that share a key. |
device |
Device for a key that is not an ag_tensor; defaults to
|
Details
The key is read as two f32 bit patterns; any two numbers work. Drawing them
with stats::runif(2) makes the stream follow set.seed. Ops
that share one key must use different streams, or they see the same
numbers.
Not differentiable: the result is a constant of the graph. For a
reparameterised sample build mu + exp(log_std) * eps from it.
Value
ag_tensor [n, batch].
Examples
set.seed(1)
eps <- ag_rand(runif(2), "normal", c(2, 5), device = "cpu")
Replay buffer
Description
A ring of capacity transitions held in one ag_buffer
on the current device, one column per transition, packed as
[obs | act | reward | terminated | truncated | next_obs], so a batch is
one ag_get_rows and its fields are row slices (obs and
act are adjacent: the critic input Q(s, a) is one slice).
next_obs must be the FINAL observation of an episode on automatic
reset, not the first observation of the next one.
Usage
ag_replay_buffer(
obs_dim,
act_dim,
capacity,
n_envs = 1L,
prioritized = FALSE,
alpha = 0.6,
eps = 1e-06
)
Arguments
obs_dim, act_dim |
Observation and action sizes. |
capacity |
Number of transitions, at least |
n_envs |
Transitions added per |
prioritized |
Use |
alpha, eps |
Passed to |
Details
Methods:
$add(obs, act, reward, terminated, truncated, next_obs)One step of
n_envstransitions:obs,next_obs[n_envs, obs_dim],act[n_envs, act_dim], the rest lengthn_envs. Prioritized: the written slots get the largest priority seen so far.$sample(batch, beta = 0.4)list(idx, weights): 0-based slots from the filled part and importance weights (all 1 when uniform). Uniform draws usesample.int(R's RNG, with replacement).$gather(idx)The batch as one packed ag_tensor
[rows, B]. Insideag_capturepassidxas an argument and listrb$bufinparams.$field(x, field),$slice(x, fields)One field, or adjacent fields, of a gathered batch.
$update_priorities(idx, td_errors)Prioritized only.
$save(path),$free(),$full()
episode_start records the slot where each transition's episode
began, for sequential batches.
Value
An ag_replay_buffer environment.
See Also
ag_replay_buffer_load, ag_per_sampler,
ag_nstep_targets
Examples
rb <- ag_replay_buffer(obs_dim = 4L, act_dim = 1L, capacity = 1000L,
prioritized = TRUE)
rb$add(matrix(0, 1, 4), matrix(1, 1, 1), 1, FALSE, FALSE, matrix(0, 1, 4))
b <- rb$sample(32L, beta = 0.4)
x <- rb$gather(b$idx)
r <- rb$field(x, "reward")
rb$update_priorities(b$idx, td_errors = rep(0.5, 32))
rb$free()
Load a replay buffer saved with $save()
Description
Restores into a new buffer on the current device. Also reads files written
by rltoolsR's replay buffer (rltoolsR.replay_buffer.v1), which are
not prioritized. As with any serialized R data, only read files you trust.
Usage
ag_replay_buffer_load(path, chunk = 65536L)
Arguments
path |
File written by |
chunk |
Columns read per part. |
Value
An ag_replay_buffer.
Reshape tensor
Description
Reshape tensor
Usage
ag_reshape(x, nrow, ncol)
Arguments
x |
ag_tensor |
nrow |
New number of rows (use -1 to infer) |
ncol |
New number of columns (use -1 to infer) |
Value
ag_tensor with new shape, same data
Create an RMS Normalization layer
Description
Normalizes each example (column) by its root mean square over the
normalized_shape features, then applies a learnable per-feature scale
(gamma) and shift (beta):
y = gamma * x / sqrt(mean(x^2) + eps) + beta.
Usage
ag_rms_norm(normalized_shape, eps = 1e-05, elementwise_affine = TRUE)
Arguments
normalized_shape |
Number of features to normalize over (the number of rows of the input). |
eps |
Small constant added to the mean square (default 1e-5). |
elementwise_affine |
Logical; when |
Details
This is the normalization of ggml_layer_rms_norm and of
ggml_layer_transformer_block (norm = "rms"), with the
same parameters and the same default eps, so weights carry over
between the two APIs. The normalization itself is a single tape node
(ggml_rms_norm / ggml_rms_norm_back); gamma and
beta are an ordinary ag_mul and ag_add.
All three run in the graph backward, so the layer can be part of an
ag_capture_step recording, as can ag_layer_norm.
Value
An ag_rms_norm environment with $forward() and
$parameters().
See Also
Examples
rn <- ag_rms_norm(4L)
x <- ag_param(matrix(rnorm(8), 4, 2)) # [features, batch]
with_grad_tape({
y <- rn$forward(x)
loss <- ag_mse_loss(y, matrix(0, 4, 2))
})
grads <- backward(loss)
Sample one index per column from logits
Description
Categorical sampling by the Gumbel-max trick:
argmax(x + G) with G = -log(-log(u)), u ~ U(0, 1),
draws index i of column j with probability softmax(x[, j])[i]. x
may be logits or log-probabilities (they differ by a per-column constant).
Usage
ag_sample_categorical(x, key = NULL, stream = 0L)
Arguments
x |
ag_tensor or numeric matrix, |
key |
|
stream |
Stream of the key, see |
Details
The noise comes from stats::runif, so set.seed reproduces the
draw, and the CPU and GPU paths see the same noise. On the GPU only the noise
goes up and only the indices come down: x stays on the device, and the
add and argmax are one launch. Like ag_argmax it is a synchronisation
point and not differentiable; get the log-probability of the drawn action with
ag_gather(ag_log_softmax(x), a).
Mask illegal entries as for ag_log_softmax: add a large but FINITE
negative number (-1e4). Gumbel noise exceeds 1e4 with probability
about exp(-1e4), so a masked entry is never drawn while one column
entry is unmasked.
Near-ties of x + G are rounded in f32 on the GPU and in double on the
CPU, so the two paths can differ in a rare column even with the same seed.
With key the noise is drawn on the device instead
(ag_rand, Gumbel) and the result is a [1, batch]
ag_tensor that stays there: nothing is uploaded but the key and nothing
comes down, so the draw can be recorded by ag_capture with the
key as an input. The numbers differ from the runif path.
Value
Integer vector of length ncol(x), 0-based; with key,
an ag_tensor [1, batch] holding the same.
Examples
set.seed(1)
ag_sample_categorical(matrix(c(0, 0, -1e4, 5, 0, 0), 3, 2))
Sample from a diagonal Gaussian
Description
mu + exp(log_std) * eps with eps ~ N(0, 1) from
stats::rnorm (so set.seed reproduces it). Differentiable in
mu and log_std – the reparameterisation used by SAC; for PPO
collection the result is just the action.
Usage
ag_sample_normal(mu, log_std)
Arguments
mu |
ag_tensor or numeric matrix, |
log_std |
ag_tensor or numeric matrix, |
Details
log_std is either the shape of mu (state-dependent) or one
column [n, 1] broadcast over the batch (state-independent). On the GPU
mu and log_std stay on the device; only eps is uploaded.
Value
ag_tensor [n, batch].
Examples
set.seed(1)
a <- ag_sample_normal(matrix(0, 2, 3), matrix(log(0.5), 2, 1))
Save an autograd module's state to disk
Description
Serializes the trainable parameters and persistent buffers of an
ag_sequential (or single ag_* layer) module as a portable
state dictionary of plain numeric matrices. This avoids serializing the live
module (environments + closures), which is brittle across ggmlR versions and
carries non-portable GPU pointers.
Usage
ag_save_model(model, path, model_fn = NULL)
Arguments
model |
An |
path |
File path to write (an RDS container). |
model_fn |
Optional zero-argument function that rebuilds the module
architecture (fresh, untrained). If supplied, it is stored in the container
so |
Details
Reconstruction requires the architecture. Either pass model_fn here so
it is stored in the file, or pass it later to ag_load_model.
Value
path, invisibly.
See Also
Examples
build <- function() ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 3L))
model <- build()
ag_save_model(model, tempfile(fileext = ".rds"), model_fn = build)
Scale tensor by a scalar constant
Description
Scale tensor by a scalar constant
Usage
ag_scale(x, scalar)
Arguments
x |
ag_tensor |
scalar |
Numeric scalar (not tracked for gradients) |
Value
ag_tensor
Create a sequential container of layers
Description
Chains layers so that forward(x) passes x through each layer
in order. parameters() collects all trainable params from all layers.
ag_train() / ag_eval() propagate mode to stateful sub-layers.
Usage
ag_sequential(...)
Arguments
... |
Layer objects (ag_linear, ag_dropout, ag_batch_norm, ag_embedding) or a single list of layers. |
Value
An ag_sequential environment
Examples
model <- ag_sequential(
ag_linear(4L, 16L, activation = "relu"),
ag_dropout(0.5),
ag_linear(16L, 2L, activation = "softmax")
)
x <- ag_tensor(matrix(runif(4 * 8), 4, 8))
out <- model$forward(x)
Write buffer columns by index
Description
Writes the columns of x into columns idx of an
ag_buffer. Inside ag_capture() the write is a
node of the recorded graph (ggml_set_rows): every replay writes on
the device, with nothing crossing the bus, and the state it leaves – a
counter, a board, a rollout column – is what the next replay reads. A
recording whose only effect is such writes may return NULL: its
replays then download nothing.
Usage
ag_set_rows(buf, idx, x)
Arguments
buf |
An |
idx |
0-based column indices: a numeric vector, or an ag_tensor with one column (f32). Inside a recording an ag_tensor (an input or a value computed on the device). |
x |
An ag_tensor or numeric matrix of shape |
Details
Outside a recording the write goes through the host
(ag_buffer_write), after any queued device work.
Inside a recording the writes run after the outputs, in the order they
were made. A recording may read columns it also writes only when the
written value depends on the read one (t <- t + 1, a board
updated from itself); otherwise the order of the read and the write is
not defined. Indices are clamped to 0..capacity-1 on the device,
as in ag_get_rows; with repeated indices which write wins is
not defined (outside a recording repeated indices are an error).
Value
buf, invisibly.
Slice a range of rows
Description
Returns rows from + 1 to from + n of x (from is
0-based), e.g. one field of a packed batch read with
ag_get_rows. On the GPU it is a view made contiguous on the
device, with nothing crossing the bus; inside ag_capture()
from and n are constants of the recording.
Usage
ag_slice_rows(x, from, n)
Arguments
x |
ag_tensor (or matrix) with at least |
from |
0-based index of the first row |
n |
Number of rows |
Details
The gradient is g in the sliced rows and zero elsewhere.
Value
ag_tensor of shape n x ncol(x)
Examples
x <- ag_tensor(matrix(1:12, 4, 3))
as.matrix(ag_slice_rows(x, 1, 2))
Update a target network from a source network
Description
ag_soft_update is the Polyak update
target <- tau * source + (1 - tau) * target
applied to every parameter; tau = 1 copies, tau = 0 changes
nothing. ag_hard_update copies the parameters directly (no
arithmetic, so the copy is exact).
Usage
ag_soft_update(target, source, tau)
ag_hard_update(target, source)
Arguments
target |
Model (anything with |
source |
Model or list of ag_param tensors, same structure. Read only. |
tau |
Interpolation factor in |
Details
Argument order: the TARGET (the model that is written) comes first,
as in ag_load_state_dict. Stable-Baselines3's
polyak_update(params, target_params, tau) and RLlib put the source
first – mind the order when porting code.
Buffers: when both arguments are models, persistent buffers
(ag_batch_norm running statistics) are COPIED from the source
whatever tau is – they are statistics, not learned weights. This
is what SB3 does (it updates them with tau = 1); the TD3 paper does
not mention it. Plain parameter lists have no buffers.
Parameters are paired by name when both sides are named, otherwise by position; the counts must be equal and every shape must match. Everything is checked before anything is written. On the device, pairs where both tensors are resident are updated in one graph per 256 parameters; the arithmetic is F32 and the result is converted to the target's type (an F32 master copy of the target, if it has one, gets the unrounded value). Other pairs are updated on the host.
Value
target, invisibly.
Examples
online <- ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 2L))
target <- ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 2L))
ag_hard_update(target, online) # start equal
ag_soft_update(target, online, 0.005) # after each training step
Softmax activation (column-wise)
Description
Applies numerically stable softmax along rows so that each column (one sample) sums to 1.
Usage
ag_softmax(x)
Arguments
x |
ag_tensor of shape |
Value
ag_tensor of the same shape as x
Fused softmax + cross-entropy loss (numerically stable)
Description
Combines softmax and CE in one op using the fused gradient (p - y) / n.
More numerically stable than chaining ag_softmax + ag_cross_entropy_loss.
Use this when your last layer outputs raw logits.
Usage
ag_softmax_cross_entropy_loss(logits, target)
Arguments
logits |
ag_tensor [classes, batch_size] raw (pre-softmax) scores |
target |
matrix [classes, batch_size] one-hot labels |
Value
scalar ag_tensor
Element-wise Unary Operations (autograd)
Description
Element-wise Unary Operations (autograd)
Usage
ag_sqrt(x)
ag_abs(x)
ag_softplus(x)
ag_relu(x)
ag_sigmoid(x)
ag_silu(x)
ag_tanh(x)
ag_log(x)
ag_exp(x)
ag_sin(x)
ag_cos(x)
Arguments
x |
ag_tensor |
Details
ag_sqrt(): x^0.5; the gradient is infinite at 0.
Returns: ag_tensor
ag_abs(): relu(x) + relu(-x); the gradient at 0 is 0.
Returns: ag_tensor
ag_softplus(): log(1 + exp(x)) in the stable form
relu(x) - log(sigmoid(|x|)): sigmoid(|x|) lies in
[0.5, 1], so nothing overflows or underflows. Gradient
sigmoid(x).
Returns: ag_tensor
ag_relu(): Applies the rectified linear unit: \max(0, x).
Returns: ag_tensor
ag_sigmoid(): Applies 1 / (1 + e^{-x}).
Returns: ag_tensor
ag_silu(): Applies x / (1 + e^{-x}), the activation of the feed-forward half of
ggml_layer_transformer_block. One node on the tape (forward
ggml_silu, backward ggml_silu_back) rather than the three of
ag_mul(x, ag_sigmoid(x)), and it runs in the graph backward, so it can
be part of an ag_capture_step recording.
Returns: ag_tensor of the same shape as x
ag_tanh():
Returns: ag_tensor
ag_log():
Returns: ag_tensor
ag_sin():
Returns: ag_tensor
Value
ag_tensor
Examples
ag_silu(ag_tensor(matrix(c(-2, 0, 2), 3, 1)))
Model weights as plain R data
Description
A copy of every parameter and persistent buffer (ag_batch_norm
running statistics) as host matrices, keyed like model$parameters().
Values are R doubles whatever the device or ag_dtype(): an f32 value
is represented exactly, so a state taken on the GPU in f32 restores on the
CPU (or back) to the same numbers. f16/bf16 weights carry only their own
precision; continuing exactly across f16 and f32 is not possible.
Usage
ag_state_dict(model)
Arguments
model |
An |
Details
Use it for checkpoints, for copying weights between models (target networks:
ag_load_state_dict(target, ag_state_dict(online))), or with
saveRDS. As with any .rds, only read files you trust.
Value
A list with parameters, buffers and
param_dtypes, plus format/version.
See Also
ag_load_state_dict, ag_save_model
Element-wise subtraction
Description
Computes A - B. Either operand may be a plain number, as in
ag_add.
Usage
ag_sub(A, B)
Arguments
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix |
Value
ag_tensor
Sum all elements (or along a dim): out = sum(x)
Description
Sum all elements (or along a dim): out = sum(x)
Usage
ag_sum(x, dim = NULL, keepdim = FALSE)
Arguments
x |
ag_tensor |
dim |
NULL (all), 1 (row-wise), or 2 (col-wise) |
keepdim |
Logical: keep size-1 dimensions |
Value
scalar (or reduced) ag_tensor
Report what the gradient tape is holding
Description
Walks the current tape and reports how many nodes it holds and how much memory those nodes keep alive, split into three categories: the operand tensors, which outlive the tape; the activations captured by the backward closures, which clearing the tape would release; and any extras recorded for the graph backward path beyond those.
Usage
ag_tape_memory(top = 10L, quiet = FALSE)
Arguments
top |
Number of operations to list in the per-op breakdown, most expensive first. Set to 0 for totals only. |
quiet |
Suppress the printed report and return the figures only. For callers that consume the numbers – tests, scripts that format their own table – rather than read them. |
Details
Storage shared between categories is counted once, and charged to the
longest-lived one. That matters for reading the result: a weight's matrix
and the a_snap an ag_matmul closure captured are the same
object, so it is reported as an operand. The activation figure is therefore
what clearing the tape would actually free, not the sum of everything the
closures reference.
The tape lives from the start of with_grad_tape until
zero_grad() clears it, which happens after the optimizer step. Call
this between backward and zero_grad() to see the peak.
Value
Invisibly, a list with nodes, bytes_total,
bytes_snapshots, bytes_fields, bytes_inputs and a
data frame by_op. Printed as a report as a side effect.
Examples
w <- ag_param(matrix(runif(64), 8, 8))
x <- ag_tensor(matrix(runif(64), 8, 8))
with_grad_tape({
h <- ag_relu(ag_matmul(w, x))
loss <- ag_mse_loss(h, matrix(0, 8, 8))
})
backward(loss)
ag_tape_memory()
Create a dynamic tensor (no gradient tracking)
Description
ag_tensor is backed by an R environment so all references to the same tensor see updates (like PyTorch tensors).
Usage
ag_tensor(
data,
device = .ag_device_state$device,
dtype = .ag_device_state$dtype
)
Arguments
data |
Numeric matrix or vector |
device |
|
dtype |
Floating-point precision: |
Value
An ag_tensor object (environment)
Move a tensor to the specified device
Description
Copies an ag_tensor to the target device, returning a new tensor.
The original tensor is not modified.
Usage
ag_to_device(tensor, device)
Arguments
tensor |
An |
device |
|
Value
A new ag_tensor on the target device (or the original if
already on the target device)
Switch a layer or sequential model to training mode
Description
Switch a layer or sequential model to training mode
Usage
ag_train(model)
Arguments
model |
An ag_sequential, ag_batch_norm, or ag_dropout layer |
Value
The model/layer (invisibly)
Transpose a tensor
Description
Transpose a tensor
Usage
ag_transpose(x)
Arguments
x |
ag_tensor |
Value
ag_tensor with rows and columns swapped
Element-wise selection by a mask
Description
B + cond * (A - B): A where cond is 1, B where it
is 0. cond is a constant (no gradient flows into it). A and
B must be finite: Inf - Inf gives NaN – mask with a
large finite number (-1e4) instead, as for ag_log_softmax.
Usage
ag_where(cond, A, B)
Arguments
cond |
logical or 0/1 numeric matrix, or an ag_tensor holding 0/1 |
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix, same shape as |
Value
ag_tensor
Examples
as.matrix(ag_where(matrix(c(TRUE, FALSE), 1, 2),
ag_tensor(matrix(1, 1, 2)), ag_tensor(matrix(5, 1, 2))))
Extract the value of an ag_tensor as an R matrix
Description
The value of an ag_tensor lives in $data only on the CPU. On
the GPU it lives in a backend buffer and $data stays NULL until
something materialises it, so reading $data directly works on one
device and returns NULL on the other. This method downloads the value
when it has to and returns the same matrix either way – use it rather than
$data in code that may run on a GPU.
Usage
## S3 method for class 'ag_tensor'
as.matrix(x, ...)
Arguments
x |
An |
... |
Ignored |
Value
A numeric matrix holding the tensor's current value.
Examples
t <- ag_tensor(matrix(1:6, nrow = 2))
as.matrix(t)
Extract the underlying matrix from a ggml_matrix
Description
Extract the underlying matrix from a ggml_matrix
Usage
## S3 method for class 'ggml_matrix'
as.matrix(x, ...)
Arguments
x |
A |
... |
Ignored; present for S3 |
Value
The plain numeric matrix.
Wrap a matrix so multiplies run on the GPU
Description
Tags a matrix as GPU-backed. In an expression like
as_gpu_matrix(A) %*% B, the multiply dispatches to
ggml_matmul (Vulkan GPU with CPU fallback) instead of R's
built-in. Only one operand needs wrapping. crossprod() and
tcrossprod() on a wrapped matrix likewise use the GPU.
Usage
as_gpu_matrix(x, device = "auto", prec = "f32")
Arguments
x |
A numeric matrix (or something coercible with |
device |
|
prec |
|
Value
A ggml_matrix.
Examples
A <- matrix(rnorm(9), 3)
B <- matrix(rnorm(9), 3)
as_gpu_matrix(A, device = "cpu") %*% B
Augment new data with predictions from a fitted ggml parsnip model
Description
Adds prediction columns to 'new_data', broom style. For classification this appends '.pred_class' plus one '.pred_<level>' probability column per class; for regression it appends '.pred'. Predictions are produced by the existing 'predict()' method for ggml parsnip models (no duplicate inference logic).
Usage
## S3 method for class 'ggmlr_parsnip_model'
augment(x, new_data, ...)
Arguments
x |
A fitted 'ggmlr_parsnip_model' (the engine object inside a parsnip fit; e.g. from 'extract_fit_engine()'). |
new_data |
A data frame of predictors (same columns used for fitting). |
... |
Unused; for generic compatibility. |
Value
'new_data' as a tibble with prediction columns appended.
Examples
ggml_set_n_threads(1L) # deterministic, single OpenMP pool
spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |>
parsnip::set_engine("ggml", backend = "cpu") |>
parsnip::set_mode("regression")
fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars)
generics::augment(parsnip::extract_fit_engine(fit_obj), mtcars)
Run backward pass from a scalar loss tensor
Description
Traverses the gradient tape in reverse and accumulates gradients into
tensor$grad for all leaf tensors with requires_grad = TRUE.
Usage
backward(loss)
Arguments
loss |
Scalar ag_tensor |
Value
Named environment: tensor id -> gradient matrix (for use by optimizer$step)
Examples
w <- ag_param(matrix(runif(4), 2, 2))
x <- ag_tensor(matrix(c(1, 2), 2, 1))
y <- ag_tensor(matrix(c(0, 1), 2, 1))
with_grad_tape({
out <- ag_matmul(w, x)
loss <- ag_mse_loss(out, y)
})
grads <- backward(loss)
Detect NaN / Inf gradients
Description
Scans the gradient environment produced by backward() and reports
every parameter whose gradient contains NaN or Inf values.
Useful as a debugging hook right after backward() — it pinpoints the
offending layer instead of leaving you with a model that silently turns to
NaN several epochs later.
Usage
check_grad_anomaly(
params,
grads,
action = c("warn", "stop", "silent"),
max_abs = NULL
)
Arguments
params |
List of ag_param tensors (same as passed to optimizer), named or not. |
grads |
Gradient environment returned by |
action |
One of |
max_abs |
Optional numeric threshold; gradients whose maximum absolute
finite value exceeds it are also reported (as |
Value
A data frame (invisibly) with one row per parameter that has a
gradient, containing columns param, n_nan, n_inf,
max_abs and status. status is one of "ok",
"nan", "inf", "nan+inf" or "large".
Parameters without a gradient are reported with status = "missing".
Examples
w <- ag_param(matrix(runif(4), 2, 2))
x <- ag_tensor(matrix(c(1, 1), 2, 1))
with_grad_tape({
out <- ag_matmul(w, x)
loss <- ag_mse_loss(out, matrix(0, 2, 1))
})
grads <- backward(loss)
check_grad_anomaly(list(w = w), grads)
Clip gradients by global L2 norm
Description
Rescales all gradients in grads so that their global L2 norm does
not exceed max_norm. Modifies the grads environment
in-place and returns the pre-clip norm.
Usage
clip_grad_norm(params, grads, max_norm)
Arguments
params |
List of ag_param tensors (same as passed to optimizer), named or not. |
grads |
Gradient environment returned by |
max_norm |
Maximum allowed global L2 norm. |
Details
Call this after backward() and before
optimizer$step().
Value
Numeric: the global L2 norm before clipping (invisibly).
Examples
w <- ag_param(matrix(runif(4), 2, 2))
x <- ag_tensor(matrix(c(1, 1), 2, 1))
with_grad_tape({
out <- ag_matmul(w, x)
loss <- ag_mse_loss(out, matrix(0, 2, 1))
})
grads <- backward(loss)
clip_grad_norm(list(w = w), grads, max_norm = 1.0)
Clip gradients element-wise by value
Description
Clamps every gradient element into
[-clip_value, clip_value]. Unlike clip_grad_norm this
does not preserve the gradient direction — each element is treated
independently. Modifies the grads environment in-place.
Usage
clip_grad_value(params, grads, clip_value)
Arguments
params |
List of ag_param tensors (same as passed to optimizer), named or not. |
grads |
Gradient environment returned by |
clip_value |
Positive numeric; maximum absolute value per element. |
Details
Call this after backward() and before
optimizer$step().
Value
Numeric: the largest absolute gradient value before clipping (invisibly). Returns 0 when no gradients were found.
See Also
Examples
w <- ag_param(matrix(runif(4), 2, 2))
x <- ag_tensor(matrix(c(1, 1), 2, 1))
with_grad_tape({
out <- ag_matmul(w, x)
loss <- ag_mse_loss(out, matrix(0, 2, 1))
})
grads <- backward(loss)
clip_grad_value(list(w = w), grads, clip_value = 0.5)
Compile a Model
Description
Configures the model for training by setting the optimizer, loss function,
and metrics. This is the keras-compatible interface; it delegates to
ggml_compile.
Usage
## S3 method for class 'ggml_sequential_model'
compile(
object,
optimizer = "adam",
loss = "categorical_crossentropy",
metrics = c("accuracy"),
...
)
## S3 method for class 'ggml_functional_model'
compile(
object,
optimizer = "adam",
loss = "categorical_crossentropy",
metrics = c("accuracy"),
...
)
Arguments
object |
A model object (e.g. |
optimizer |
Character: |
loss |
Character: |
metrics |
Character vector of metrics (default |
... |
Additional arguments passed to |
Value
The compiled model (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(10, activation = "softmax", input_shape = 4)
model <- compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
Dequantize rows
Description
Convert quantized data back to float values. One function per quantization type:
- Basic
q4_0,q4_1,q5_0,q5_1,q8_0.- K-quants
q2_Kthroughq8_K; better quality/size tradeoffs.- Ternary
tq1_0,tq2_0; extreme compression formats.- IQ
iq2_xxs,iq2_xs,iq2_s,iq3_xxs,iq3_s,iq4_nl,iq4_xs,iq1_s,iq1_m; high compression with importance-matrix-aware quantization.- Other
mxfp4(microscaling FP4),q1_0(n_elementsmust be a multiple of 128),q2_0andnvfp4(multiples of 64).
Usage
dequantize_row_q4_0(raw_data, n_elements)
dequantize_row_q4_1(raw_data, n_elements)
dequantize_row_q5_0(raw_data, n_elements)
dequantize_row_q5_1(raw_data, n_elements)
dequantize_row_q8_0(raw_data, n_elements)
dequantize_row_q2_K(raw_data, n_elements)
dequantize_row_q3_K(raw_data, n_elements)
dequantize_row_q4_K(raw_data, n_elements)
dequantize_row_q5_K(raw_data, n_elements)
dequantize_row_q6_K(raw_data, n_elements)
dequantize_row_q8_K(raw_data, n_elements)
dequantize_row_tq1_0(raw_data, n_elements)
dequantize_row_tq2_0(raw_data, n_elements)
dequantize_row_iq2_xxs(raw_data, n_elements)
dequantize_row_iq2_xs(raw_data, n_elements)
dequantize_row_iq2_s(raw_data, n_elements)
dequantize_row_iq3_xxs(raw_data, n_elements)
dequantize_row_iq3_s(raw_data, n_elements)
dequantize_row_iq4_nl(raw_data, n_elements)
dequantize_row_iq4_xs(raw_data, n_elements)
dequantize_row_iq1_s(raw_data, n_elements)
dequantize_row_iq1_m(raw_data, n_elements)
dequantize_row_mxfp4(raw_data, n_elements)
dequantize_row_q1_0(raw_data, n_elements)
dequantize_row_q2_0(raw_data, n_elements)
dequantize_row_nvfp4(raw_data, n_elements)
Arguments
raw_data |
Raw vector containing quantized data |
n_elements |
Number of elements to dequantize |
Value
Numeric vector of dequantized values
See Also
quantize_data, ggmlR-quantization
Data-parallel training across multiple GPUs
Description
Runs synchronous data-parallel training:
-
make_model()is calledn_gputimes to create one independent model replica per GPU (each with its own parameters). Each iteration: the current data item is forwarded through every replica in parallel; gradients are computed via
backward().Gradients are averaged across all replicas (element-wise mean).
One optimizer step is taken on replica 0; updated weights are then broadcast to replicas 1 … N-1 so all replicas stay in sync.
Usage
dp_train(
make_model,
data,
loss_fn = NULL,
forward_fn = NULL,
target_fn = NULL,
n_gpu = NULL,
n_iter = 10L,
lr = 0.001,
max_norm = Inf,
verbose = 10L
)
Arguments
make_model |
A zero-argument function that returns a model object with
at least |
data |
A list of training samples. Each element is passed directly to
|
loss_fn |
A function |
forward_fn |
Optional function |
target_fn |
Optional function |
n_gpu |
Number of GPU replicas (default: all available Vulkan devices, minimum 1). |
n_iter |
Number of training iterations (passes over |
lr |
Learning rate for Adam optimizer (default 1e-3). |
max_norm |
Gradient clipping threshold (default |
verbose |
Print loss every |
Details
Because all replicas live in the same R process and ag_param uses
environment (reference) semantics, no IPC or NCCL is required — weight
synchronisation is a simple in-place copy.
Value
A list with:
paramsNamed list of final parameters (from replica 0).
loss_historyNumeric vector of per-iteration mean loss.
modelReplica 0 model object.
Examples
make_model <- function() {
W <- ag_param(matrix(rnorm(4), 2, 2))
list(
forward = function(x) ag_matmul(W, x),
parameters = function() list(W = W)
)
}
data <- lapply(1:8, function(i) matrix(rnorm(2), 2, 1))
result <- dp_train(
make_model = make_model,
data = data,
loss_fn = function(out, tgt) ag_mse_loss(out, tgt),
target_fn = function(s) s,
n_gpu = 1L,
n_iter = 10L,
lr = 1e-3,
verbose = FALSE
)
Evaluate a Model
Description
Generic for computing loss and metrics on test data.
Usage
evaluate(object, ...)
## S3 method for class 'ggml_sequential_model'
evaluate(object, x, y, batch_size = NULL, ...)
## S3 method for class 'ggml_functional_model'
evaluate(object, x, y, batch_size = NULL, ...)
Arguments
object |
A trained model object. |
... |
Additional arguments passed to |
x |
Test data. |
y |
Test labels. |
batch_size |
Batch size. |
Value
A named list with loss and metric values.
Relationship to the generics package
generics::evaluate declares its first argument as x, so S3
dispatch happens on whatever is bound to x. With the keras3 argument
order evaluate(model, x, y) that works positionally, but naming the
data explicitly – evaluate(model, x = x, y = y) – binds the *data*
to x, dispatches on it, and fails with "no applicable method".
Declaring the generic here with object as its first argument fixes
that and also resolves the S3 generic/method consistency warning from
R CMD check.
The trade-off: attaching both ggmlR and keras3 makes the two evaluate
generics mask one another, and the later-attached package wins. Qualify the
call (ggmlR::evaluate(...)) if both are loaded. compile() and
fit() are unaffected – generics declares those with
object first, so they are still re-exported from there.
Train a Model
Description
Trains the model on data for a fixed number of epochs. This is the
keras-compatible interface; it delegates to ggml_fit.
Usage
## S3 method for class 'ggml_sequential_model'
fit(
object,
x,
y,
epochs = 1L,
batch_size = 32L,
validation_split = 0,
validation_data = NULL,
verbose = 1L,
shuffle = TRUE,
callbacks = list(),
...
)
## S3 method for class 'ggml_functional_model'
fit(
object,
x,
y,
epochs = 1L,
batch_size = 32L,
validation_split = 0,
validation_data = NULL,
verbose = 1L,
shuffle = TRUE,
callbacks = list(),
...
)
Arguments
object |
A compiled model object. |
x |
Training data. Matrix, array, or list of matrices (multi-input). |
y |
Training labels (matrix, one-hot encoded for classification). |
epochs |
Number of training epochs (default 1). |
batch_size |
Batch size (default 32). |
validation_split |
Fraction of data for validation (default 0). |
validation_data |
Optional |
verbose |
0 = silent, 1 = progress (default 1). |
shuffle |
Shuffle the training data (default |
callbacks |
List of callback objects (default |
... |
Additional arguments passed to |
Value
The trained model (invisibly), with model$history.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(10, activation = "softmax", input_shape = 4)
model <- compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
# model <- fit(model, x_train, y_train, epochs = 5, batch_size = 32)
Backend functions
Description
Low-level access to the ggml backend API: devices, registries, buffer types, buffers, events, graph plans and asynchronous transfers.
Devices
ggml_backend_dev_count, ggml_backend_dev_get,
ggml_backend_dev_by_name, ggml_backend_dev_by_type,
ggml_backend_dev_name, ggml_backend_dev_description,
ggml_backend_dev_memory, ggml_backend_dev_type,
ggml_backend_dev_get_props, ggml_backend_dev_supports_op,
ggml_backend_dev_supports_buft, ggml_backend_dev_offload_op,
ggml_backend_dev_buffer_type, ggml_backend_dev_host_buffer_type,
ggml_backend_dev_init, ggml_backend_get_device,
ggml_backend_meta_device, ggml_backend_device_register.
Device type constants: ggml_backend_device_type_cpu,
ggml_backend_device_type_gpu, ggml_backend_device_type_igpu,
ggml_backend_device_type_accel.
Backend initialization and execution
ggml_backend_init_by_name, ggml_backend_init_by_type,
ggml_backend_init_best, ggml_backend_synchronize,
ggml_backend_graph_compute_async,
ggml_backend_graph_plan_create, ggml_backend_graph_plan_free,
ggml_backend_graph_plan_compute.
Registry and dynamic loading
ggml_backend_reg_count, ggml_backend_reg_get,
ggml_backend_reg_by_name, ggml_backend_reg_name,
ggml_backend_reg_dev_count, ggml_backend_reg_dev_get,
ggml_backend_register, ggml_backend_load,
ggml_backend_unload, ggml_backend_load_all.
Buffer types and buffers
ggml_backend_buft_name, ggml_backend_buft_get_alignment,
ggml_backend_buft_get_max_size, ggml_backend_buft_is_host,
ggml_backend_buffer_clear, ggml_backend_buffer_set_usage,
ggml_backend_buffer_get_usage, ggml_backend_buffer_reset,
ggml_backend_buffer_is_host,
ggml_backend_multi_buffer_alloc_buffer,
ggml_backend_buffer_is_multi_buffer,
ggml_backend_multi_buffer_set_usage.
Buffer usage constants: ggml_backend_buffer_usage_any,
ggml_backend_buffer_usage_weights,
ggml_backend_buffer_usage_compute.
Events and asynchronous transfers
ggml_backend_event_new, ggml_backend_event_free,
ggml_backend_event_record, ggml_backend_event_synchronize,
ggml_backend_event_wait, ggml_backend_tensor_set_async,
ggml_backend_tensor_get_async, ggml_backend_tensor_copy_async.
Optimization (ggml-opt) functions
Description
Training through the ggml-opt API: datasets, optimizer contexts, results and fit loops.
High-level training
ggml_fit_opt, ggml_fit_opt_multi,
ggml_fit_opt_gen, ggml_opt_fit,
ggml_opt_fit_multi, ggml_opt_epoch,
ggml_opt_eval, ggml_trainer,
ggml_snapshot_free.
Constants, datasets and results
ggml_opt_loss_type (loss and optimizer type constants),
ggml_opt_dataset (dataset functions),
ggml_opt_result (result functions).
Optimizer context
ggml_opt_default_params, ggml_opt_init,
ggml_opt_init_for_fit, ggml_opt_init_for_fit_multi,
ggml_opt_free, ggml_opt_reset,
ggml_opt_static_graphs, ggml_opt_prepare_alloc,
ggml_opt_alloc, ggml_opt_grad_acc,
ggml_opt_set_lr, ggml_opt_get_lr,
ggml_opt_context_optimizer_type, ggml_opt_optimizer_name.
Graph tensors
ggml_opt_inputs, ggml_opt_outputs,
ggml_opt_outputs_i, ggml_opt_labels,
ggml_opt_loss, ggml_opt_loss_i,
ggml_opt_n_loss, ggml_opt_pred,
ggml_opt_ncorrect.
Quantization functions
Description
Low-level quantization and dequantization of raw data for all ggml quant types.
Details
quantize_dataQuantize rows with optional importance matrix (
quantize_q4_0(),quantize_q2_K(), ...).quantize_row_refReference row quantizers (
quantize_row_q4_0_ref(), ...).dequantize_rowDequantize rows (
dequantize_row_q4_0(), ...).- IQ lookup tables
iq2xs_init_impl,iq2xs_free_impl,iq3xs_init_impl,iq3xs_free_impl.- Block layout
Absolute Value (Graph)
Description
Creates a graph node for element-wise absolute value: |x|
Usage
ggml_abs_inplace(ctx, a)
ggml_abs(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_abs_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the abs operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(-2, -1, 1, 2))
result <- ggml_abs(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [2, 1, 1, 2]
ggml_free(ctx)
Element-wise Addition (Graph)
Description
Creates a graph node for element-wise addition. Must be computed using ggml_build_forward_expand() and ggml_graph_compute().
Usage
ggml_add(ctx, a, b)
ggml_add_inplace(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Details
ggml_add_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the addition operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
ggml_set_f32(b, c(5, 4, 3, 2, 1))
result <- ggml_add(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
ggml_set_f32(b, c(5, 4, 3, 2, 1))
result <- ggml_add_inplace(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Add Scalar to Tensor (Graph)
Description
Creates a graph node for adding a scalar (1-element tensor) to all elements of a tensor. This is more efficient than creating a full tensor of the same value.
Usage
ggml_add1(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Input tensor |
b |
Scalar tensor (1-element tensor) |
Value
Tensor representing the operation a + b (broadcasted)
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
scalar <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
ggml_set_f32(scalar, 10)
result <- ggml_add1(ctx, a, scalar)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Add Relative Position Bias (Graph)
Description
Adds width and height relative-position bias to a.
Usage
ggml_add_rel_pos(ctx, a, pw, ph)
Arguments
ctx |
GGML context |
a |
Input tensor |
pw |
Width relative-position tensor |
ph |
Height relative-position tensor |
Value
Tensor with added relative-position bias
Apply a Layer Object to a Tensor Node
Description
Applies a ggml_layer object (created with ggml_dense(),
ggml_lstm(), etc.) to a ggml_tensor_node. Applying the
same layer object to multiple tensor nodes produces shared weights –
the identity of the layer object (layer$layer_id) is used as the
sharing key, not its name.
Usage
ggml_apply(tensor, layer)
Arguments
tensor |
A |
layer |
A |
Value
A new ggml_tensor_node.
Examples
encoder <- ggml_dense(64L, activation = "relu")
x1 <- ggml_input(shape = 32L)
x2 <- ggml_input(shape = 32L)
out1 <- x1 |> ggml_apply(encoder)
out2 <- x2 |> ggml_apply(encoder) # shared weights
model <- ggml_model(inputs = list(x1, x2),
outputs = list(out1, out2))
Arange (Graph)
Description
Creates a 1D F32 tensor with values from start (inclusive) to
stop (exclusive) in steps of step.
Usage
ggml_arange(ctx, start, stop, step = 1)
Arguments
ctx |
GGML context |
start |
Start value (inclusive) |
stop |
Stop value (exclusive) |
step |
Step between values (default 1) |
Value
1D F32 tensor
Check if Two Tensors Have the Same Layout
Description
Compares two tensors to check if they have identical type, shape, and strides. Tensors with the same layout can be used interchangeably for memory operations.
Usage
ggml_are_same_layout(a, b)
Arguments
a |
External pointer to first tensor |
b |
External pointer to second tensor |
Value
Logical indicating if tensors have identical layout
See Also
Other tensor:
ggml_get_op_params()
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4)
b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4)
same <- ggml_are_same_layout(a, b) # TRUE
ggml_free(ctx)
Compare Tensor Shapes
Description
Checks if two tensors have the same shape.
Usage
ggml_are_same_shape(a, b)
Arguments
a |
First tensor |
b |
Second tensor |
Value
TRUE if shapes are identical, FALSE otherwise
Compare Tensor Strides
Description
Check if two tensors have the same stride pattern. Useful for determining if tensors can share operations.
Usage
ggml_are_same_stride(a, b)
Arguments
a |
First tensor |
b |
Second tensor |
Value
Logical indicating if strides are identical
See Also
Other tensor_layout:
ggml_can_repeat(),
ggml_count_equal()
Argmax (Graph)
Description
Creates a graph node that finds the index of the maximum value. CRITICAL for token generation in LLMs.
Usage
ggml_argmax(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor with argmax indices
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 5, 3, 2, 4))
result <- ggml_argmax(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_i32(result) # 1 (0-indexed)
ggml_free(ctx)
Argsort - Get Sorting Indices (Graph)
Description
Returns indices that would sort the tensor rows. Each row is sorted independently.
Usage
ggml_argsort(ctx, a, order = GGML_SORT_ORDER_ASC)
Arguments
ctx |
GGML context |
a |
Input tensor to sort (F32) |
order |
Sort order: GGML_SORT_ORDER_ASC (0) or GGML_SORT_ORDER_DESC (1) |
Value
Tensor of I32 indices that would sort each row
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Create tensor with values to sort
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(3, 1, 4, 1, 5))
# Get indices for ascending sort
indices <- ggml_argsort(ctx, a, GGML_SORT_ORDER_ASC)
graph <- ggml_build_forward_expand(ctx, indices)
ggml_graph_compute(ctx, graph)
result <- ggml_get_i32(indices)
# result: [1, 3, 0, 2, 4] (0-indexed positions for sorted order)
ggml_free(ctx)
Create a Multi-Head Attention Layer Object
Description
Returns a reusable layer object for use with ggml_apply, the
scaled dot-product attention of Attention Is All You Need with
n_heads heads computed in parallel. Applying the same object to
several tensor nodes shares one set of projection weights, which is how an
encoder block is reused across a stack.
Usage
ggml_attention(
d_model,
n_heads = 1L,
causal = FALSE,
bias = TRUE,
rope = FALSE,
rope_base = 10000,
dropout = 0,
name = NULL,
trainable = TRUE
)
Arguments
d_model |
Integer, model width. Must be divisible by |
n_heads |
Integer, number of attention heads (default 1). |
causal |
Logical: mask out positions after the current one, so a query
attends only to keys at or before it (default |
bias |
Logical: add a bias to the output projection (default
|
rope |
Logical: apply rotary position embedding to queries and keys
(default |
rope_base |
Rotation base for |
dropout |
Dropout rate applied to the attention weights during training (default 0, no dropout). |
name |
Optional character name. |
trainable |
Logical; whether weights are updated during training. |
Details
For the pipe-style form that appends attention to a graph
(x |> ggml_layer_attention(...)), see
ggml_layer_attention.
Value
A ggml_layer object.
Shapes
The parent must be a sequence node of shape c(seq_len, d_model) –
the same layout ggml_layer_lstm takes – and the output has
that same shape, so attention blocks stack directly.
Self- and cross-attention
Applied to a single node the layer is self-attention: queries, keys and
values all come from that node. Applied to a list of two nodes,
ggml_apply(list(query, context), attn), it is cross-attention:
queries come from the first, keys and values from the second, which may
have a different sequence length (but must share d_model). This is
the decoder-to-encoder path of a full transformer.
Weights
Four projections W_q, W_k, W_v, W_o, each
d_model x d_model, plus an output bias b_o when
bias = TRUE. Glorot-uniform initialised, zero bias.
See Also
ggml_layer_attention, ggml_apply
Examples
# One attention block shared between two sequences.
attn <- ggml_attention(d_model = 32L, n_heads = 4L)
x1 <- ggml_input(shape = c(10L, 32L))
x2 <- ggml_input(shape = c(10L, 32L))
o1 <- ggml_apply(x1, attn)
o2 <- ggml_apply(x2, attn) # shared weights
stopifnot(identical(o1$layer_id, o2$layer_id))
# Cross-attention: queries from x1, keys/values from a longer context.
ctx <- ggml_input(shape = c(15L, 32L))
o3 <- ggml_apply(list(x1, ctx), ggml_attention(32L, 4L))
Allocate Context Tensors to Backend
Description
Allocates all tensors in a GGML context to a specific backend. Returns a buffer that must be freed when no longer needed.
Usage
ggml_backend_alloc_ctx_tensors(ctx, backend)
Arguments
ctx |
GGML context |
backend |
Backend handle |
Value
Backend buffer object
Backend Buffer Operations
Description
Backend Buffer Operations
Usage
ggml_backend_buffer_clear(buffer, value = 0L)
ggml_backend_buffer_set_usage(buffer, usage)
ggml_backend_buffer_get_usage(buffer)
ggml_backend_buffer_reset(buffer)
ggml_backend_buffer_is_host(buffer)
ggml_backend_buffer_free(buffer)
ggml_backend_buffer_get_size(buffer)
ggml_backend_buffer_name(buffer)
Arguments
buffer |
External pointer to buffer |
value |
Byte value to fill with (default 0) |
usage |
Usage constant (use ggml_backend_buffer_usage_* functions) |
Details
ggml_backend_buffer_set_usage():
Returns: NULL invisibly
ggml_backend_buffer_get_usage():
Returns: Usage constant
ggml_backend_buffer_reset():
Returns: NULL invisibly
ggml_backend_buffer_is_host():
Returns: Logical indicating if buffer is in host memory
ggml_backend_buffer_free(): Frees a backend buffer and all associated memory.
Returns: No return value, called for side effects
ggml_backend_buffer_get_size(): Returns the total size of a backend buffer.
Returns: Size in bytes
ggml_backend_buffer_name(): Returns the name/type of a backend buffer.
Returns: Character string with buffer name
Value
NULL invisibly
See Also
Backend Buffer Usage Constants
Description
Backend Buffer Usage Constants
Usage
ggml_backend_buffer_usage_any()
ggml_backend_buffer_usage_weights()
ggml_backend_buffer_usage_compute()
Details
ggml_backend_buffer_usage_weights():
Returns: Integer constant for weights buffer usage
ggml_backend_buffer_usage_compute():
Returns: Integer constant for compute buffer usage
Value
Integer constant for any buffer usage
See Also
Backend Buffer Type Information
Description
Backend Buffer Type Information
Usage
ggml_backend_buft_name(buft)
ggml_backend_buft_get_alignment(buft)
ggml_backend_buft_get_max_size(buft)
ggml_backend_buft_is_host(buft)
Arguments
buft |
External pointer to buffer type |
Details
ggml_backend_buft_get_alignment():
Returns: Numeric alignment in bytes
ggml_backend_buft_get_max_size():
Returns: Numeric maximum single-allocation size in bytes
ggml_backend_buft_is_host():
Returns: Logical, TRUE if buffers of this type live in host memory
Value
Character name of the buffer type
See Also
Initialize CPU Backend
Description
Creates a new CPU backend instance for graph computation.
Usage
ggml_backend_cpu_init()
Value
Backend pointer
Set CPU Backend Threads
Description
Sets the number of threads for CPU backend computation.
Usage
ggml_backend_cpu_set_n_threads(backend, n_threads)
Arguments
backend |
CPU backend pointer |
n_threads |
Number of threads |
Value
NULL invisibly
Backend Device Buffer Types
Description
Returns the buffer type a device allocates from by default. This is the
buffer type the device itself reports as supported, so it is the natural
argument to ggml_backend_dev_supports_buft.
Usage
ggml_backend_dev_buffer_type(device)
ggml_backend_dev_host_buffer_type(device)
Arguments
device |
External pointer to device |
Details
The buffer type is owned by the backend and lives for as long as the backend does; the returned external pointer has no finalizer and must not be freed.
ggml_backend_dev_host_buffer_type(): Returns the buffer type used for pinned host memory, which allows faster
host-device transfers. Many devices — including the CPU backend — have no
such buffer type and return NULL.
As with ggml_backend_dev_buffer_type, the returned pointer is
owned by the backend and must not be freed.
Returns: External pointer to the host buffer type, or NULL if unavailable
Value
External pointer to the buffer type, or NULL if the device has none
See Also
Examples
dev <- ggml_backend_dev_get(0)
buft <- ggml_backend_dev_buffer_type(dev)
ggml_backend_buft_name(buft)
ggml_backend_dev_supports_buft(dev, buft) # TRUE: a device supports its own
Backend Device Lookup
Description
Backend Device Lookup
Usage
ggml_backend_dev_count()
ggml_backend_dev_get(index)
ggml_backend_dev_by_name(name)
ggml_backend_dev_by_type(type)
Arguments
index |
Device index (0-based) |
name |
Device name |
type |
Device type (use ggml_backend_device_type_* functions) |
Details
ggml_backend_dev_get():
Returns: External pointer to device, or NULL if not found
ggml_backend_dev_by_name():
Returns: External pointer to device, or NULL if not found
ggml_backend_dev_by_type():
Returns: External pointer to first device of given type, or NULL if not found
Value
Number of devices
See Also
Backend Initialization
Description
Backend Initialization
Usage
ggml_backend_dev_init(device, params = NULL)
ggml_backend_init_by_name(name, params = NULL)
ggml_backend_init_by_type(type, params = NULL)
ggml_backend_init_best()
Arguments
device |
External pointer to device |
params |
Optional parameters string |
name |
Backend name (e.g. "CPU", "Vulkan") |
type |
Device type constant |
Details
ggml_backend_dev_init():
Returns: External pointer to backend, or NULL on failure
ggml_backend_init_by_type():
Returns: External pointer to backend, or NULL on failure
ggml_backend_init_best():
Returns: External pointer to backend (GPU if available, otherwise CPU)
Value
External pointer to backend, or NULL on failure
See Also
Backend Device Properties
Description
Backend Device Properties
Usage
ggml_backend_dev_name(device)
ggml_backend_dev_description(device)
ggml_backend_dev_memory(device)
ggml_backend_dev_type(device)
ggml_backend_dev_get_props(device)
Arguments
device |
External pointer to device |
Details
ggml_backend_dev_description():
Returns: Device description
ggml_backend_dev_memory():
Returns: Named numeric vector with 'free' and 'total' memory in bytes
ggml_backend_dev_type():
Returns: Device type constant
ggml_backend_dev_get_props():
Returns: List with name, description, memory_free, memory_total, type, device_id, caps
Value
Device name
See Also
Backend Device Capability Queries
Description
Backend Device Capability Queries
Usage
ggml_backend_dev_supports_op(device, op)
ggml_backend_dev_supports_buft(device, buft)
ggml_backend_dev_offload_op(device, op)
Arguments
device |
External pointer to device |
op |
External pointer to tensor/operation |
buft |
External pointer to buffer type |
Details
ggml_backend_dev_supports_buft():
Returns: Logical indicating support
ggml_backend_dev_offload_op():
Returns: Logical indicating if operation should be offloaded
Value
Logical indicating support
See Also
Register a device
Description
Dynamically registers a new device in the global registry. This is an advanced function for custom backend development.
Usage
ggml_backend_device_register(device)
Arguments
device |
External pointer to device |
Value
NULL invisibly
See Also
Backend Device Type Constants
Description
Backend Device Type Constants
Usage
ggml_backend_device_type_cpu()
ggml_backend_device_type_gpu()
ggml_backend_device_type_igpu()
ggml_backend_device_type_accel()
Details
ggml_backend_device_type_gpu():
Returns: Integer constant for GPU device type
ggml_backend_device_type_igpu():
Returns: Integer constant for integrated GPU device type
ggml_backend_device_type_accel():
Returns: Integer constant for accelerator device type (e.g. BLAS, AMX)
Value
Integer constant for CPU device type
See Also
Backend Events
Description
Backend Events
Usage
ggml_backend_event_new(device)
ggml_backend_event_free(event)
ggml_backend_event_record(event, backend)
ggml_backend_event_synchronize(event)
ggml_backend_event_wait(backend, event)
Arguments
device |
External pointer to device |
event |
External pointer to event |
backend |
External pointer to backend |
Details
ggml_backend_event_free():
Returns: NULL invisibly
ggml_backend_event_record():
Returns: NULL invisibly
ggml_backend_event_synchronize():
Returns: NULL invisibly
ggml_backend_event_wait():
Returns: NULL invisibly
Value
External pointer to event, or NULL on failure
See Also
Free Backend
Description
Releases resources associated with a backend.
Usage
ggml_backend_free(backend)
Arguments
backend |
Backend pointer |
Value
NULL invisibly
Get device from backend
Description
Get device from backend
Usage
ggml_backend_get_device(backend)
Arguments
backend |
External pointer to backend |
Value
External pointer to device
See Also
Compute Graph with Backend
Description
Executes computation graph using specified backend.
Usage
ggml_backend_graph_compute(backend, graph)
Arguments
backend |
Backend pointer |
graph |
Graph pointer |
Value
Status code (0 = success)
Compute graph asynchronously
Description
Starts graph computation without blocking. Use ggml_backend_synchronize() to wait for completion.
Usage
ggml_backend_graph_compute_async(backend, graph)
Arguments
backend |
External pointer to backend |
graph |
External pointer to computation graph |
Value
Integer status code (0 = success)
See Also
Examples
cpu <- ggml_backend_cpu_init()
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
b <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, b)
ggml_set_f32(a, rnorm(100))
# Start async computation
status <- ggml_backend_graph_compute_async(cpu, graph)
# Do other work while computation runs...
ggml_backend_synchronize(cpu)
ggml_backend_free(cpu)
ggml_free(ctx)
Backend Graph Plans
Description
Backend Graph Plans
Usage
ggml_backend_graph_plan_create(backend, graph)
ggml_backend_graph_plan_free(backend, plan)
ggml_backend_graph_plan_compute(backend, plan)
Arguments
backend |
External pointer to backend |
graph |
External pointer to computation graph |
plan |
External pointer to plan |
Details
ggml_backend_graph_plan_free():
Returns: NULL invisibly
ggml_backend_graph_plan_compute():
Returns: Status code (0 = success)
Value
External pointer to plan, or NULL on failure
See Also
Dynamic Backend Loading
Description
Dynamic Backend Loading
Usage
ggml_backend_load(path)
ggml_backend_unload(reg)
ggml_backend_load_all()
Arguments
path |
Path to dynamic library |
reg |
External pointer to registry |
Details
ggml_backend_unload():
Returns: NULL invisibly
ggml_backend_load_all():
Returns: NULL invisibly
Value
External pointer to registry, or NULL on failure
See Also
Create a Meta Backend Device
Description
Creates a "meta" device that wraps multiple "simple" backend devices for
tensor parallelism. Each tensor is split across the wrapped devices
according to the result of split_fn, which is called by ggml when
weight buffers are allocated.
Usage
ggml_backend_meta_device(devs, split_fn, env = environment(split_fn))
Arguments
devs |
A list of |
split_fn |
A function |
env |
An environment in which to evaluate |
Details
The split function is invoked with two arguments:
- tensor_info
a named list with fields
name(character),type(integer ggml_type enum),ne(numeric vector of dimensions),op(integer op enum),flags(integer).- n_devs
the number of simple devices wrapped by the meta backend.
It must return a named list with:
- axis
integer; one of 0..3 to split along a tensor axis, 10 for
MIRRORED(full copy on each device), 11 forPARTIAL(each device has a partial sum), or 98/99 forNONE/UNKNOWN.- ne
integer or numeric vector of length
n_segments * n_devsgiving the per-segment, per-device slice size along the split axis.- n_segments
integer; usually 1, larger for fused tensors like QKV.
If split_fn errors or returns an unparseable result, the meta
backend silently falls back to MIRRORED for that tensor and stops
calling the callback (sticky error). This is intentional: a misbehaving
callback would otherwise spray errors for every tensor in the model.
Note: with a single device this is a degenerate (no-op) configuration — useful for testing but provides no parallelism benefit. The feature is experimental and the API may change.
Value
External pointer to the meta ggml_backend_dev_t.
See Also
Backend Multi-buffers
Description
Creates a buffer that combines multiple backend buffers into one. Useful for managing memory across different backends.
Usage
ggml_backend_multi_buffer_alloc_buffer(buffers)
ggml_backend_buffer_is_multi_buffer(buffer)
ggml_backend_multi_buffer_set_usage(buffer, usage)
Arguments
buffers |
List of backend buffer external pointers |
buffer |
External pointer to buffer |
usage |
Usage constant (from ggml_backend_buffer_usage_*) |
Details
ggml_backend_buffer_is_multi_buffer():
Returns: Logical indicating if buffer is a multi-buffer
ggml_backend_multi_buffer_set_usage():
Returns: NULL invisibly
Value
External pointer to multi-buffer
See Also
Examples
cpu <- ggml_backend_cpu_init()
ctx1 <- ggml_init(1024, no_alloc = TRUE)
ctx2 <- ggml_init(2048, no_alloc = TRUE)
a <- ggml_new_tensor_1d(ctx1, GGML_TYPE_F32, 10)
b <- ggml_new_tensor_1d(ctx2, GGML_TYPE_F32, 20)
buf1 <- ggml_backend_alloc_ctx_tensors(ctx1, cpu)
buf2 <- ggml_backend_alloc_ctx_tensors(ctx2, cpu)
multi <- ggml_backend_multi_buffer_alloc_buffer(list(buf1, buf2))
ggml_backend_buffer_free(multi)
ggml_backend_free(cpu)
ggml_free(ctx1)
ggml_free(ctx2)
Get Backend Name
Description
Returns the name of the backend (e.g., "CPU").
Usage
ggml_backend_name(backend)
Arguments
backend |
Backend pointer |
Value
Character string name
Backend Registry
Description
Backend Registry
Usage
ggml_backend_reg_count()
ggml_backend_reg_get(index)
ggml_backend_reg_by_name(name)
ggml_backend_reg_name(reg)
ggml_backend_reg_dev_count(reg)
ggml_backend_reg_dev_get(reg, index)
Arguments
index |
Registry index (0-based) |
name |
Registry name |
reg |
External pointer to registry |
Details
ggml_backend_reg_get():
Returns: External pointer to registry, or NULL if not found
ggml_backend_reg_by_name():
Returns: External pointer to registry, or NULL if not found
ggml_backend_reg_name():
Returns: Registry name
ggml_backend_reg_dev_count():
Returns: Number of devices
ggml_backend_reg_dev_get():
Returns: External pointer to device
Value
Number of registered backends
See Also
Register a backend
Description
Dynamically registers a new backend in the global registry. This is an advanced function for custom backend development.
Usage
ggml_backend_register(reg)
Arguments
reg |
External pointer to backend registry |
Value
NULL invisibly
See Also
Allocate graph on scheduler
Description
Allocates memory for a graph across the scheduler's backends. Must be called before computing the graph.
Usage
ggml_backend_sched_alloc_graph(sched, graph)
Arguments
sched |
Scheduler pointer |
graph |
Graph pointer |
Value
Logical indicating success
Free backend scheduler
Description
Releases resources associated with the backend scheduler.
Usage
ggml_backend_sched_free(sched)
Arguments
sched |
Scheduler pointer from ggml_backend_sched_new() |
Value
NULL (invisible)
Examples
cpu <- ggml_backend_cpu_init()
sched <- ggml_backend_sched_new(list(cpu))
ggml_backend_sched_free(sched)
ggml_backend_free(cpu)
Scheduler Information
Description
Returns the number of backends managed by the scheduler.
Usage
ggml_backend_sched_get_n_backends(sched)
ggml_backend_sched_get_backend(sched, index = 0L)
ggml_backend_sched_get_n_splits(sched)
ggml_backend_sched_get_n_copies(sched)
Arguments
sched |
Scheduler pointer |
index |
Backend index (0-based) |
Details
ggml_backend_sched_get_backend(): Returns a specific backend from the scheduler by index.
Returns: Backend pointer
ggml_backend_sched_get_n_splits(): Returns the number of splits in the last computed graph.
Higher numbers indicate more distribution across backends.
Returns: Integer count of splits
ggml_backend_sched_get_n_copies(): Returns the number of tensor copies made in the last computed graph.
Copies occur when data needs to be transferred between backends.
Returns: Integer count of copies
Value
Integer count of backends
Scheduler Graph Compute
Description
Computes a graph by distributing work across multiple backends. This is the main function for multi-GPU computation.
Usage
ggml_backend_sched_graph_compute(sched, graph)
ggml_backend_sched_graph_compute_async(sched, graph)
Arguments
sched |
Scheduler pointer |
graph |
Graph pointer |
Details
ggml_backend_sched_graph_compute_async(): Computes a graph asynchronously across backends.
Use ggml_backend_sched_synchronize() to wait for completion.
Returns: Status code (0 = success)
Value
Status code (0 = success)
Examples
# Multi-GPU example
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) {
gpu1 <- ggml_vulkan_init(0)
gpu2 <- ggml_vulkan_init(1)
sched <- ggml_backend_sched_new(list(gpu1, gpu2))
ctx <- ggml_init(64 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10000)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10000)
ggml_set_f32(a, rnorm(10000))
ggml_set_f32(b, rnorm(10000))
c <- ggml_add(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, c)
# Reserve memory
ggml_backend_sched_reserve(sched, graph)
# Compute using both GPUs
ggml_backend_sched_graph_compute(sched, graph)
result <- ggml_get_f32(c)
cat("Splits:", ggml_backend_sched_get_n_splits(sched), "\n")
cat("Copies:", ggml_backend_sched_get_n_copies(sched), "\n")
ggml_free(ctx)
ggml_backend_sched_free(sched)
ggml_vulkan_free(gpu1)
ggml_vulkan_free(gpu2)
}
Create a new backend scheduler
Description
Creates a scheduler that can distribute computation across multiple backends (GPUs, CPU). A CPU backend is automatically added as a fallback. Backends with lower index have higher priority.
Usage
ggml_backend_sched_new(backends, parallel = TRUE, graph_size = 2048)
Arguments
backends |
List of backend pointers (from ggml_vulkan_init() or ggml_backend_cpu_init()). Note: A CPU backend is automatically added, so you only need to specify GPU backends. |
parallel |
Logical, whether to run backends in parallel (default: TRUE) |
graph_size |
Expected maximum graph size (default: 2048) |
Value
Scheduler pointer
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) {
# Create two GPU backends (CPU is added automatically)
gpu1 <- ggml_vulkan_init(0)
gpu2 <- ggml_vulkan_init(1)
# Create scheduler with both GPUs + CPU (automatic)
sched <- ggml_backend_sched_new(list(gpu1, gpu2), parallel = TRUE)
# The scheduler now has 3 backends: GPU1, GPU2, CPU
cat("Backends:", ggml_backend_sched_get_n_backends(sched), "\\n")
# Use scheduler...
# Cleanup
ggml_backend_sched_free(sched)
ggml_vulkan_free(gpu1)
ggml_vulkan_free(gpu2)
}
Reserve memory for scheduler
Description
Pre-allocates memory based on a measurement graph. This should be called before using the scheduler to compute graphs.
Usage
ggml_backend_sched_reserve(sched, graph)
Arguments
sched |
Scheduler pointer |
graph |
Graph pointer to measure memory requirements |
Value
Logical indicating success
Examples
cpu <- ggml_backend_cpu_init()
sched <- ggml_backend_sched_new(list(cpu))
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1000)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1000)
c <- ggml_add(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, c)
ggml_backend_sched_reserve(sched, graph)
ggml_backend_sched_free(sched)
ggml_backend_free(cpu)
ggml_free(ctx)
Reset scheduler
Description
Resets the scheduler, deallocating all tensors. Must be called before changing node backends or allocating a new graph.
Usage
ggml_backend_sched_reset(sched)
Arguments
sched |
Scheduler pointer |
Value
NULL (invisible)
Scheduler Tensor Backend Assignment
Description
Manually assigns a specific tensor to run on a specific backend. This overrides automatic scheduling.
Usage
ggml_backend_sched_set_tensor_backend(sched, tensor, backend)
ggml_backend_sched_get_tensor_backend(sched, tensor)
Arguments
sched |
Scheduler pointer |
tensor |
Tensor pointer |
backend |
Backend pointer to assign tensor to |
Details
ggml_backend_sched_get_tensor_backend(): Returns which backend a tensor is assigned to.
Assignments are only meaningful once the scheduler has actually placed the
graph — that is, after ggml_backend_sched_graph_compute.
After ggml_backend_sched_reserve alone, or for a tensor from a
different graph, the assignment may not exist and this returns NULL.
Returns: Backend pointer, or NULL if the tensor is not assigned to one
Value
NULL (invisible)
Synchronize scheduler
Description
Waits for all asynchronous operations to complete.
Usage
ggml_backend_sched_synchronize(sched)
Arguments
sched |
Scheduler pointer |
Value
NULL (invisible)
Trace Every Node a Scheduler Computes
Description
Turns on a per-node trace: for each node the scheduler evaluates, its name, op, shape and value summary (sum/min/max) are printed to stderr. Nodes are identified by NAME rather than index, so traces from two backends can be diffed even when their graphs enumerate nodes differently – which is what makes this useful for finding where a CPU and a GPU run diverge.
Usage
ggml_backend_sched_trace(sched, enable = TRUE)
Arguments
sched |
Backend scheduler |
enable |
Logical; turn tracing on or off |
Details
Off by default; leaving it on makes computation much slower.
Value
NULL, invisibly
Synchronize backend
Description
Synchronize backend
Usage
ggml_backend_synchronize(backend)
Arguments
backend |
External pointer to backend |
Value
NULL invisibly
See Also
Backend Tensor Get and Sync
Description
Gets tensor data from a backend with synchronization.
Usage
ggml_backend_tensor_get_and_sync(backend, tensor, offset = 0, size)
Arguments
backend |
Backend pointer (or NULL for CPU) |
tensor |
Tensor pointer |
offset |
Byte offset (default 0) |
size |
Number of bytes to read |
Value
Raw vector with tensor data
Get First Float from Backend Tensor
Description
Reads the first f32 element from a backend tensor.
Usage
ggml_backend_tensor_get_f32_first(tensor)
Arguments
tensor |
Tensor pointer |
Value
Float value
Asynchronous Tensor Transfers
Description
Asynchronous Tensor Transfers
Usage
ggml_backend_tensor_set_async(backend, tensor, data, offset = 0, size = NULL)
ggml_backend_tensor_get_async(backend, tensor, offset = 0, size)
ggml_backend_tensor_copy_async(backend_src, backend_dst, src, dst)
Arguments
backend |
External pointer to backend |
tensor |
External pointer to tensor |
data |
Numeric or integer vector |
offset |
Byte offset (default 0) |
size |
Number of bytes to copy |
backend_src |
Source backend |
backend_dst |
Destination backend |
src |
Source tensor |
dst |
Destination tensor |
Details
ggml_backend_tensor_get_async():
Returns: Double vector of ceiling(size / 8) elements carrying the raw bytes (reinterpret with writeBin/readBin); bytes past size are zero. Filled only after ggml_backend_synchronize().
ggml_backend_tensor_copy_async():
Returns: NULL invisibly
Value
NULL invisibly
See Also
Backend Tensor Data Access
Description
Sets tensor data using the backend API. This works with tensors allocated on any backend, not just CPU.
Usage
ggml_backend_tensor_set_data(tensor, data, offset = 0)
ggml_backend_tensor_get_data(tensor, offset = 0, n_elements = NULL)
Arguments
tensor |
Tensor pointer |
data |
R vector with data to set |
offset |
Byte offset (default: 0) |
n_elements |
Number of elements to retrieve (NULL for all) |
Details
Integer and logical data are accepted for float tensors and
converted element by element to float (an integer NA becomes
NaN). An I32 tensor (token indices, say) takes
integers or whole-valued doubles; NA, fractions and values outside
the 32-bit range stop with an error rather than being truncated.
An F64 tensor takes doubles as they are. An I64 tensor
takes integers or whole-valued doubles within \pm 2^{53}
(where doubles are exact); anything else stops with an error.
ggml_backend_tensor_get_data(): Gets tensor data using the backend API. This works with tensors
allocated on any backend, not just CPU.
Returns: R vector with tensor data: integer for I32, double otherwise. I64 values come back as doubles, with a warning when any exceeds 2^{53} in magnitude (precision is lost there).
Value
No return value, called for side effects
Create a Batch Normalization Layer Object
Description
Create a Batch Normalization Layer Object
Usage
ggml_batch_norm(eps = 1e-05, name = NULL, trainable = TRUE)
Arguments
eps |
Small constant for numerical stability (default 1e-5). |
name |
Optional character name. |
trainable |
Logical. |
Value
A ggml_layer object.
Add the Backward Pass to a Computation Graph
Description
Extends graph in place with the nodes that compute gradients of its
output with respect to every tensor marked by ggml_set_param.
After computing the graph, read a gradient with
ggml_graph_get_grad.
Usage
ggml_build_backward_expand(ctx, graph)
Arguments
ctx |
GGML context |
graph |
Graph from |
Details
The graph must come from ggml_build_forward_expand_grads:
ggml_build_forward_expand produces one without gradient
storage, and that is rejected rather than allowed to abort inside ggml.
Value
NULL, invisibly; graph is modified in place
The full sequence
Every step is required; skipping one gives either an error or – for the last – gradients that are silently all zero.
-
ggml_set_paramon each tensor to differentiate build the forward ops
-
ggml_set_losson the output -
ggml_build_backward_expand() -
ggml_graph_reset– seedsd(loss)/d(loss) = 1 -
ggml_graph_get_gradto read a gradient
Which ops are differentiable
Gradients flow only through operations that implement a backward pass. Most
do, but not all – ggml_clamp is a notable exception, and among
the state-space ops only ggml_ssm_conv is differentiable.
See Also
ggml_build_forward_expand_grads,
ggml_graph_get_grad, ggml_set_param
Build Forward Computation Graph
Description
Creates a computation graph by expanding backwards from the output tensor
Usage
ggml_build_forward_expand(ctx, tensor)
Arguments
ctx |
GGML context |
tensor |
Output tensor of the computation |
Value
Graph object (external pointer)
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_set_f32(a, 1:10)
ggml_set_f32(b, 11:20)
c <- ggml_add(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, c)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(c)
ggml_free(ctx)
Build a Forward Graph That Can Be Differentiated
Description
Like ggml_build_forward_expand, but the graph also reserves
storage for gradients, which is what ggml_build_backward_expand
requires. A graph from the plain forward builder carries none, so passing it
to the backward builder is an error.
Usage
ggml_build_forward_expand_grads(ctx, tensor, graph_size = 2048L)
Arguments
ctx |
GGML context |
tensor |
Output tensor of the computation (usually a scalar loss) |
graph_size |
Node capacity of the graph (default 2048) |
Details
Mark the tensors to differentiate with respect to using
ggml_set_param before building the graph.
Value
Graph object (external pointer)
Graph size
graph_size is the node capacity. The backward pass appends nodes to
the forward ones, so it needs more room than a forward-only graph: roughly
three times the forward node count is a safe starting point. Too small a
value fails while the graph is being built, not silently.
See Also
ggml_build_backward_expand,
ggml_graph_get_grad
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
ggml_set_f32(b, c(5, 6, 7, 8))
ggml_set_param(a)
loss <- ggml_sum(ctx, ggml_mul(ctx, a, b))
ggml_set_loss(loss)
graph <- ggml_build_forward_expand_grads(ctx, loss)
ggml_build_backward_expand(ctx, graph)
ggml_graph_reset(graph) # seeds d(loss)/d(loss) = 1
ggml_graph_compute(ctx, graph)
# d(sum(a*b))/da == b
ggml_get_f32(ggml_graph_get_grad(graph, a))
ggml_free(ctx)
Early stopping callback
Description
Stops training when the monitored metric does not improve.
Usage
ggml_callback_early_stopping(
monitor = "val_loss",
patience = 5,
min_delta = 0,
mode = "auto"
)
Arguments
monitor |
Metric to monitor: "val_loss", "val_accuracy", "train_loss", "train_accuracy" |
patience |
Number of epochs with no improvement before stopping |
min_delta |
Minimum change to qualify as improvement |
mode |
"min" (lower is better) or "max" (higher is better). "auto" infers from monitor name. |
Value
List with on_train_begin and on_epoch_end functions; the state resets at the start of every fit
See Also
Other callbacks:
ggml_schedule_cosine_decay(),
ggml_schedule_reduce_on_plateau(),
ggml_schedule_step_decay()
Check If Tensor Can Be Repeated
Description
Check if tensor a can be repeated (broadcast) to match tensor b.
Used for broadcasting operations.
Usage
ggml_can_repeat(a, b)
Arguments
a |
Source tensor (smaller) |
b |
Target tensor (larger or same size) |
Value
Logical indicating if a can be repeated to match b
See Also
Other tensor_layout:
ggml_are_same_stride(),
ggml_count_equal()
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8)
ggml_can_repeat(a, b) # TRUE - a can broadcast along dim 1
ggml_free(ctx)
Cast a Tensor to Another Type
Description
Returns a tensor with the same shape as a but of the given type.
Unlike ggml_cpy there is no need to allocate the destination
tensor first. Some ggml operations require a specific type – for example
ggml_conv_1d builds its im2col in F16 and so needs an F16
kernel – and this is the direct way to supply one.
Usage
ggml_cast(ctx, a, type)
Arguments
ctx |
GGML context |
a |
Source tensor |
type |
Target type constant (e.g. |
Value
Tensor of the requested type with a's shape
See Also
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
a16 <- ggml_cast(ctx, a, GGML_TYPE_F16)
ggml_free(ctx)
Numeric Type Conversion
Description
Converts values numerically (truncating or rounding) rather than
reinterpreting the bits as ggml_cast does. Supports
F32<->I32, F32<->I16, F32<->I8, F16<->F32 and BF16<->F32.
Usage
ggml_cast_numeric(ctx, a, type)
Arguments
ctx |
GGML context |
a |
Source tensor |
type |
Target type constant (e.g. |
Value
Tensor of the requested type with a's shape
See Also
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
i <- ggml_cast_numeric(ctx, a, GGML_TYPE_I32)
ggml_free(ctx)
Ceiling (Graph)
Description
Creates a graph node for element-wise ceiling: ceil(x)
Usage
ggml_ceil_inplace(ctx, a)
ggml_ceil(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_ceil_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the ceil operation
Clamp (Graph)
Description
Creates a graph node for clamping values to a range: clamp(x, min, max)
Usage
ggml_clamp(ctx, a, min_val, max_val)
ggml_clamp_inplace(ctx, a, min_val, max_val)
Arguments
ctx |
GGML context |
a |
Input tensor |
min_val |
Minimum value |
max_val |
Maximum value |
Details
ggml_clamp_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor with values clamped to [min_val, max_val]
Inference only
GGML_OP_CLAMP has no backward pass. A graph that needs a gradient
through it therefore cannot be differentiated: the backward pass aborts with
"unsupported ggml op for backward pass: CLAMP" rather than returning
an error. This matters inside
ggml_layer_custom, whose gradients flow only through
operations that implement a backward pass.
For a trainable equivalent, express the clamp with operations that do:
min(x, hi) is x - ggml_relu(ctx, x - hi) and max(x, lo)
is lo + ggml_relu(ctx, x - lo).
See Also
ggml_relu for the differentiable formulation above.
Column to 1D Signal (Graph)
Description
Inverse of a 1D im2col: scatter-adds the columns of a GEMM result back into a 1D signal, as in a transposed convolution computed by matrix multiply.
Usage
ggml_col2im_1d(ctx, a, s0, oc, p0 = 0L)
Arguments
ctx |
GGML context |
a |
Column tensor |
s0 |
Stride |
oc |
Output channels |
p0 |
Padding cropped from both ends |
Value
Tensor [T_out, oc], T_out = (T_in - 1) * s0 + K - 2 * p0
Compile a Sequential Model
Description
Configures the model for training: infers shapes, creates backend. Weight tensors are created at training time when batch_size is known.
Usage
## S3 method for class 'ggml_functional_model'
ggml_compile(
model,
optimizer = "adam",
loss = "categorical_crossentropy",
metrics = c("accuracy"),
backend = "auto",
loss_weights = NULL,
weight_decay = 0,
predict_precision = "default"
)
ggml_compile(
model,
optimizer = "adam",
loss = "categorical_crossentropy",
metrics = c("accuracy"),
backend = "auto",
loss_weights = NULL,
weight_decay = 0,
predict_precision = "default"
)
## S3 method for class 'ggml_sequential_model'
ggml_compile(
model,
optimizer = "adam",
loss = "categorical_crossentropy",
metrics = c("accuracy"),
backend = "auto",
loss_weights = NULL,
weight_decay = 0,
predict_precision = "default"
)
Arguments
model |
A ggml_sequential_model object |
optimizer |
Optimizer name: "adam" or "sgd" |
loss |
Loss function name: |
metrics |
Character vector of metrics to report from
|
backend |
Backend to use: "auto" (GPU if available, else CPU), "cpu", or "vulkan" |
loss_weights |
Optional per-output weights for a multi-output model;
the optimized total is |
weight_decay |
Decoupled weight decay for the optimizer, in |
predict_precision |
Matrix-multiply precision for prediction:
|
Value
The compiled model (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_max_pooling_2d(c(2, 2)) |>
ggml_layer_flatten() |>
ggml_layer_dense(10, activation = "softmax")
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
Concatenate Tensors (Graph)
Description
Concatenates two tensors along a specified dimension. CRITICAL for KV-cache operations in transformers.
Usage
ggml_concat(ctx, a, b, dim = 0)
Arguments
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (must match a in all dimensions except the concat dim) |
dim |
Dimension along which to concatenate (0-3) |
Value
Concatenated tensor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 3)
b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 2)
ggml_set_f32(a, rnorm(12))
ggml_set_f32(b, rnorm(8))
# Concatenate along dimension 1: result is 4x5
c <- ggml_concat(ctx, a, b, 1)
graph <- ggml_build_forward_expand(ctx, c)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Make Contiguous (Graph)
Description
Makes a tensor contiguous in memory. Required after permute/transpose before some operations.
Usage
ggml_cont(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Contiguous tensor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4)
ggml_set_f32(a, 1:12)
transposed <- ggml_transpose(ctx, a)
contiguous <- ggml_cont(ctx, transposed)
ggml_free(ctx)
1D Convolution (Graph)
Description
Applies 1D convolution to input data.
Usage
ggml_conv_1d(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)
ggml_conv_1d_dw(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)
Arguments
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride (default 1) |
p0 |
Padding (default 0) |
d0 |
Dilation (default 1) |
Details
ggml_conv_1d_dw(): Applies depthwise 1D convolution: each input channel is convolved with its
own kernel.
Returns: Convolved tensor
Value
Convolved tensor
Create a Reusable Conv1D Layer Object
Description
Builds a layer object to be applied with ggml_apply,
which is how a single set of convolution weights is shared between several
inputs. This is the convolution counterpart of ggml_dense.
Usage
ggml_conv_1d_layer(
filters,
kernel_size,
activation = NULL,
strides = 1L,
padding = "valid",
name = NULL,
trainable = TRUE
)
Arguments
filters |
Number of output filters. |
kernel_size |
Integer kernel size. |
activation |
Activation function name or NULL. |
strides |
Integer stride (default 1). |
padding |
|
name |
Optional character name. |
trainable |
Logical. |
Details
For the ordinary pipe-style form that appends a conv layer to a graph
(x |> ggml_layer_conv_1d(...)), see ggml_layer_conv_1d.
Value
A ggml_layer object.
See Also
Examples
# One convolution shared between two sequence inputs.
shared <- ggml_conv_1d_layer(filters = 4L, kernel_size = 3L)
x1 <- ggml_input(shape = c(16L, 2L))
x2 <- ggml_input(shape = c(16L, 2L))
o1 <- ggml_apply(x1, shared)
o2 <- ggml_apply(x2, shared)
stopifnot(identical(o1$layer_id, o2$layer_id)) # same weights
2D Convolution (Graph)
Description
Applies 2D convolution to input data.
Usage
ggml_conv_2d(ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L)
ggml_conv_2d_direct(
ctx,
a,
b,
s0 = 1L,
s1 = 1L,
p0 = 0L,
p1 = 0L,
d0 = 1L,
d1 = 1L
)
ggml_conv_2d_dw(
ctx,
a,
b,
s0 = 1L,
s1 = 1L,
p0 = 0L,
p1 = 0L,
d0 = 1L,
d1 = 1L
)
ggml_conv_2d_dw_direct(
ctx,
a,
b,
s0 = 1L,
s1 = 1L,
p0 = 0L,
p1 = 0L,
d0 = 1L,
d1 = 1L
)
Arguments
ctx |
GGML context |
a |
Convolution kernel tensor [KW, KH, IC, OC] |
b |
Input data tensor [W, H, C, N] |
s0 |
Stride dimension 0 (default 1) |
s1 |
Stride dimension 1 (default 1) |
p0 |
Padding dimension 0 (default 0) |
p1 |
Padding dimension 1 (default 0) |
d0 |
Dilation dimension 0 (default 1) |
d1 |
Dilation dimension 1 (default 1) |
Details
ggml_conv_2d_direct(): Applies 2D convolution using the direct algorithm (no im2col).
Returns: Convolved tensor
ggml_conv_2d_dw(): Applies depthwise 2D convolution: each input channel is convolved with its
own kernel. Uses the im2col-based path.
Returns: Convolved tensor
ggml_conv_2d_dw_direct(): Direct depthwise 2D convolution without an explicit im2col intermediate.
Returns: Convolved tensor
Value
Convolved tensor
Create a Reusable Conv2D Layer Object
Description
Builds a layer object to be applied with ggml_apply,
which is how a single set of convolution weights is shared between several
inputs. This is the convolution counterpart of ggml_dense.
Usage
ggml_conv_2d_layer(
filters,
kernel_size,
activation = NULL,
strides = c(1L, 1L),
padding = "valid",
name = NULL,
trainable = TRUE
)
Arguments
filters |
Number of output filters. |
kernel_size |
Integer or length-2 integer vector. |
activation |
Activation function name or NULL. |
strides |
Integer or length-2 integer vector (default 1). |
padding |
|
name |
Optional character name. |
trainable |
Logical. |
Details
For the ordinary pipe-style form that appends a conv layer to a graph
(x |> ggml_layer_conv_2d(...)), see ggml_layer_conv_2d.
Value
A ggml_layer object.
See Also
Examples
# One convolution shared between two inputs.
shared <- ggml_conv_2d_layer(filters = 4L, kernel_size = 3L, activation = "relu")
x1 <- ggml_input(shape = c(8L, 8L, 1L))
x2 <- ggml_input(shape = c(8L, 8L, 1L))
o1 <- ggml_apply(x1, shared)
o2 <- ggml_apply(x2, shared)
stopifnot(identical(o1$layer_id, o2$layer_id)) # same weights
3D Convolution (Graph)
Description
3D convolution built from ggml_im2col_3d and a matrix
multiply. The kernel type sets the im2col type, so an F16 kernel computes
in F16.
Usage
ggml_conv_3d(ctx, a, b, IC, s = 1L, p = 0L, d = 1L)
ggml_conv_3d_direct(
ctx,
a,
b,
s = 1L,
p = 0L,
d = 1L,
n_channels,
n_batch,
n_channels_out
)
Arguments
ctx |
GGML context |
a |
Kernel tensor |
b |
Input tensor |
IC |
Number of input channels |
s |
Stride: one value for all axes or |
p |
Padding, same form as |
d |
Dilation, same form as |
n_channels |
Number of input channels (IC) |
n_batch |
Batch size (N) |
n_channels_out |
Number of output channels (OC) |
Details
ggml_conv_3d_direct(): 3D convolution as a single op (GGML_OP_CONV_3D), without the
im2col buffer of ggml_conv_3d. Same layout and result.
Returns: F32 tensor [OW, OH, OD, OC * N]
Value
Tensor [OW, OH, OD, OC * N]
See Also
Transposed 1D Convolution (Graph)
Description
Applies transposed 1D convolution (deconvolution) to input data.
Usage
ggml_conv_transpose_1d(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)
Arguments
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride (default 1) |
p0 |
Padding (default 0) |
d0 |
Dilation (default 1) |
Value
Transposed convolved tensor
Transposed 2D Convolution, zero padding (Graph)
Description
Applies transposed 2D convolution (deconvolution) with zero padding.
Usage
ggml_conv_transpose_2d_p0(ctx, a, b, stride = 1L)
Arguments
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
stride |
Stride (default 1) |
Value
Transposed convolved tensor
Cosine (Graph)
Description
Creates a graph node for element-wise cosine: cos(x)
Usage
ggml_cos(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the cos operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(0, pi/3, pi/2, pi))
result <- ggml_cos(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [1, 0.5, 0, -1]
ggml_free(ctx)
Count Equal Elements (Graph)
Description
Creates a graph node that counts equal elements between two tensors. Useful for accuracy computation.
Usage
ggml_count_equal(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Value
Tensor containing the count of equal elements
See Also
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat()
Examples
ctx <- ggml_init(16 * 1024 * 1024)
pred <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 100)
labels <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 100)
# ... set values ...
correct <- ggml_count_equal(ctx, pred, labels)
graph <- ggml_build_forward_expand(ctx, correct)
ggml_graph_compute(ctx, graph)
# correct now contains count of matching elements
ggml_free(ctx)
Element-wise Addition (CPU Direct)
Description
Performs element-wise addition of two tensors using direct CPU computation. Returns the result as an R numeric vector. Does NOT use computation graphs.
Usage
ggml_cpu_add(a, b)
Arguments
a |
First tensor (must be F32 type) |
b |
Second tensor (must be F32 type, same size as a) |
Value
Numeric vector containing the element-wise sum
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
ggml_set_f32(b, c(5, 4, 3, 2, 1))
ggml_cpu_add(a, b)
ggml_free(ctx)
CPU Feature Detection
Description
ggml_cpu_features() returns a named list of all CPU feature
detection results. Useful for diagnostics and optimizing computation.
Usage
ggml_cpu_has_sse3()
ggml_cpu_has_ssse3()
ggml_cpu_has_avx()
ggml_cpu_has_avx_vnni()
ggml_cpu_has_avx2()
ggml_cpu_has_bmi2()
ggml_cpu_has_f16c()
ggml_cpu_has_fma()
ggml_cpu_has_avx512()
ggml_cpu_has_avx512_vbmi()
ggml_cpu_has_avx512_vnni()
ggml_cpu_has_avx512_bf16()
ggml_cpu_has_amx_int8()
ggml_cpu_has_neon()
ggml_cpu_has_arm_fma()
ggml_cpu_has_fp16_va()
ggml_cpu_has_dotprod()
ggml_cpu_has_matmul_int8()
ggml_cpu_has_sve()
ggml_cpu_get_sve_cnt()
ggml_cpu_has_sme()
ggml_cpu_has_sme2()
ggml_cpu_has_riscv_v()
ggml_cpu_get_rvv_vlen()
ggml_cpu_has_vsx()
ggml_cpu_has_vxe()
ggml_cpu_has_wasm_simd()
ggml_cpu_has_llamafile()
ggml_cpu_features()
Details
Each ggml_cpu_has_*() function checks a single CPU feature:
- x86
SSE3, SSSE3, AVX, AVX-VNNI, AVX2 (256-bit SIMD), BMI2, F16C, FMA, AVX-512, AVX-512 VBMI / VNNI / BF16, AMX-INT8.
- ARM
NEON, ARM FMA, FP16 vector arithmetic, dot product, INT8 matrix multiply, SVE, SME, SME2.
- Other
RISC-V Vector, PowerPC VSX, IBM z/Architecture VXE, WebAssembly SIMD, llamafile optimizations.
ggml_cpu_get_sve_cnt() and ggml_cpu_get_rvv_vlen() return
the SVE / RISC-V vector length in bytes (0 if not supported).
Value
ggml_cpu_features(): named list with feature names and
values. ggml_cpu_has_*(): logical. ggml_cpu_get_*():
integer vector length in bytes.
Examples
ggml_cpu_has_avx2()
features <- ggml_cpu_features()
print(features)
# On typical x86-64: sse3=TRUE, avx=TRUE, avx2=TRUE, ...
Element-wise Multiplication (CPU Direct)
Description
Performs element-wise multiplication of two tensors using direct CPU computation. Returns the result as an R numeric vector. Does NOT use computation graphs.
Usage
ggml_cpu_mul(a, b)
Arguments
a |
First tensor (must be F32 type) |
b |
Second tensor (must be F32 type, same size as a) |
Value
Numeric vector containing the element-wise product
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
ggml_set_f32(b, c(2, 2, 2, 2, 2))
ggml_cpu_mul(a, b)
ggml_free(ctx)
Copy Tensor with Type Conversion (Graph)
Description
Copies tensor a into tensor b, performing type conversion if needed. The tensors must have the same number of elements. CRITICAL for type casting operations (e.g., F32 to F16).
Usage
ggml_cpy(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Source tensor |
b |
Destination tensor (defines output type and shape) |
Value
Tensor representing the copy operation (returns b with a's data)
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Create F32 tensor
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
ggml_set_f32(a, rnorm(100))
# Create F16 tensor for output
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F16, 100)
# Copy with F32 -> F16 conversion
result <- ggml_cpy(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Cross-Entropy Loss (Graph)
Description
Creates a graph node computing the cross-entropy loss between logits
a and labels b. The op applies its own softmax, so a
must be logits, not probabilities.
Usage
ggml_cross_entropy_loss(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Logits |
b |
Labels, same shape as |
Details
Differentiable: the backward pass is wired into the graph, so this can be used as a training loss in a hand-built graph.
Value
A scalar tensor holding the loss
See Also
Cross-Entropy Loss Backward (Graph)
Description
Gradient node for ggml_cross_entropy_loss.
Usage
ggml_cross_entropy_loss_back(ctx, a, b, c)
Arguments
ctx |
GGML context |
a |
Upstream gradient |
b |
Logits from the forward pass |
c |
Labels from the forward pass |
Details
ggml_build_backward_expand() inserts this automatically, so it is
rarely needed directly; it is exposed for hand-built backward graphs.
Value
Gradient w.r.t. the logits
See Also
GPU cross product (drop-in for crossprod)
Description
Computes t(A) %*% B (or t(A) %*% A when B is
NULL) on the Vulkan GPU, returning an R matrix, with a CPU fallback.
See ggml_matmul for the precision and dispatch notes.
Usage
ggml_crossprod(A, B = NULL, device = "auto", prec = "f32")
Arguments
A |
A numeric matrix. |
B |
A numeric matrix with |
device, prec |
See |
Value
t(A) %*% B as a numeric matrix.
Cumulative Sum (Graph)
Description
Creates a graph node computing the cumulative sum along the first dimension.
Usage
ggml_cumsum(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor of the same shape holding the running sum
Custom Operation Node
Description
Creates a graph node whose value is computed by a C kernel registered under
name. The kernel is looked up in ggmlR's custom-op registry; supplying
an unregistered name is an error, listing the names that are available.
Usage
ggml_custom(
ctx,
name,
args = NULL,
ne,
type = GGML_TYPE_F32,
n_tasks = GGML_N_TASKS_MAX
)
Arguments
ctx |
GGML context |
name |
Name of a registered custom op (character scalar) |
args |
List of input tensors, or NULL for none. At most 9 tensors. |
ne |
Output shape as a numeric vector of up to 4 dimensions, innermost first (ggml column-major order). Missing dimensions default to 1. |
type |
Output data type (default |
n_tasks |
Number of threads the kernel may be split across, or
|
Details
Kernels are registered from C, not from R. A package that links against
ggmlR registers its kernels at load time via the ggmlR_register_custom_op
callable; see system.file("include", "ggmlR.h", package = "ggmlR") for
the C-side contract and a worked example. R callbacks are not supported: the
kernel may run off the main thread, where calling into R is unsafe.
The output tensor is newly allocated with the given type and shape. Input
tensors passed in args become the node's sources and are visible to
the kernel as dst->src[0], dst->src[1], and so on.
Value
Tensor pointer for the custom node
Backend
Custom ops run on the CPU backend only, because the kernel is a host function pointer that a GPU backend cannot execute. Computing a graph that contains a custom node raises an error when the target backend is not the CPU, or when a scheduler has no CPU backend to fall back to. Keep custom nodes out of graphs you intend to run entirely on Vulkan.
See Also
ggml_custom_inplace, ggml_custom_ops
Examples
# "row_median" is one of ggmlR's built-in kernels; see ggml_custom_ops().
ggml_set_n_threads(1L)
ctx <- ggml_init(16 * 1024 * 1024)
x <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 5, 2)
ggml_set_f32(x, c(5, 1, 3, 2, 4,
9, 7, 8, 6, 10))
# One median per row: the output collapses ne[1] to 1.
med <- ggml_custom(ctx, "row_median", args = list(x), ne = c(1, 2))
graph <- ggml_build_forward_expand(ctx, med)
ggml_graph_compute(ctx, graph)
ggml_get_f32(med) # 3, 8
ggml_free(ctx)
In-Place Custom Operation Node
Description
Creates a custom-op node that writes into a view of a rather than into
a freshly allocated tensor. The output has the shape and type of a,
and a itself is the node's first source; tensors in args follow
it, so the kernel sees a as dst->src[0] and the first extra
argument as dst->src[1].
Usage
ggml_custom_inplace(ctx, a, name, args = NULL, n_tasks = GGML_N_TASKS_MAX)
Arguments
ctx |
GGML context |
a |
Tensor to write into |
name |
Name of a registered custom op (character scalar) |
args |
List of additional input tensors, or NULL for none. At most 8
tensors, one fewer than |
n_tasks |
Number of threads the kernel may be split across, or
|
Details
See ggml_custom for how kernels are registered and for the
CPU-only backend restriction, which applies here too.
Value
Tensor pointer for the custom node (a view of a)
See Also
Examples
# "clip_inplace" takes its bounds as a 2-element tensor, since a custom node
# carries no scalar parameters of its own.
ggml_set_n_threads(1L)
ctx <- ggml_init(16 * 1024 * 1024)
x <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(x, c(-3, -1, 0, 1, 3))
bounds <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 2)
ggml_set_f32(bounds, c(-1, 1))
y <- ggml_custom_inplace(ctx, x, "clip_inplace", args = list(bounds))
graph <- ggml_build_forward_expand(ctx, y)
ggml_graph_compute(ctx, graph)
ggml_get_f32(y) # -1, -1, 0, 1, 1
ggml_free(ctx)
List Registered Custom Ops
Description
Returns the names of the C kernels currently registered with ggmlR, in registration order. Useful for checking that a downstream package registered its kernels as expected.
Usage
ggml_custom_ops()
Value
Character vector of registered custom op names, empty if none
Built-in kernels
ggmlR registers three kernels at load time. Each one exercises a different path through the custom-op API, and they double as worked examples:
"row_median"Median of each row. One input; the output shape is chosen by the caller as
ne[1] = 1, collapsing the row. Usene = c(1, nrow)for ann x nrowinput."row_permute"out[i] = x[perm[i]]within each row, withperm0-based. Two inputs: F32 data and an I32 index tensor. Indices outside the row yield 0. This is notggml_get_rows, which permutes whole rows rather than elements inside one."clip_inplace"Clamps values into
[lo, hi], writing into the input. For use withggml_custom_inplace; the bounds are passed as a 2-element F32 tensor, since a custom node carries no scalar parameters of its own.ggml_clampdoes the same thing as a graph op – this kernel exists as the in-place example.
See Also
ggml_custom, ggml_custom_inplace
Examples
ggml_custom_ops()
Default MLP builder for classification and regression
Description
Constructs an uncompiled sequential multi-layer perceptron suitable as a
starting point for tabular classification or regression. This is the default
model_fn used by LearnerClassifGGML and LearnerRegrGGML
when the user does not supply a custom builder, and it is also exported for
direct use or as a template for user-defined builders.
Usage
ggml_default_mlp(
n_features,
n_out,
task_type = c("classif", "regr"),
hidden_layers = c(128L, 64L),
activation = "relu",
dropout = 0.2
)
Arguments
n_features |
Integer. Number of input features. Required. |
n_out |
Integer. Number of output units. For classification this is the number of classes; for regression this is typically 1. |
task_type |
Character. One of |
|
Integer vector. Widths of the hidden dense layers.
Default | |
activation |
Character. Activation applied to each hidden layer.
Default |
dropout |
Numeric in |
Details
The returned model is not compiled: the caller is responsible for
calling ggml_compile with the appropriate loss
("categorical_crossentropy" for classification, "mse" for
regression) before training.
The final layer is chosen based on task_type:
-
"classif"— dense withunits = n_outand softmax activation. -
"regr"— dense withunits = n_outand no activation (identity / linear output).
Value
An uncompiled ggml_sequential_model object. Call
ggml_compile before ggml_fit.
See Also
ggml_model_sequential, ggml_layer_dense,
ggml_layer_dropout, ggml_compile
Examples
## Not run:
# 3-class classifier on 20 features
model <- ggml_default_mlp(
n_features = 20L,
n_out = 3L,
task_type = "classif",
hidden_layers = c(64L, 32L),
dropout = 0.1
)
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
# Single-output regressor
reg <- ggml_default_mlp(
n_features = 10L,
n_out = 1L,
task_type = "regr"
)
reg <- ggml_compile(reg, optimizer = "adam", loss = "mse")
## End(Not run)
Create a Dense Layer Object
Description
Returns a reusable layer object for use with ggml_apply().
Applying the same object to multiple tensor nodes shares weights.
Usage
ggml_dense(
units,
activation = NULL,
name = NULL,
trainable = TRUE,
time_distributed = FALSE
)
Arguments
units |
Number of output units. |
activation |
Activation function name or NULL. |
name |
Optional character name. |
trainable |
Logical; whether weights are updated during training. |
time_distributed |
Logical; apply the kernel independently at every
position of a sequence input rather than flattening it (default
|
Value
A ggml_layer object.
Examples
encoder <- ggml_dense(64L, activation = "relu")
x1 <- ggml_input(shape = 32L)
x2 <- ggml_input(shape = 32L)
out1 <- x1 |> ggml_apply(encoder)
out2 <- x2 |> ggml_apply(encoder) # shared weights
Diagonal Matrix (Graph)
Description
Creates a diagonal matrix from a vector. For vector a[n], produces matrix with a on the diagonal.
Usage
ggml_diag(ctx, a)
Arguments
ctx |
GGML context |
a |
Input vector tensor |
Value
Diagonal matrix tensor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3)
ggml_set_f32(a, c(1, 2, 3))
d <- ggml_diag(ctx, a) # 3x3 diagonal matrix
graph <- ggml_build_forward_expand(ctx, d)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Diagonal Mask with -Inf (Graph)
Description
Creates a graph node that sets elements above the diagonal to -Inf. This is used for causal (autoregressive) attention masking.
Usage
ggml_diag_mask_inf(ctx, a, n_past)
ggml_diag_mask_inf_inplace(ctx, a, n_past)
Arguments
ctx |
GGML context |
a |
Input tensor (typically attention scores) |
n_past |
Number of past tokens (shifts the diagonal). Use 0 for standard causal masking where position i can only attend to positions <= i. |
Details
In causal attention, we want each position to only attend to itself and previous positions. Setting future positions to -Inf ensures that after softmax, they contribute 0 attention weight.
The n_past parameter allows for KV-cache scenarios where the diagonal needs to be shifted to account for previously processed tokens.
ggml_diag_mask_inf_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor with same shape as input, elements above diagonal set to -Inf
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Create attention scores matrix
scores <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4)
ggml_set_f32(scores, rep(1, 16))
# Apply causal mask
masked <- ggml_diag_mask_inf(ctx, scores, 0)
graph <- ggml_build_forward_expand(ctx, masked)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Diagonal Mask with Zero (Graph)
Description
Creates a graph node that sets elements above the diagonal to 0. Alternative to -Inf masking for certain use cases.
Usage
ggml_diag_mask_zero(ctx, a, n_past)
Arguments
ctx |
GGML context |
a |
Input tensor |
n_past |
Number of past tokens |
Value
Tensor with same shape as input, elements above diagonal set to 0
Element-wise Division (Graph)
Description
Creates a graph node for element-wise division.
Usage
ggml_div(ctx, a, b)
ggml_div_inplace(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First tensor (numerator) |
b |
Second tensor (denominator, same shape as a) |
Details
ggml_div_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the division operation (a / b)
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(10, 20, 30, 40, 50))
ggml_set_f32(b, c(2, 2, 2, 2, 2))
result <- ggml_div(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
DeepSeek V4 Hyper-Connection Combination Matrix (Graph)
Description
Builds the per-token 4 \times 4 stream-mixing matrix of DeepSeek V4
hyper-connections. With 0-based indices: logits
mixes[8 + dst + 4 * src, t] * scale[2] + base[8 + dst + 4 * src]; a
softmax over dst with eps added to each entry; then each
dst row divided by eps plus its sum over src; and for
iterations 1 to n_iter - 1, each src column divided by
eps plus its sum over dst, followed by the row step again
(Sinkhorn style).
Usage
ggml_dsv4_hc_comb(ctx, mixes, scale, base, eps, n_iter)
Arguments
ctx |
GGML context |
mixes |
F32 |
scale |
F32 vector of length |
base |
F32 vector of length 24 |
eps |
Added to every entry after the softmax |
n_iter |
Number of normalisation iterations (positive) |
Value
F32 tensor [4, 4, n_tokens] indexed [dst, src, t]
DeepSeek V4 Hyper-Connection Post-Mix (Graph)
Description
Expands a layer output back into the streams:
result[i, dst, t] = x[i, t] * post[dst, t] + sum_src residual[i, src, t]
* comb[dst, src, t]; with comb = NULL the identity mix
x[i, t] * post[dst, t] + residual[i, dst, t].
Usage
ggml_dsv4_hc_post(ctx, x, residual, post, comb = NULL)
Arguments
ctx |
GGML context |
x |
F32 |
residual |
F32 |
post |
F32 |
comb |
F32 |
Value
F32 tensor [n_embd, hc, n_tokens]
DeepSeek V4 Hyper-Connection Pre-Mix (Graph)
Description
Collapses the hyper-connection streams into one: with weights,
result[i, t] = sum_h x[i, h, t] * weights[h, t]; with gate
(the per-element gated form), result[i, t] = scale * sum_h x[i, h, t]
* sigmoid(gate[i, h, t]). Give exactly one of weights and gate.
Usage
ggml_dsv4_hc_pre(ctx, x, weights = NULL, gate = NULL, scale = 1)
Arguments
ctx |
GGML context |
x |
F32 |
weights |
F32 |
gate |
F32 with the shape of |
scale |
Output scale for the gated form |
Value
F32 tensor [n_embd, n_tokens]
Duplicate Tensor (Graph)
Description
Creates a graph node that copies a tensor. This is a graph operation that must be computed using ggml_build_forward_expand() and ggml_graph_compute(). Unlike ggml_dup_tensor which just allocates, this creates a copy operation in the graph.
Usage
ggml_dup(ctx, a)
ggml_dup_inplace(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_dup_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the copy operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
b <- ggml_dup(ctx, a)
graph <- ggml_build_forward_expand(ctx, b)
ggml_graph_compute(ctx, graph)
ggml_get_f32(b)
ggml_free(ctx)
Duplicate Tensor
Description
Creates a copy of a tensor with the same shape and type
Usage
ggml_dup_tensor(ctx, tensor)
Arguments
ctx |
GGML context |
tensor |
Tensor to duplicate |
Value
New tensor pointer with same shape
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
b <- ggml_dup_tensor(ctx, a)
ggml_nelements(b)
ggml_free(ctx)
Get Element Size
Description
Returns the size of a single element in the tensor.
Usage
ggml_element_size(tensor)
Arguments
tensor |
Tensor pointer |
Value
Element size in bytes
ELU Activation (Graph)
Description
Creates a graph node for ELU (Exponential Linear Unit) activation. ELU(x) = x if x > 0, else alpha * (exp(x) - 1) where alpha = 1.
Usage
ggml_elu_inplace(ctx, a)
ggml_elu(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_elu_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the ELU operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
r <- ggml_elu(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r)
ggml_free(ctx)
Create an Embedding Layer Object
Description
Create an Embedding Layer Object
Usage
ggml_embedding(vocab_size, dim, name = NULL, trainable = TRUE)
Arguments
vocab_size |
Number of distinct tokens. |
dim |
Embedding dimension. |
name |
Optional character name. |
trainable |
Logical. |
Value
A ggml_layer object.
Estimate Required Memory
Description
Helper function to estimate memory needed for a tensor
Usage
ggml_estimate_memory(type = GGML_TYPE_F32, ne0, ne1 = 1, ne2 = 1, ne3 = 1)
Arguments
type |
Tensor type (GGML_TYPE_F32, etc) |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 (optional) |
ne2 |
Size of dimension 2 (optional) |
ne3 |
Size of dimension 3 (optional) |
Value
Estimated memory in bytes
Examples
# For 1000x1000 F32 matrix
ggml_estimate_memory(GGML_TYPE_F32, 1000, 1000)
Evaluate a Trained Model
Description
Evaluate a Trained Model
Usage
## S3 method for class 'ggml_functional_model'
ggml_evaluate(model, x, y, batch_size = 32L, ...)
ggml_evaluate(model, ...)
## S3 method for class 'ggml_sequential_model'
ggml_evaluate(
model,
x,
y,
batch_size = 32,
sample_weight = NULL,
class_weight = NULL,
...
)
Arguments
model |
A trained ggml_sequential_model |
x |
Test data |
y |
Test labels (one-hot encoded) |
batch_size |
Batch size for evaluation |
... |
Additional arguments (ignored). |
sample_weight |
Numeric vector of per-sample weights (length = nrow(x)). |
class_weight |
Named vector of weights per class, e.g. c("0"=1, "1"=10). Cannot be used with sample_weight. |
Value
Named list with loss and accuracy. A multi-output
functional model additionally reports each head as <output>_loss.
Note the deliberate difference from the training history, which names the
same quantities train_<output>_loss and val_<output>_loss:
there the prefix distinguishes the two phases (and keeps an output named
val_x from colliding with another output's validation column),
whereas evaluation has no phases to tell apart, so the bare name is used.
Examples
ggml_set_n_threads(1L) # deterministic, single OpenMP pool
n <- 128
x <- matrix(runif(n * 4), nrow = n, ncol = 4)
y <- matrix(0, nrow = n, ncol = 2)
for (i in seq_len(n)) { y[i, if (sum(x[i,]) > 2) 1L else 2L] <- 1 }
model <- ggml_model_sequential() |>
ggml_layer_dense(8, activation = "relu") |>
ggml_layer_dense(2, activation = "softmax")
model$input_shape <- 4L
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
model <- ggml_fit(model, x, y, epochs = 5, batch_size = 32, verbose = 0)
# Basic evaluation
result <- ggml_evaluate(model, x, y, batch_size = 32)
# With sample_weight
sw <- runif(n, 0.5, 1.5)
result <- ggml_evaluate(model, x, y, batch_size = 32, sample_weight = sw)
# With class_weight
result <- ggml_evaluate(model, x, y, batch_size = 32,
class_weight = c("0" = 1, "1" = 2))
Exponential (Graph)
Description
Creates a graph node for element-wise exponential: exp(x)
Usage
ggml_exp_inplace(ctx, a)
ggml_exp(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_exp_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the exp operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3)
ggml_set_f32(a, c(0, 1, 2))
result <- ggml_exp(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [1, e, e^2]
ggml_free(ctx)
Exponential Minus One (Graph)
Description
Creates a graph node for element-wise e^x - 1, accurate for small
|x| where exp(x) - 1 loses precision to cancellation.
Usage
ggml_expm1(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the expm1 operation
Extract a feature-by-cell matrix from a single-cell container
Description
Pulls an expression matrix out of a Seurat object, a
SingleCellExperiment, a sparse dgCMatrix or a plain
matrix, returning a dense numeric matrix with features in rows and
cells in columns — the layout the GPU engine expects. Optional
genes/cells subsetting happens before any sparse-to-dense
materialisation.
Usage
ggml_extract(x, assay = NULL, layer = "data", genes = NULL, cells = NULL, ...)
## S3 method for class 'matrix'
ggml_extract(x, assay = NULL, layer = "data", genes = NULL, cells = NULL, ...)
## S3 method for class 'dgCMatrix'
ggml_extract(
x,
assay = NULL,
layer = "data",
genes = NULL,
cells = NULL,
keep_sparse = FALSE,
...
)
## S3 method for class 'Seurat'
ggml_extract(
x,
assay = NULL,
layer = "data",
genes = NULL,
cells = NULL,
keep_sparse = FALSE,
...
)
## S3 method for class 'SingleCellExperiment'
ggml_extract(
x,
assay = NULL,
layer = "logcounts",
genes = NULL,
cells = NULL,
keep_sparse = FALSE,
...
)
Arguments
x |
A |
assay |
Assay to read from. Seurat: defaults to the object's default assay. Ignored for bare matrices. |
layer |
Layer / slot to read. Seurat v5: a layer name (default
|
genes |
Optional character/integer vector selecting feature rows. |
cells |
Optional character/integer vector selecting cell columns. |
... |
Passed to methods. |
keep_sparse |
If |
Details
For a SingleCellExperiment the layer argument names an
assay (default "logcounts", the log-normalised matrix); it is read
with SummarizedExperiment::assay().
Value
A dense numeric matrix, features x cells.
Fill (Graph)
Description
Creates a graph node: a tensor of the shape and type of a with every
element set to c. The values of a are not read.
Usage
ggml_fill(ctx, a, c)
Arguments
ctx |
GGML context |
a |
Tensor giving the shape and type |
c |
Fill value |
Value
Tensor representing the fill operation
Train a Model (dispatcher)
Description
Dispatcher: if the first argument is a ggml_sequential_model, delegates
to the Keras-style high-level API (ggml_fit_sequential); otherwise
delegates to the low-level optimizer loop (ggml_fit_opt).
Usage
## S3 method for class 'ggml_functional_model'
ggml_fit(
model,
x,
y,
epochs = 1L,
batch_size = 32L,
validation_split = 0,
validation_data = NULL,
verbose = 1L,
shuffle = TRUE,
callbacks = list(),
...
)
ggml_fit(model, ...)
## S3 method for class 'ggml_sequential_model'
ggml_fit(model, ...)
## Default S3 method:
ggml_fit(model, ...)
Arguments
model |
A compiled model object. |
x |
Training data (matrix or array). |
y |
Training labels (matrix, one-hot encoded). |
epochs |
Number of training epochs (default: 1). |
batch_size |
Batch size (default: 32). |
validation_split |
Fraction of data for validation (default: 0). |
validation_data |
Optional list(x_val, y_val). Overrides validation_split. |
verbose |
0 = silent, 1 = progress (default: 1). |
shuffle |
Shuffle the data (default |
callbacks |
List of callback objects, e.g.
Both phases carry a prefix on purpose: it keeps an output named
|
... |
Arguments passed to the appropriate implementation. |
Details
Keras-style (Sequential model):
- model
A compiled
ggml_sequential_model- x
Training data (matrix or array)
- y
Training labels (matrix, one-hot encoded for classification)
- epochs
Number of training epochs (default: 1)
- batch_size
Batch size (default: 32)
- validation_split
Fraction of data for validation (default: 0)
- validation_data
Optional list(x_val, y_val) for validation. Overrides validation_split.
- class_weight
Named vector of weights per class, e.g. c("0"=1, "1"=10). Cannot be used with sample_weight.
- sample_weight
Per-sample weights: a numeric vector of length
nrow(x), or – formean_squared_erroronly – anrow(x)byncol(y)matrix acting as a per-output loss mask. A mask trains only the outputs whose weight is non-zero, which is what a Q-model needs: 1 on the action that has a TD target, 0 elsewhere. The denominator is unchanged (sum(w*(pred-y)^2)/nelements), so masking does not renormalise over the active outputs. Cannot be used with class_weight.- verbose
0 = silent, 1 = progress (default: 1)
- shuffle
Shuffle the data (default: TRUE). Shuffled once before the train/validation split, then the training portion each epoch; the validation portion stays fixed. FALSE for time series or exactly reproducible runs.
- callbacks
List of callback objects (early stopping, LR schedules)
- generator
Function of no arguments yielding
list(x, y)for one batch, orNULLwhen exhausted – an alternative toxandythat keeps only the current batch in memory. Seeggml_fit_generator, to which the call is forwarded along withsteps_per_epoch,initial_epoch,validation_generatorandvalidation_steps. The arguments that are defined in terms of a whole dataset –validation_split,validation_data,class_weightandsample_weight– are rejected rather than ignored, andshuffleis inert because the generator owns its own order.
Low-level (optimizer loop):
- sched
Backend scheduler
- ctx_compute
Compute context
- inputs
Input tensor
- outputs
Output tensor
- dataset
Dataset from
ggml_opt_dataset_init()- loss_type
Loss type (default: MSE)
- optimizer
Optimizer type (default: AdamW)
- nepoch
Number of epochs (default: 10)
- nbatch_logical
Logical batch size (default: 32)
- val_split
Validation fraction (default: 0)
- shuffle
Shuffle the data (default: TRUE)
- callbacks
List of callback objects
- silent
Suppress output (default: FALSE)
Value
For Sequential models: the trained model (invisibly).
For the low-level API: a data frame with columns
epoch, train_loss, train_accuracy,
val_loss, val_accuracy.
See Also
Examples
ggml_set_n_threads(1L) # deterministic, single OpenMP pool
n <- 128
x <- matrix(runif(n * 4), nrow = n, ncol = 4)
y <- matrix(0, nrow = n, ncol = 2)
for (i in seq_len(n)) { y[i, if (sum(x[i,]) > 2) 1L else 2L] <- 1 }
model <- ggml_model_sequential() |>
ggml_layer_dense(8, activation = "relu") |>
ggml_layer_dense(2, activation = "softmax")
model$input_shape <- 4L
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
# Basic training
model <- ggml_fit(model, x, y, epochs = 5, batch_size = 32, verbose = 0)
# With validation_data
x_val <- matrix(runif(32 * 4), nrow = 32, ncol = 4)
y_val <- matrix(0, nrow = 32, ncol = 2)
for (i in seq_len(32)) { y_val[i, if (sum(x_val[i,]) > 2) 1L else 2L] <- 1 }
model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32,
validation_data = list(x_val, y_val), verbose = 0)
# With class_weight (useful for imbalanced classes)
model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32,
class_weight = c("0" = 1, "1" = 2), verbose = 0)
# With sample_weight
sw <- runif(n, 0.5, 1.5)
model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32,
sample_weight = sw, verbose = 0)
Train a Sequential Model from a Generator
Description
Streaming counterpart of ggml_fit for sequential models:
batches are pulled from a generator instead of being sliced out of a matrix
held in memory. Reachable either directly or as
ggml_fit(model, generator = ...).
Usage
ggml_fit_generator(
model,
generator,
steps_per_epoch = NULL,
epochs = 1L,
batch_size = 32L,
initial_epoch = 0L,
validation_generator = NULL,
validation_steps = NULL,
calibration_generator = NULL,
calibration_steps = NULL,
verbose = 1,
callbacks = list()
)
Arguments
model |
A compiled |
generator |
Function of no arguments yielding |
steps_per_epoch |
Batches per epoch. Required for a generator that never
returns |
epochs |
Number of epochs to run |
batch_size |
Samples per batch. The graph is built for a fixed batch, so the generator must yield exactly this many; a short batch is an error rather than being padded. |
initial_epoch |
Epoch number to start counting from. Shifts the
numbering seen by callbacks and learning-rate schedules; it does not
restore optimizer state (see |
validation_generator |
Optional generator for validation, same contract. |
validation_steps |
Validation batches per epoch. |
calibration_generator |
Generator used to recompute batch-norm running
statistics after training. Defaults to |
calibration_steps |
Batches to draw for that calibration. |
verbose |
0 to silence per-epoch messages |
callbacks |
List of callback lists, as for |
Details
Batches are given in the same R layout that x and y use; the
column-major conversion ggml wants happens internally, which is also what
lets a recorded batch feed batch-norm calibration unchanged.
Value
The trained model, invisibly.
See Also
ggml_fit_opt_gen for the low-level loop this wraps.
Fit model with R-side epoch loop and callbacks
Description
Trains a model epoch by epoch in R, allowing callbacks for early stopping and learning rate scheduling. Optimizer state (momentum) is preserved across all epochs.
Usage
ggml_fit_opt(
sched,
ctx_compute,
inputs,
outputs,
dataset,
loss_type = ggml_opt_loss_type_mse(),
optimizer = ggml_opt_optimizer_type_adamw(),
nepoch = 10L,
nbatch_logical = 32L,
val_split = 0,
shuffle = TRUE,
shuffle_all = shuffle,
callbacks = list(),
silent = FALSE,
loss_mask_ne0 = NULL,
weight_decay = 0
)
Arguments
sched |
Backend scheduler |
ctx_compute |
Compute context (for temporary tensors) |
inputs |
Input tensor with shape [ne_datapoint, batch_size] |
outputs |
Output tensor with shape [ne_label, batch_size] |
dataset |
Dataset created with 'ggml_opt_dataset_init()' |
loss_type |
Loss type (default: MSE) |
optimizer |
Optimizer type (default: AdamW) |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size (for gradient accumulation) |
val_split |
Fraction of data for validation (0.0 to 1.0) |
shuffle |
Shuffle the training portion each epoch (default
|
shuffle_all |
Shuffle the whole dataset once before the
train/validation split is taken, so the validation portion is a random
sample rather than the tail of the input. Defaults to |
callbacks |
List of callback lists. Each element may have 'on_train_begin(logs, state)' (once, before the first epoch), 'on_epoch_begin(epoch, logs, state)' and/or 'on_epoch_end(epoch, logs, state)'. Built-in factories: 'ggml_callback_early_stopping()', 'ggml_schedule_step_decay()', 'ggml_schedule_cosine_decay()', 'ggml_schedule_reduce_on_plateau()'. 'state' is a mutable environment with fields: 'stop' (set TRUE to stop training), 'lr_ud', 'nepoch'. |
silent |
Whether to suppress per-epoch progress output |
loss_mask_ne0 |
Width of the weighted-MSE weight tensor: |
weight_decay |
Decoupled weight decay in |
Value
Data frame with columns epoch, train_loss, train_accuracy, val_loss, val_accuracy
See Also
Examples
if (FALSE) {
history <- ggml_fit_opt(sched, ctx_compute, inputs, outputs, dataset,
nepoch = 50, val_split = 0.2,
callbacks = list(
ggml_callback_early_stopping(monitor = "val_loss", patience = 5),
ggml_schedule_cosine_decay()
))
}
Train from a generator, one batch at a time
Description
Streaming counterpart of ggml_fit_opt. Instead of a
ggml_opt_dataset holding every sample, batches are pulled from a
generator function, so only the current batch is ever in memory. This is what
makes three things possible that the dataset path cannot express: data larger
than RAM, augmentation that must differ per epoch, and generated sources
(RL rollouts, simulators) that have no fixed dataset at all.
Usage
ggml_fit_opt_gen(
sched,
ctx_compute,
inputs,
outputs,
generator,
steps_per_epoch = NULL,
epochs = 1L,
initial_epoch = 0L,
validation_generator = NULL,
validation_steps = NULL,
loss_type = ggml_opt_loss_type_mse(),
optimizer = ggml_opt_optimizer_type_adamw(),
nbatch_logical = NULL,
callbacks = list(),
silent = FALSE,
loss_mask_ne0 = NULL,
weight_decay = 0
)
Arguments
sched |
Backend scheduler |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor of the built graph |
outputs |
Output tensor of the built graph |
generator |
Function of no arguments returning one batch, or
|
steps_per_epoch |
Number of batches that make up one epoch. Required
when |
epochs |
Number of epochs to run. |
initial_epoch |
Epoch number to start counting from, as in Keras. This
shifts the numbering seen by callbacks and learning-rate schedules only;
it does not restore optimizer state. A fresh |
validation_generator |
Optional generator used for validation after each epoch, following the same contract. Evaluated without a backward pass. |
validation_steps |
Number of validation batches per epoch. Required when
|
loss_type |
Loss type constant |
optimizer |
Optimizer type constant |
nbatch_logical |
Logical batch size, for gradient accumulation |
callbacks |
List of callback lists, as for |
silent |
Suppress per-epoch messages |
loss_mask_ne0 |
Width of the per-output loss mask, or NULL |
weight_decay |
Decoupled weight decay in |
Value
A history data frame, one row per epoch, as
ggml_fit_opt returns.
The generator contract
generator is a function of no arguments. Each call returns one batch;
returning NULL ends the epoch. The same contract covers both finite
and infinite sources – an infinite generator simply never returns
NULL, and steps_per_epoch then becomes required, because
nothing else can say where an epoch ends.
A finite generator must restart itself. Training runs several
epochs over the same source, so a generator that ends must reset its own
state before returning NULL, leaving the next call to yield the first
batch again. Nothing else can do it: a bare closure has no handle to reset
from. An epoch that receives no batch at all stops training with a warning
rather than looping over nothing.
batch_gen <- function(x, y, batch_size) {
i <- 0L
nb <- nrow(x) %/% batch_size
function() {
if (i >= nb) {
i <<- 0L # reset, so the next epoch starts over
return(NULL)
}
idx <- (i * batch_size + 1L):((i + 1L) * batch_size)
i <<- i + 1L
list(x[idx, , drop = FALSE], y[idx, , drop = FALSE])
}
}
A batch is list(inputs = list(...), labels = list(...)). For the
common single-input case a plain list(x, y) is accepted as well. Each
element is a numeric vector or array already in ggml layout, holding exactly
one physical batch.
The generator is called from R, so it may do anything R can: read a shard from disk, augment the previous batch, or sample a replay buffer that a simulator is filling.
Gradient accumulation
When nbatch_logical exceeds the graph's physical batch size, the
optimizer accumulates gradients over several generator batches and steps once
per logical batch. The period is tracked inside opt_ctx; the generator
is unaware of it and simply keeps producing physical batches.
See Also
ggml_fit_opt for the in-memory dataset path.
Fit a multi-output model with an R-side epoch loop and callbacks
Description
Multi-output counterpart of ggml_fit_opt(). Each output head has its
own labels, loss type and weight; the optimized total is
sum(loss_weights[i] * loss_i). Because the epoch loop runs in R, this
path supports callbacks (early stopping, LR schedulers) and the
shuffle/shuffle_all controls, which the single C call
ggml_opt_fit_multi() cannot offer.
Usage
ggml_fit_opt_multi(
sched,
ctx_compute,
inputs,
outputs,
dataset,
loss_types,
loss_weights = rep(1, length(outputs)),
labels_offs = NULL,
head_names = NULL,
optimizer = ggml_opt_optimizer_type_adamw(),
nepoch = 10L,
nbatch_logical = 32L,
val_split = 0,
shuffle = TRUE,
shuffle_all = shuffle,
callbacks = list(),
silent = FALSE,
weight_decay = 0
)
Arguments
sched |
Backend scheduler |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor with shape [ne_datapoint, batch_size] |
outputs |
List of output tensors, one per head |
dataset |
Dataset with data and concatenated labels |
loss_types |
Integer vector of loss type constants, one per head |
loss_weights |
Numeric vector of head weights (default: all 1) |
labels_offs |
Numeric vector of 0-based label offsets, one per head.
Defaults to laying the heads out back to back in head order, using each
head's |
head_names |
Character vector naming the heads, one per head, used for
the per-head keys in |
optimizer |
Optimizer type (default: AdamW) |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size (for gradient accumulation) |
val_split |
Fraction of data for validation (0.0 to 1.0) |
shuffle |
Shuffle the training portion each epoch (default
|
shuffle_all |
Shuffle the whole dataset once before the
train/validation split is taken, so the validation portion is a random
sample rather than the tail of the input. Defaults to |
callbacks |
List of callback lists, as for |
silent |
Whether to suppress per-epoch progress output |
weight_decay |
Decoupled weight decay in |
Details
All heads share the dataset's datapoints. The dataset's labels hold every
head's labels concatenated along the first dimension: head i occupies
labels_offs[i] + seq_len(width_i) of each label row.
Value
Data frame with columns epoch, train_loss, train_accuracy, val_loss,
val_accuracy – one row per epoch actually run, so a callback that stops
early shortens it. The per-head metrics come as attributes rather than
columns, since they are matrices, each [nepoch_run x n_head] with columns
named after the heads: attr(h, "head_loss"),
attr(h, "head_accuracy"), attr(h, "val_head_loss") and
attr(h, "val_head_accuracy"), plus attr(h, "head_names").
See Also
Examples
if (FALSE) {
history <- ggml_fit_opt_multi(sched, ctx_compute, inputs, list(out_a, out_b),
dataset, loss_types = c(ggml_opt_loss_type_mse(), ggml_opt_loss_type_mse()),
nepoch = 50, val_split = 0.2,
callbacks = list(ggml_callback_early_stopping(monitor = "val_loss", patience = 5)))
attr(history, "head_loss")
}
Flash Attention Backward (Graph)
Description
Backward pass for ggml_flash_attn_ext: given the gradient of
the attention output, produces the gradients of Q, K and V in one pass.
Usage
ggml_flash_attn_back(ctx, q, k, v, mask, d, scale)
Arguments
ctx |
GGML context |
q |
Query tensor |
k |
Key tensor |
v |
Value tensor |
mask |
Attention mask |
d |
Gradient of the forward output, shape |
scale |
Softmax scale used in the forward pass (e.g. |
Details
Upstream ggml ships this operation as a stub that aborts, left behind when
flash_attn was replaced by flash_attn_ext; ggmlR implements it
so attention can be trained through the fused op. It is also wired into the
graph autodiff, so ggml_build_backward_expand reaches it on its
own — calling this directly is only needed when building a backward graph
by hand.
The three gradients come back packed head to tail in one contiguous 1D
tensor: grad_q, then grad_k, then grad_v, each padded to
GGML_MEM_ALIGN. Take a view of the slice you need.
Not supported: max_bias (ALiBi), logit_softcap and attention
sinks. Q, K, V and d must be F32.
Value
Tensor holding grad_q, grad_k and grad_v packed contiguously
See Also
ggml_flash_attn_ext for the forward pass.
Flash Attention (Graph)
Description
Creates a graph node for Flash Attention computation. This is a memory-efficient implementation of scaled dot-product attention.
Usage
ggml_flash_attn_ext(
ctx,
q,
k,
v,
mask = NULL,
scale,
max_bias = 0,
logit_softcap = 0
)
Arguments
ctx |
GGML context |
q |
Query tensor of shape [head_dim, n_head, n_tokens, batch] |
k |
Key tensor of shape [head_dim, n_head_kv, n_kv, batch] |
v |
Value tensor of shape [head_dim, n_head_kv, n_kv, batch] |
mask |
Optional attention mask tensor (NULL for no mask). For causal attention, use ggml_diag_mask_inf instead. |
scale |
Attention scale factor, typically 1/sqrt(head_dim) |
max_bias |
Maximum ALiBi bias (0.0 to disable ALiBi) |
logit_softcap |
Logit soft-capping value (0.0 to disable). Used by some models like Gemma 2. |
Details
Flash Attention computes: softmax(Q * K^T / scale + mask) * V
Key features: - Memory efficient: O(n) instead of O(n^2) memory for attention matrix - Supports grouped-query attention (GQA) when n_head_kv < n_head - Supports multi-query attention (MQA) when n_head_kv = 1 - Optional ALiBi (Attention with Linear Biases) for position encoding - Optional logit soft-capping for numerical stability
Value
Attention output tensor of shape [head_dim, n_head, n_tokens, batch]
Examples
ctx <- ggml_init(64 * 1024 * 1024)
head_dim <- 64
n_head <- 8
n_head_kv <- 2 # GQA with 4:1 ratio
seq_len <- 32
q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head, seq_len, 1)
k <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head_kv, seq_len, 1)
v <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head_kv, seq_len, 1)
ggml_set_f32(q, rnorm(head_dim * n_head * seq_len))
ggml_set_f32(k, rnorm(head_dim * n_head_kv * seq_len))
ggml_set_f32(v, rnorm(head_dim * n_head_kv * seq_len))
# Scale = 1/sqrt(head_dim)
scale <- 1.0 / sqrt(head_dim)
# Compute attention
out <- ggml_flash_attn_ext(ctx, q, k, v, NULL, scale, 0.0, 0.0)
graph <- ggml_build_forward_expand(ctx, out)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Flash Attention Settings
Description
Sets the precision a ggml_flash_attn_ext() node computes in.
GGML_PREC_F32 raises accuracy at some cost in speed.
Usage
ggml_flash_attn_ext_set_n_kv_max(a, n_kv_max)
ggml_flash_attn_ext_set_prec(tensor, prec)
ggml_flash_attn_ext_get_prec(tensor)
ggml_flash_attn_ext_add_sinks(tensor, sinks)
Arguments
a |
FLASH_ATTN_EXT node: the result of |
n_kv_max |
Upper bound on finite mask entries per row (non-negative) |
tensor |
A flash-attention node |
prec |
Precision constant |
sinks |
Sink logits, one per head |
Details
ggml_flash_attn_ext_set_n_kv_max(): Declares that every row of the mask of an existing
ggml_flash_attn_ext() node has at most n_kv_max finite
entries (the rest are -Inf). The Vulkan backend may then gather only
those K/V positions instead of scanning the whole sequence; the CPU backend
ignores the bound. 0 disables it. The bound is a promise: rows with
more finite entries give wrong results on Vulkan.
Modifies the node in place and returns it.
Returns: a
ggml_flash_attn_ext_get_prec():
Returns: The precision constant currently set on the node
ggml_flash_attn_ext_add_sinks(): Adds per-head "sink" logits, an always-available extra key that lets a query
attend to nothing in particular.
Returns: The tensor (for chaining)
Value
The tensor (for chaining)
See Also
Floor (Graph)
Description
Creates a graph node for element-wise floor: floor(x)
Usage
ggml_floor_inplace(ctx, a)
ggml_floor(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_floor_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the floor operation
Free GGML context
Description
Free GGML context
Usage
ggml_free(ctx)
Arguments
ctx |
Context pointer |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
ggml_free(ctx)
Fix or Release Batch-Norm Statistics
Description
ggml_freeze_bn_stats() fixes the running mean and variance of
batch-norm layers: they then normalize with those stored statistics during
training as well as at prediction, and ggml_fit no longer
recalibrates them. gamma and beta still train unless the layer
is also frozen with ggml_freeze_weights – the usual fine-tuning
setting when batches are small. ggml_unfreeze_bn_stats() returns the
layers to batch statistics in training and recalibration after a fit, which
is the default.
Usage
ggml_freeze_bn_stats(
model,
from = 1L,
to = length(model$layers),
layer_names = NULL,
prefix = NULL,
...
)
ggml_unfreeze_bn_stats(
model,
from = 1L,
to = length(model$layers),
layer_names = NULL,
prefix = NULL,
...
)
Arguments
model |
A model object (ggml_sequential_model or ggml_functional_model) |
from |
Integer index of the first layer to freeze (default: 1) |
to |
Integer index of the last layer to freeze (default: last layer) |
layer_names |
Character vector of layer names to freeze (overrides from/to) |
prefix |
Character vector: freeze every layer whose name starts with one
of these – a whole transformer block, say: |
... |
Additional arguments passed to methods |
Details
Freezing the weights alone does not fix the statistics: a frozen batch-norm
still normalizes each training batch by its own mean and variance, and its
running statistics are still recalibrated after ggml_fit().
The fixed state is saved by ggml_save_model. Only sequential
models carry running statistics; the functional batch_norm is a
per-sample RMS normalization (see ggml_layer_batch_norm),
already independent of the batch, and is refused with a message.
Value
The model with the selected batch-norm layers changed. Selected layers of other types are skipped; a selection with no batch-norm layer at all is an error.
Freeze Layer Weights
Description
Sets trainable = FALSE on layers, preventing their weights from being
updated during training. Layers are chosen by layer_names and/or
prefix; without either, by the index range from / to
(sequential models), or all of them.
Usage
ggml_freeze_weights(
model,
from = 1L,
to = length(model$layers),
layer_names = NULL,
prefix = NULL,
...
)
Arguments
model |
A model object (ggml_sequential_model or ggml_functional_model) |
from |
Integer index of the first layer to freeze (default: 1) |
to |
Integer index of the last layer to freeze (default: last layer) |
layer_names |
Character vector of layer names to freeze (overrides from/to) |
prefix |
Character vector: freeze every layer whose name starts with one
of these – a whole transformer block, say: |
... |
Additional arguments passed to methods |
Value
The model with selected layers frozen.
See Also
ggml_freeze_bn_stats to fix batch-norm statistics,
which freezing the weights does not do.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(64, activation = "relu") |>
ggml_layer_dense(10, activation = "softmax")
# Freeze all layers
model <- ggml_freeze_weights(model)
# Freeze only the first layer
model <- ggml_freeze_weights(model, from = 1, to = 1)
Graph Allocator
Description
Creates a new graph allocator for efficient memory management. The allocator can automatically allocate and reuse memory for graph tensors.
Usage
ggml_gallocr_new()
ggml_gallocr_free(galloc)
ggml_gallocr_reserve(galloc, graph)
ggml_gallocr_alloc_graph(galloc, graph)
ggml_gallocr_get_buffer_size(galloc, buffer_id = 0L)
ggml_gallocr_new_buft(buft)
Arguments
galloc |
Graph allocator object |
graph |
Graph object |
buffer_id |
Buffer ID (default: 0 for single-buffer allocator) |
buft |
Buffer type (e.g. from |
Details
ggml_gallocr_free(): Frees a graph allocator and all associated buffers.
Returns: No return value, called for side effects
ggml_gallocr_reserve(): Pre-allocates memory for a graph. This is optional but recommended
when running the same graph multiple times to avoid reallocation.
Returns: TRUE on success, FALSE on failure
ggml_gallocr_alloc_graph(): Allocates memory for all tensors in the computation graph.
This must be called before computing the graph.
Returns: TRUE on success, FALSE on failure
ggml_gallocr_get_buffer_size(): Returns the size of the buffer used by the graph allocator.
Returns: Size in bytes
ggml_gallocr_new_buft(): Like ggml_gallocr_new, but allocates from the given buffer type
rather than the CPU default – so a graph can be allocated in GPU memory.
Returns: Graph allocator object (external pointer)
Value
Graph allocator object (external pointer)
Examples
ctx <- ggml_init(16 * 1024 * 1024)
galloc <- ggml_gallocr_new()
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, b)
# Allocate graph
ggml_gallocr_alloc_graph(galloc, graph)
ggml_gallocr_free(galloc)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
galloc <- ggml_gallocr_new()
# Create graph
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, b)
# Allocate and compute
ggml_gallocr_alloc_graph(galloc, graph)
ggml_graph_compute(ctx, graph)
ggml_gallocr_free(galloc)
ggml_free(ctx)
Gated Delta Net (Linear Attention Recurrence)
Description
The gated delta rule used by linear-attention layers such as Qwen3-Next
and Kimi (KDA). For each head the recurrent state S (S_v x S_v)
is decayed by the gate, corrected towards v along k with step
beta, and read out with q (scaled by 1/sqrt(S_v)).
Usage
ggml_gated_delta_net(ctx, q, k, v, g, beta, state, K = 1L)
Arguments
ctx |
GGML context |
q |
Queries |
k |
Keys |
v |
Values |
g |
Gate (log-decay), scalar per head or per channel (KDA) |
beta |
Update step per head |
state |
Initial recurrent state |
K |
Number of state snapshots kept after the outputs |
Value
The packed result tensor
Shapes
-
q,k—[S_v, H_k, n_tokens, n_seqs], rows contiguous;H_vmust be a multiple ofH_k -
v—[S_v, H_v, n_tokens, n_seqs], rows contiguous -
g—[1, H_v, n_tokens, n_seqs](scalar gate) or[S_v, H_v, n_tokens, n_seqs](KDA), contiguous -
beta—[1, H_v, n_tokens, n_seqs], contiguous -
state— initial state,S_v * S_v * H_v * n_seqselements
All inputs are F32. The Vulkan backend runs S_v of 16, 32, 64 or 128;
other sizes fall back to the CPU.
Packed result
A 2D tensor [S_v * H_v, n_tokens * n_seqs + K * S_v * n_seqs]: the
outputs (S_v * H_v * n_tokens * n_seqs elements) followed by K
state snapshots, most recent first (slot 0 is the final state, slot s
the state s tokens before the end). Slots older than the sequence are
left untouched.
Inference only
GGML_OP_GATED_DELTA_NET has no backward pass in ggml.
See Also
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Gated Linear Attention
Description
Linear attention with a per-token gate (the GLA / RetNet family). Shares the
RWKV recurrence machinery: keys, values and queries drive a running state
that g gates, and scale multiplies the query.
Usage
ggml_gated_linear_attn(ctx, k, v, q, g, state, scale)
Arguments
ctx |
GGML context |
k |
Keys, |
v |
Values, |
q |
Queries, |
g |
Gate, |
state |
Initial state, |
scale |
Scalar applied to the queries (usually |
Value
A tensor [S * H, n_tokens + S * n_seqs]
Shapes
k, v, q and g are [S, H, n_tokens];
state holds S * S * H * n_seqs elements. The result is
[S * H, n_tokens + S * n_seqs], outputs then final state – see
ggml_rwkv_output.
n_tokens counts the tokens of all sequences together, so it
must be a multiple of n_seqs.
Backend support
CPU only. Unlike the RWKV and SSM ops, GGML_OP_GATED_LINEAR_ATTN has
no Vulkan shader, so a graph containing it falls back to the CPU for this
node. It has no backward pass either, so it is inference-only.
See Also
ggml_rwkv_wkv7, ggml_rwkv_output
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Backward Pass of Gated Linear Attention
Description
Gradients of ggml_gated_linear_attn, packed in forward source
order: [d_k | d_v | d_q | d_g | d_state].
Usage
ggml_gated_linear_attn_back(ctx, k, v, q, g, state, grad, scale)
Arguments
ctx |
GGML context |
k, v, q, g, state |
The tensors passed to the forward op |
grad |
Gradient w.r.t. the packed forward result |
scale |
The scale the forward op was built with |
Value
A flat tensor holding the five gradients, concatenated
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
GELU Backward (Graph)
Description
Gradient of the GELU activation: dx = dy * dgelu(x). Wired into the
autodiff engine, so ggml_compile(activation = "gelu") trains – this
is a ggmlR extension, upstream ggml has no GELU backward and aborts instead.
Usage
ggml_gelu_back(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Gradient tensor from upstream ( |
b |
Forward input tensor ( |
Details
The derivative is that of the tanh approximation ggml_gelu
computes,
g(x) = 0.5x(1 + tanh(u)) with u = c*x*(1 + a*x^2). The quick
and erf variants (ggml_gelu_quick, ggml_gelu_erf)
are different functions with different derivatives and remain without a
backward rule.
Value
Gradient tensor for the input
See Also
Exact GELU Activation (Graph)
Description
Creates a graph node for exact GELU using the error function (erf). GELU(x) = x * 0.5 * (1 + erf(x / sqrt(2))). More accurate than approximate GELU but potentially slower on some backends.
Usage
ggml_gelu_erf(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the exact GELU operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
r <- ggml_gelu_erf(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r)
ggml_free(ctx)
GELU Activation (Graph)
Description
Creates a graph node for GELU (Gaussian Error Linear Unit) activation. CRITICAL for GPT models.
Usage
ggml_gelu_inplace(ctx, a)
ggml_gelu(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_gelu_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the GELU operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
result <- ggml_gelu(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
GELU Quick Activation (Graph)
Description
Creates a graph node for fast approximation of GELU. Faster than standard GELU with minimal accuracy loss.
Usage
ggml_gelu_quick(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the GELU quick operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
result <- ggml_gelu_quick(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result)
ggml_free(ctx)
Get F32 Data
Description
Get F32 Data
Usage
ggml_get_f32(tensor)
Arguments
tensor |
Tensor |
Value
Numeric vector
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(t, c(1, 2, 3, 4, 5))
ggml_get_f32(t)
ggml_free(ctx)
Get First Tensor from Context
Description
Get First Tensor from Context
Usage
ggml_get_first_tensor(ctx)
Arguments
ctx |
GGML context |
Value
Tensor pointer or NULL
Get I32 Data
Description
Gets integer data from an I32 tensor (e.g., from ggml_argmax)
Usage
ggml_get_i32(tensor)
Arguments
tensor |
Tensor of type GGML_TYPE_I32 |
Value
Integer vector
Examples
ctx <- ggml_init(1024 * 1024)
pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 10)
ggml_set_i32(pos, 0:9)
ggml_get_i32(pos)
ggml_free(ctx)
Get a Layer from a Sequential Model
Description
Retrieves a layer by name or by integer index (1-based).
Usage
ggml_get_layer(model, index = NULL, name = NULL)
Arguments
model |
A ggml_sequential_model object |
index |
Integer index of the layer (1-based), or NULL |
name |
Character name of the layer, or NULL |
Value
The layer list object
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(64, activation = "relu", name = "hidden") |>
ggml_layer_dense(10, activation = "softmax", name = "output")
ggml_get_layer(model, index = 1)
ggml_get_layer(model, name = "output")
Get Maximum Tensor Size
Description
Returns the maximum tensor size that can be allocated in the context
Usage
ggml_get_max_tensor_size(ctx)
Arguments
ctx |
GGML context |
Value
Maximum tensor size in bytes
Examples
ctx <- ggml_init(1024 * 1024)
ggml_get_max_tensor_size(ctx)
ggml_free(ctx)
Get Context Memory Size
Description
Returns the total memory pool size of the context
Usage
ggml_get_mem_size(ctx)
Arguments
ctx |
GGML context |
Value
Total memory size in bytes
Examples
ctx <- ggml_init(1024 * 1024)
ggml_get_mem_size(ctx)
ggml_free(ctx)
Get Number of Threads
Description
Get the current number of threads for GGML operations
Usage
ggml_get_n_threads()
Value
Number of threads
Examples
ggml_get_n_threads()
Get Tensor Name
Description
Retrieves the name of a tensor.
Usage
ggml_get_name(tensor)
Arguments
tensor |
Tensor pointer |
Value
Character string name or NULL if not set
Get Next Tensor from Context
Description
Get Next Tensor from Context
Usage
ggml_get_next_tensor(ctx, tensor)
Arguments
ctx |
GGML context |
tensor |
Current tensor |
Value
Next tensor pointer or NULL
Get No Allocation Mode
Description
Check if no-allocation mode is enabled
Usage
ggml_get_no_alloc(ctx)
Arguments
ctx |
GGML context |
Value
Logical indicating if no_alloc is enabled
Examples
ctx <- ggml_init(1024 * 1024)
ggml_get_no_alloc(ctx)
ggml_free(ctx)
Operation Parameters
Description
Returns the raw op_params bytes from a tensor. These parameters control operation-specific behavior (e.g., precision, mode).
Usage
ggml_get_op_params(tensor)
ggml_set_op_params(tensor, params)
ggml_get_op_params_i32(tensor, index)
ggml_set_op_params_i32(tensor, index, value)
ggml_get_op_params_f32(tensor, index)
ggml_set_op_params_f32(tensor, index, value)
Arguments
tensor |
External pointer to tensor |
params |
Raw vector of parameters (max 64 bytes) |
index |
0-based index (0-15 for 64-byte op_params) |
value |
Numeric value to set |
Details
ggml_set_op_params(): Sets the raw op_params bytes for a tensor.
Returns: NULL invisibly
ggml_get_op_params_i32(): Gets a single int32 value from tensor op_params at given index.
Returns: Integer value
ggml_set_op_params_i32(): Sets a single int32 value in tensor op_params at given index.
Returns: NULL invisibly
ggml_get_op_params_f32(): Gets a single float value from tensor op_params at given index.
Returns: Numeric value
ggml_set_op_params_f32(): Sets a single float value in tensor op_params at given index.
Returns: NULL invisibly
Value
Raw vector of op_params bytes
See Also
Other tensor:
ggml_are_same_layout()
Get Relative Position (Graph)
Description
Gathers relative-position rows for relative-position attention bias.
Usage
ggml_get_rel_pos(ctx, a, qh, kh)
Arguments
ctx |
GGML context |
a |
Input tensor |
qh |
Query height |
kh |
Key height |
Value
Relative-position tensor
Get Rows by Indices (Graph)
Description
Creates a graph node that extracts rows from a tensor by index. This is commonly used for embedding lookup in LLMs.
Usage
ggml_get_rows(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Data tensor of shape [n_embd, n_rows, ...] - the embedding table |
b |
Index tensor (int32) of shape [n_indices] - which rows to extract |
Details
This operation is fundamental for embedding lookup in transformers: given a vocabulary embedding matrix and token indices, it retrieves the corresponding embedding vectors.
Value
Tensor of shape [n_embd, n_indices, ...] containing the selected rows
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Create embedding matrix: 10 tokens, 4-dim embeddings
embeddings <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 10)
ggml_set_f32(embeddings, rnorm(40))
# Token indices to look up
indices <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 3)
ggml_set_i32(indices, c(0L, 5L, 2L))
# Get embeddings for tokens 0, 5, 2
result <- ggml_get_rows(ctx, embeddings, indices)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Get Rows Backward (Graph)
Description
Backward pass for ggml_get_rows operation. Accumulates gradients at the original row positions.
Usage
ggml_get_rows_back(ctx, a, b, c)
Arguments
ctx |
GGML context |
a |
Gradient of get_rows output |
b |
Index tensor (same as forward pass) |
c |
Reference tensor defining output shape |
Value
Gradient tensor for the embedding matrix
Gated Linear Unit Operations (Graph)
Description
Creates a graph node for GLU operation with specified gating type. GLU splits the input tensor in half along the first dimension, applies an activation to the first half (x), and multiplies it with the second half (gate).
Usage
ggml_glu(ctx, a, op, swapped = FALSE)
ggml_reglu(ctx, a)
ggml_geglu(ctx, a)
ggml_swiglu(ctx, a)
ggml_geglu_quick(ctx, a)
ggml_glu_split(ctx, a, b, op)
ggml_reglu_split(ctx, a, b)
ggml_geglu_split(ctx, a, b)
ggml_swiglu_split(ctx, a, b)
ggml_swiglu_clamp(ctx, a, b = NULL, limit)
Arguments
ctx |
GGML context |
a |
Input tensor (first dimension must be even) |
op |
GLU operation type (GGML_GLU_OP_REGLU, GGML_GLU_OP_GEGLU, etc.) |
swapped |
If TRUE, swap x and gate halves (default FALSE) |
b |
Gate tensor (same shape as a) |
limit |
Clamp bound (positive) |
Details
Formula: output = activation(x) * gate where x and gate are the two halves of the input tensor.
ggml_reglu(): Creates a graph node for ReGLU operation.
ReGLU uses ReLU as the activation function on the first half.
Formula: output = ReLU(x) * gate Returns: Tensor with half the first dimension of input
ggml_geglu(): Creates a graph node for GeGLU operation.
GeGLU uses GELU as the activation function on the first half.
CRITICAL for models like GPT-NeoX and Falcon.
Formula: output = GELU(x) * gate Returns: Tensor with half the first dimension of input
ggml_swiglu(): Creates a graph node for SwiGLU operation.
SwiGLU uses SiLU (Swish) as the activation function on the first half.
CRITICAL for LLaMA, Mistral, and many modern LLMs.
Formula: output = SiLU(x) * gate Returns: Tensor with half the first dimension of input
ggml_geglu_quick(): Creates a graph node for fast GeGLU approximation.
Uses faster but less accurate GELU approximation for gating.
Returns: Tensor with half the first dimension of input
ggml_glu_split(): Creates a graph node for GLU with separate input and gate tensors.
Unlike standard GLU which splits a single tensor, this takes two separate tensors.
Returns: Tensor with same shape as input tensors
ggml_reglu_split(): Creates a graph node for ReGLU with separate input and gate tensors.
Formula: output = ReLU(a) * b Returns: Tensor with same shape as input tensors
ggml_geglu_split(): Creates a graph node for GeGLU with separate input and gate tensors.
Formula: output = GELU(a) * b Returns: Tensor with same shape as input tensors
ggml_swiglu_split(): Creates a graph node for SwiGLU with separate input and gate tensors.
Formula: output = SiLU(a) * b Returns: Tensor with same shape as input tensors
ggml_swiglu_clamp(): SwiGLU whose inputs are clamped before they meet: the gate from above, the
value from both sides, silu(min(a, limit)) * clamp(b, -limit, limit).
Returns: Tensor; same shape as a, or half its dim 0 when b is NULL
Value
Tensor with shape [n/2, ...] where n is the first dimension of input
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Create tensor with 10 columns (will be split into 5 + 5)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 4)
ggml_set_f32(a, rnorm(40))
# Apply SwiGLU
r <- ggml_glu(ctx, a, GGML_GLU_OP_SWIGLU, FALSE)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # Shape: 5x4
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3)
ggml_set_f32(a, rnorm(24))
r <- ggml_reglu(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # Shape: 4x3
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3)
ggml_set_f32(a, rnorm(24))
r <- ggml_geglu(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # Shape: 4x3
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3)
ggml_set_f32(a, rnorm(24))
r <- ggml_swiglu(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # Shape: 4x3
ggml_free(ctx)
Append a Node to a Computation Graph
Description
Adds tensor to graph as a node, without expanding its
dependencies. For the usual case – adding a tensor together with everything
it depends on – use ggml_graph_expand.
The node is marked for computation (in the C API that is left to the caller).
Usage
ggml_graph_add_node(graph, tensor)
Arguments
graph |
Graph to add to |
tensor |
Tensor to append |
Value
NULL, invisibly; graph is modified in place
See Also
Clear a Computation Graph
Description
Removes every node from graph, keeping its allocated capacity so it
can be rebuilt without a fresh allocation.
Usage
ggml_graph_clear(graph)
Arguments
graph |
Graph to clear |
Value
NULL, invisibly; graph is modified in place
Compute Computation Graph
Description
Executes the computation graph using CPU backend
Usage
ggml_graph_compute(ctx, graph)
Arguments
ctx |
GGML context |
graph |
Graph object created by ggml_build_forward_expand |
Value
No return value, called for side effects
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_set_f32(a, 1:10)
ggml_set_f32(b, 11:20)
c <- ggml_add(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, c)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(c)
ggml_free(ctx)
Compute Graph with Context (Alternative Method)
Description
Computes the computation graph using the context-based method. This is an alternative to ggml_graph_compute() that uses ggml_graph_plan() and ggml_graph_compute() internally.
Usage
ggml_graph_compute_with_ctx(ctx, graph, n_threads = 0L)
Arguments
ctx |
GGML context |
graph |
Graph object created by ggml_build_forward_expand |
n_threads |
Number of threads to use (0 for auto-detect, default: 0) |
Value
No return value, called for side effects
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_set_f32(a, 1:10)
c <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, c)
ggml_graph_compute_with_ctx(ctx, graph)
result <- ggml_get_f32(c)
ggml_free(ctx)
Export Graph to DOT Format
Description
Exports the computation graph to a DOT file for visualization. The DOT file can be converted to an image using Graphviz tools.
Usage
ggml_graph_dump_dot(graph, leafs = NULL, filename)
Arguments
graph |
Graph object |
leafs |
Optional graph with leaf tensors (NULL for none) |
filename |
Output filename (should end with .dot) |
Value
No return value, called for side effects
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, b)
ggml_graph_dump_dot(graph, NULL, tempfile(fileext = ".dot"))
ggml_free(ctx)
Add Another Root to an Existing Computation Graph
Description
ggml_build_forward_expand() always creates a fresh graph, so it can
only express a single root. A model with several independent output
branches needs every output expanded into the same graph: an output that is
unreachable from the first root never enters the graph, the scheduler never
assigns it a buffer, and reading it back fails. This appends tensor
and its ancestors to a graph that already exists.
Usage
ggml_graph_expand(graph, tensor)
Arguments
graph |
Graph object returned by |
tensor |
Additional output tensor to expand into |
Value
NULL, invisibly. graph is modified in place.
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, 1:4)
ggml_set_f32(b, 5:8)
# Two independent outputs -- neither reachable from the other.
o1 <- ggml_add(ctx, a, a)
o2 <- ggml_add(ctx, b, b)
graph <- ggml_build_forward_expand(ctx, o1)
ggml_graph_expand(graph, o2)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Graph Inspection
Description
Returns the number of computation nodes in the graph
Usage
ggml_graph_get_grad(graph, node)
ggml_graph_get_grad_acc(graph, node)
ggml_graph_n_nodes(graph)
ggml_graph_node(graph, i)
ggml_graph_overhead()
ggml_graph_get_tensor(graph, name)
Arguments
graph |
Graph object |
node |
Tensor whose gradient is wanted |
i |
Node index (0-based, negative indices count from end) |
name |
Character string with tensor name |
Details
ggml_graph_get_grad(): Returns the tensor holding d(output)/d(node) after
ggml_build_backward_expand has added the backward pass and the
graph has been computed.
Returns NULL when the node has no gradient in this graph – because it
was never marked with ggml_set_param, or because nothing
differentiable connects it to the output.
Returns: The gradient tensor, or NULL
ggml_graph_node(): Gets a specific node (tensor) from the computation graph by index
Returns: Tensor pointer
ggml_graph_overhead(): Returns the memory overhead required for a computation graph
Returns: Size in bytes
ggml_graph_get_tensor(): Finds a tensor in the computation graph by its name
Returns: Tensor pointer or NULL if not found
Value
For ggml_graph_get_grad_acc(), the gradient accumulator
tensor, or NULL
Integer number of nodes
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_add(ctx, a, a)
graph <- ggml_build_forward_expand(ctx, b)
# Get the last node (output)
output <- ggml_graph_node(graph, -1)
ggml_free(ctx)
Add a Tensor to a Graph Without Computing It
Description
Adds tensor and its parents to graph in dependency order, like
ggml_graph_expand, but without marking them for computation.
They are computed only once a node added with ggml_graph_expand (or
ggml_build_forward_expand) reaches them. Use it to fix where
nodes appear in the graph without forcing them to run.
Usage
ggml_graph_order(graph, tensor)
Arguments
graph |
Graph pointer |
tensor |
Tensor to add |
Value
NULL, invisibly
Print Graph Information
Description
Prints debug information about the computation graph
Usage
ggml_graph_print(graph)
Arguments
graph |
Graph object |
Value
No return value, called for side effects
Reset Graph (for backpropagation)
Description
Zeroes every gradient in the graph and seeds the loss node's own gradient
with 1 – the value the chain rule starts from. Call this after
ggml_build_backward_expand and before
ggml_graph_compute: without it every gradient computes as
zero, since the backward pass multiplies through a seed that was never set.
Usage
ggml_graph_reset(graph)
Arguments
graph |
Graph object with gradients allocated |
Details
Requires a graph built by ggml_build_forward_expand_grads;
an inference-only graph has no gradients to reset and is rejected.
Value
No return value, called for side effects
See Also
ggml_build_backward_expand,
ggml_build_forward_expand_grads
Graph Copies and Views
Description
Copies the contents of src into dst, which must already have
room for them. Unlike ggml_graph_dup this allocates nothing.
Usage
ggml_graph_view(graph, i0, i1)
ggml_graph_dup(ctx, graph, force_grads = FALSE)
ggml_graph_cpy(src, dst)
Arguments
graph |
Graph to duplicate |
i0 |
Start index (0-based, inclusive) |
i1 |
End index (exclusive) |
ctx |
GGML context to allocate the copy in |
force_grads |
Logical; give the copy gradient storage (default
|
src |
Source graph |
dst |
Destination graph |
Details
ggml_graph_view(): Creates a view of a portion of a computation graph, containing nodes
from index i0 to i1 (exclusive). The view shares the underlying nodes
but does not include leaf tensors or gradients.
Returns: External pointer to graph view
ggml_graph_dup(): Copies graph into ctx. With force_grads = TRUE the copy
carries gradient storage even when the original does not, which is one way to
turn a plain forward graph into one that
ggml_build_backward_expand accepts.
Returns: A new graph object (external pointer)
Value
NULL, invisibly; dst is modified in place
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
b <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, b)
n_nodes <- ggml_graph_n_nodes(graph)
view <- ggml_graph_view(graph, 0, n_nodes)
ggml_free(ctx)
Group Normalization (Graph)
Description
Creates a graph node for group normalization. Normalizes along ne0*ne1*n_groups dimensions. Used in Stable Diffusion and other image generation models.
Usage
ggml_group_norm(ctx, a, n_groups, eps = 1e-05)
ggml_group_norm_inplace(ctx, a, n_groups, eps = 1e-05)
Arguments
ctx |
GGML context |
a |
Input tensor |
n_groups |
Number of groups to divide channels into |
eps |
Epsilon for numerical stability (default 1e-5) |
Details
ggml_group_norm_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the group norm operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# 4 channels, 2 groups (2 channels per group)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8)
ggml_set_f32(a, rnorm(32))
result <- ggml_group_norm(ctx, a, n_groups = 2)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8)
ggml_set_f32(a, rnorm(32))
result <- ggml_group_norm_inplace(ctx, a, n_groups = 2)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Create a GRU Layer Object
Description
Create a GRU Layer Object
Usage
ggml_gru(
units,
return_sequences = FALSE,
activation = "tanh",
recurrent_activation = "sigmoid",
name = NULL,
trainable = TRUE
)
Arguments
units |
Integer, number of hidden units. |
return_sequences |
Logical. |
activation |
Candidate activation (default |
recurrent_activation |
Gate activation (default |
name |
Optional character name. |
trainable |
Logical. |
Value
A ggml_layer object.
Hard Sigmoid Activation (Graph)
Description
Creates a graph node for Hard Sigmoid activation. HardSigmoid(x) = ReLU6(x + 3) / 6 = min(max(0, x + 3), 6) / 6. A computationally efficient approximation of the sigmoid function.
Usage
ggml_hardsigmoid(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the Hard Sigmoid operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-4, -1, 0, 1, 4))
r <- ggml_hardsigmoid(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # [0, 0.333, 0.5, 0.667, 1]
ggml_free(ctx)
Hard Swish Activation (Graph)
Description
Creates a graph node for Hard Swish activation. HardSwish(x) = x * ReLU6(x + 3) / 6 = x * min(max(0, x + 3), 6) / 6. Used in MobileNetV3 and other efficient architectures.
Usage
ggml_hardswish(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the Hard Swish operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-4, -1, 0, 1, 4))
r <- ggml_hardswish(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r)
ggml_free(ctx)
Image to Column (Graph)
Description
Transforms image data into column format for efficient convolution. This is a low-level operation used internally by convolution implementations.
Usage
ggml_im2col(
ctx,
a,
b,
s0,
s1,
p0,
p1,
d0,
d1,
is_2D = TRUE,
dst_type = GGML_TYPE_F16
)
Arguments
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride dimension 0 |
s1 |
Stride dimension 1 |
p0 |
Padding dimension 0 |
p1 |
Padding dimension 1 |
d0 |
Dilation dimension 0 |
d1 |
Dilation dimension 1 |
is_2D |
Whether this is a 2D operation (default TRUE) |
dst_type |
Output type (default GGML_TYPE_F16) |
Value
Transformed tensor in column format
3D Image to Column (Graph)
Description
Unfolds 3D patches of the input into columns, the first step of a 3D
convolution computed by matrix multiply (see ggml_conv_3d).
Usage
ggml_im2col_3d(ctx, a, b, IC, s = 1L, p = 0L, d = 1L, dst_type = GGML_TYPE_F16)
Arguments
ctx |
GGML context |
a |
Kernel tensor |
b |
Input tensor |
IC |
Number of input channels |
s |
Stride: one value for all axes or |
p |
Padding, same form as |
d |
Dilation, same form as |
dst_type |
Output type, |
Value
Tensor [KW * KH * KD * IC, OW, OH, OD * N]
See Also
Column to Image, the backward of im2col (Graph)
Description
Gradient of ggml_im2col with respect to its image input:
every column element is added back to the image position it was read from,
so overlapping windows (stride smaller than the kernel) sum; positions read
from the zero padding are dropped. With dY the gradient of a
convolution's output and W its kernel, the input gradient is
ggml_im2col_back(W^T dY, W, dim(x)).
Usage
ggml_im2col_back(ctx, a, b, ne, s0, s1, p0, p1, d0, d1, is_2D = TRUE)
Arguments
ctx |
GGML context |
a |
Gradient of the im2col output |
b |
Convolution kernel |
ne |
Shape of the im2col input, 4 values: |
s0 |
Stride dimension 0 |
s1 |
Stride dimension 1 |
p0 |
Padding dimension 0 |
p1 |
Padding dimension 1 |
d0 |
Dilation dimension 0 |
d1 |
Dilation dimension 1 |
is_2D |
Whether this is a 2D operation (default TRUE) |
Details
The gradient comes first, the kernel second: that is the order the CPU and
Vulkan kernels read and the order ggml's own backward pass uses (upstream's
parameter comments in ggml.h had them swapped).
Every input pixel gathers the taps that read it, with no atomics, so the
result does not depend on scheduling.
Value
F32 tensor of shape ne
See Also
Initialize GGML context
Description
Initialize GGML context
Usage
ggml_init(mem_size = 16 * 1024 * 1024, no_alloc = FALSE)
Arguments
mem_size |
Memory size in bytes |
no_alloc |
If TRUE, don't allocate memory for tensors (default: FALSE) |
Value
GGML context pointer
Examples
ctx <- ggml_init(1024 * 1024)
ggml_free(ctx)
Create Context with Auto-sizing
Description
Creates a context with automatically calculated size based on planned tensors.
Usage
ggml_init_auto(..., extra_mb = 10, type = GGML_TYPE_F32, no_alloc = FALSE)
Arguments
... |
Named arguments with tensor dimensions (integer vectors) |
extra_mb |
Extra megabytes to add (default: 10) |
type |
Tensor type (default: GGML_TYPE_F32) |
no_alloc |
If TRUE, don't allocate memory for tensors (default: FALSE) |
Value
GGML context
Examples
ctx <- ggml_init_auto(mat1 = c(1000L, 1000L), mat2 = c(1000L, 1000L))
ggml_free(ctx)
Inject a single-cell result back into its container
Description
Writes a ggml_result back into the appropriate slot of a Seurat
or SingleCellExperiment object, returning the updated object. The
destination depends on
result$metadata$kind: a dimensionality reduction (the default) goes into
a DimReduc; a "transform" (normalize / scale) overwrites an assay
layer; a "graph" (neighbors) writes <assay>_nn and
<assay>_snn Graph objects into @graphs, exactly where
Seurat's FindNeighbors puts them so FindClusters can consume them.
Component standard deviations and the backend used are recorded alongside so
downstream tools and the user can see how the result was produced.
Usage
ggml_inject(
x,
result,
reduction_name = "ggml",
key = "GGML_",
assay = NULL,
...
)
## S3 method for class 'Seurat'
ggml_inject(
x,
result,
reduction_name = "ggml",
key = "GGML_",
assay = NULL,
...
)
## S3 method for class 'SingleCellExperiment'
ggml_inject(
x,
result,
reduction_name = "ggml",
key = "GGML_",
assay = NULL,
...
)
Arguments
x |
A |
result |
A |
reduction_name |
Name of the reduction slot to create, e.g.
|
key |
Column-name prefix for the embedding, e.g. |
assay |
Assay to associate the reduction with (Seurat). Defaults to the object's default assay. |
... |
Passed to methods. |
Value
The updated container.
Declare a Functional API Input Tensor
Description
Creates a symbolic input node for the Functional API. The node records
only the shape of one sample (without batch dimension); actual
memory is allocated when ggml_compile() is called.
Usage
ggml_input(shape, name = NULL, dtype = "float32")
Arguments
shape |
Integer vector describing the shape of a single sample.
For flat feature vectors use a scalar, e.g. |
name |
Optional character name for the input tensor. |
dtype |
Data type of the input: |
Value
A ggml_tensor_node object.
Examples
x <- ggml_input(shape = 64L)
x <- ggml_input(shape = c(28L, 28L, 1L), name = "image")
x <- ggml_input(shape = 10L, dtype = "int32") # token indices
Check if GGML is available
Description
Check if GGML is available
Usage
ggml_is_available()
Value
TRUE if GGML library is loaded
Examples
ggml_is_available()
Tensor Contiguity Checks
Description
Returns TRUE if tensor data is stored contiguously in memory
Usage
ggml_is_contiguous_0(tensor)
ggml_is_contiguous_1(tensor)
ggml_is_contiguous_2(tensor)
ggml_is_contiguous_to_1(tensor)
ggml_is_contiguous_to_2(tensor)
ggml_is_contiguous_to_3(tensor)
ggml_is_contiguously_allocated(tensor)
ggml_is_contiguous_channels(tensor)
ggml_is_contiguous_rows(tensor)
ggml_is_contiguous(tensor)
Arguments
tensor |
Tensor pointer |
Details
ggml_is_contiguous_0(): Check if tensor is contiguous. Same as ggml_is_contiguous.
Returns: Logical indicating contiguity
ggml_is_contiguous_1(): Check if tensor is contiguous for dimensions >= 1.
Allows non-contiguous first dimension.
Returns: Logical indicating contiguity for dims >= 1
ggml_is_contiguous_2(): Check if tensor is contiguous for dimensions >= 2.
Allows non-contiguous first two dimensions.
Returns: Logical indicating contiguity for dims >= 2
ggml_is_contiguous_to_1(): Check whether the first 1 dimension(s) of a tensor are packed (dims
0 .. 1 - 1 in ggml order); the higher dimensions may have any stride.
Returns: Logical indicating contiguity for dims < 1
ggml_is_contiguous_to_2(): Check whether the first 2 dimension(s) of a tensor are packed (dims
0 .. 2 - 1 in ggml order); the higher dimensions may have any stride.
Returns: Logical indicating contiguity for dims < 2
ggml_is_contiguous_to_3(): Check whether the first 3 dimension(s) of a tensor are packed (dims
0 .. 3 - 1 in ggml order); the higher dimensions may have any stride.
Returns: Logical indicating contiguity for dims < 3
ggml_is_contiguously_allocated(): Check if tensor data is contiguously allocated in memory.
Different from contiguous layout - this checks the actual allocation.
Returns: Logical indicating if data is contiguously allocated
ggml_is_contiguous_channels(): Check if tensor has contiguous channels (important for CNN operations).
Data for each channel should be stored contiguously.
Returns: Logical indicating channel-wise contiguity
ggml_is_contiguous_rows(): Check if tensor has contiguous rows (important for matrix operations).
Each row should be stored contiguously in memory.
Returns: Logical indicating row-wise contiguity
Value
Logical
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_is_contiguous(t)
ggml_free(ctx)
Check if Tensor is Permuted
Description
Returns TRUE if tensor dimensions have been permuted
Usage
ggml_is_permuted(tensor)
Arguments
tensor |
Tensor pointer |
Value
Logical
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20)
ggml_is_permuted(t)
ggml_free(ctx)
Check if Tensor is Transposed
Description
Returns TRUE if tensor has been transposed
Usage
ggml_is_transposed(tensor)
Arguments
tensor |
Tensor pointer |
Value
Logical
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20)
ggml_is_transposed(t)
ggml_free(ctx)
L2 Normalization (Graph)
Description
Creates a graph node for L2 normalization (unit norm). Normalizes vectors to unit length: x / ||x||_2. Used in RWKV v7 and embedding normalization.
Usage
ggml_l2_norm(ctx, a, eps = 1e-05)
ggml_l2_norm_inplace(ctx, a, eps = 1e-05)
Arguments
ctx |
GGML context |
a |
Input tensor |
eps |
Epsilon for numerical stability (default 1e-5) |
Details
ggml_l2_norm_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the L2 norm operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(3, 0, 0, 4)) # Length = 5
result <- ggml_l2_norm(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [0.6, 0, 0, 0.8] unit vector
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(3, 0, 0, 4))
result <- ggml_l2_norm_inplace(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Element-wise Addition of Two Tensor Nodes
Description
Adds two (or more) tensor nodes element-wise. All tensors must have the same shape. This is the functional equivalent of a residual / skip connection.
Usage
ggml_layer_add(tensors, name = NULL)
Arguments
tensors |
A list of |
name |
Optional character name for the layer. |
Value
A new ggml_tensor_node representing the sum.
Examples
x <- ggml_input(shape = 64L)
a <- x |> ggml_layer_dense(64, activation = "relu")
b <- x |> ggml_layer_dense(64)
out <- ggml_layer_add(list(a, b))
Add a Multi-Head Attention Layer
Description
Scaled dot-product attention with n_heads heads, in the pipe-style
form that appends a layer to a functional graph. For the reusable
layer object that shares one set of weights between several
applications – how an encoder block is reused across a stack – see
ggml_attention.
Usage
ggml_layer_attention(
x,
d_model,
n_heads = 1L,
causal = FALSE,
context = NULL,
mask = NULL,
rope = FALSE,
rope_base = 10000,
dropout = 0,
bias = TRUE,
name = NULL,
trainable = TRUE
)
Arguments
x |
A |
d_model |
Integer, model width. Must be divisible by |
n_heads |
Integer, number of attention heads (default 1). |
causal |
Logical: mask out positions after the current one, making the
layer autoregressive (default |
context |
Optional |
mask |
Optional |
rope |
Logical: apply rotary position embedding to the queries and keys
(default |
rope_base |
Rotation base for |
dropout |
Dropout rate applied to the attention weights during training (default 0, no dropout). used by LLaMA-family models). Larger values stretch the rotation over longer contexts. |
bias |
Logical: add a bias to the output projection (default
|
name |
Optional layer name. |
trainable |
Logical. |
Value
A new ggml_tensor_node of shape c(seq_len, d_model).
Shapes
x must be a sequence node of shape c(seq_len, d_model), the
same layout ggml_layer_lstm takes. The output has that same
shape, so attention blocks stack directly and a residual connection
(ggml_layer_add(list(x, attn_out))) needs no reshaping.
How it is computed
All heads are computed in one batched pass rather than in a loop: the
projections are reshaped to [d_head, n_heads, seq, batch] and
permuted so that ggml_mul_mat() batches over the head and batch axes.
The node count is therefore independent of n_heads.
Cross-attention
Pass a list of two nodes, ggml_layer_attention(list(query, context),
...), to take queries from the first and keys/values from the second. The
two may differ in sequence length but must share d_model. A single
node is self-attention.
See Also
ggml_attention for the shared-weights layer object.
Examples
# A transformer encoder block: attention + residual, then a feed-forward
# sublayer + residual.
x <- ggml_input(shape = c(10L, 32L))
attn <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
h <- ggml_layer_add(list(x, attn))
ff <- h |> ggml_layer_dense(32L, activation = "relu")
out <- ggml_layer_add(list(h, ff))
# GPT-style causal self-attention.
dec <- x |> ggml_layer_attention(32L, n_heads = 4L, causal = TRUE)
# Padding mask, fed as a second input alongside the sequence.
pad <- ggml_input(shape = c(10L, 10L), name = "mask")
enc <- x |> ggml_layer_attention(32L, n_heads = 4L, mask = pad)
Add Batch Normalization Layer
Description
Applies normalization: RMS-normalizes the input, then scales by gamma
and shifts by beta (both learnable). Uses ggml_rms_norm which
supports backward pass for training.
Usage
ggml_layer_batch_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)
Arguments
model |
A ggml_sequential_model object |
eps |
Small constant for numerical stability (default 1e-5) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
Value
The model object with the batch_norm layer appended (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(128, input_shape = 784) |>
ggml_layer_batch_norm() |>
ggml_layer_dense(10, activation = "softmax")
Concatenate Tensor Nodes Along an Axis
Description
Concatenates two or more tensor nodes along the specified axis.
Usage
ggml_layer_concatenate(tensors, axis = 0L, name = NULL)
Arguments
tensors |
A list of |
axis |
Integer axis along which to concatenate (0-based, ggml convention).
Default |
name |
Optional character name for the layer. |
Value
A new ggml_tensor_node representing the concatenated tensor.
Examples
x <- ggml_input(shape = 32L)
y <- ggml_input(shape = 32L)
out <- ggml_layer_concatenate(list(x, y), axis = 0L)
Add 1D Convolution Layer
Description
Add 1D Convolution Layer
Usage
ggml_layer_conv_1d(
model,
filters,
kernel_size,
activation = NULL,
input_shape = NULL,
strides = 1L,
padding = "valid",
name = NULL,
trainable = TRUE
)
Arguments
model |
A ggml_sequential_model object |
filters |
Number of output filters |
kernel_size |
Integer kernel size |
activation |
Activation function name: "relu", "sigmoid", "tanh", "softmax", or NULL |
input_shape |
Input shape c(L, C) - required for first layer only (length, channels) |
strides |
Integer stride (default 1) |
padding |
"valid" (no padding) or "same" (preserve length) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
Value
The model object with the conv_1d layer appended (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_1d(32, 3, activation = "relu",
input_shape = c(100, 1))
Add 2D Convolution Layer
Description
Add 2D Convolution Layer
Usage
ggml_layer_conv_2d(
model,
filters,
kernel_size,
activation = NULL,
input_shape = NULL,
strides = c(1L, 1L),
padding = "valid",
name = NULL,
trainable = TRUE
)
Arguments
model |
A ggml_sequential_model object |
filters |
Number of output filters |
kernel_size |
Integer or vector of 2 integers for kernel height and width |
activation |
Activation function name: "relu", "sigmoid", "tanh", "softmax", or NULL |
input_shape |
Input shape c(H, W, C) - required for first layer only |
strides |
Integer or vector of 2 integers for stride |
padding |
"valid" (no padding) or "same" (preserve spatial dims) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
Value
The model object with the conv_2d layer appended (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1))
Define a Custom Layer From an R Function
Description
Turns an arbitrary composition of ggml operations into a reusable functional
layer. ggml_layer_custom() does not build anything itself — it
returns a layer function that behaves like any other
ggml_layer_*() and can be piped into a model.
Usage
ggml_layer_custom(forward, name = NULL, output_shape = NULL)
Arguments
forward |
A function of |
name |
Optional character name for the layer. Used as the base for auto-generated node names. |
output_shape |
Output shape excluding the batch dimension. Defaults to
|
Details
The forward function is called once while the compute graph is being
built, with the active ggml_context and the parent tensor. Because
the returned tensor becomes an ordinary graph node, gradients flow through a
custom layer automatically — as long as every operation used inside
forward implements a backward pass.
Value
A layer function of (x, name = NULL) that appends a custom
node to the graph and returns the new ggml_tensor_node.
Weights
Custom layers are stateless: forward maps activations to activations
and owns no trainable parameters. Put learnable weights in a neighbouring
ggml_layer_dense(), or use the autograd API (ag_*) when a layer
needs its own parameters.
Examples
# Mish activation: x * tanh(softplus(x))
layer_mish <- ggml_layer_custom(
name = "mish",
forward = function(ctx, x) ggml_mul(ctx, x, ggml_tanh(ctx, ggml_softplus(ctx, x)))
)
inp <- ggml_input(shape = 8L)
out <- inp |>
ggml_layer_dense(16L) |>
layer_mish() |>
ggml_layer_dense(2L, activation = "softmax")
model <- ggml_model(inputs = inp, outputs = out)
Add Dense (Fully Connected) Layer
Description
Add Dense (Fully Connected) Layer
Usage
ggml_layer_dense(
model,
units,
activation = NULL,
input_shape = NULL,
name = NULL,
trainable = TRUE,
time_distributed = FALSE
)
Arguments
model |
A ggml_sequential_model object |
units |
Number of output units |
activation |
Activation function name: "relu", "sigmoid", "tanh", "softmax", or NULL |
input_shape |
Integer or integer vector specifying the input shape (only needed for the first layer) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
time_distributed |
Logical; apply the kernel per position of a sequence
input rather than flattening it (default |
Value
The model object with the dense layer appended (invisibly).
Time-distributed application
By default a dense layer flattens its input, so a sequence node of shape
c(seq_len, features) becomes a single vector of units and the
sequence axis is gone. With time_distributed = TRUE the same kernel
is applied independently at every position instead, sharing one set of
weights across the sequence, and the output keeps its length:
c(seq_len, units). This is the semantics of Keras'
TimeDistributed, and it is what the position-wise feed-forward
sublayer of a transformer block needs – without it, a dense layer after
ggml_layer_attention would collapse every position into one
vector.
It is computed as a single batched ggml_mul_mat() over the combined
position/batch axes, not as a loop over positions.
Functional API only: a sequential model carries a flat running shape, so there is no sequence axis to distribute over.
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_flatten() |>
ggml_layer_dense(128, activation = "relu")
# Position-wise feed-forward, as in a transformer block.
x <- ggml_input(shape = c(10L, 32L))
ff <- x |> ggml_layer_dense(64L, activation = "relu", time_distributed = TRUE)
Add Dropout Layer
Description
Applies dropout regularization. During training, multiplies all activations
by (1 - rate) (deterministic expected-value scaling).
During inference (training = FALSE), the layer is an identity (no change).
Usage
ggml_layer_dropout(
model,
rate,
stochastic = FALSE,
name = NULL,
trainable = FALSE
)
Arguments
model |
A |
rate |
Dropout rate in |
stochastic |
Logical. If |
name |
Optional layer name. |
trainable |
Ignored for dropout (no weights); kept for API consistency. |
Value
The model with the dropout layer appended, or a new tensor node.
Difference from Keras / inverted dropout
Keras implements inverted dropout: during training it applies a random
Bernoulli mask and scales surviving activations up by
1 / (1 - rate), so the expected value of each unit is preserved and
no scaling is needed at inference.
This implementation uses deterministic scaling (multiply by
(1 - rate) at training, identity at inference) — equivalent in
expected value but without stochastic noise. Consequences:
No random mask → the regularization signal is weaker (no co-adaptation breaking).
Activations at training are scaled down, not up — the magnitude seen by subsequent layers differs from Keras behaviour.
Results are fully deterministic and reproducible without setting a seed.
Note
With stochastic = TRUE the Bernoulli mask is regenerated once
per epoch (not per batch), because ggml_opt_fit processes all
batches inside a single C call. This is weaker than per-batch dropout
but stronger than the deterministic variant.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(128, activation = "relu", input_shape = 784L) |>
ggml_layer_dropout(0.5, stochastic = TRUE) |>
ggml_layer_dense(10, activation = "softmax")
Add Embedding Layer
Description
Looks up dense vectors for integer token indices. The input must be an
integer matrix of 0-based indices in [0, vocab_size - 1] (use
ggml_input(shape, dtype = "int32") in Functional mode).
Usage
ggml_layer_embedding(
model,
vocab_size,
dim,
name = NULL,
trainable = TRUE,
input_shape = NULL
)
Arguments
model |
A |
vocab_size |
Number of distinct tokens (vocabulary size). |
dim |
Embedding dimension (vector length per token). |
name |
Optional layer name. |
trainable |
Logical; whether embedding weights are updated during training. |
input_shape |
Sequence length, for an embedding that is the first layer
of a sequential model (its input is then a matrix of token indices,
one row per sample). Ignored in the functional API, where
|
Value
The model with the embedding layer appended, or a new tensor node.
Axis order
The layer's output shape is c(seq_len, dim) – one row per position,
as in Keras – which is the same convention a declared sequence input
(ggml_input(shape = c(seq_len, features))) uses. So the output feeds
ggml_layer_attention, ggml_layer_gru and
ggml_layer_lstm directly, with no axis juggling in between.
The underlying tensor is [dim, seq_len, N]: ggml is column-major, so
an R shape c(a, b) is the tensor [b, a, N] throughout the
package. That matters only when reading raw output tensors –
ggml_layer_flatten() gives the same vector either way.
Index validation
Indices must be in [0, vocab_size - 1]. Out-of-range values cause
undefined behaviour inside the ggml kernel (no bounds check is performed at
the R level).
Examples
inp <- ggml_input(shape = 10L, dtype = "int32")
out <- inp |>
ggml_layer_embedding(vocab_size = 1000L, dim = 32L) |>
ggml_layer_flatten() |>
ggml_layer_dense(10L, activation = "softmax")
model <- ggml_model(inputs = inp, outputs = out)
Add Flatten Layer
Description
Flattens the spatial dimensions into a single vector per sample.
Usage
ggml_layer_flatten(model, name = NULL, trainable = TRUE)
Arguments
model |
A ggml_sequential_model object |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Value
The model object with the flatten layer appended (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_flatten()
Global Average Pooling for 2D Feature Maps
Description
Reduces a [H, W, C] feature map to [C] by averaging all
spatial positions per channel. Equivalent to Keras
GlobalAveragePooling2D().
Usage
ggml_layer_global_average_pooling_2d(model, name = NULL, trainable = TRUE)
Arguments
model |
A |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Value
Updated model or a new ggml_tensor_node.
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_global_average_pooling_2d() |>
ggml_layer_dense(10, activation = "softmax")
Global Max Pooling for 2D Feature Maps
Description
Reduces a [H, W, C] feature map to [C] by taking the maximum
value per channel across all spatial positions. Equivalent to Keras
GlobalMaxPooling2D().
Usage
ggml_layer_global_max_pooling_2d(model, name = NULL, trainable = TRUE)
Arguments
model |
A |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Value
Updated model or a new ggml_tensor_node.
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_global_max_pooling_2d() |>
ggml_layer_dense(10, activation = "softmax")
Add a GRU Layer
Description
Gated Recurrent Unit recurrent layer. Implemented as an unrolled computation graph (BPTT).
Usage
ggml_layer_gru(
model,
units,
return_sequences = FALSE,
activation = "tanh",
recurrent_activation = "sigmoid",
input_shape = NULL,
name = NULL,
trainable = TRUE
)
Arguments
model |
A |
units |
Integer, number of hidden units. |
return_sequences |
Logical; return all hidden states or only the last. |
activation |
Activation for the candidate hidden state ( |
recurrent_activation |
Activation for z/r gates ( |
input_shape |
Input shape |
name |
Optional layer name. |
trainable |
Logical. |
Value
Updated model or a new ggml_tensor_node.
Weight layout
-
W_zh[input_size, 2*units]— input kernel for z and r gates. -
U_zh[units, 2*units]— recurrent kernel for z and r. -
b_zh[2*units]— bias for z and r. -
W_n[input_size, units]— input kernel for candidate. -
U_n[units, units]— recurrent kernel for candidate. -
b_n[units]— bias for candidate.
Examples
model <- ggml_model_sequential() |>
ggml_layer_gru(64L, input_shape = c(10L, 32L)) |>
ggml_layer_dense(10L, activation = "softmax")
Add Layer Normalization Layer
Description
Normalizes each sample over its own features – subtract the mean, divide by
the standard deviation – then scales by gamma and shifts by beta (both
learnable). Like ggml_layer_rms_norm it keeps no running
statistics and does not couple the samples in a batch, but unlike it, it does
centre the input. This is the normalization of the original transformer.
Usage
ggml_layer_layer_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)
Arguments
model |
A |
eps |
Small constant for numerical stability (default 1e-5) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
Value
The model with the layer appended, or a new ggml_tensor_node
in the functional API.
See Also
ggml_layer_rms_norm, ggml_layer_batch_norm
Examples
x <- ggml_input(shape = c(10L, 32L))
h <- x |> ggml_layer_layer_norm()
h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
Add an LSTM Layer
Description
Long Short-Term Memory recurrent layer. Implemented as an unrolled computation graph (BPTT) so that ggml's automatic differentiation works without any C extensions.
Usage
ggml_layer_lstm(
model,
units,
return_sequences = FALSE,
activation = "tanh",
recurrent_activation = "sigmoid",
input_shape = NULL,
name = NULL,
trainable = TRUE
)
Arguments
model |
A |
units |
Integer, number of hidden units. |
return_sequences |
Logical; if |
activation |
Activation for the cell gate (default |
recurrent_activation |
Activation for the recurrent step (default
|
input_shape |
Input shape |
name |
Optional layer name. |
trainable |
Logical. |
Value
Updated model or a new ggml_tensor_node.
Weight layout
-
W_gates[input_size, 4*units]— input kernel for all four gates (i, f, g, o) concatenated. -
U_gates[units, 4*units]— recurrent kernel. -
b_gates[4*units]— bias.
Input / output shapes
Input: [seq_len, input_size] per sample (R row-major), or a 3-D
array [N, seq_len, input_size]. In the Functional API the input
node shape should be c(seq_len, input_size).
Output (Sequential): [units] per sample when
return_sequences = FALSE (default), or c(seq_len, units)
when return_sequences = TRUE.
Examples
model <- ggml_model_sequential() |>
ggml_layer_lstm(64L, input_shape = c(10L, 32L)) |>
ggml_layer_dense(10L, activation = "softmax")
Add 2D Max Pooling Layer
Description
Add 2D Max Pooling Layer
Usage
ggml_layer_max_pooling_2d(
model,
pool_size = c(2L, 2L),
strides = NULL,
name = NULL,
trainable = TRUE
)
Arguments
model |
A ggml_sequential_model object |
pool_size |
Integer or vector of 2 integers for pool height and width |
strides |
Integer or vector of 2 integers (defaults to pool_size) |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Value
The model object with the max pooling layer appended (invisibly).
Examples
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_max_pooling_2d(c(2, 2))
Add Permute Layer
Description
Reorder a sample's axes. The batch axis is not part of dims and is
never moved – dims indexes the per-sample shape, exactly as
input_shape does.
Usage
ggml_layer_permute(
model,
dims,
input_shape = NULL,
name = NULL,
trainable = TRUE
)
Arguments
model |
A ggml_sequential_model object, or a |
dims |
Integer vector, a permutation of |
input_shape |
Input shape, required for the first layer only. |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Value
The model object with the permute layer appended (invisibly), or a
ggml_tensor_node in the functional API.
Argument order
dims follows base R's aperm: dims[i] names
the source axis that ends up at position i. So
dims = c(2, 1) means "the new first axis is the old second one".
This is deliberately the inverse of the underlying
ggml_permute, whose arguments are destination
positions; the layer converts between the two so that callers can think in
the R idiom. Read ggml_permute itself as "source axis i goes
to position p_i" – mixing the two conventions up silently inverts
every non-trivial permutation.
Typical use
Reordering a sample's axes to match what the next layer reads: a
c(features, seq_len) tensor into the c(seq_len, features) that
attention and the recurrent layers expect, or the reverse.
Note that ggml_layer_embedding does not need this –
it already reports c(seq_len, dim) and feeds attention, GRU and LSTM
directly. Adding a permute there transposes the underlying tensor and breaks
the graph.
See Also
ggml_permute for the low-level op and its
destination-position convention.
Examples
# Swap a c(features, seq_len) sample into c(seq_len, features).
model <- ggml_model_sequential() |>
ggml_layer_permute(c(2, 1), input_shape = c(32, 16))
Add a Learned Positional Embedding
Description
Adds a learned vector to each position of a sequence, so that attention –
which is order-blind on its own – can tell one position from another. The
table is [d_model, seq_len], one row per position, added to the input
and broadcast over the batch.
Usage
ggml_layer_positional_embedding(model, name = NULL, trainable = TRUE)
Arguments
model |
A |
name |
Optional character name for the layer. |
trainable |
Logical; whether the table is updated during training. |
Details
Unlike ggml_layer_embedding there are no indices to look up:
the position IS the place in the sequence, so this layer takes the sequence
itself and returns it with the positional term added. The shape is
unchanged.
Value
The model with the layer appended, or a new ggml_tensor_node
of the same shape as the input.
See Also
ggml_layer_attention, ggml_layer_embedding
Examples
# Without this the encoder cannot distinguish "ab" from "ba".
x <- ggml_input(shape = c(10L, 32L))
h <- x |> ggml_layer_positional_embedding()
h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
Add Reshape Layer
Description
Reinterpret a sample's shape without moving any data. The batch axis is not
part of shape and is never reshaped – shape describes one
sample, exactly as input_shape does.
Usage
ggml_layer_reshape(
model,
shape,
input_shape = NULL,
name = NULL,
trainable = TRUE
)
Arguments
model |
A ggml_sequential_model object, or a |
shape |
Integer vector, the new per-sample shape. At most one entry may
be |
input_shape |
Input shape, required for the first layer only. |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Value
The model object with the reshape layer appended (invisibly), or a
ggml_tensor_node in the functional API.
Inferred axis
One entry of shape may be -1, in which case it is computed
from the element count: ggml_layer_reshape(c(-1, 32)) on a
c(4, 8, 16) input gives c(16, 32). At most one axis may be
-1, and the remaining sizes must divide the element count exactly.
Reshape versus permute
Reshape only relabels the axes; the elements keep their memory order. To
actually reorder axes – swapping c(a, b) into c(b, a), say –
use ggml_layer_permute. Using reshape for that silently
interleaves the data instead.
Non-contiguous input
ggml_reshape_*() requires contiguous data, and a permuted tensor is
not contiguous. The layer inserts ggml_cont when needed, so it
works after a permute; the cost is one copy.
See Also
ggml_layer_permute to reorder axes,
ggml_layer_flatten to collapse them all.
Examples
# Split a flat feature vector into a sequence of 32-wide steps.
model <- ggml_model_sequential() |>
ggml_layer_dense(512, input_shape = 128) |>
ggml_layer_reshape(c(-1, 32))
Add RMS Normalization Layer
Description
Normalizes each sample by the root mean square of its features, then scales
by gamma and shifts by beta (both learnable). Unlike
ggml_layer_batch_norm it does not subtract the mean and keeps
no running statistics, so it behaves identically in training and inference
and does not couple the samples in a batch. This is the normalization used
by transformer blocks.
Usage
ggml_layer_rms_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)
Arguments
model |
A |
eps |
Small constant for numerical stability (default 1e-5) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
Value
The model with the layer appended, or a new ggml_tensor_node
in the functional API.
See Also
Examples
# Functional API: the normalization of a transformer block.
x <- ggml_input(shape = c(10L, 32L))
h <- x |> ggml_layer_rms_norm()
h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
Pool a Sequence Down to One Vector
Description
Collapses the sequence axis, turning c(seq_len, d_model) into
d_model: the step that lets a classification or regression head sit
on top of an encoder. ggml_layer_flatten() also produces a flat
vector, but keeps every position separately (seq_len * d_model
features), so its width – and the head above it – depends on the sequence
length. Pooling does not.
Usage
ggml_layer_sequence_pooling(model, mode = c("mean", "first"), name = NULL)
Arguments
model |
A |
mode |
|
name |
Optional character name for the layer. |
Details
mode = "mean" averages over the positions; every position
contributes equally, which is the usual choice for an encoder without a
dedicated summary token. mode = "first" takes position 1 and ignores
the rest – the CLS-token convention, where attention is expected to have
gathered what matters into that position.
Value
The model with the layer appended, or a new ggml_tensor_node
of shape d_model.
See Also
ggml_layer_flatten, ggml_layer_attention
Examples
# An encoder with a regression head: the head's width is d_model, whatever
# the sequence length turns out to be.
x <- ggml_input(shape = c(10L, 32L))
h <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
h <- h |> ggml_layer_sequence_pooling()
y <- h |> ggml_layer_dense(1L)
Add a Transformer Encoder Block
Description
The standard pre-LN encoder block, in one call: normalize, attend, add the residual, normalize again, run the position-wise feed-forward, add the second residual. Composed from the existing layers – there is no new node type – so the pieces can still be wired by hand when a block needs to differ.
Usage
ggml_layer_transformer_block(
x,
d_model,
n_heads = 1L,
ff_dim = NULL,
activation = "silu",
norm = c("rms", "layer"),
causal = FALSE,
mask = NULL,
rope = FALSE,
dropout = 0,
attn_dropout = dropout,
name = NULL
)
Arguments
x |
A |
d_model |
Integer, model width. Must match the input's feature
dimension and be divisible by |
n_heads |
Integer, number of attention heads (default 1). |
ff_dim |
Integer, width of the feed-forward hidden layer. Defaults to
|
activation |
Feed-forward activation (default |
norm |
|
causal |
Logical: causal self-attention (default |
mask |
Optional attention mask node, |
rope |
Logical: rotary position embedding on queries and keys
(default |
dropout |
Dropout rate applied to each sublayer's output before it is added back to the residual (default 0, meaning no dropout). |
attn_dropout |
Dropout rate for the attention weights specifically;
defaults to |
name |
Optional prefix for the layers inside the block. |
Value
A new ggml_tensor_node of shape c(seq_len, d_model),
so blocks stack directly.
Pre-LN vs post-LN
Normalization comes BEFORE each sublayer, not after, which is what makes deep stacks trainable without a warmup schedule: the residual path stays un-normalized from input to output, so the gradient reaches the early layers undiminished. The original 2017 transformer normalized after; every large model since does it this way.
See Also
ggml_layer_attention,
ggml_layer_sequence_pooling
Examples
# A two-block encoder with a pooled regression head.
x <- ggml_input(shape = c(10L, 32L))
h <- x |> ggml_layer_positional_embedding()
for (i in 1:2) h <- h |> ggml_layer_transformer_block(32L, n_heads = 4L)
y <- h |> ggml_layer_sequence_pooling() |> ggml_layer_dense(1L)
Leaky ReLU Activation (Graph)
Description
Creates a graph node for Leaky ReLU activation. LeakyReLU(x) = x if x > 0, else negative_slope * x. Unlike standard ReLU, Leaky ReLU allows a small gradient for negative values.
Usage
ggml_leaky_relu(ctx, a, negative_slope = 0.01, inplace = FALSE)
Arguments
ctx |
GGML context |
a |
Input tensor |
negative_slope |
Slope for negative values (default: 0.01) |
inplace |
If TRUE, operation is performed in-place (default: FALSE) |
Value
Tensor representing the Leaky ReLU operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
r <- ggml_leaky_relu(ctx, a, negative_slope = 0.1)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # [-0.2, -0.1, 0, 1, 2]
ggml_free(ctx)
DSA Lightning Indexer (Graph)
Description
Scores every key position for every query token as used by DeepSeek sparse
attention: s[k, t] = \sum_h \max(q_{h,t} \cdot k_k, 0)\, w[h, t] + m[k, t].
Usage
ggml_lightning_indexer(ctx, q, k, weights, mask)
Arguments
ctx |
GGML context |
q |
Queries, F32 |
k |
Keys |
weights |
Per-head weights, F32 |
mask |
Additive mask, F16 |
Value
F32 tensor [n_kv, n_batch, 1, ne3]
Load a Full Model (Architecture + Weights)
Description
Restores a model previously saved with ggml_save_model(). The
returned model is compiled and ready for ggml_predict() /
ggml_evaluate(). Call ggml_fit() again to continue training.
Usage
ggml_load_model(path, backend = "auto")
Arguments
path |
File path to an RDS file written by |
backend |
Backend selection: |
Value
A compiled model object.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(16L, activation = "relu", input_shape = 4L) |>
ggml_layer_dense(2L, activation = "softmax")
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
x <- matrix(runif(64 * 4), 64, 4)
y <- matrix(c(rep(c(1,0), 32), rep(c(0,1), 32)), 64, 2)
model <- ggml_fit(model, x, y, epochs = 1L, batch_size = 32L, verbose = 0L)
tmp <- tempfile(fileext = ".rds")
ggml_save_model(model, tmp)
model2 <- ggml_load_model(tmp)
Load Model Weights from File
Description
Loads previously saved weights into a compiled model. The model architecture must match the saved weights (same layer types, sizes, and shapes).
Usage
ggml_load_weights(model, path)
Arguments
model |
A compiled ggml_sequential_model (same architecture as saved) |
path |
File path to load weights from |
Value
The model with loaded weights.
Natural Logarithm (Graph)
Description
Creates a graph node for element-wise natural logarithm: log(x)
Usage
ggml_log_inplace(ctx, a)
ggml_log(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_log_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the log operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3)
ggml_set_f32(a, c(1, exp(1), exp(2)))
result <- ggml_log(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [0, 1, 2]
ggml_free(ctx)
Logging and Abort Callbacks
Description
Redirects GGML log messages to R's message system: - INFO/DEBUG messages go to stdout (via Rprintf) - WARN/ERROR messages go to stderr (via REprintf)
Usage
ggml_log_set_r()
ggml_log_set_default()
ggml_log_is_r_enabled()
ggml_set_abort_callback_r()
ggml_set_abort_callback_default()
ggml_abort_is_r_enabled()
Details
ggml_log_set_default(): Restores GGML to default logging behavior (stderr output).
Returns: NULL invisibly
ggml_log_is_r_enabled():
Returns: Logical indicating if R-compatible logging is active
ggml_set_abort_callback_r(): Converts GGML abort calls into R errors (via Rf_error).
This allows R to catch GGML failures with tryCatch.
Returns: NULL invisibly
ggml_set_abort_callback_default(): Restores GGML to default abort behavior (prints to stderr and aborts).
Returns: NULL invisibly
ggml_abort_is_r_enabled():
Returns: Logical indicating if R-compatible abort handling is active
Value
NULL invisibly
Examples
ggml_log_set_r()
# Now GGML messages will appear in R console
ggml_set_abort_callback_r()
# Now GGML aborts will become R errors
result <- tryCatch({
# ... ggml operations that might fail ...
}, error = function(e) {
message("GGML error caught: ", e$message)
})
Create an LSTM Layer Object
Description
Create an LSTM Layer Object
Usage
ggml_lstm(
units,
return_sequences = FALSE,
activation = "tanh",
recurrent_activation = "sigmoid",
name = NULL,
trainable = TRUE
)
Arguments
units |
Integer, number of hidden units. |
return_sequences |
Logical. |
activation |
Cell gate activation (default |
recurrent_activation |
Recurrent gate activation (default |
name |
Optional character name. |
trainable |
Logical. |
Value
A ggml_layer object.
Marshal a ggmlR model to an in-memory container
Description
Serializes a trained sequential or functional ggmlR model into a self-describing raw container suitable for transport between R sessions or parallel workers (e.g. for mlr3 parallel resampling and tuning).
Usage
ggml_marshal_model(model)
Arguments
model |
A compiled |
Details
The container wraps the bytes produced by ggml_save_model
together with a format tag, schema version, package/R versions, a SHA-256
integrity checksum, and a timestamp. Autograd modules are not
supported in this version and cause the function to signal an error; the
mlr3 learners catch this and fall back to marshaled = FALSE.
Value
A named list with class "ggmlR_marshaled" containing the
serialized payload and metadata. Pass it to
ggml_unmarshal_model to reconstruct the model.
See Also
ggml_unmarshal_model, ggml_save_model
GPU matrix multiply (drop-in for %*%)
Description
Computes A %*% B on the Vulkan GPU and returns an ordinary R matrix,
with a transparent CPU fallback. A drop-in accelerator for a large matrix
multiply: no autograd, no wrapper types required — plain matrices in and out.
Usage
ggml_matmul(A, B, device = "auto", prec = "f32")
Arguments
A, B |
Numeric matrices with |
device |
|
prec |
|
Details
Precision: R multiplies in double precision (f64); a GPU offers f32 at best,
so the GPU result never matches R to full double precision. prec = "f32"
(the default) requests the f32 accumulation kernel, but how close the result
actually lands depends on the Vulkan driver: some accumulate mul_mat in
f16 regardless (e.g. RADV / Mesa), giving a relative error around
1e-3 either way. Treat the GPU path as a fast, approximate multiply —
typically 1e-3, better on drivers with true f32 accumulation — not a
bit-for-bit replacement for %*%. prec = "f16" only ever lowers
precision; use it when speed matters more than the last digits.
Value
The product A %*% B as a numeric matrix.
See Also
ggml_crossprod, ggml_tcrossprod,
as_gpu_matrix
Examples
A <- matrix(rnorm(4), 2); B <- matrix(rnorm(4), 2)
ggml_matmul(A, B, device = "cpu")
GPU double-precision matrix multiply
Description
Computes A %*% B on the Vulkan GPU in full double precision
(fp64), returning an ordinary R matrix, with a transparent CPU fallback.
Unlike ggml_matmul (which is f32/f16 and approximate), this
matches R's %*% to machine precision (~1e-15), because the
kernel accumulates in double throughout.
Usage
ggml_matmul_f64(A, B, device = "auto")
Arguments
A, B |
Numeric matrices with |
device |
|
Details
fp64 on the GPU is only worthwhile on hardware with fast double throughput:
data-centre cards (NVIDIA Tesla P100/V100, AMD Instinct) run fp64 at 1:2 of
fp32, whereas consumer GPUs cripple it (RDNA ~1:16, GeForce ~1:64), where the
CPU (multithreaded double BLAS) is usually faster. Use this when you need
bit-accurate double results on a capable GPU, or for numerically sensitive
work where the f32 path's error is unacceptable; otherwise prefer
ggml_matmul. Requires the shaderFloat64 device feature — without
it (or without a GPU), the multiply silently falls back to the CPU.
Value
The product A %*% B as a numeric matrix, accurate to
double precision.
See Also
ggml_matmul (faster f32/f16 approximate path).
Examples
A <- matrix(rnorm(4), 2); B <- matrix(rnorm(4), 2)
ggml_matmul_f64(A, B, device = "cpu")
A GPU-backed matrix wrapper
Description
An S4 class holding an ordinary numeric matrix, tagged so that %*%,
crossprod() and tcrossprod() run on the GPU (see
ggml_matmul). Wrap a matrix with as_gpu_matrix.
Usage
## S4 method for signature 'ggml_matrix,ANY'
x %*% y
## S4 method for signature 'ANY,ggml_matrix'
x %*% y
## S4 method for signature 'ggml_matrix,ggml_matrix'
x %*% y
## S4 method for signature 'ggml_matrix'
crossprod(x, y)
## S4 method for signature 'ggml_matrix'
tcrossprod(x, y)
## S4 method for signature 'ggml_matrix'
dim(x)
## S4 method for signature 'ggml_matrix'
show(object)
Arguments
x, y |
Operands; at least one is a |
object |
A |
Value
%*%, crossprod and tcrossprod return a plain
numeric matrix (the GPU result); dim an integer vector.
Slots
dataThe underlying numeric matrix.
deviceDispatch preference:
"auto","gpu"or"cpu".precPrecision of the GPU path:
"f32"or"f16".
Mean (Graph)
Description
Creates a graph node that computes the mean of all elements.
Usage
ggml_mean(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Scalar tensor with the mean
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(2, 4, 6, 8, 10))
result <- ggml_mean(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # 6
ggml_free(ctx)
Tensor-parallel backend over several devices (meta backend)
Description
Wraps several devices into one ggml "meta" backend. Weights allocated in it
are laid out across the devices by split; a graph computed on it runs
on every device at once, each on its own share, and the meta backend
inserts the cross-device reductions itself. That is graph-level tensor
parallelism: unlike ggml_vulkan_split_mul_mat, which splits a
single matrix and gathers immediately, a column-split layer followed by a
row-split one exchanges nothing in between.
Usage
ggml_meta_backend(devices, split = list(), default = "mirror")
Arguments
devices |
Vulkan device ids (0-based integers), device names
( |
split |
Named list of rules, or |
default |
Rule for tensors |
Details
Split rules, by tensor name (ggml_set_name):
"mirror"a full copy on every device (the default).
"rows"split along ggml axis 1 (
ne[1]): for a weight[n_in, n_out]the output features – column parallel."cols"split along ggml axis 0 (
ne[0]): the input features – row parallel; the partial sums are reduced."axis2","axis3"split along
ne[2]/ne[3].list(axis = , weights = )any of the above with uneven shares, e.g.
list(axis = "rows", weights = c(0.7, 0.3)).
split may also be a function(name, ne) returning a rule (or
NULL for the default).
Only tensors in buffers allocated on the returned backend are asked for a
rule (weights and inputs); intermediate results get their layout from the
op that produces them. Use the backend through a scheduler whose last
backend is the CPU, e.g.
ggml_backend_sched_new(list(meta, ggml_backend_cpu_init())).
A rule that cannot be applied (the axis is shorter than the number of
devices, or not a multiple of the type's block size) falls back to
"mirror" for that tensor with a warning. Rules for names that never
show up are not an error; see attr(backend, "meta_split_log"), an
environment whose applied element records, by name, the rule each
tensor received, and whose warnings element collects the fallback
messages. The rules are evaluated from C while the buffer is allocated, in
a context of its own: those warnings reach the console, but not a
withCallingHandlers() / tryCatch() around the allocation –
check warnings instead.
A tensor that ends up holding per-device partial sums (a matrix product
whose inputs are both split along the reduced axis, e.g. "cols" on
the weight and on its input) is reduced before its first consumer. It cannot
be read itself, and a graph must not end in one: read a tensor computed from
it. ggml_tp_forward takes care of the latter.
The data path between devices is host staging (no Vulkan backend offers a
device-to-device all-reduce), so this pays off only when a model does not
fit on one device – see vignette("multi-gpu").
Value
A ggml_backend_t external pointer. Free it with
ggml_backend_free. It keeps the meta device alive.
See Also
ggml_backend_meta_device for the raw callback API,
ggml_vulkan_split_mul_mat, ggml_pp_forward
Other meta backend:
GGML_SPLIT_AXIS,
ggml_tp_forward()
Examples
cpu <- "CPU"
meta <- ggml_meta_backend(c(cpu, cpu), split = list(w = "rows"))
ggml_backend_free(meta)
Create a Functional Model
Description
Assembles a ggml_functional_model from symbolic input and output
nodes produced by ggml_input() and ggml_layer_*() calls.
Usage
ggml_model(inputs, outputs)
Arguments
inputs |
A |
outputs |
A |
Value
A ggml_functional_model object.
Examples
x <- ggml_input(shape = 64L)
out <- x |> ggml_layer_dense(10, activation = "softmax")
model <- ggml_model(inputs = x, outputs = out)
Backend a fitted ggml model actually ran on
Description
Reports the backend the model was actually compiled onto, making a
silent backend = "auto" fallback to CPU (when no GPU is available)
inspectable. Works on a raw sequential/functional model or a fitted parsnip
engine object.
Usage
ggml_model_backend(object, verbose = FALSE)
Arguments
object |
A compiled/fitted |
verbose |
If |
Value
If verbose = FALSE, a length-1 character: the backend in use
("vulkan" or "cpu"). If verbose = TRUE, a list with:
requested (what was asked: "auto"/"cpu"/"vulkan"),
used ("vulkan"/"cpu"), device (GPU device
description, or "cpu") and fallback (logical: TRUE when
a non-CPU backend was requested but CPU was used instead).
See Also
Create a Sequential Neural Network Model
Description
Creates an empty sequential model that layers can be added to using
pipe (|>) operators.
Usage
ggml_model_sequential()
Value
A ggml_sequential_model object
Examples
## Not run:
model <- ggml_model_sequential() |>
ggml_layer_conv_2d(32, c(3,3), activation = "relu",
input_shape = c(28, 28, 1)) |>
ggml_layer_max_pooling_2d(c(2,2)) |>
ggml_layer_flatten() |>
ggml_layer_dense(128, activation = "relu") |>
ggml_layer_dense(10, activation = "softmax")
## End(Not run)
Element-wise Multiplication (Graph)
Description
Creates a graph node for element-wise multiplication.
Usage
ggml_mul(ctx, a, b)
ggml_mul_inplace(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Details
ggml_mul_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the multiplication operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
ggml_set_f32(b, c(2, 2, 2, 2, 2))
result <- ggml_mul(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Matrix Multiplication (Graph)
Description
Creates a graph node for matrix multiplication. CRITICAL for LLM operations. For matrices A (m x n) and B (n x p), computes C = A * B (m x p).
Usage
ggml_mul_mat(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First matrix tensor |
b |
Second matrix tensor |
Value
Tensor representing the matrix multiplication
Examples
ctx <- ggml_init(1024 * 1024)
A <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 3)
B <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 2)
ggml_set_f32(A, 1:12)
ggml_set_f32(B, 1:8)
C <- ggml_mul_mat(ctx, A, B)
graph <- ggml_build_forward_expand(ctx, C)
ggml_graph_compute(ctx, graph)
ggml_get_f32(C)
ggml_free(ctx)
Matrix Multiplication with Expert Selection (Graph)
Description
Indirect matrix multiplication for Mixture of Experts architectures. Selects expert weights based on indices and performs batched matmul.
Usage
ggml_mul_mat_id(ctx, as, b, ids)
Arguments
ctx |
GGML context |
as |
Stacked expert weight matrices [n_embd, n_ff, n_experts] |
b |
Input tensor |
ids |
Expert selection indices tensor (I32) |
Value
Output tensor after expert-selected matrix multiplication
Examples
ctx <- ggml_init(64 * 1024 * 1024)
# 4 experts, each with 8x16 weights (small for example)
experts <- ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 8, 16, 4)
ggml_set_f32(experts, rnorm(8 * 16 * 4))
input <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 2)
ggml_set_f32(input, rnorm(16))
# Select expert 0 for token 0, expert 2 for token 1
ids <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 2)
ggml_set_i32(ids, c(0L, 2L))
output <- ggml_mul_mat_id(ctx, experts, input, ids)
graph <- ggml_build_forward_expand(ctx, output)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Set a Matrix Multiplication Hint (Graph)
Description
Marks an existing ggml_mul_mat() node with a hint about its first
operand. With GGML_HINT_SRC0_IS_HADAMARD both backends replace the
product with a fast Walsh-Hadamard transform of b and never read
a: the CPU for any power-of-two row length, Vulkan for row length
64, 128, 256 or 512 (other lengths fall back to a plain matmul there). The
hint is a promise: if a is not the normalized Hadamard matrix, the
result differs from the plain product. Requires F32 b and result.
Modifies the node in place and returns it.
Usage
ggml_mul_mat_set_hint(a, hint)
Arguments
a |
MUL_MAT node: the result of |
hint |
One of |
Value
a
See Also
Get Number of Dimensions
Description
Returns the number of dimensions of a tensor
Usage
ggml_n_dims(tensor)
Arguments
tensor |
Tensor pointer |
Value
Integer number of dimensions (1-4)
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20)
ggml_n_dims(t)
ggml_free(ctx)
Get Number of Bytes
Description
Get Number of Bytes
Usage
ggml_nbytes(tensor)
Arguments
tensor |
Tensor |
Value
Integer number of bytes
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_nbytes(t)
ggml_free(ctx)
Negation (Graph)
Description
Creates a graph node for element-wise negation: -x
Usage
ggml_neg_inplace(ctx, a)
ggml_neg(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_neg_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the negation operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, -2, 3, -4))
result <- ggml_neg(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [-1, 2, -3, 4]
ggml_free(ctx)
Get Number of Elements
Description
Get Number of Elements
Usage
ggml_nelements(tensor)
Arguments
tensor |
Tensor |
Value
Integer number of elements
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20)
ggml_nelements(t)
ggml_free(ctx)
Create Scalar F32 Tensor
Description
Creates a 1-element tensor containing a single float value. Useful for scalar operations, learning rates, and other scalar floats.
Usage
ggml_new_f32(ctx, value)
Arguments
ctx |
GGML context |
value |
Numeric value |
Value
Tensor pointer (1-element F32 tensor)
Examples
ctx <- ggml_init(1024 * 1024)
scalar <- ggml_new_f32(ctx, 3.14)
ggml_get_f32(scalar)
ggml_free(ctx)
Create Scalar I32 Tensor
Description
Creates a 1-element tensor containing a single integer value. Useful for indices, counters, and other scalar integer operations.
Usage
ggml_new_i32(ctx, value)
Arguments
ctx |
GGML context |
value |
Integer value |
Value
Tensor pointer (1-element I32 tensor)
Examples
ctx <- ggml_init(1024 * 1024)
scalar <- ggml_new_i32(ctx, 42)
ggml_get_i32(scalar)
ggml_free(ctx)
Create New Tensors
Description
Generic tensor constructor for creating tensors with 1-4 dimensions. This is more flexible than the ggml_new_tensor_Nd functions.
Usage
ggml_new_tensor_1d(ctx, type = GGML_TYPE_F32, ne0)
ggml_new_tensor_2d(ctx, type = GGML_TYPE_F32, ne0, ne1)
ggml_new_tensor_3d(ctx, type = GGML_TYPE_F32, ne0, ne1, ne2)
ggml_new_tensor_4d(ctx, type = GGML_TYPE_F32, ne0, ne1, ne2, ne3)
ggml_new_tensor(ctx, type = GGML_TYPE_F32, n_dims, ne)
Arguments
ctx |
GGML context |
type |
Data type (GGML_TYPE_F32, etc.) |
ne0 |
Size of dimension 0 (number of elements / rows) |
ne1 |
Size of dimension 1 (number of columns) |
ne2 |
Size of dimension 2 |
ne3 |
Size of dimension 3 |
n_dims |
Number of dimensions (1-4) |
ne |
Numeric vector of dimension sizes |
Details
ggml_new_tensor_1d(), ggml_new_tensor_2d(),
ggml_new_tensor_3d(), ggml_new_tensor_4d(): create a tensor
with 1 to 4 dimensions of sizes ne0 .. ne3.
ggml_new_tensor_3d():
Returns: Tensor pointer
ggml_new_tensor_4d():
Returns: Tensor pointer
Value
Tensor pointer
Examples
ctx <- ggml_init(1024 * 1024)
tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_nelements(tensor)
tensor <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20)
ggml_nelements(tensor)
ggml_free(ctx)
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 10, 20, 30)
ggml_nelements(t)
ggml_free(ctx)
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 8, 8, 3, 2)
ggml_nelements(t)
ggml_free(ctx)
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor(ctx, GGML_TYPE_F32, 3, c(10, 20, 30))
ggml_nelements(t)
ggml_free(ctx)
Layer Normalization (Graph)
Description
Creates a graph node for layer normalization. Normalizes input to zero mean and unit variance.
Usage
ggml_norm(ctx, a, eps = 1e-05)
ggml_norm_inplace(ctx, a, eps = 1e-05)
Arguments
ctx |
GGML context |
a |
Input tensor |
eps |
Epsilon value for numerical stability (default: 1e-5) |
Details
ggml_norm_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the layer normalization operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_norm(ctx, a, eps = 1e-5)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result) # Normalized values
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_norm_inplace(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Layer Norm Backward (Graph)
Description
Creates a graph node for the backward pass of ggml_norm, i.e.
of LayerNorm without gamma/beta. Upstream ggml has no backward for
GGML_OP_NORM; ggmlR adds it with CPU and Vulkan kernels, which is what
makes ggml_layer_layer_norm trainable.
Usage
ggml_norm_back(ctx, a, b, eps = 1e-05)
Arguments
ctx |
GGML context |
a |
Gradient tensor (dy from the forward output) |
b |
Input tensor (x from the forward pass) |
eps |
Epsilon for numerical stability (default 1e-5) |
Value
Tensor representing the gradient with respect to input
See Also
Get Number of Rows
Description
Returns the number of rows in a tensor (product of all dimensions except ne[0]).
Usage
ggml_nrows(tensor)
Arguments
tensor |
Tensor pointer |
Value
Number of rows
Check if Operation Can Be Done In-place
Description
Returns whether a GGML operation can reuse memory from its source tensors. This is useful for memory optimization.
Usage
ggml_op_can_inplace(op)
Arguments
op |
Operation code (integer) |
Value
Logical indicating if operation supports in-place execution
Examples
# Check if operation code 1 (ADD) can be in-place
can_inplace <- ggml_op_can_inplace(1L)
Operation Information
Description
Returns the string name of a GGML operation.
Usage
ggml_op_name(op)
ggml_op_symbol(op)
ggml_unary_op_name(op)
ggml_op_desc(tensor)
ggml_get_unary_op(tensor)
Arguments
op |
GGML operation constant |
tensor |
Tensor pointer |
Details
ggml_op_symbol(): Returns the mathematical symbol for a GGML operation.
Returns: Character string with operation symbol
ggml_unary_op_name(): Returns the string name of a GGML unary operation.
Returns: Character string with operation name
ggml_op_desc(): Returns a description of the operation that produces a tensor.
Returns: Character string describing the operation
ggml_get_unary_op(): Returns the unary operation type for a unary operation tensor.
Returns: Integer unary operation type
Value
Character string with operation name
Supported single-cell operations
Description
Returns the registry of operations the single-cell adapter can dispatch. Use
this to check capabilities (and required parameters) before building a
ggml_task — capability is declared, never discovered at runtime.
Usage
ggml_ops_registry(op = NULL)
Arguments
op |
Optional operation name. If supplied, returns that single entry (or
|
Value
A list describing the operation(s): op, params
(required parameter names) and desc.
Examples
ggml_ops_registry()
ggml_ops_registry("embed")
Allocate graph for evaluation
Description
Must be called before ggml_opt_eval. Allocates forward or forward+backward graph.
Usage
ggml_opt_alloc(opt_ctx, backward = TRUE)
Arguments
opt_ctx |
External pointer to optimizer context |
backward |
Whether to allocate backward graph (for training) |
Value
NULL invisibly
See Also
Get optimizer type from context
Description
Get optimizer type from context
Usage
ggml_opt_context_optimizer_type(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
Integer optimizer type constant
See Also
Optimization datasets
Description
Create, inspect and batch a dataset for ggml-opt training.
Usage
ggml_opt_dataset_init(
type_data,
type_label,
ne_datapoint,
ne_label,
ndata,
ndata_shard = 1
)
ggml_opt_dataset_free(dataset)
ggml_opt_dataset_ndata(dataset)
ggml_opt_dataset_data(dataset)
ggml_opt_dataset_labels(dataset)
ggml_opt_dataset_weights(dataset, ne0 = 1L)
ggml_opt_dataset_shuffle(opt_ctx, dataset, idata = -1)
ggml_opt_dataset_get_batch(dataset, data_batch, labels_batch = NULL, ibatch)
ggml_opt_dataset_get_batch_head(
dataset,
data_batch,
labels_batch,
labels_off,
ibatch
)
Arguments
type_data |
GGML type for data tensor (e.g., GGML_TYPE_F32) |
type_label |
GGML type for label tensor (e.g., GGML_TYPE_F32) |
ne_datapoint |
Number of elements per datapoint |
ne_label |
Number of elements per label (0 if no labels) |
ndata |
Total number of datapoints |
ndata_shard |
Shard size for shuffling (1 is fine for most cases) |
dataset |
External pointer to dataset |
ne0 |
Width of the weight tensor: 1 for per-datapoint weights, or the label width for a per-output mask. |
opt_ctx |
External pointer to optimizer context |
idata |
Number of datapoints to shuffle (-1 for all) |
data_batch |
Tensor to receive data batch. For
|
labels_batch |
Tensor to receive labels batch (can be NULL for
|
ibatch |
Batch index, 0-based |
labels_off |
Offset of this head within a label row, 0-based, in elements |
Details
ggml_opt_dataset_init()Creates a dataset with the given data and label types.
ggml_opt_dataset_free()Releases memory associated with a dataset.
ggml_opt_dataset_ndata()Number of datapoints.
ggml_opt_dataset_data()The underlying data tensor, shape
[ne_datapoint, ndata].ggml_opt_dataset_labels()The underlying labels tensor, shape
[ne_label, ndata], or NULL if no labels.ggml_opt_dataset_weights()The loss weights tensor; see section Loss weights.
ggml_opt_dataset_shuffle()Shuffles the dataset using the RNG from the optimizer context.
ggml_opt_dataset_get_batch()Copies a batch of data and labels to the provided tensors.
ggml_opt_dataset_get_batch_head()Multi-output counterpart of
ggml_opt_dataset_get_batch(). The dataset's labels hold every head's labels concatenated along the first dimension; this copies just the slice belonging to one head.
Value
ggml_opt_dataset_init(): external pointer to dataset.
ggml_opt_dataset_ndata(): number of datapoints.
ggml_opt_dataset_data(), ggml_opt_dataset_labels(),
ggml_opt_dataset_weights(): external pointer to the tensor (labels
may be NULL). ggml_opt_dataset_free(),
ggml_opt_dataset_shuffle(), ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(): NULL invisibly.
Loss weights
ggml_opt_dataset_weights() returns the (lazily allocated) loss
weights tensor with shape [ne0, ndata]. The first call allocates it;
fill it via ggml_backend_tensor_set_data(). Used together with
ggml_opt_loss_type_weighted_mse() (see ggml_opt_loss_type).
ne0 = 1 gives one weight per datapoint, broadcast over the outputs –
the original behaviour. ne0 = ne_label gives a per-output-element
mask, which is what a Q-model needs: weight 1 on the action that has a
TD target, 0 elsewhere, so only that coordinate is trained.
The mask does not change the denominator: the loss stays
sum(w * (pred - y)^2) / nelements(pred), so zeroed elements drop out
of the numerator without renormalising over the active coordinates.
Calling this again with a different ne0 than the allocation is an
error rather than a silent return of the cached tensor – both shapes
broadcast, so a mismatch would otherwise weight the wrong thing unnoticed.
See Also
ggmlR-optimization for the full list of training functions.
Get default optimizer parameters
Description
Returns a list with default optimization parameters.
Usage
ggml_opt_default_params(sched, loss_type)
Arguments
sched |
Backend scheduler |
loss_type |
Loss type constant |
Value
List with loss_type, build_type, opt_period, optimizer
See Also
Run one training epoch
Description
Performs training on the front portion of the dataset and evaluation on the back portion. This gives more control than ggml_opt_fit.
Usage
ggml_opt_epoch(
opt_ctx,
dataset,
result_train = NULL,
result_eval = NULL,
idata_split,
callback_train = TRUE,
callback_eval = TRUE
)
Arguments
opt_ctx |
External pointer to optimizer context |
dataset |
External pointer to dataset |
result_train |
Result object to accumulate training stats (or NULL) |
result_eval |
Result object to accumulate evaluation stats (or NULL) |
idata_split |
Data index at which to split training and evaluation |
callback_train |
Callback for training: TRUE for progress bar, FALSE for none, or a function(train, ibatch, ibatch_max, t_start_us, result) |
callback_eval |
Callback for evaluation: TRUE for progress bar, FALSE for none, or a function(train, ibatch, ibatch_max, t_start_us, result) |
Value
NULL invisibly
See Also
Examples
# Requires full optimizer setup - see ggml_opt_fit() for simpler API
if (FALSE) {
result_train <- ggml_opt_result_init()
result_eval <- ggml_opt_result_init()
ggml_opt_epoch(opt_ctx, dataset, result_train, result_eval,
idata_split = 900, callback_train = TRUE)
ggml_opt_result_free(result_train)
ggml_opt_result_free(result_eval)
}
Evaluate model
Description
Performs forward pass, optionally increments result, and does backward pass if allocated.
Usage
ggml_opt_eval(opt_ctx, result = NULL)
Arguments
opt_ctx |
External pointer to optimizer context |
result |
External pointer to result object (optional) |
Value
NULL invisibly
See Also
Fit model to dataset
Description
High-level function to train a model on a dataset. This is the recommended way to train models.
Usage
ggml_opt_fit(
sched,
ctx_compute,
inputs,
outputs,
dataset,
loss_type = ggml_opt_loss_type_mse(),
optimizer = ggml_opt_optimizer_type_adamw(),
nepoch = 1,
nbatch_logical = 32,
val_split = 0,
silent = FALSE
)
Arguments
sched |
Backend scheduler |
ctx_compute |
Compute context (for temporary tensors) |
inputs |
Input tensor with shape [ne_datapoint, batch_size] |
outputs |
Output tensor with shape [ne_label, batch_size] |
dataset |
Dataset created with ggml_opt_dataset_init |
loss_type |
Loss type (default: MSE) |
optimizer |
Optimizer type (default: AdamW) |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size (for gradient accumulation) |
val_split |
Fraction of data for validation (0.0 to 1.0) |
silent |
Whether to suppress progress output |
Value
NULL invisibly
See Also
Examples
# Full training requires building a computation graph
# See package vignettes for complete examples
if (FALSE) {
cpu <- ggml_backend_cpu_init()
sched <- ggml_backend_sched_new(list(cpu))
dataset <- ggml_opt_dataset_init(GGML_TYPE_F32, GGML_TYPE_F32, 10, 1, 1000)
# ... build model graph with ctx_compute, inputs, outputs ...
ggml_opt_fit(sched, ctx_compute, inputs, outputs, dataset,
nepoch = 10, val_split = 0.1)
ggml_opt_dataset_free(dataset)
ggml_backend_sched_free(sched)
ggml_backend_free(cpu)
}
Fit a multi-output model
Description
Trains a model with several output heads. Each head has its own labels,
loss type and weight; the optimized total is
sum(loss_weights[i] * loss_i).
Usage
ggml_opt_fit_multi(
sched,
ctx_compute,
inputs,
outputs,
dataset,
loss_types,
loss_weights = rep(1, length(outputs)),
labels_offs = NULL,
optimizer = ggml_opt_optimizer_type_adamw(),
nepoch = 1,
nbatch_logical = 32,
val_split = 0,
silent = FALSE
)
Arguments
sched |
Backend scheduler |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor |
outputs |
List of output tensors, one per head |
dataset |
Dataset with data and concatenated labels |
loss_types |
Integer vector of loss type constants, one per head |
loss_weights |
Numeric vector of head weights (default: all 1) |
labels_offs |
Numeric vector of label offsets, 0-based, one per head |
optimizer |
Optimizer type constant |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size |
val_split |
Fraction of the data used for validation |
silent |
Suppress progress output |
Details
All heads share the dataset's datapoints. The dataset's labels hold every
head's labels concatenated along the first dimension: head i occupies
labels_offs[i] + seq_len(width_i) of each label row.
Value
Named list with train_loss, train_accuracy, val_loss, val_accuracy, head_loss (a nepoch x nhead matrix of unweighted per-head losses) and head_accuracy (same shape; NA for heads whose loss is not cross-entropy)
See Also
ggml_fit_opt_multi, which this wraps and which adds
callbacks and shuffling control
Free optimizer context
Description
Releases memory associated with an optimizer context.
Usage
ggml_opt_free(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
NULL invisibly
See Also
Get current learning rate from optimizer context
Description
Get current learning rate from optimizer context
Usage
ggml_opt_get_lr(lr_ud)
Arguments
lr_ud |
LR userdata pointer (from 'ggml_opt_init_for_fit()$lr_ud') |
Value
Named numeric vector with 'adamw' and 'sgd' LR values
See Also
Get gradient accumulator for a tensor
Description
Returns the gradient accumulator tensor for a node from the forward graph.
Usage
ggml_opt_grad_acc(opt_ctx, node)
Arguments
opt_ctx |
External pointer to optimizer context |
node |
External pointer to tensor node |
Value
External pointer to gradient accumulator tensor, or NULL if not found
See Also
Initialize optimizer context
Description
Creates a new optimizer context for training.
Usage
ggml_opt_init(
sched,
loss_type,
optimizer = ggml_opt_optimizer_type_adamw(),
opt_period = 1L,
ctx_compute = NULL,
inputs = NULL,
outputs = NULL
)
Arguments
sched |
Backend scheduler |
loss_type |
Loss type (use ggml_opt_loss_type_* functions) |
optimizer |
Optimizer type (use ggml_opt_optimizer_type_* functions) |
opt_period |
Gradient accumulation steps before optimizer step |
ctx_compute |
Compute context for static graph mode (or NULL) |
inputs |
Input tensor for static graph mode (or NULL) |
outputs |
Output tensor for static graph mode (or NULL) |
Value
External pointer to optimizer context
See Also
Initialize optimizer context for R-side epoch loop
Description
Returns a list with 'opt_ctx' and 'lr_ud' (learning rate userdata pointer). Use 'ggml_opt_set_lr()' to update LR between epochs. The optimizer state (momentum) is preserved across epochs.
Usage
ggml_opt_init_for_fit(
sched,
loss_type,
optimizer = ggml_opt_optimizer_type_adamw(),
opt_period = 1L,
ctx_compute = NULL,
inputs = NULL,
outputs = NULL,
loss_mask_ne0 = NULL,
weight_decay = 0
)
Arguments
sched |
Backend scheduler |
loss_type |
Loss type constant |
optimizer |
Optimizer type constant |
opt_period |
Gradient accumulation period |
ctx_compute |
Compute context (for static graphs) |
inputs |
Input tensor (for static graphs) |
outputs |
Output tensor (for static graphs) |
loss_mask_ne0 |
Width of the weighted-MSE weight tensor: |
weight_decay |
Decoupled weight decay, in |
Value
List with elements 'opt_ctx' and 'lr_ud'
See Also
Initialize a multi-output optimizer context for an R-side epoch loop
Description
Multi-output counterpart of ggml_opt_init_for_fit(). Returns the same
opt_ctx/lr_ud pair, so ggml_opt_set_lr() and hence LR
schedulers work identically; in addition it records each head's loss type,
weight and label offset, which ggml_opt_fit_multi() would otherwise
set internally.
Usage
ggml_opt_init_for_fit_multi(
sched,
loss_types,
loss_weights,
labels_offs,
optimizer = ggml_opt_optimizer_type_adamw(),
opt_period = 1L,
ctx_compute = NULL,
inputs = NULL,
outputs = NULL,
weight_decay = 0
)
Arguments
sched |
Backend scheduler |
loss_types |
Integer vector of loss type constants, one per head |
loss_weights |
Numeric vector of head weights, one per head |
labels_offs |
Numeric vector of 0-based label offsets, one per head |
optimizer |
Optimizer type constant |
opt_period |
Gradient accumulation period |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor |
outputs |
List of output tensors, one per head |
weight_decay |
As for |
Value
List with elements 'opt_ctx' and 'lr_ud'
See Also
Get inputs tensor from optimizer context
Description
Get inputs tensor from optimizer context
Usage
ggml_opt_inputs(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
External pointer to inputs tensor
See Also
Get labels tensor from optimizer context
Description
Get labels tensor from optimizer context
Usage
ggml_opt_labels(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
External pointer to labels tensor
See Also
Get loss tensor from optimizer context
Description
Get loss tensor from optimizer context
Usage
ggml_opt_loss(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
External pointer to loss tensor
See Also
Get one head's loss scalar, before weighting
Description
Returns the head's own loss, before it is multiplied by its
loss_weights entry and summed into the total. ggml_opt_loss()
returns the weighted total that is actually optimized.
Usage
ggml_opt_loss_i(opt_ctx, ihead)
Arguments
opt_ctx |
External pointer to optimizer context |
ihead |
Output head index, 1-based |
Value
External pointer to the head's loss tensor
See Also
Loss and optimizer type constants
Description
Integer constants selecting the loss function and the optimizer for
ggml_opt_init, ggml_opt_fit and related
training functions.
Usage
ggml_opt_loss_type_mean()
ggml_opt_loss_type_sum()
ggml_opt_loss_type_cross_entropy()
ggml_opt_loss_type_mse()
ggml_opt_loss_type_weighted_mse()
ggml_opt_loss_type_mae()
ggml_opt_loss_type_huber()
ggml_opt_loss_type_binary_cross_entropy()
ggml_opt_optimizer_type_adamw()
ggml_opt_optimizer_type_sgd()
Details
Loss types:
ggml_opt_loss_type_mean()Custom loss: reduces outputs to their mean value.
ggml_opt_loss_type_sum()Custom loss: reduces outputs to their sum.
ggml_opt_loss_type_cross_entropy()Cross entropy, for classification. Softmaxes its input across the class axis, so it expects logits.
ggml_opt_loss_type_mse()Mean squared error, for regression.
ggml_opt_loss_type_weighted_mse()Per-datapoint weighted MSE:
sum(w * (pred - y)^2) / nelements, wherewis a per-sample weight supplied viaggml_opt_dataset_weights.ggml_opt_loss_type_mae()Mean absolute error (L1),
mean(|pred - y|). Its gradient issgn(pred - y), so a far-off datapoint pulls no harder than a near one – which is what makes it more robust to outliers than MSE, at the cost of a gradient that does not vanish as the fit improves.ggml_opt_loss_type_huber()Huber (smooth L1) loss with
delta = 1(fixed, the usual default):0.5 * e^2where|e| <= 1and|e| - 0.5beyond it. Quadratic near zero, so unlike MAE the gradient vanishes at the optimum, and linear far out, so unlike MSE a single outlier cannot dominate the batch.ggml_opt_loss_type_binary_cross_entropy()Element-wise binary cross-entropy,
mean(-[y*log(p) + (1-y)*log(1-p)]). Unlike cross entropy, it treats every output as an independent Bernoulli and expects probabilities – so the model's last layer should end in a sigmoid. It is the loss for multi-label targets and for a single-unit binary output. Inputs are clamped away from 0 and 1 internally, sincelog(0)would otherwise poison the whole batch.
Optimizer types:
ggml_opt_optimizer_type_adamw()AdamW: Adam with weight decay, recommended for most tasks.
ggml_opt_optimizer_type_sgd()Stochastic gradient descent: simpler but may require tuning.
Value
Integer constant.
See Also
ggmlR-optimization for the full list of training functions.
Number of output heads in an optimizer context
Description
Number of output heads in an optimizer context
Usage
ggml_opt_n_loss(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
Integer number of output heads (1 for single-output models)
See Also
Get number of correct predictions tensor
Description
Get number of correct predictions tensor
Usage
ggml_opt_ncorrect(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
External pointer to ncorrect tensor
See Also
Get optimizer name
Description
Get optimizer name
Usage
ggml_opt_optimizer_name(optimizer_type)
Arguments
optimizer_type |
Integer optimizer type constant |
Value
Character string with optimizer name
See Also
Get outputs tensor from optimizer context
Description
Get outputs tensor from optimizer context
Usage
ggml_opt_outputs(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
External pointer to outputs tensor
See Also
Get one output head's tensors from an optimizer context
Description
Per-head counterparts of ggml_opt_outputs() and friends, for
multi-output models. ihead is 1-based; head 1 is what the
single-head accessors return.
Usage
ggml_opt_outputs_i(opt_ctx, ihead)
ggml_opt_labels_i(opt_ctx, ihead)
ggml_opt_loss_weights_i(opt_ctx, ihead)
ggml_opt_pred_i(opt_ctx, ihead)
ggml_opt_ncorrect_i(opt_ctx, ihead)
Arguments
opt_ctx |
External pointer to optimizer context |
ihead |
Output head index, 1-based |
Value
External pointer to the tensor, or NULL if this head has none
See Also
Get predictions tensor from optimizer context
Description
Get predictions tensor from optimizer context
Usage
ggml_opt_pred(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
External pointer to predictions tensor
See Also
Prepare allocation for non-static graphs
Description
Must be called before ggml_opt_alloc when not using static graphs. Sets up the optimizer context with the computation graph and input/output tensors.
Usage
ggml_opt_prepare_alloc(opt_ctx, ctx_compute, graph, inputs, outputs)
Arguments
opt_ctx |
External pointer to optimizer context |
ctx_compute |
Compute context for temporary tensors |
graph |
Computation graph (from ggml_build_forward_expand) |
inputs |
Input tensor |
outputs |
Output tensor |
Value
NULL invisibly
See Also
Reset optimizer context
Description
Resets gradients to zero, initializes loss, and optionally resets optimizer state.
Usage
ggml_opt_reset(opt_ctx, optimizer = FALSE)
Arguments
opt_ctx |
External pointer to optimizer context |
optimizer |
Whether to also reset optimizer state (momentum, etc.) |
Value
NULL invisibly
See Also
Optimization results
Description
A result object accumulates training statistics (loss, accuracy, predictions) over an epoch.
Usage
ggml_opt_result_init()
ggml_opt_result_free(result)
ggml_opt_result_reset(result)
ggml_opt_result_ndata(result)
ggml_opt_result_loss(result)
ggml_opt_result_accuracy(result)
ggml_opt_result_n_loss(result)
ggml_opt_result_loss_i(result, ihead)
ggml_opt_result_accuracy_i(result, ihead)
ggml_opt_result_pred(result)
Arguments
result |
External pointer to result object |
ihead |
Output head index, 1-based |
Details
ggml_opt_result_init()Creates a new result object.
ggml_opt_result_free(),ggml_opt_result_reset()Free or reset a result.
ggml_opt_result_ndata()Number of datapoints processed.
ggml_opt_result_loss(),ggml_opt_result_accuracy()-
The weighted total the optimizer minimizes, and its accuracy.
ggml_opt_result_pred()Predictions as an integer vector; its length equals the number of datapoints processed.
ggml_opt_result_n_loss()Number of output heads accumulated. Zero before the first epoch has run, so callers must treat a freshly reset result as holding no per-head data yet.
ggml_opt_result_loss_i(),ggml_opt_result_accuracy_i()-
Per-head counterparts of
ggml_opt_result_loss()andggml_opt_result_accuracy(): they report a single head on its own, before weighting, for the training history. Heads whose loss is not cross-entropy have no accuracy; those returnNA.
Value
ggml_opt_result_init(): external pointer to result object.
ggml_opt_result_free(), ggml_opt_result_reset(): NULL
invisibly. ggml_opt_result_ndata(): number of datapoints.
ggml_opt_result_loss(), ggml_opt_result_accuracy(),
ggml_opt_result_loss_i(), ggml_opt_result_accuracy_i():
named numeric vector with the value and its 'uncertainty'.
ggml_opt_result_n_loss(): integer number of heads.
ggml_opt_result_pred(): integer vector of predictions.
See Also
ggmlR-optimization for the full list of training functions.
Set learning rate in optimizer context
Description
Updates the LR used for subsequent backward passes. Can be called between epochs to implement LR scheduling.
Usage
ggml_opt_set_lr(lr_ud, adamw_lr = NA, sgd_lr = NA)
Arguments
lr_ud |
LR userdata pointer (from 'ggml_opt_init_for_fit()$lr_ud') |
adamw_lr |
New AdamW learning rate (NA to keep current) |
sgd_lr |
New SGD learning rate (NA to keep current) |
Value
NULL invisibly
See Also
Check if using static graphs
Description
Check if using static graphs
Usage
ggml_opt_static_graphs(opt_ctx)
Arguments
opt_ctx |
External pointer to optimizer context |
Value
Logical indicating if graphs are statically allocated
See Also
AdamW Optimizer Step (Graph Node)
Description
Builds a node that applies one AdamW step when the graph is computed:
m <- beta1 * m + (1 - beta1) * grad v <- beta2 * v + (1 - beta2) * grad^2 a <- a * (1 - alpha * wd) - alpha * (m * beta1h) / (sqrt(v * beta2h) + eps)
a, m and v are updated in place; the returned
tensor is a view of a. Because one kernel reads and writes each
element, the update is free of the read-before-write ordering that a chain of
separate ops would need. With wd = 0 the decay factor is exactly 1.
Usage
ggml_opt_step_adamw(ctx, a, grad, m, v, adamw_params)
Arguments
ctx |
GGML context |
a |
Weight tensor, F32, contiguous. It need not be marked with
|
grad |
Gradient, same shape as |
m |
First moment, same shape as |
v |
Second moment, same shape as |
adamw_params |
F32 tensor of 7 elements:
|
Details
Precision: the kernel receives beta2 as a float and forms
1 - beta2 itself, so the weight of grad^2 in v is
1 - fl(beta2), not 1 - beta2. Its relative error is
9.5e-7 for 0.99, 1.29e-5 for 0.999 and 1.66e-4 for 0.9999 – the closer
beta2 is to 1, the larger. The effect on the weight is at most about
half of that. Expect agreement with a double-precision Adam to that level,
not to the last bit.
Bias correction is passed in, not derived: for step t,
beta1h = 1 / (1 - beta1^t) and beta2h = 1 / (1 - beta2^t).
Only F32 is supported (the CPU and Vulkan kernels are F32-only); any other type or a shape mismatch is an R error.
Value
A view of a; expand it into a graph to run the step
See Also
ggml_set_param, optimizer_adam
Outer Product (Graph)
Description
Computes the outer product of two vectors: C = a * b^T For vectors a[m] and b[n], produces matrix C[m, n].
Usage
ggml_out_prod(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First vector tensor |
b |
Second vector tensor |
Value
Matrix tensor representing the outer product
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3))
ggml_set_f32(b, c(1, 2, 3, 4))
c <- ggml_out_prod(ctx, a, b) # Result: 3x4 matrix
graph <- ggml_build_forward_expand(ctx, c)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Outer Product with Row Sums (Graph)
Description
ggml_out_prod(ctx, a, b) with one more column: the sums of a
over its second dimension. For a of shape [m, k] and b
of shape [n, k] (ggml order, ne[0] first) the result is
[m, n + 1]: columns 1..n are out_prod(a, b), column
n + 1 is rowSums of a in R terms. With a the
output gradient of a dense layer and b its input, these are the
weight and the bias gradients in one pass. F32 only; dimensions 3 and 4
broadcast as in ggml_out_prod.
Usage
ggml_out_prod_rowsum(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Tensor |
b |
Tensor |
Value
Tensor [m, n + 1, ...]
See Also
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 5)
b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 2, 5)
ggml_set_f32(a, as.numeric(1:15))
ggml_set_f32(b, rep(1, 10))
r <- ggml_out_prod_rowsum(ctx, a, b) # [3, 3]; column 3 = row sums of a
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Pad Tensor with Zeros (Graph)
Description
Pads tensor dimensions with zeros on the right side. Useful for aligning tensor sizes in attention operations.
Usage
ggml_pad(ctx, a, p0 = 0L, p1 = 0L, p2 = 0L, p3 = 0L)
Arguments
ctx |
GGML context |
a |
Input tensor to pad |
p0 |
Padding for dimension 0 (default 0) |
p1 |
Padding for dimension 1 (default 0) |
p2 |
Padding for dimension 2 (default 0) |
p3 |
Padding for dimension 3 (default 0) |
Value
Padded tensor with shape [ne0+p0, ne1+p1, ne2+p2, ne3+p3]
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 5, 3)
ggml_set_f32(a, 1:15)
# Pad to 8x4
b <- ggml_pad(ctx, a, 3, 1) # Add 3 zeros to dim0, 1 to dim1
graph <- ggml_build_forward_expand(ctx, b)
ggml_graph_compute(ctx, graph)
# Result shape: [8, 4]
ggml_free(ctx)
Pad a tensor with zeros on both sides
Description
Like ggml_pad, but each dimension can also be padded before its
first element.
Usage
ggml_pad_ext(ctx, a, lp = c(0L, 0L, 0L, 0L), rp = c(0L, 0L, 0L, 0L))
Arguments
ctx |
GGML context |
a |
Input tensor to pad |
lp |
Integer vector of length 4: zeros added before dimensions 0..3 |
rp |
Integer vector of length 4: zeros added after dimensions 0..3 |
Value
Padded tensor with shape [ne0+lp0+rp0, ne1+lp1+rp1, ...]
Reflective 1D Padding (Graph)
Description
Pads the first dimension of a tensor using reflection of its values.
Usage
ggml_pad_reflect_1d(ctx, a, p0, p1)
Arguments
ctx |
GGML context |
a |
Input tensor |
p0 |
Left padding |
p1 |
Right padding |
Value
Padded tensor
Permute Tensor Dimensions (Graph)
Description
Permutes the tensor dimensions according to specified axes. CRITICAL for attention mechanisms in transformers.
Usage
ggml_permute(ctx, a, axis0, axis1, axis2, axis3)
Arguments
ctx |
GGML context |
a |
Input tensor |
axis0 |
New position for axis 0 |
axis1 |
New position for axis 1 |
axis2 |
New position for axis 2 |
axis3 |
New position for axis 3 |
Value
Permuted tensor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Create 4D tensor: (2, 3, 4, 5)
t <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 2, 3, 4, 5)
# Swap axes 0 and 1: result shape (3, 2, 4, 5)
t_perm <- ggml_permute(ctx, t, 1, 0, 2, 3)
ggml_free(ctx)
1D Pooling (Graph)
Description
Applies 1D pooling operation for downsampling.
Usage
ggml_pool_1d(ctx, a, op, k0, s0 = k0, p0 = 0L)
GGML_OP_POOL_MAX
GGML_OP_POOL_AVG
Arguments
ctx |
GGML context |
a |
Input tensor |
op |
Pool operation constant (see details) |
k0 |
Kernel size (window size) |
s0 |
Stride; must equal |
p0 |
Padding; must be 0 (the default) |
Format
An object of class integer of length 1.
An object of class integer of length 1.
Details
Pool operation constants:
-
GGML_OP_POOL_MAX(0): Max pooling - takes maximum value in each window -
GGML_OP_POOL_AVG(1): Average pooling - takes mean of values in each window
Value
Pooled tensor with reduced dimensions
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 8)
ggml_set_f32(a, c(1, 3, 2, 4, 5, 2, 8, 1))
# Max pooling with kernel 2, stride 2
max_pool <- ggml_pool_1d(ctx, a, GGML_OP_POOL_MAX, k0 = 2)
# Result: [3, 4, 5, 8] (max of each pair)
# Average pooling with kernel 2, stride 2
avg_pool <- ggml_pool_1d(ctx, a, GGML_OP_POOL_AVG, k0 = 2)
# Result: [2, 3, 3.5, 4.5] (mean of each pair)
ggml_free(ctx)
2D Pooling (Graph)
Description
Applies 2D pooling operation.
Usage
ggml_pool_2d(ctx, a, op, k0, k1, s0 = k0, s1 = k1, p0 = 0, p1 = 0)
Arguments
ctx |
GGML context |
a |
Input tensor |
op |
Pool operation: GGML_OP_POOL_MAX (0) or GGML_OP_POOL_AVG (1) |
k0 |
Kernel size dimension 0 |
k1 |
Kernel size dimension 1 |
s0 |
Stride dimension 0 (default = k0) |
s1 |
Stride dimension 1 (default = k1) |
p0 |
Padding dimension 0 (default 0) |
p1 |
Padding dimension 1 (default 0) |
Value
Pooled tensor
2D Pooling Backward (Graph)
Description
Gradient of ggml_pool_2d with respect to its input: spreads
the gradient of the pooled output back over the input window (evenly for
average pooling, to the window maximum for max pooling).
Usage
ggml_pool_2d_back(ctx, a, af, op, k0, k1, s0 = k0, s1 = k1, p0 = 0, p1 = 0)
Arguments
ctx |
GGML context |
a |
Gradient of the pooled output (F32) |
af |
Forward input of the pooling (F32); the result has its shape |
op |
Pool operation: GGML_OP_POOL_MAX (0) or GGML_OP_POOL_AVG (1) |
k0 |
Kernel size dimension 0 |
k1 |
Kernel size dimension 1 |
s0 |
Stride dimension 0 (default = k0) |
s1 |
Stride dimension 1 (default = k1) |
p0 |
Padding dimension 0 (default 0) |
p1 |
Padding dimension 1 (default 0) |
Value
Gradient tensor with the shape of af
Remove the Last Layer from a Sequential Model
Description
Removes the last layer from the model. The model must not be compiled.
Usage
ggml_pop_layer(model)
Arguments
model |
A ggml_sequential_model object |
Value
The model with the last layer removed.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(64, activation = "relu") |>
ggml_layer_dense(10, activation = "softmax")
model <- ggml_pop_layer(model)
length(model$layers) # 1
Pipeline + data parallel forward pass (PP x DP)
Description
Runs data parallelism over the batch across replicas of a pipeline-parallel
model: the batch is split into contiguous shards, one per replica, and each
replica runs the whole model as a device-by-layer pipeline on its own
set of GPUs (via ggml_pp_forward). This is the pipeline
counterpart of ggml_tp_dp_forward — use it when the model is too
large for one card (split it by layers) and you also want to raise throughput
by replicating the pipeline across independent GPU sets.
Usage
ggml_pp_dp_forward(
make_stages,
x,
replicas,
out_ncol,
mem_per_stage = 16L * 1024L * 1024L
)
Arguments
make_stages |
A function |
x |
Input activation matrix, |
replicas |
A list of integer vectors, each the GPU-index pipeline for one
replica, e.g. |
out_ncol |
Number of columns |
mem_per_stage |
Bytes of ggml context metadata per stage (see
|
Details
On a 4-GPU box the natural layout is PP=2 x DP=2: replica A pipelines
its layers across GPUs c(0, 1), replica B across c(2, 3), and the
two replicas each handle half the batch with no cross-replica traffic.
Value
The M x N result matrix, the batch-concatenation of each
replica's pipeline output.
See Also
ggml_pp_forward, ggml_tp_dp_forward.
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 4) {
K <- 64L
W1 <- matrix(rnorm(K * K), K); W2 <- matrix(rnorm(K * K), K)
make_stages <- function(devices, m) {
mk <- function(dev, Wt, relu) list(
device = dev, in_shape = c(K, m),
build = function(ctx, input) {
w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K)
z <- ggml_mul_mat(ctx, w, input)
list(output = if (relu) ggml_relu(ctx, z) else z,
set_weights = function() ggml_backend_tensor_set_data(w, as.numeric(Wt)))
})
list(mk(devices[1], W1, TRUE), mk(devices[2], W2, FALSE))
}
X <- matrix(rnorm(8 * K), nrow = 8) # batch of 8
Y <- ggml_pp_dp_forward(make_stages, X, replicas = list(c(0, 1), c(2, 3)),
out_ncol = K)
}
Pipeline-parallel forward pass across per-device layer stages
Description
Runs a forward pass split BY LAYERS across devices (pipeline parallelism), the
complement of ggml_vulkan_split_mul_mat's split-by-matrix tensor
parallelism. Each stage owns a contiguous block of the model's layers on
one GPU; the activation tensor is handed from one stage to the next exactly once
per pass (a single cross-device copy), versus TP's per-layer gather. This suits
models too large for one card's VRAM: at ~1 GB/s host staging the single
handoff costs only ~10-20 ms per pass.
Usage
ggml_pp_forward(stages, x, out_shape, mem_per_stage = 16L * 1024L * 1024L)
Arguments
stages |
A list of stage descriptors (see Details), in pipeline order. |
x |
A numeric vector/array of input activations for the first stage,
laid out in ggml (column-major) order matching |
out_shape |
Integer vector: the ggml |
mem_per_stage |
Bytes of ggml context metadata to reserve per stage (default 16 MiB) — raise it for stages with very many ops. |
Details
Each stage is a list with:
devicePhysical GPU index (0-based) the stage runs on.
buildA function
function(ctx, input)that builds this stage's sub-graph from theinputtensor using theggml_*ops and returns either the output tensor, or a listlist(output = <tensor>, set_weights = function() ...). Theset_weightsclosure (if given) is called AFTER the stage's tensors are allocated on the device, and is where weight tensors created insidebuildget their values (viaggml_backend_tensor_set_data).in_shapeInteger vector: the ggml
neshape of this stage's input tensor (fastest dim first). For stage 1 this must match the suppliedx; for later stages it must match the previous stage's output shape.
Value
A numeric vector of the final stage's output activations (ggml order).
See Also
ggml_tp_dp_forward for the tensor-parallel counterpart.
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) {
K <- 32L; M <- 4L
W1 <- matrix(rnorm(K * K), K); W2 <- matrix(rnorm(K * K), K)
stage <- function(dev, Wt) list(
device = dev, in_shape = c(K, M),
build = function(ctx, input) {
w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K)
ggml_mul_mat(ctx, w, input) # (weights set inside build via attr, see vignette)
})
# See inst/examples/pp_pipeline.R for a complete runnable stage definition.
}
Get Predictions from a Trained Model
Description
Runs forward pass on input data and returns prediction probabilities
(or raw output values for regression). Unlike ggml_evaluate(), this
does not require labels.
Usage
## S3 method for class 'ggml_functional_model'
ggml_predict(model, x, batch_size = 32L, matmul_precision = NULL, ...)
ggml_predict(model, ...)
## S3 method for class 'ggml_sequential_model'
ggml_predict(model, x, batch_size = 32L, matmul_precision = NULL, ...)
Arguments
model |
A trained ggml_sequential_model |
x |
Input data (matrix or array) |
batch_size |
Batch size for inference |
matmul_precision |
|
... |
Additional arguments (ignored). |
Value
Matrix of predictions with shape [N, output_units]
Precision
On a Vulkan device with fp16 support the matrix multiplies keep their
tiles in f16 by default. Results then carry a relative error near 1e-3
that depends on the batch size (a single row takes an exact f32 path,
larger batches the tiled one), so the same input can predict slightly
differently in a batch of 1 and a batch of 64. "f32" marks every
multiply GGML_PREC_F32 and selects f32 tiles: the result then
matches the CPU to about 1e-6, at a cost in matmul throughput that depends
on the device and driver. The CPU backend is exact either way.
Which setting wins: the matmul_precision argument of the call, then
the model's predict_precision, then "default".
Predict Classes from a Trained Model
Description
Returns predicted class indices (1-based) by applying argmax
to the output of ggml_predict().
Usage
ggml_predict_classes(model, x, batch_size = 32L)
Arguments
model |
A trained ggml_sequential_model |
x |
Input data (matrix or array) |
batch_size |
Batch size for inference |
Value
Integer vector of predicted class indices (1-based)
Print Context Memory Status
Description
Helper to print memory usage information
Usage
ggml_print_mem_status(ctx)
Arguments
ctx |
GGML context |
Value
List with total, used, free memory (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
ggml_print_mem_status(ctx)
ggml_free(ctx)
Print Objects in Context
Description
Debug function to print all objects (tensors) in the context
Usage
ggml_print_objects(ctx)
Arguments
ctx |
GGML context |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_print_objects(ctx)
ggml_free(ctx)
Get Quantization Block Info
Description
Returns information about a quantization type including name, type size, block size, and whether it's quantized.
Usage
ggml_quant_block_info(type)
Arguments
type |
GGML type constant |
Value
List with type_name, type_size, block_size, is_quantized
See Also
Quantize Data Chunk
Description
Quantizes a chunk of floating-point data to a lower precision format.
Usage
ggml_quantize_chunk(type, src, nrows, n_per_row)
Arguments
type |
Target GGML type (e.g., GGML_TYPE_Q4_0) |
src |
Source numeric vector (F32 data) |
nrows |
Number of rows |
n_per_row |
Number of elements per row |
Value
Raw vector containing quantized data
Examples
# Quantize 256 floats to Q8_0 (block size 32)
data <- rnorm(256)
quantized <- ggml_quantize_chunk(GGML_TYPE_Q8_0, data, 1, 256)
ggml_quantize_free() # Clean up
Free Quantization Resources
Description
Frees any memory allocated by quantization. Call at end of program to avoid memory leaks.
Usage
ggml_quantize_free()
Value
NULL invisibly
Initialize Quantization Tables
Description
Initializes quantization tables for a given type. Called automatically by ggml_quantize_chunk, but can be called manually.
Usage
ggml_quantize_init(type)
Arguments
type |
GGML type (e.g., GGML_TYPE_Q4_0) |
Value
NULL invisibly
Check if Quantization Requires Importance Matrix
Description
Some quantization types require an importance matrix for optimal quality.
Usage
ggml_quantize_requires_imatrix(type)
Arguments
type |
GGML type |
Value
TRUE if importance matrix is required
Counter-Based Random Numbers (Graph)
Description
An F32 tensor of random numbers computed on the backend the graph runs on
(Philox4x32-10). Element i depends only on the key, stream
and i: a recorded graph draws new numbers when the key tensor's data
changes, and the CPU and Vulkan kernels produce the same uniform bits.
Usage
ggml_rand(ctx, key, dist = c("uniform", "gumbel", "normal"), dims, stream = 0L)
Arguments
ctx |
GGML context |
key |
F32 tensor with at least 2 elements; the bit patterns of the first two are the 64-bit key |
dist |
|
dims |
Shape of the result, 1 to 4 positive integers |
stream |
Non-negative integer separating nodes that share a key |
Details
"uniform" is (k + 0.5) / 2^23 for a 23-bit integer
k, so it lies strictly inside (0, 1) and is bit-identical across
backends. "gumbel" is -log(-log(u)) and "normal" is
Box-Muller on two uniforms; both go through log / cos /
sqrt and may differ between backends in the last bits.
The key is read as raw bits, so any two floats work, e.g. two draws of
stats::runif (which makes the stream follow set.seed). Give
every ggml_rand node sharing one key its own stream.
Value
F32 tensor of shape dims
ReLU Activation (Graph)
Description
Creates a graph node for ReLU (Rectified Linear Unit) activation: max(0, x)
Usage
ggml_relu_inplace(ctx, a)
ggml_relu(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_relu_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the ReLU operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
result <- ggml_relu(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Repeat (Graph)
Description
Creates a graph node that repeats tensor 'a' to match shape of tensor 'b'.
Usage
ggml_repeat(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Tensor to repeat |
b |
Target tensor (defines output shape) |
Value
Tensor with repeated values
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 1, 2)
ggml_set_f32(a, c(1, 2))
b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2)
result <- ggml_repeat(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [1, 1, 1, 2, 2, 2]
ggml_free(ctx)
Repeat Backward (Graph)
Description
Backward pass for repeat operation - sums repetitions back to original shape. Used for gradient computation during training.
Usage
ggml_repeat_back(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Input tensor (gradients from repeated tensor) |
b |
Target shape tensor (original tensor before repeat) |
Value
Tensor with summed gradients matching shape of b
Reset GGML Context
Description
Clears all tensor allocations in the context memory pool. The context can be reused without recreating it. This is more efficient than free + init for temporary operations.
Usage
ggml_reset(ctx)
Arguments
ctx |
GGML context pointer |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
ggml_reset(ctx)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 200)
ggml_free(ctx)
Reshape Tensor (Graph)
Description
Reshapes tensor to 1D with ne0 elements
Usage
ggml_reshape_1d(ctx, a, ne0)
ggml_reshape_2d(ctx, a, ne0, ne1)
ggml_reshape_3d(ctx, a, ne0, ne1, ne2)
ggml_reshape_4d(ctx, a, ne0, ne1, ne2, ne3)
Arguments
ctx |
GGML context |
a |
Input tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
ne3 |
Size of dimension 3 |
Details
ggml_reshape_2d(): Reshapes tensor to 2D with shape (ne0, ne1)
Returns: Reshaped tensor
ggml_reshape_3d(): Reshapes tensor to 3D with shape (ne0, ne1, ne2)
Returns: Reshaped tensor
ggml_reshape_4d(): Reshapes tensor to 4D with shape (ne0, ne1, ne2, ne3)
Returns: Reshaped tensor
Value
Reshaped tensor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4)
ggml_set_f32(a, 1:12)
result <- ggml_reshape_1d(ctx, a, 12)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 12)
ggml_set_f32(a, 1:12)
result <- ggml_reshape_2d(ctx, a, 3, 4)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 24)
ggml_set_f32(a, 1:24)
result <- ggml_reshape_3d(ctx, a, 2, 3, 4)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 120)
ggml_set_f32(a, 1:120)
result <- ggml_reshape_4d(ctx, a, 2, 3, 4, 5)
ggml_free(ctx)
Construct a single-cell result
Description
A ggml_result is the contract object returned by the dispatch layer
and consumed by the injection layer. The embedding is stored cell-by-component
(cells in rows), ready to drop into reducedDim() / a Seurat reduction.
Usage
ggml_result(embedding, metadata = list(), timings = numeric(0))
Arguments
embedding |
A numeric matrix, cells in rows, components in columns. |
metadata |
Named list (e.g. |
timings |
Named numeric vector of elapsed seconds per stage. |
Value
An object of class ggml_result.
RMS Normalization (Graph)
Description
Creates a graph node for RMS (Root Mean Square) normalization. Normalizes by x / sqrt(mean(x^2) + eps). CRITICAL for LLaMA models.
Usage
ggml_rms_norm(ctx, a, eps = 1e-05)
ggml_rms_norm_inplace(ctx, a, eps = 1e-05)
Arguments
ctx |
GGML context |
a |
Input tensor |
eps |
Epsilon value for numerical stability (default: 1e-5) |
Details
ggml_rms_norm_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the RMS normalization operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_rms_norm(ctx, a, eps = 1e-5)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result)
# sqrt(mean(output^2)) should be ~1
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_rms_norm_inplace(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
RMS Norm Backward (Graph)
Description
Creates a graph node for backward pass of RMS normalization. Used in training for computing gradients.
Usage
ggml_rms_norm_back(ctx, a, b, eps = 1e-05)
Arguments
ctx |
GGML context |
a |
Gradient tensor (dy from the forward output) |
b |
Input tensor (x from the forward pass) |
eps |
Epsilon for numerical stability (default 1e-5) |
Value
Tensor representing the gradient with respect to input
Roll (Graph)
Description
Circularly shifts tensor elements along dimensions 0..3.
Usage
ggml_roll(ctx, a, shift0 = 0L, shift1 = 0L, shift2 = 0L, shift3 = 0L)
Arguments
ctx |
GGML context |
a |
Input tensor |
shift0, shift1, shift2, shift3 |
Shift amount along each dimension |
Value
Rolled tensor
Rotary Position Embedding (Graph)
Description
Creates a graph node for RoPE (Rotary Position Embedding). RoPE is the dominant position encoding method in modern LLMs like LLaMA, Mistral, and many others.
Usage
ggml_rope(ctx, a, b, n_dims, mode = 0L)
ggml_rope_inplace(ctx, a, b, n_dims, mode = 0L)
ggml_rope_ext(
ctx,
a,
b,
c = NULL,
n_dims,
mode = 0L,
n_ctx_orig = 0L,
freq_base = 10000,
freq_scale = 1,
ext_factor = 0,
attn_factor = 1,
beta_fast = 32,
beta_slow = 1
)
ggml_rope_set_offset(a, n_offs)
ggml_rope_ext_inplace(
ctx,
a,
b,
c = NULL,
n_dims,
mode = 0L,
n_ctx_orig = 0L,
freq_base = 10000,
freq_scale = 1,
ext_factor = 0,
attn_factor = 1,
beta_fast = 32,
beta_slow = 1
)
ggml_rope_multi(
ctx,
a,
b,
c = NULL,
n_dims,
sections = c(0L, 0L, 0L, 0L),
mode = 0L,
n_ctx_orig = 0L,
freq_base = 10000,
freq_scale = 1,
ext_factor = 0,
attn_factor = 1,
beta_fast = 32,
beta_slow = 1
)
ggml_rope_multi_inplace(
ctx,
a,
b,
c = NULL,
n_dims,
sections = c(0L, 0L, 0L, 0L),
mode = 0L,
n_ctx_orig = 0L,
freq_base = 10000,
freq_scale = 1,
ext_factor = 0,
attn_factor = 1,
beta_fast = 32,
beta_slow = 1
)
ggml_rope_ext_back(
ctx,
a,
b,
c = NULL,
n_dims,
mode = 0L,
n_ctx_orig = 0L,
freq_base = 10000,
freq_scale = 1,
ext_factor = 0,
attn_factor = 1,
beta_fast = 32,
beta_slow = 1
)
ggml_rope_multi_back(
ctx,
a,
b,
c = NULL,
n_dims,
sections = c(0L, 0L, 0L, 0L),
mode = 0L,
n_ctx_orig = 0L,
freq_base = 10000,
freq_scale = 1,
ext_factor = 0,
attn_factor = 1,
beta_fast = 32,
beta_slow = 1
)
Arguments
ctx |
GGML context |
a |
Input tensor of shape [head_dim, n_head, seq_len, batch] |
b |
Position tensor (int32) of shape [seq_len] containing position indices |
n_dims |
Number of dimensions to apply rotation to (usually head_dim) |
mode |
RoPE mode: GGML_ROPE_TYPE_NORM (0), GGML_ROPE_TYPE_NEOX (2), etc. |
c |
Optional frequency factors tensor (NULL for default) |
n_ctx_orig |
Original context length the model was trained on |
freq_base |
Base frequency for RoPE (default 10000 for most models) |
freq_scale |
Frequency scale factor (1.0 = no scaling) |
ext_factor |
YaRN extension factor (0.0 to disable) |
attn_factor |
Attention scale factor (typically 1.0) |
beta_fast |
YaRN parameter for fast dimensions |
beta_slow |
YaRN parameter for slow dimensions |
n_offs |
Number of leading dims to leave unrotated (non-negative, even) |
sections |
Integer vector of length 4 specifying dimension sections for MRoPE |
Details
RoPE encodes position information by rotating pairs of dimensions in the embedding space. The rotation angle depends on position and dimension index.
Key benefits of RoPE: - Relative position information emerges naturally from rotation - Better extrapolation to longer sequences than absolute embeddings - No additional parameters needed
ggml_rope_inplace() is the in-place variant: it modifies its input and returns a view of it.
ggml_rope_ext(): Creates a graph node for extended RoPE with frequency scaling parameters.
Supports context extension techniques like YaRN, Linear Scaling, etc.
Returns: Tensor with extended RoPE applied
This extended version supports various context extension techniques:
- **Linear Scaling**: Set freq_scale = original_ctx / new_ctx - **YaRN**: Set ext_factor > 0 with appropriate beta_fast/beta_slow - **NTK-aware**: Adjust freq_base for NTK-style scaling
Common freq_base values: - LLaMA 1/2: 10000 - LLaMA 3: 500000 - Mistral: 10000 - Phi-3: 10000
ggml_rope_set_offset(): Restricts an existing RoPE node to the dims [n_offs, n_offs + n_dims)
of each row; the dims before and after are passed through unrotated.
Modifies the node in place and returns it. Errors when n_offs + n_dims
exceeds the row length ne[0].
Returns: a
ggml_rope_ext_inplace() is the in-place variant: it modifies its input and returns a view of it.
ggml_rope_multi(): Creates a graph node for multi-dimensional RoPE (MRoPE) used in vision transformers.
Supports separate rotation for different positional dimensions (e.g., height, width, time).
Returns: Tensor with multi-dimensional RoPE applied
ggml_rope_multi_inplace() is the in-place variant: it modifies its input and returns a view of it.
ggml_rope_ext_back(): Backward pass for extended RoPE (Rotary Position Embedding).
Used during training to compute gradients through RoPE.
Returns: Gradient tensor for the input
ggml_rope_multi_back(): Gradient node for ggml_rope_multi. Takes the same geometry
arguments as the forward op, which is what makes the rotation invertible.
Returns: Gradient w.r.t. the rotated input
Value
Tensor with same shape as input, with rotary embeddings applied
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Query tensor: head_dim=8, n_head=4, seq_len=16, batch=1
q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 8, 4, 16, 1)
ggml_set_f32(q, rnorm(8 * 4 * 16))
# Position indices
pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 16)
ggml_set_i32(pos, 0:15)
# Apply RoPE
q_rope <- ggml_rope(ctx, q, pos, 8, GGML_ROPE_TYPE_NORM)
graph <- ggml_build_forward_expand(ctx, q_rope)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 64, 8, 32, 1)
ggml_set_f32(q, rnorm(64 * 8 * 32))
pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 32)
ggml_set_i32(pos, 0:31)
# Standard RoPE with default freq_base
q_rope <- ggml_rope_ext(ctx, q, pos, NULL,
n_dims = 64, mode = 0L,
n_ctx_orig = 4096,
freq_base = 10000, freq_scale = 1.0,
ext_factor = 0.0, attn_factor = 1.0,
beta_fast = 32, beta_slow = 1)
graph <- ggml_build_forward_expand(ctx, q_rope)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Round Half to Even (Graph)
Description
Creates a graph node for element-wise rounding with ties to the even
neighbour (banker's rounding): round_even(2.5) = 2. Unlike
ggml_round, which sends ties away from zero.
Usage
ggml_round_even(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the round_even operation
Round (Graph)
Description
Creates a graph node for element-wise rounding: round(x)
Usage
ggml_round_inplace(ctx, a)
ggml_round(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_round_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the round operation
Run a single-cell task on the GGML backend
Description
Validates a ggml_task against ggml_ops_registry,
resolves the compute device, invokes the registered engine and returns a
ggml_result. This is the dispatch layer: the only place that
decides GPU vs CPU. device = "auto" (the task default) uses Vulkan when
a GPU is available and CPU otherwise; device = "vulkan" on a machine
with no GPU degrades to CPU with a message rather than failing.
Usage
ggml_run(task, backend = NULL, ...)
Arguments
task |
A |
backend |
Optional override of |
... |
Additional parameters merged over |
Value
A ggml_result.
See Also
Views Into a Packed RWKV / GLA Result
Description
ggml_rwkv_wkv6, ggml_rwkv_wkv7 and
ggml_gated_linear_attn all return one
[S * H, n_tokens + S * n_seqs] tensor holding the per-token outputs
followed by the final state. These helpers view either half, so the column
arithmetic lives in one place rather than at every call site.
Usage
ggml_rwkv_output(ctx, result, k)
ggml_rwkv_state(ctx, result, k, state)
Arguments
ctx |
GGML context |
result |
The tensor returned by one of the RWKV / GLA ops |
k |
The |
state |
The |
Details
Both are graph views sharing memory with result and add no
computation. k must be the same tensor the op was built with, since
S, H and n_tokens are read from it.
Value
For ggml_rwkv_output(), a tensor [S * H, n_tokens]
For ggml_rwkv_state(), a tensor [S * H, S * n_seqs]
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
RWKV-6 WKV Recurrence
Description
The weighted key-value recurrence of RWKV-6: a linear-attention-style scan
whose decay is per-token (td) on top of a per-channel bonus
(tf). Runs in O(n_tokens) rather than the O(n_tokens^2) of ordinary
attention, which is what makes RWKV a recurrent model at inference time.
Usage
ggml_rwkv_wkv6(ctx, k, v, r, tf, td, state)
Arguments
ctx |
GGML context |
k |
Keys, |
v |
Values, |
r |
Receptance, |
tf |
Per-channel decay bonus, |
td |
Per-token decay, |
state |
Initial state, |
Value
A tensor [S * H, n_tokens + S * n_seqs]
Shapes
With head size S, H heads, n_tokens tokens and
n_seqs sequences: k, v, r and td are
[S, H, n_tokens], tf is [S, H], and state holds
S * S * H * n_seqs elements (typically [S * S * H, n_seqs]).
All must be contiguous.
n_tokens counts the tokens of all sequences together, so it
must be a multiple of n_seqs: the kernel derives the per-sequence
length as n_tokens / n_seqs and restarts the state at each boundary.
Packed result
The result is [S * H, n_tokens + S * n_seqs]: the per-token outputs
occupy the first n_tokens columns and the final state the remaining
S * n_seqs. ggml_rwkv_output and
ggml_rwkv_state view either half.
Inference only
GGML_OP_RWKV_WKV6 has no backward pass in ggml.
See Also
ggml_rwkv_wkv7, ggml_rwkv_output
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Backward Pass of the RWKV-6 WKV Recurrence
Description
Gradients of ggml_rwkv_wkv6 with respect to all six of its
inputs, given grad, the gradient of the loss with respect to the
recurrence's packed forward result.
Usage
ggml_rwkv_wkv6_back(ctx, k, v, r, tf, td, state, grad)
Arguments
ctx |
GGML context |
k, v, r, tf, td, state |
The tensors passed to the forward recurrence |
grad |
Gradient w.r.t. the packed forward result |
Details
Training reaches this automatically through the backward graph; it is exported so the gradient can be checked against a numeric one.
Value
A flat tensor holding the six gradients, concatenated
Packed result
All six gradients come back in one flat tensor, in the order the forward op
takes its inputs: [d_k | d_v | d_r | d_tf | d_td | d_state].
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
RWKV-7 WKV Recurrence
Description
The RWKV-7 ("Goose") recurrence. Where RWKV-6 decays the state by a vector,
RWKV-7 applies a rank-1 update built from a and b alongside the
decay w, giving the state a richer transition.
Usage
ggml_rwkv_wkv7(ctx, r, w, k, v, a, b, state)
Arguments
ctx |
GGML context |
r |
Receptance, |
w |
Decay, |
k |
Keys, |
v |
Values, |
a |
Removal (state update term), |
b |
Addition (state update term), |
state |
Initial state, |
Value
A tensor [S * H, n_tokens + S * n_seqs]
Shapes
r, w, k, v, a and b are
[S, H, n_tokens]; state holds S * S * H * n_seqs
elements. All must be contiguous. The result is
[S * H, n_tokens + S * n_seqs], outputs then final state – see
ggml_rwkv_output.
n_tokens counts the tokens of all sequences together, so it
must be a multiple of n_seqs.
Inference only
GGML_OP_RWKV_WKV7 has no backward pass in ggml.
See Also
ggml_rwkv_wkv6, ggml_rwkv_output
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Backward Pass of the RWKV-7 WKV Recurrence
Description
Gradients of ggml_rwkv_wkv7 with respect to all seven of its
inputs, packed in forward source order:
[d_r | d_w | d_k | d_v | d_a | d_b | d_state].
Usage
ggml_rwkv_wkv7_back(ctx, r, w, k, v, a, b, state, grad)
Arguments
ctx |
GGML context |
r, w, k, v, a, b, state |
The tensors passed to the forward recurrence |
grad |
Gradient w.r.t. the packed forward result |
Value
A flat tensor holding the seven gradients, concatenated
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Save a Full Model (Architecture + Weights)
Description
Saves both the architecture and trained weights of a model to an RDS file.
Unlike ggml_save_weights(), which requires the model to be manually
reconstructed before loading, ggml_save_model() saves everything
needed to restore the model with a single call to ggml_load_model().
Usage
ggml_save_model(model, path)
Arguments
model |
A trained |
path |
File path (typically |
Value
The model (invisibly).
Supported model types
-
ggml_sequential_model— input shape, layer configs, trained weights, and compilation settings are all saved. -
ggml_functional_model— input/output node graphs (pure R lists, no ggml pointers) and trainednode_weightsare saved.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(16L, activation = "relu", input_shape = 4L) |>
ggml_layer_dense(2L, activation = "softmax")
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
x <- matrix(runif(64 * 4), 64, 4)
y <- matrix(c(rep(c(1,0), 32), rep(c(0,1), 32)), 64, 2)
model <- ggml_fit(model, x, y, epochs = 1L, batch_size = 32L, verbose = 0L)
tmp <- tempfile(fileext = ".rds")
ggml_save_model(model, tmp)
model2 <- ggml_load_model(tmp)
Save Model Weights to File
Description
Saves the trained weights of a sequential model to an RDS file. The file includes both weights and architecture metadata for validation when loading.
Usage
ggml_save_weights(model, path)
Arguments
model |
A trained ggml_sequential_model |
path |
File path to save weights (typically with .rds extension) |
Value
The model (invisibly).
Scale and Shift by Constants (Graph)
Description
Creates a graph node computing s * a + b for scalar s and
b. Unlike ggml_add1() with ggml_new_f32(), this adds a
constant without materialising a data-carrying tensor, so it is usable while
building a graph in a no_alloc context.
Usage
ggml_scale_bias(ctx, a, s, b)
Arguments
ctx |
GGML context |
a |
Input tensor |
s |
Scale factor (numeric scalar) |
b |
Bias added after scaling (numeric scalar) |
Value
Tensor representing s * a + b
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_scale_bias(ctx, a, 2.0, 1.0)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [3, 5, 7, 9]
ggml_free(ctx)
Scale (Graph)
Description
Creates a graph node for scaling tensor by a scalar: x * s
Usage
ggml_scale_inplace(ctx, a, s)
ggml_scale(ctx, a, s)
Arguments
ctx |
GGML context |
a |
Input tensor |
s |
Scalar value to multiply by |
Details
ggml_scale_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the scaled values
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_scale(ctx, a, 2.0)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [2, 4, 6, 8]
ggml_free(ctx)
Scatter elements into a copy of a tensor (ONNX ScatterElements)
Description
Copies 'data', then writes each element of 'updates' into the copy at the position given by the matching element of 'indices' along 'axis'. Every other coordinate is the element's own, so 'updates' and 'indices' share a shape and only their position along 'axis' is redirected.
Usage
ggml_scatter_elements(ctx, data, updates, indices, reduction = 0L, axis = 0L)
Arguments
ctx |
A ggml context |
data |
Base tensor (F32); the result starts as a copy of it |
updates |
Values to write (F32), same shape as 'indices' |
indices |
Destination positions along 'axis' (I32) |
reduction |
0 to overwrite, 1 to add |
axis |
Axis to scatter along, in ggml dim order (0 = ne[0]) |
Value
A new tensor with the shape of 'data'
Examples
ctx <- ggml_init(16 * 1024 * 1024)
base <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) # 4 rows of 3
ggml_set_f32(base, rep(0, 12))
upd <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2)
ggml_set_f32(upd, c(1, 2, 3, 4, 5, 6))
idx <- ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 3, 2)
ggml_set_i32(idx, c(3L, 3L, 3L, 0L, 0L, 0L)) # rows 3 and 0
out <- ggml_scatter_elements(ctx, base, upd, idx, 0L, 1L)
ggml_free(ctx)
Cosine annealing LR scheduler
Description
Anneals LR from initial value to 'eta_min' following a cosine curve.
Usage
ggml_schedule_cosine_decay(eta_min = 0, T_max = NULL)
Arguments
eta_min |
Minimum LR at end of schedule |
T_max |
Total number of epochs (defaults to nepoch from fit state) |
Value
List with on_train_begin and on_epoch_begin functions; eta_max is re-read from the LR at the start of every fit
See Also
Other callbacks:
ggml_callback_early_stopping(),
ggml_schedule_reduce_on_plateau(),
ggml_schedule_step_decay()
Reduce on plateau LR scheduler
Description
Reduces LR when a metric stops improving.
Usage
ggml_schedule_reduce_on_plateau(
monitor = "val_loss",
factor = 0.5,
patience = 5,
min_lr = 1e-07,
min_delta = 1e-04,
mode = "auto"
)
Arguments
monitor |
Metric to monitor: "val_loss", "train_loss", etc. |
factor |
Factor to reduce LR by |
patience |
Epochs with no improvement before reducing |
min_lr |
Minimum LR |
min_delta |
Minimum change to qualify as improvement |
mode |
"min" or "max". "auto" infers from monitor name. |
Value
List with on_train_begin and on_epoch_end functions; the state resets at the start of every fit
See Also
Other callbacks:
ggml_callback_early_stopping(),
ggml_schedule_cosine_decay(),
ggml_schedule_step_decay()
Step decay LR scheduler
Description
Reduces LR by a factor every 'step_size' epochs.
Usage
ggml_schedule_step_decay(step_size = 10, gamma = 0.1)
Arguments
step_size |
Reduce LR every this many epochs |
gamma |
Multiplicative factor of LR reduction |
Value
List with on_epoch_begin function
See Also
Other callbacks:
ggml_callback_early_stopping(),
ggml_schedule_cosine_decay(),
ggml_schedule_reduce_on_plateau()
Set Tensor Region (Graph)
Description
Copies tensor b into tensor a at a specified offset. This allows writing to a portion of a tensor.
Usage
ggml_set(ctx, a, b, nb1, nb2, nb3, offset)
Arguments
ctx |
GGML context |
a |
Destination tensor |
b |
Source tensor (data to copy) |
nb1 |
Stride for dimension 1 (in bytes) |
nb2 |
Stride for dimension 2 (in bytes) |
nb3 |
Stride for dimension 3 (in bytes) |
offset |
Byte offset in destination tensor |
Value
Tensor representing the set operation
Set 1D Tensor Region (Graph)
Description
Simplified 1D version of ggml_set. Copies tensor b into tensor a starting at offset.
Usage
ggml_set_1d(ctx, a, b, offset)
Arguments
ctx |
GGML context |
a |
Destination tensor |
b |
Source tensor |
offset |
Byte offset in destination tensor |
Value
Tensor representing the set operation
Set 2D Tensor Region (Graph)
Description
Simplified 2D version of ggml_set.
Usage
ggml_set_2d(ctx, a, b, nb1, offset)
Arguments
ctx |
GGML context |
a |
Destination tensor |
b |
Source tensor |
nb1 |
Stride for dimension 1 (in bytes) |
offset |
Byte offset in destination tensor |
Value
Tensor representing the set operation
Set F32 Data
Description
Set F32 Data
Usage
ggml_set_f32(tensor, data)
Arguments
tensor |
Tensor |
data |
Numeric vector |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(t, c(1, 2, 3, 4, 5))
ggml_get_f32(t)
ggml_free(ctx)
N-dimensional Element Access
Description
Gets a single f32 value from the tensor at position [i0, i1, i2, i3]. Works with any tensor type (auto-converts to float).
Usage
ggml_set_f32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0, value)
ggml_get_f32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0)
ggml_get_i32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0)
ggml_set_i32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0, value)
Arguments
tensor |
Tensor pointer |
i0, i1, i2, i3 |
Indices (0-based) |
value |
Float value to set |
Details
ggml_set_f32_nd(): Sets a single f32 value in the tensor at position [i0, i1, i2, i3].
This is a direct data write, not a graph operation.
Returns: NULL (invisible)
ggml_get_i32_nd(): Gets a single i32 value from the tensor at position [i0, i1, i2, i3].
Returns: Integer value
ggml_set_i32_nd(): Sets a single i32 value in the tensor at position [i0, i1, i2, i3].
Returns: NULL (invisible)
Value
Float value
Set I32 Data
Description
Sets integer data in an I32 tensor. Used for indices (ggml_get_rows) and position tensors (ggml_rope).
Usage
ggml_set_i32(tensor, data)
Arguments
tensor |
Tensor of type GGML_TYPE_I32 |
data |
Integer vector |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 10)
ggml_set_i32(pos, 0:9)
ggml_get_i32(pos)
ggml_free(ctx)
Set Number of Threads
Description
Set the number of threads for GGML operations
Usage
ggml_set_n_threads(n_threads)
Arguments
n_threads |
Number of threads to use |
Value
Number of threads set
Examples
# Use 4 threads
ggml_set_n_threads(4)
# Use all available cores
ggml_set_n_threads(parallel::detectCores())
Set Tensor Name
Description
Assigns a name to a tensor. Useful for debugging and graph visualization.
Usage
ggml_set_name(tensor, name)
Arguments
tensor |
Tensor pointer |
name |
Character string name |
Value
The tensor (for chaining)
Set No Allocation Mode
Description
When enabled, tensor creation will not allocate memory for data. Useful for creating computation graphs without allocating storage.
Usage
ggml_set_no_alloc(ctx, no_alloc)
Arguments
ctx |
GGML context |
no_alloc |
Logical, TRUE to disable allocation |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
ggml_set_no_alloc(ctx, TRUE)
ggml_get_no_alloc(ctx)
ggml_set_no_alloc(ctx, FALSE)
ggml_free(ctx)
Set OpenMP Thread Count
Description
Directly calls omp_set_num_threads() to limit OpenMP parallelism. Useful in tests to comply with CRAN policy on core usage.
Usage
ggml_set_omp_threads(n)
Arguments
n |
Number of threads |
Value
NULL invisibly
Tensor Role Flags
Description
Tensor Role Flags
Usage
ggml_set_param(tensor)
ggml_set_loss(tensor)
ggml_set_input(tensor)
ggml_set_output(tensor)
Arguments
tensor |
Tensor pointer |
Details
ggml_set_param(): Marks a tensor as a trainable parameter for backpropagation.
The optimizer will compute gradients for this tensor during training.
Returns: The tensor (for chaining)
ggml_set_loss(): Marks a tensor as the quantity to differentiate.
ggml_build_backward_expand requires the graph to contain one –
together with at least one ggml_set_param tensor, these are the
two ends of the backward pass.
A graph missing either is rejected with an error rather than aborting inside ggml. Returns: The tensor (for chaining)
ggml_set_output():
Returns: The tensor (for chaining)
Value
The tensor (for chaining)
Set Rows by Index (Graph)
Description
Writes the rows of b into the rows of a named by c:
row i of b goes to row c[i] of a (0-based),
converted to the type of a. The inverse of ggml_get_rows.
Rows of a not named in c keep their values; overlapping
destinations give an undefined result.
Usage
ggml_set_rows(ctx, a, b, c)
Arguments
ctx |
GGML context |
a |
Destination tensor |
b |
F32 or F16 source rows |
c |
I32 or I64 row indices |
Value
A view of a holding the result
See Also
Set the random seed for reproducible ggmlR runs
Description
Fixes the random seed used by ggmlR for everything that is stochastic:
weight initialisation (sequential, functional and autograd layers),
dropout masks (training-time),
data shuffling in the autograd dataloader / training loops.
Usage
ggml_set_seed(seed)
Arguments
seed |
A single integer (or value coercible to integer) used as the RNG
seed. |
Details
This is a thin wrapper around set.seed: all randomness in
ggmlR is produced by the base R RNG, so a fixed seed gives identical starting
weights, dropout masks and batch ordering across runs. It is the single point
of control used by the mlr3 learners (seed hyperparameter) and
the parsnip "ggml" engine (seed engine argument).
GPU note: this controls the random *inputs* to the computation, not the floating-point arithmetic itself. GPU (Vulkan) kernels are run-to-run stable on a given device/driver for the standard forward/backward paths, but ggmlR does not guarantee bit-for-bit identical results across different devices, drivers or backends (CPU vs Vulkan). Reproducibility is at the level of training dynamics, not exact bits.
Value
Invisibly returns seed.
See Also
Examples
ggml_set_seed(42)
a <- runif(3)
ggml_set_seed(42)
b <- runif(3)
identical(a, b) # TRUE
Set Tensor to Zero
Description
Sets all elements of a tensor to zero. This is more efficient than manually setting all elements.
Usage
ggml_set_zero(tensor)
Arguments
tensor |
Tensor to zero out |
Value
NULL (invisible)
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_set_f32(t, 1:10)
ggml_set_zero(t)
ggml_get_f32(t)
ggml_free(ctx)
Sign Function (Graph)
Description
Creates a graph node for element-wise sign function. sgn(x) = -1 if x < 0, 0 if x == 0, 1 if x > 0
Usage
ggml_sgn(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the sign operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -0.5, 0, 0.5, 2))
r <- ggml_sgn(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # c(-1, -1, 0, 1, 1)
ggml_free(ctx)
Sigmoid Activation (Graph)
Description
Creates a graph node for sigmoid activation: 1 / (1 + exp(-x))
Usage
ggml_sigmoid_inplace(ctx, a)
ggml_sigmoid(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_sigmoid_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the sigmoid operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
result <- ggml_sigmoid(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
SiLU Backward (Graph)
Description
Computes the backward pass for SiLU (Swish) activation. Used during training for gradient computation.
Usage
ggml_silu_back(ctx, a, b)
Arguments
ctx |
GGML context |
a |
Gradient tensor from upstream (dy) |
b |
Forward input tensor (x) |
Value
Gradient tensor for the input
SiLU Activation (Graph)
Description
Creates a graph node for SiLU (Sigmoid Linear Unit) activation, also known as Swish. CRITICAL for LLaMA models.
Usage
ggml_silu_inplace(ctx, a)
ggml_silu(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_silu_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the SiLU operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
result <- ggml_silu(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Sine (Graph)
Description
Creates a graph node for element-wise sine: sin(x)
Usage
ggml_sin(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the sin operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(0, pi/6, pi/2, pi))
result <- ggml_sin(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [0, 0.5, 1, 0]
ggml_free(ctx)
Release a Snapshot Made by a Trainer
Description
Frees the weights buffer and scheduler of a model returned by a
ggml_trainer's $snapshot(). Calling it twice is
harmless. The snapshot must not be used afterwards.
Usage
ggml_snapshot_free(model)
Arguments
model |
A snapshot from |
Value
NULL, invisibly.
See Also
Softmax Operations (Graph)
Description
Creates a graph node for softmax operation. CRITICAL for attention mechanisms.
Usage
ggml_soft_max(ctx, a)
ggml_soft_max_inplace(ctx, a)
ggml_soft_max_ext(ctx, a, mask = NULL, scale = 1, max_bias = 0)
ggml_soft_max_ext_inplace(ctx, a, mask = NULL, scale = 1, max_bias = 0)
ggml_soft_max_ext_back_inplace(ctx, a, b, scale = 1, max_bias = 0)
ggml_soft_max_ext_back(ctx, a, b, scale = 1, max_bias = 0)
ggml_soft_max_add_sinks(tensor, sinks)
Arguments
ctx |
GGML context |
a |
Input tensor |
mask |
Optional attention mask tensor (F16 or F32). NULL for no mask. Shape must be broadcastable to input tensor. |
scale |
Scaling factor, typically 1/sqrt(head_dim) |
max_bias |
Maximum ALiBi bias (0.0 to disable ALiBi) |
b |
Gradient tensor from upstream |
tensor |
A softmax node |
sinks |
Sink logits, one per head |
Details
ggml_soft_max_inplace() is the in-place variant: it modifies its input and returns a view of it.
ggml_soft_max_ext(): Creates a graph node for fused softmax operation with optional masking
and ALiBi (Attention with Linear Biases) support.
Computes: softmax(a * scale + mask * (ALiBi slope))
CRITICAL for efficient attention computation in transformers.
Returns: Tensor representing the scaled and masked softmax
This extended softmax is commonly used in transformer attention: 1. Scale attention scores by 1/sqrt(d_k) for numerical stability 2. Apply attention mask (e.g., causal mask, padding mask) 3. Optionally apply ALiBi position bias 4. Compute softmax
All these operations are fused for efficiency.
ggml_soft_max_ext_inplace() is the in-place variant: it modifies its input and returns a view of it.
ggml_soft_max_ext_back_inplace() is the in-place variant: it modifies its input and returns a view of it.
ggml_soft_max_ext_back(): Backward pass for extended softmax operation.
Returns: Gradient tensor for the input
ggml_soft_max_add_sinks(): The ggml_soft_max_ext counterpart of
ggml_flash_attn_ext_add_sinks.
Returns: The tensor (for chaining)
Value
Tensor representing the softmax operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_soft_max(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result)
# Output sums to 1.0
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_soft_max_inplace(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
ctx <- ggml_init(16 * 1024 * 1024)
scores <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 10)
ggml_set_f32(scores, rnorm(100))
attn <- ggml_soft_max_ext(ctx, scores, NULL, 1.0, max_bias = 0.0)
graph <- ggml_build_forward_expand(ctx, attn)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
Softplus Activation (Graph)
Description
Creates a graph node for Softplus activation. Softplus(x) = log(1 + exp(x)). A smooth approximation of ReLU.
Usage
ggml_softplus_inplace(ctx, a)
ggml_softplus(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_softplus_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the Softplus operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
r <- ggml_softplus(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r)
ggml_free(ctx)
Square (Graph)
Description
Creates a graph node for element-wise squaring: x^2
Usage
ggml_sqr_inplace(ctx, a)
ggml_sqr(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_sqr_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the square operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 2, 3, 4))
result <- ggml_sqr(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [1, 4, 9, 16]
ggml_free(ctx)
Square Root (Graph)
Description
Creates a graph node for element-wise square root: sqrt(x)
Usage
ggml_sqrt_inplace(ctx, a)
ggml_sqrt(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_sqrt_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the sqrt operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4)
ggml_set_f32(a, c(1, 4, 9, 16))
result <- ggml_sqrt(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [1, 2, 3, 4]
ggml_free(ctx)
State-Space Convolution (Mamba)
Description
Depthwise causal convolution over a sequence, the convolution branch of a
Mamba block. Each of the d_inner channels is convolved with its own
d_conv-tap kernel; the leading d_conv - 1 positions of
sx are the carried-over context from the previous chunk, which is
what makes the op resumable across chunks.
Usage
ggml_ssm_conv(ctx, sx, c)
Arguments
ctx |
GGML context |
sx |
Input with carried context, |
c |
Convolution kernels, |
Value
A tensor [d_inner, n_t, n_s]
Shapes
sx is [d_conv - 1 + n_t, d_inner, n_s] and c is
[d_conv, d_inner]; the result is [d_inner, n_t, n_s], where
n_t is the number of tokens per sequence and n_s the number of
sequences. All tensors are F32.
Inference only
GGML_OP_SSM_CONV has no backward pass in ggml, so a graph containing
it cannot be trained – see ggml_ssm_scan.
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Backward Pass of the State-Space Convolution
Description
Gradients of ggml_ssm_conv with respect to both of its inputs,
given grad, the gradient of the loss with respect to the convolution's
output.
Usage
ggml_ssm_conv_back(ctx, sx, c, grad)
Arguments
ctx |
GGML context |
sx |
The |
c |
The |
grad |
Gradient w.r.t. the output, shaped like it: |
Details
Training reaches this automatically through the backward graph – an
ggml_ssm_conv node is differentiable, unlike the other
state-space and RWKV ops. It is exported so the gradient can be checked
directly against a numeric one.
Value
A flat tensor holding d_sx followed by d_c
Packed result
Both gradients come back in one flat tensor, [d_sx | d_c]: the first
ggml_nelements(sx) elements are the gradient with respect to
sx, the remaining ggml_nelements(c) the gradient with respect
to c. A single op returns a single tensor, and packing them keeps the
convolution from being run once per input.
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Selective State-Space Scan (Mamba / Mamba-2)
Description
The recurrent core of a Mamba block: a linear state-space recurrence whose
transition is modulated per token by dt, B and C.
ids selects which state row each sequence continues from, so several
sequences can share one state buffer.
Usage
ggml_ssm_scan(ctx, s, x, dt, A, B, C, ids, K = 1L)
Arguments
ctx |
GGML context |
s |
Initial states, |
x |
Inputs, |
dt |
Per-token timestep, |
A |
State decay, |
B |
Input projection, |
C |
Output projection, same shape as |
ids |
Sequence-to-state mapping, |
K |
Number of state snapshots kept after the outputs (ggml 0.27):
slot 0 is the final state, slot |
Value
A flat tensor: outputs followed by the K state snapshots
(with K = 1, the final states)
Shapes
With d_state the state width, head_dim the per-head channel
count, n_head heads, n_seq_tokens tokens and n_seqs
sequences:
-
s—[d_state, head_dim, n_head, n_seqs_total], contiguous -
x—[head_dim, n_head, n_seq_tokens, n_seqs] -
dt—[n_head, n_seq_tokens, n_seqs], contiguous -
A—[1, n_head](Mamba-2) or[d_state, n_head](Mamba-1) -
B,C—[d_state, n_group, n_seq_tokens, n_seqs], same shape -
ids—[n_seqs], typeGGML_TYPE_I32, 0-based
Packed result
The result is a flat tensor holding the outputs followed by the final
states: ggml_nelements(x) output elements, then
d_state * head_dim * n_head * n_seqs state elements. Use
ggml_ssm_scan_output and ggml_ssm_scan_state to
view either half rather than computing the offsets by hand – getting them
wrong silently reads the state as output.
Inference only
GGML_OP_SSM_SCAN has no backward pass in ggml. A graph containing it
runs forward on CPU and Vulkan, but cannot be trained: ggml_fit() and
the autograd engine will not produce gradients for it.
See Also
ggml_ssm_scan_output, ggml_ssm_scan_state,
ggml_ssm_conv
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Backward Pass of the Selective State-Space Scan
Description
Gradients of ggml_ssm_scan with respect to all six of its
differentiable inputs, given grad, the gradient of the loss with
respect to the scan's packed forward result.
Usage
ggml_ssm_scan_back(ctx, s, x, dt, A, B, C, ids, grad)
Arguments
ctx |
GGML context |
s |
Initial states, as passed to the forward scan |
x |
Inputs, as passed to the forward scan |
dt |
Per-token timestep, as passed to the forward scan |
A |
State decay, as passed to the forward scan |
B |
Input projection, as passed to the forward scan |
C |
Output projection, as passed to the forward scan |
ids |
Sequence-to-state mapping, as passed to the forward scan |
grad |
Gradient w.r.t. the packed forward result |
Details
Training reaches this automatically through the backward graph. It is exported so the gradient can be checked directly against a numeric one.
Value
A flat tensor holding the six gradients, concatenated
Packed result
All six gradients come back in one flat tensor, in the same order the scan
takes its inputs: [d_s | d_x | d_dt | d_A | d_B | d_C]. ids is
integer and has no gradient. A single op returns a single tensor, and packing
them keeps the recurrence from being replayed once per input.
Cost
The forward pass does not keep its intermediate states, so this replays the recurrence before walking it back – roughly twice the forward work, and no extra memory beyond one state buffer.
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_output()
Views Into a Packed ssm_scan Result
Description
ggml_ssm_scan returns the outputs and the final states in one
flat tensor. These helpers view either half with the right shape, so the
offset arithmetic lives in one place: ggml_ssm_scan_output() yields
the outputs shaped like x, and ggml_ssm_scan_state() the final
states shaped like the leading n_seqs rows of s.
Usage
ggml_ssm_scan_output(ctx, result, x)
ggml_ssm_scan_state(ctx, result, s, ids)
Arguments
ctx |
GGML context |
result |
The tensor returned by |
x |
The |
s |
The |
ids |
The |
Details
Both are graph views sharing memory with result; they add no
computation. x, s and ids must be the same tensors the
scan was built with, since their shapes are what the offsets are derived
from.
Value
For ggml_ssm_scan_output(), a tensor shaped like x
For ggml_ssm_scan_state(), a tensor
[d_state, head_dim, n_head, n_seqs]
See Also
Other state-space:
ggml_gated_delta_net(),
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back()
Step Function (Graph)
Description
Creates a graph node for element-wise step function. step(x) = 0 if x <= 0, 1 if x > 0 Also known as the Heaviside step function.
Usage
ggml_step(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the step operation
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -0.5, 0, 0.5, 2))
r <- ggml_step(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r) # c(0, 0, 0, 1, 1)
ggml_free(ctx)
Element-wise Subtraction (Graph)
Description
Creates a graph node for element-wise subtraction.
Usage
ggml_sub(ctx, a, b)
ggml_sub_inplace(ctx, a, b)
Arguments
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Details
ggml_sub_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the subtraction operation (a - b)
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(5, 4, 3, 2, 1))
ggml_set_f32(b, c(1, 1, 1, 1, 1))
result <- ggml_sub(ctx, a, b)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Sum (Graph)
Description
Creates a graph node that computes the sum of all elements.
Usage
ggml_sum(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Scalar tensor with the sum
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(1, 2, 3, 4, 5))
result <- ggml_sum(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # 15
ggml_free(ctx)
Sum Rows (Graph)
Description
Creates a graph node that computes the sum along rows.
Usage
ggml_sum_rows(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor with row sums
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2)
ggml_set_f32(a, c(1, 2, 3, 4, 5, 6))
result <- ggml_sum_rows(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
output <- ggml_get_f32(result) # [6, 15]
ggml_free(ctx)
Tanh Activation (Graph)
Description
Creates a graph node for hyperbolic tangent activation.
Usage
ggml_tanh_inplace(ctx, a)
ggml_tanh(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Details
ggml_tanh_inplace() is the in-place variant: it modifies its input and returns a view of it.
Value
Tensor representing the tanh operation
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5)
ggml_set_f32(a, c(-2, -1, 0, 1, 2))
result <- ggml_tanh(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
ggml_get_f32(result)
ggml_free(ctx)
Construct a single-cell compute task
Description
A ggml_task is the contract object passed from the extraction layer to
the dispatch layer. It bundles the operation name, the dense feature matrix
(features in rows, cells in columns — the single-cell convention), the
operation parameters and the requested device. It performs no computation.
Usage
ggml_task(op, matrix, params = list(), device = c("auto", "vulkan", "cpu"))
Arguments
op |
Operation name; must be registered in
|
matrix |
A numeric |
params |
Named list of operation parameters (e.g. |
device |
|
Value
An object of class ggml_task.
See Also
GPU transposed cross product (drop-in for tcrossprod)
Description
Computes A %*% t(B) (or A %*% t(A) when B is
NULL) on the Vulkan GPU, returning an R matrix, with a CPU fallback.
See ggml_matmul for the precision and dispatch notes.
Usage
ggml_tcrossprod(A, B = NULL, device = "auto", prec = "f32")
Arguments
A |
A numeric matrix. |
B |
A numeric matrix with |
device, prec |
See |
Value
A %*% t(B) as a numeric matrix.
Copy Tensor Data
Description
Copies raw data from src tensor to dst tensor (must be same size).
Usage
ggml_tensor_copy(dst, src)
Arguments
dst |
Destination tensor |
src |
Source tensor |
Value
NULL (invisible)
Raw Data Pointer of a Tensor
Description
Returns the address of the tensor's data buffer as a numeric value. Useful for checking whether two tensors alias the same memory – a view and its source share a pointer.
Usage
ggml_tensor_data_ptr(tensor)
Arguments
tensor |
Tensor pointer |
Value
The data address, as a double
Get Tensor Strides (nb)
Description
Returns the byte strides for each dimension of the tensor.
Usage
ggml_tensor_nb(tensor)
Arguments
tensor |
Tensor pointer |
Value
Numeric vector of 4 stride values (nb0, nb1, nb2, nb3)
Count Tensors in Context
Description
Counts the number of tensors allocated in a context.
Usage
ggml_tensor_num(ctx)
Arguments
ctx |
GGML context |
Value
Number of tensors
Get Tensor Overhead
Description
Returns the memory overhead (metadata) for each tensor in bytes
Usage
ggml_tensor_overhead()
Value
Size in bytes
Examples
ggml_tensor_overhead()
Fill Tensor with Scalar
Description
Sets all elements of a f32 tensor to a single value.
Usage
ggml_tensor_set_f32_scalar(tensor, value)
Arguments
tensor |
Tensor pointer |
value |
Float value to fill with |
Value
NULL (invisible)
Get Tensor Shape
Description
Returns the shape of a tensor as a numeric vector of 4 elements (ne0, ne1, ne2, ne3)
Usage
ggml_tensor_shape(tensor)
Arguments
tensor |
Tensor pointer |
Value
Numeric vector of length 4 with dimensions
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20)
ggml_tensor_shape(t)
ggml_free(ctx)
Get Tensor Type
Description
Returns the data type of a tensor as an integer code
Usage
ggml_tensor_type(tensor)
Arguments
tensor |
Tensor pointer |
Value
Integer type code (0 = F32, 1 = F16, etc.)
Examples
ctx <- ggml_init(1024 * 1024)
t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_tensor_type(t)
ggml_free(ctx)
Test GGML
Description
Runs GGML library self-test and prints version info.
Usage
ggml_test()
Value
TRUE if test passed
Examples
ggml_test()
Compare the AdamW Optimizer Step Across Backends
Description
Runs ggml_opt_step_adamw() for several steps on backend and on
the CPU from the same starting weights and the same gradients, and reports the
weight checksum after each step.
Usage
ggml_test_adamw_steps(backend)
Arguments
backend |
Backend pointer to compare against the CPU. |
Details
This needs its own function rather than a case in
ggml_test_backend_ops: the op updates the weights and both
moment buffers in place, so step N depends on every step before it, and the
bias-correction terms it receives change every step. Comparing a single
execution proves nothing here – the first Adam step agrees bit-for-bit on
both backends and the disagreement only starts at the second.
Value
A data frame with step, w_cpu, w_backend and
abs_diff, or a one-column data frame with note when the op
could not be run.
See Also
Examples
if (ggml_vulkan_available()) {
be <- ggml_vulkan_init(0)
print(ggml_test_adamw_steps(be))
ggml_backend_free(be)
}
Compare Backend Operations Against the CPU
Description
Runs each supported operation on backend and on the CPU with identical
inputs, and reports the normalized mean squared error between the two. This is
the differential test used to find operations whose accelerated implementation
disagrees with the reference one.
Usage
ggml_test_backend_ops(backend, filter = NULL)
Arguments
backend |
Backend pointer, e.g. from |
filter |
Optional substring; only operations whose name contains it are
run. |
Details
Each case is executed three times against the same allocated graph and the final run is the one compared, so faults that only appear when a graph is re-executed are visible. That matters: both backend defects found in this package so far were of that kind – the first run was exact and later ones were not.
Value
A data frame with columns op, nmse and note.
nmse is NA when the case could not be run, and note
then says why.
Examples
if (ggml_vulkan_available()) {
be <- ggml_vulkan_init(0)
res <- ggml_test_backend_ops(be)
res[res$nmse > 1e-6, ] # operations that disagree
ggml_backend_free(be)
}
Timing Utilities
Description
Initializes the GGML timing system. Call this once at the beginning of the program before using ggml_time_ms() or ggml_time_us().
Usage
ggml_time_init()
ggml_time_ms()
ggml_time_us()
ggml_cycles()
ggml_cycles_per_ms()
Details
ggml_time_ms(): Returns the current time in milliseconds since the timer was initialized.
Returns: Numeric value representing milliseconds
ggml_time_us(): Returns the current time in microseconds since the timer was initialized.
More precise than ggml_time_ms() for micro-benchmarking.
Returns: Numeric value representing microseconds
ggml_cycles(): Returns the current CPU cycle count. Useful for low-level benchmarking.
Returns: Numeric value representing CPU cycles
ggml_cycles_per_ms(): Returns an estimate of CPU cycles per millisecond.
Useful for converting cycle counts to time.
Returns: Numeric value representing cycles per millisecond
Value
NULL (invisible)
Examples
ggml_time_init()
start <- ggml_time_ms()
Sys.sleep(0.01)
elapsed <- ggml_time_ms() - start
ggml_time_init()
start <- ggml_time_ms()
Sys.sleep(0.01)
elapsed <- ggml_time_ms() - start
ggml_time_init()
start <- ggml_time_us()
Sys.sleep(0.001)
elapsed <- ggml_time_us() - start
ggml_cycles()
ggml_cycles_per_ms()
Timestep Embedding (Graph Operation)
Description
Creates sinusoidal timestep embeddings as used in diffusion models. Reference: CompVis/stable-diffusion util.py timestep_embedding
Usage
ggml_timestep_embedding(ctx, timesteps, dim, max_period = 10000L)
Arguments
ctx |
GGML context |
timesteps |
Input tensor of timestep values [N] |
dim |
Embedding dimension |
max_period |
Maximum period for sinusoidal embedding (default 10000) |
Value
Tensor of shape [N, dim] with timestep embeddings
Top-K Indices (Graph)
Description
Returns the indices of top K elements per row. Useful for sampling strategies in language models (top-k sampling). Note: the resulting indices are in no particular order within top-k.
Usage
ggml_top_k(ctx, a, k)
Arguments
ctx |
GGML context |
a |
Input tensor (F32) |
k |
Number of top elements to return per row |
Value
Tensor containing I32 indices of top-k elements (not values)
Examples
ctx <- ggml_init(16 * 1024 * 1024)
# Logits from model output
logits <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
ggml_set_f32(logits, rnorm(100))
# Get top 5 logits for sampling
top5 <- ggml_top_k(ctx, logits, 5)
graph <- ggml_build_forward_expand(ctx, top5)
ggml_graph_compute(ctx, graph)
ggml_free(ctx)
TPxDP hybrid matrix multiply across replicas of Vulkan device groups
Description
Computes Y = X %*% t(W) as a hybrid of tensor parallelism and data
parallelism: the weight matrix W is replicated across replicas
device groups (data parallelism over the batch X), and within each group
the weight rows are split across that group's GPUs (tensor parallelism, via
ggml_vulkan_split_mul_mat). Each replica computes a contiguous
shard of the batch rows; the shards are concatenated back into the full result.
Usage
ggml_tp_dp_forward(
W,
X,
replicas,
weights = NULL,
transport = c("host-staging", "opaque-fd", "device-group")
)
Arguments
W |
Weight matrix, |
X |
Activation matrix, |
replicas |
A list of integer vectors, each a group of physical GPU indices
(0-based) forming one replica, e.g. |
weights |
Optional numeric vector (length = group size) giving the
per-device TP row weighting inside each group (applied to every group).
|
transport |
Cross-device TP gather transport (see
|
Details
This mirrors the inference layout for the 4x P100 target (2 replicas x TP=2): there is no cross-replica communication, so throughput scales with the replica count while each replica's cross-device TP gather stays local to its group.
Value
The M x N result matrix, equal to X %*% t(W) up to the
GPU's floating-point accumulation.
See Also
ggml_vulkan_split_mul_mat for the single-group TP path.
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 4) {
W <- matrix(rnorm(2048 * 64), nrow = 2048)
X <- matrix(rnorm(8 * 64), nrow = 8) # batch of 8
# 2 replicas x TP=2: batch split 4/4, weights split within {0,1} and {2,3}
Y <- ggml_tp_dp_forward(W, X, replicas = list(c(0, 1), c(2, 3)))
max(abs(Y - X %*% t(W)))
}
Tensor-parallel forward pass of a whole graph (meta backend)
Description
Runs forward once over several devices with graph-level tensor
parallelism: the weights are laid out across the devices by split
(see ggml_meta_backend), every op runs on all devices on its
own share, and the reductions between devices are inserted automatically.
The counterpart of ggml_pp_forward (split by layers) and
ggml_vulkan_split_mul_mat (one matrix at a time).
Usage
ggml_tp_forward(
weights,
forward,
x,
devices,
split = list(),
default = "mirror",
type = GGML_TYPE_F32,
mem_size = 16L * 1024L * 1024L
)
Arguments
weights |
Named list of numeric arrays (or vectors, as 1-D tensors),
at most 4 dimensions each; |
forward |
|
x |
Numeric array (or vector) of input data; |
devices, split, default |
See |
type |
Storage type of the weights: |
mem_size |
Bytes of ggml context metadata for the weights and for the graph (each); raise it for graphs with very many ops. |
Details
Shapes follow ggml: an R array's dim() is the tensor's ne,
fastest dimension first – the layout ggml_backend_tensor_set_data
uses. A weight W with dim(W) = c(n_in, n_out) and an input with
dim(x) = c(n_in, batch) give ggml_mul_mat(ctx, w$W, input) of
shape c(n_out, batch), i.e. t(W) %*% x in R terms.
The weights get the names of the weights list, so split rules
refer to them directly; the input tensor is named "input" (mirrored
unless split says otherwise). Intermediate tensors take their layout
from the ops. The classic MLP pattern is split = list(w1 = "rows",
w2 = "cols"): the hidden layer stays split, and the only reduction is
after w2.
Value
Numeric array with the output; dim() is the output tensor's
ne without trailing unit dimensions (a plain vector for 1-D).
See Also
ggml_meta_backend, ggml_pp_forward,
ggml_vulkan_split_mul_mat
Other meta backend:
GGML_SPLIT_AXIS,
ggml_meta_backend()
Examples
n_in <- 16L; n_hid <- 32L; n_out <- 8L; batch <- 4L
W1 <- array(rnorm(n_in * n_hid), c(n_in, n_hid))
W2 <- array(rnorm(n_hid * n_out), c(n_hid, n_out))
X <- array(rnorm(n_in * batch), c(n_in, batch))
Y <- ggml_tp_forward(
weights = list(w1 = W1, w2 = W2),
forward = function(ctx, input, w)
ggml_mul_mat(ctx, w$w2, ggml_relu(ctx, ggml_mul_mat(ctx, w$w1, input))),
x = X, devices = c("CPU", "CPU"),
split = list(w1 = "rows", w2 = "cols"))
max(abs(Y - t(W2) %*% pmax(t(W1) %*% X, 0)))
A Training Context You Step Yourself
Description
Builds the graph and optimizer context for a compiled model and hands them
back, so a training loop can be written in R rather than handed to
ggml_fit. The context lives until $free(), keeping
weights, Adam moments and the compute graph across as many steps as the loop
takes.
Usage
ggml_trainer(
model,
batch_size = 32L,
nbatch_logical = NULL,
loss_mask = NULL,
weight_decay = NULL
)
Arguments
model |
A compiled |
batch_size |
Samples per step. The graph is built for exactly this many;
every batch handed to |
nbatch_logical |
Logical batch size for gradient accumulation. Defaults
to |
loss_mask |
|
weight_decay |
Decoupled weight decay in |
Details
This is the level below ggml_fit_opt_gen. Reach for it when the
loop is not a pass over data: reinforcement learning, where experience comes
from a replay buffer the agent is still filling and a target network is
refreshed every so many steps; curriculum schemes that change the data
distribution as they go; data generated fresh for every step; or anything
where "epoch" is not the unit of progress.
Do not emulate this by calling ggml_fit once per batch: each
such call builds a new optimizer context, so the Adam moments are reset and
every step is the optimizer's first.
Value
An object of class ggml_trainer: a list of functions
$step(x, y, sample_weight = NULL),
$eval(x, y, sample_weight = NULL),
$predict(x, batch_size = NULL, matmul_precision = NULL), $snapshot(),
$update_target(target, tau = 1), $model(),
$calibrate_bn(x), $set_lr(lr), $n_steps(), $free().
Using it
tr <- ggml_trainer(model, batch_size = 32L)
on.exit(tr$free())
for (i in seq_len(10000L)) {
b <- sample_batch(replay, 32L) # your own bookkeeping
loss <- tr$step(b$x, b$y)
if (i %% 500L == 0L) target <- tr$model() # refresh a target network
}
model <- tr$model() # weights back into an R model object
tr$free()
What a step does
$step(x, y, sample_weight) runs one forward and backward pass and
returns that step's loss. Gradient accumulation, if nbatch_logical
asked for it, is tracked inside the context: the optimizer updates the
weights only on the period boundary, and the intervening steps only
accumulate. $eval(x, y, sample_weight) is the same without the
backward pass, for measuring on held-out data.
Loss weights and masks
With loss_mask set, the loss is weighted mean squared error,
sum(w * (pred - y)^2) / (batch * outputs), and $step() and
$eval() take a sample_weight matrix:
-
loss_mask = 1:sample_weightis[batch, 1](or a vector of lengthbatch) – one weight per sample, as in Keras. -
loss_mask = <outputs>:sample_weightis[batch, outputs]– one weight per output element. This is not Keras'sample_weight: a 0/1 matrix is a mask that trains only the outputs whose label is known (e.g. one action of a Q-network), and a 0 removes that element's gradient exactly. The loss is not renormalised by the number of active elements.
Omitting sample_weight on a weighted trainer means all ones. Only
loss = "mse" supports weights.
Models
Sequential models and functional models (ggml_model) with
one input and one output are supported. Functional models with
several inputs or outputs are refused. For a functional model with dropout
the dropout masks are redrawn before every $step().
Predicting during training
$predict(x, batch_size) returns predictions from the current weights,
as ggml_predict would, in eval mode (no dropout). It runs on a
scheduler of its own, created on first use: the weights are copied into a
separate buffer for the call and released after it, so neither the training
graph's buffers nor the optimizer state are touched. batch_size
defaults to the trainer's; x may have any number of rows.
Target networks
$snapshot() returns an independent copy of the model: its own weight
tensors in its own buffer and its own scheduler, sharing nothing mutable with
the trainer. Later steps do not move it, ggml_predict on it
never touches the training path, and it outlives $free(). It is built
in eval mode, so dropout is off in it. Release it with
ggml_snapshot_free (a finaliser is only a backstop); after that
ggml_predict and $update_target on it are refused with an error.
$update_target(target, tau = 1) moves a snapshot towards the current
weights, \theta' \leftarrow \tau\theta + (1 - \tau)\theta':
tau = 1 is the hard copy of DQN, a small tau the Polyak
average of DDPG/SAC. Non-trainable buffers (a sequential batch-norm's
running mean and variance) are copied whole, never averaged. The target
must have the same weights, by name and shape, or it is refused. The update
goes through host memory, one read and one write per tensor.
Lifetime
The context holds GPU buffers and must be released with $free();
calling it twice is harmless. A trainer that goes out of scope without it is
cleaned up when garbage collected, but that moment is not predictable, so
free it yourself. After $free() every other method errors rather than
touching released memory.
The model object passed in is unchanged – R semantics being what they are,
the trainer works on its own copy of the weights. $model() returns a
model carrying the current ones.
The returned model shares the trainer's backend scheduler, and predicting
with it re-lays the scheduler's buffers for another graph. The trainer
therefore re-allocates its own graph on the first step after any
$model() call; weights and Adam moments are kept. $predict()
needs none of this. Predicting with the
original model object while the trainer is alive is not covered by
this and can crash the session on the next step – take the model from
$model() instead.
Batch norm
A step normalizes each batch by its own statistics and does not update the
running ones a sequential batch_norm predicts with; ggml_fit()
recalibrates them after training, a trainer does not. Call
$calibrate_bn(x) with representative inputs (up to 1024 rows are
used) before predicting, snapshotting or saving – or fix the statistics
with ggml_freeze_bn_stats. Creating a trainer for a model
with unfixed batch-norm layers warns once.
See Also
ggml_fit_opt_gen for a generator-driven epoch loop,
ggml_fit for the ordinary in-memory path.
Examples
ggml_set_n_threads(1L) # deterministic, single OpenMP pool
n <- 64
x <- matrix(runif(n * 4), nrow = n, ncol = 4)
y <- matrix(0, nrow = n, ncol = 2)
for (i in seq_len(n)) { y[i, if (sum(x[i, ]) > 2) 1L else 2L] <- 1 }
model <- ggml_model_sequential() |>
ggml_layer_dense(8, activation = "relu") |>
ggml_layer_dense(2, activation = "softmax")
model$input_shape <- 4L
model <- ggml_compile(model, optimizer = "adam",
loss = "categorical_crossentropy")
tr <- ggml_trainer(model, batch_size = 16L)
for (i in 1:4) {
idx <- ((i - 1L) * 16L + 1L):(i * 16L)
tr$step(x[idx, , drop = FALSE], y[idx, , drop = FALSE])
}
model <- tr$model()
tr$free()
Training history of a fitted ggml model
Description
Returns the per-epoch loss / accuracy curve recorded during
ggml_fit, in a tidy data frame. This is the standard accessor
for the loss curve; it works on a raw sequential/functional model or on a
fitted parsnip engine object (e.g. from extract_fit_engine()).
Usage
ggml_training_history(object, format = c("wide", "long"), ...)
Arguments
object |
A fitted |
format |
|
... |
Unused; for extensibility. |
Value
A data frame (tibble if tibble is installed). Wide columns:
epoch, train_loss, train_accuracy, and
val_loss / val_accuracy when a validation split was used.
Returns NULL with a warning if the model has no recorded history
(e.g. not yet fitted).
See Also
Transpose (Graph)
Description
Creates a graph node for matrix transpose operation.
Usage
ggml_transpose(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor (2D matrix) |
Value
Tensor representing the transposed matrix
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2)
ggml_set_f32(a, 1:6)
result <- ggml_transpose(ctx, a)
graph <- ggml_build_forward_expand(ctx, result)
ggml_graph_compute(ctx, graph)
shape <- ggml_tensor_shape(result) # [2, 3]
ggml_free(ctx)
Truncate (Graph)
Description
Creates a graph node for element-wise truncation toward zero: trunc(x)
Usage
ggml_trunc(ctx, a)
Arguments
ctx |
GGML context |
a |
Input tensor |
Value
Tensor representing the trunc operation
Type Information
Description
Returns the string name of a GGML type.
Usage
ggml_type_size(type)
ggml_type_name(type)
ggml_type_sizef(type)
ggml_blck_size(type)
ggml_is_quantized(type)
ggml_ftype_to_ggml_type(ftype)
Arguments
type |
GGML type constant (e.g., GGML_TYPE_F32) |
ftype |
File type constant |
Details
ggml_type_size(): Returns the size in bytes for all elements in a block for a given type.
Returns: Size in bytes
ggml_type_sizef(): Returns the size in bytes of a GGML type as a floating-point number.
For quantized types, this is the average bytes per element.
Returns: Numeric size in bytes (can be fractional for quantized types)
ggml_blck_size(): Returns the block size for a GGML type. Quantized types process
data in blocks (e.g., 32 elements for Q4_0).
Returns: Integer block size
ggml_is_quantized(): Returns TRUE if the GGML type is a quantized format.
Returns: Logical indicating if type is quantized
ggml_ftype_to_ggml_type(): Converts a file type (ftype) to the corresponding GGML type.
Used when loading quantized models.
Returns: Integer GGML type
Value
Character string with type name
Examples
ggml_type_name(GGML_TYPE_F32) # "f32"
ggml_type_name(GGML_TYPE_Q4_0) # "q4_0"
ggml_type_sizef(GGML_TYPE_F32) # 4.0
ggml_type_sizef(GGML_TYPE_F16) # 2.0
ggml_blck_size(GGML_TYPE_F32) # 1
ggml_blck_size(GGML_TYPE_Q4_0) # 32
ggml_is_quantized(GGML_TYPE_F32) # FALSE
ggml_is_quantized(GGML_TYPE_Q4_0) # TRUE
Unfreeze Layer Weights
Description
Sets trainable = TRUE on layers, chosen as in
ggml_freeze_weights: by layer_names and/or
prefix, else by from / to, else all of them.
Usage
ggml_unfreeze_weights(
model,
from = 1L,
to = length(model$layers),
layer_names = NULL,
prefix = NULL,
...
)
Arguments
model |
A model object (ggml_sequential_model or ggml_functional_model) |
from |
Integer index of the first layer to unfreeze (default: 1) |
to |
Integer index of the last layer to unfreeze (default: last layer) |
layer_names |
Character vector of layer names to unfreeze (overrides from/to) |
prefix |
Character vector: unfreeze every layer whose name starts with
one of these. With |
... |
Additional arguments passed to methods |
Value
The model with selected layers unfrozen.
Examples
model <- ggml_model_sequential() |>
ggml_layer_dense(64, activation = "relu") |>
ggml_layer_dense(10, activation = "softmax")
model <- ggml_freeze_weights(model)
model <- ggml_unfreeze_weights(model, from = 2) # unfreeze last layer only
Unmarshal a ggmlR model from an in-memory container
Description
Reconstructs a ggmlR model previously produced by
ggml_marshal_model. Validates the container's format tag,
schema version, and (if digest is installed) the SHA-256 checksum of
the payload before deserializing.
Usage
ggml_unmarshal_model(x, backend = NULL)
Arguments
x |
A |
backend |
Backend selection passed through to
|
Value
A compiled ggmlR model object (sequential or functional).
See Also
ggml_marshal_model, ggml_load_model
Upscale Tensor (Graph)
Description
Upscales tensor by multiplying ne0 and ne1 by scale factor. Supports different interpolation modes for image upscaling.
Usage
ggml_upscale(ctx, a, scale_factor, mode = 0L)
GGML_SCALE_MODE_NEAREST
GGML_SCALE_MODE_BILINEAR
GGML_SCALE_MODE_BICUBIC
Arguments
ctx |
GGML context |
a |
Input tensor (typically 2D or 4D for images) |
scale_factor |
Integer scale factor (e.g., 2 = double size) |
mode |
Scale mode constant (see details) |
Format
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
Details
Scale mode constants:
-
GGML_SCALE_MODE_NEAREST(0): Nearest neighbor interpolation - fastest, pixelated -
GGML_SCALE_MODE_BILINEAR(1): Bilinear interpolation - smooth, good balance -
GGML_SCALE_MODE_BICUBIC(2): Bicubic interpolation - smoothest, most compute
Value
Upscaled tensor with dimensions multiplied by scale_factor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
img <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 8)
ggml_set_f32(img, rnorm(64))
# Nearest neighbor (fastest, pixelated)
up_nearest <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_NEAREST)
# Bilinear (smooth)
up_bilinear <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_BILINEAR)
# Bicubic (smoothest)
up_bicubic <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_BICUBIC)
graph <- ggml_build_forward_expand(ctx, up_nearest)
ggml_graph_compute(ctx, graph)
# Result is 16x16
ggml_free(ctx)
Get Used Memory
Description
Returns the amount of memory currently used in the context
Usage
ggml_used_mem(ctx)
Arguments
ctx |
GGML context |
Value
Used memory in bytes
Examples
ctx <- ggml_init(1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
ggml_used_mem(ctx)
ggml_free(ctx)
Get GGML version
Description
Get GGML version
Usage
ggml_version()
Value
Character string with GGML version
Examples
ggml_version()
Tensor Views (Graph)
Description
Creates a 1D view of a tensor starting at a byte offset. The view shares memory with the source tensor.
Usage
ggml_view_1d(ctx, a, ne0, offset = 0)
ggml_view_2d(ctx, a, ne0, ne1, nb1, offset = 0)
ggml_view_3d(ctx, a, ne0, ne1, ne2, nb1, nb2, offset = 0)
ggml_view_4d(ctx, a, ne0, ne1, ne2, ne3, nb1, nb2, nb3, offset = 0)
ggml_view_5d(ctx, a, ne0, ne1, ne2, ne3, ne4, nb1, nb2, nb3, nb4, offset = 0)
Arguments
ctx |
GGML context |
a |
Source tensor |
ne0 |
Number of elements in the view |
offset |
Byte offset from the start of tensor data |
ne1 |
Size of dimension 1 |
nb1 |
Stride for dimension 1 (in bytes) |
ne2 |
Size of dimension 2 |
nb2 |
Stride for dimension 2 (in bytes) |
ne3 |
Size of dimension 3 |
nb3 |
Stride for dimension 3 (in bytes) |
ne4 |
Size of dimension 4 |
nb4 |
Stride for dimension 4 (in bytes) |
Details
ggml_view_2d(): Creates a 2D view of a tensor starting at a byte offset.
The view shares memory with the source tensor.
Returns: View tensor
ggml_view_3d(): Creates a 3D view of a tensor starting at a byte offset.
The view shares memory with the source tensor.
Returns: View tensor
ggml_view_4d(): Creates a 4D view of a tensor starting at a byte offset.
The view shares memory with the source tensor.
CRITICAL for KV-cache operations in transformers.
Returns: View tensor
ggml_view_5d(): Creates a 5D view of a tensor starting at a byte offset. The view shares
memory with the source tensor. With strides that are not the natural
contiguous ones it reorders axes without a copy – e.g. swapping dims 3
and 4, which ggml_permute cannot do (it only moves dims 0-3).
Returns: View tensor
Value
View tensor
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100)
# View elements 10-19 (offset = 10 * 4 bytes = 40)
v <- ggml_view_1d(ctx, a, 10, 40)
ggml_free(ctx)
ctx <- ggml_init(1024 * 1024)
x <- ggml_new_tensor(ctx, GGML_TYPE_F32, 5L, c(4, 3, 2, 5, 6))
nb <- ggml_tensor_nb(x)
# dims 3 and 4 swapped: [4, 3, 2, 6, 5]
v <- ggml_view_5d(ctx, x, 4, 3, 2, 6, 5, nb[2], nb[3], nb[5], nb[4])
ggml_free(ctx)
View Tensor
Description
Creates a view of the tensor (shares data, no copy)
Usage
ggml_view_tensor(ctx, src)
Arguments
ctx |
GGML context |
src |
Source tensor |
Value
View tensor (shares data with src)
Examples
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
view <- ggml_view_tensor(ctx, a)
# view shares data with a
ggml_free(ctx)
Check if Vulkan support is available
Description
Returns TRUE if the package was compiled with Vulkan support. To enable Vulkan, install libvulkan-dev and glslc, then reinstall ggmlR.
Usage
ggml_vulkan_available()
Value
Logical indicating if Vulkan is available
Examples
ggml_vulkan_available()
Vulkan Device Information
Description
Returns the number of available Vulkan-capable GPU devices.
Usage
ggml_vulkan_device_count()
ggml_vulkan_list_devices()
ggml_vulkan_device_description(device = 0L)
ggml_vulkan_device_memory(device = 0L)
ggml_vulkan_device_caps(device = 0L)
Arguments
device |
Device index (0-based) |
Details
ggml_vulkan_list_devices(): Returns detailed information about all available Vulkan devices.
Returns: List of device information (index, name, memory)
ggml_vulkan_device_description(): Returns a human-readable description of the specified Vulkan device.
Returns: Character string with device description
ggml_vulkan_device_memory(): Returns free and total memory for the specified Vulkan device.
Returns: Named list with 'free' and 'total' memory in bytes, and 'max_tensor_size': the largest single tensor in bytes the device can bind (NA if unknown). Ops with a larger input or output do not run on this device and fall back to CPU, so callers that split work (e.g. tiled VAE decode) should keep every tensor below it, not only the total below 'free'.
ggml_vulkan_device_caps(): Returns hardware capabilities for the specified Vulkan device.
Returns: Named list: coopmat_support, coopmat1_fa_support, fp16, subgroup_size, subgroup_no_shmem
Value
Integer count of Vulkan devices (0 if Vulkan not available)
Examples
if (ggml_vulkan_available()) {
ggml_vulkan_device_count()
}
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
devices <- ggml_vulkan_list_devices()
print(devices)
}
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
ggml_vulkan_device_description(0)
}
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
mem <- ggml_vulkan_device_memory(0)
cat("Free:", mem$free / 1e9, "GB\n")
cat("Total:", mem$total / 1e9, "GB\n")
}
Probe Vulkan device groups (NVLink / multi-GPU peer access)
Description
Enumerates Vulkan device groups (VK_KHR_device_group, also known as
Linked Device Adapter / LDA) and probes whether the driver reports direct
peer memory access between the physical GPUs in each group.
Usage
ggml_vulkan_device_groups()
Details
A device group with more than one device and peer copy/generic memory features is the prerequisite for true GPU-to-GPU transfers routed over NVLink (or PCIe peer-to-peer) through a single device-group logical device — as opposed to sharing memory as an opaque fd between independent devices, which the driver may route through host memory.
This is a diagnostic only: it does not create any long-lived device group. On a machine with a single GPU it reports zero multi-device groups. Device groups for compute are effectively an NVIDIA feature; AMD/RADV typically reports only single-device groups.
Value
A named list with n_groups (integer, number of device groups
reported by the driver) and report (character, a human-readable
per-group diagnostic including peer memory features). Use cat() on
report to read it.
Examples
if (ggml_vulkan_available()) {
g <- ggml_vulkan_device_groups()
cat(g$report)
}
Is the Vulkan hard-exit path compiled in?
Description
Reports whether this build of ggmlR was compiled with
-DGGML_VK_HARD_EXIT, which is what makes
ggml_vulkan_shutdown(hard = TRUE) actually call _exit().
Usage
ggml_vulkan_hard_exit_available()
Details
The hard-exit path is disabled by default: CRAN Repository Policy
forbids a package from terminating the user's R session, so the released
package must not link _exit(). Builds from source can opt in with
R CMD INSTALL . --configure-args="--enable-hard-exit" (on Windows, set
Sys.setenv(GGML_VK_HARD_EXIT = "1") before installing, because R there
ignores configure.args).
When it is not compiled in, ggml_vulkan_shutdown(hard = TRUE) performs
the normal teardown and emits a warning rather than silently
ignoring the request.
Value
TRUE if the hard-exit path is available, otherwise FALSE.
See Also
Examples
ggml_vulkan_hard_exit_available()
Vulkan Backend Lifecycle
Description
Creates a Vulkan backend for the specified device. The backend must be freed with ggml_vulkan_free() when done.
Usage
ggml_vulkan_init(device = 0L)
ggml_vulkan_free(backend)
ggml_vulkan_is_backend(backend)
ggml_vulkan_backend_name(backend)
Arguments
device |
Device index (0-based, default 0) |
backend |
Vulkan backend pointer from ggml_vulkan_init() |
Details
ggml_vulkan_free(): Releases resources associated with the Vulkan backend.
Returns: NULL (invisible)
ggml_vulkan_is_backend(): Returns TRUE if the given backend is a Vulkan backend.
Returns: Logical indicating if backend is Vulkan
ggml_vulkan_backend_name(): Returns the name of the Vulkan backend (includes device info).
Returns: Character string with backend name
Value
Vulkan backend pointer
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
backend <- ggml_vulkan_init(0)
print(ggml_vulkan_backend_name(backend))
ggml_vulkan_free(backend)
}
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
backend <- ggml_vulkan_init(0)
ggml_vulkan_free(backend)
}
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
vk_backend <- ggml_vulkan_init(0)
cpu_backend <- ggml_backend_cpu_init()
ggml_vulkan_is_backend(vk_backend) # TRUE
ggml_vulkan_is_backend(cpu_backend) # FALSE
ggml_vulkan_free(vk_backend)
ggml_backend_free(cpu_backend)
}
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) {
backend <- ggml_vulkan_init(0)
print(ggml_vulkan_backend_name(backend))
ggml_vulkan_free(backend)
}
Opaque-fd device-to-device P2P self-test
Description
Exercises the VK_KHR_external_memory_fd transport used by Vulkan tensor
parallelism to move data between GPUs. A byte pattern is written on the source
device, its memory is exported as an opaque fd and imported on the destination
device, copied into a local buffer there, then read back and compared.
Usage
ggml_vulkan_p2p_selftest(
src_device,
dst_device,
bytes = 64L * 1024L * 1024L,
iters = 50L,
transport = c("host-staging", "opaque-fd", "device-group")
)
Arguments
src_device |
Source GPU index (0-based). |
dst_device |
Destination GPU index (0-based). Equal to |
bytes |
Transfer size in bytes (default 64 MiB). |
iters |
Number of copies to time for the bandwidth measurement (default 50). |
transport |
Cross-device transport to exercise: |
Details
When src_device == dst_device the test runs in loopback mode
(export and import on the same GPU) — a sanity check of the fd mechanism that
touches no inter-device link. When the devices differ it runs
cross-device: after verifying correctness it times iters
device-to-device copies and reports the achieved bandwidth.
Interpreting bandwidth: a measured rate above the PCIe 3.0 x16 ceiling (~16 GB/s) is empirical evidence that a faster physical link (e.g. NVLink) carried the bytes. It is not a claim that Vulkan used an NVLink API — Vulkan exposes no call to query the route, so the conclusion is inferred from the rate alone, never asserted from the API.
Value
A named list: status (integer, 0 = data verified, <0 = failure),
gbps (numeric, measured cross-device bandwidth; 0 for loopback or on
failure) and report (character diagnostic, incl. the NVLink-vs-PCIe
inference).
Examples
if (ggml_vulkan_available()) {
# loopback sanity check on device 0
r <- ggml_vulkan_p2p_selftest(0L, 0L)
cat(r$report)
# cross-device P2P (requires >= 2 GPUs)
if (ggml_vulkan_device_count() >= 2) {
r <- ggml_vulkan_p2p_selftest(0L, 1L)
cat(r$report)
}
}
Explicitly tear down the Vulkan instance and devices
Description
Waits for all Vulkan devices to go idle, then destroys the devices and the Vulkan instance. Call this at the end of a script that used multiple GPUs (tensor / pipeline parallelism), before the process exits.
Usage
ggml_vulkan_shutdown(hard = FALSE, status = 0L)
Arguments
hard |
Logical. If |
status |
Integer process exit code used only when |
Details
Why this exists: with several Vulkan devices active, R's own teardown at
process exit can run after the Vulkan loader / Mesa ICD libraries have
been unmapped, so the device destructors call into unmapped memory and the
process crashes with a segfault after your results were already printed.
Calling ggml_vulkan_shutdown() while the process is still alive runs the
teardown while the loader is still mapped, avoiding that crash. It is only
needed for exit cleanliness — your computed results are unaffected either way.
Idempotent and a no-op if Vulkan was never initialized. After this call the next Vulkan operation transparently re-initializes the instance from scratch, so it is safe to call mid-session too.
The plain teardown alone does not always win the exit-time race: with backends
created inside pipeline / tensor-parallel forwards, the loader-static-destruction
segfault (address ending f1ba0) can still fire flakily after your
results are printed, because no R exit hook runs before R unmaps the loader.
For a guaranteed-clean exit, pass hard = TRUE as the last
statement of a standalone script: after teardown it calls _exit(0),
terminating the process immediately without running R's / the C runtime's exit
handlers or unmapping any shared library — so there is no loader teardown phase
left to crash. Because it bypasses R's normal shutdown, only use
hard = TRUE at the very end of a script, never mid-session or in a
package/interactive context.
Value
Invisibly NULL. Does not return when hard = TRUE and
the hard-exit path is compiled in (see the section above).
Hard exit is opt-in at build time
The _exit() path is compiled out by default, because CRAN
Repository Policy forbids a package from terminating the user's R session. In
such a build hard = TRUE performs the normal teardown and emits a
warning — it is never silently ignored — so the exit-time race
described above can still fire. To compile it in, build from source with
R CMD INSTALL . --configure-args="--enable-hard-exit" (on Windows set
Sys.setenv(GGML_VK_HARD_EXIT = "1") first, since R ignores
configure.args there). Check the current build with
ggml_vulkan_hard_exit_available().
See Also
ggml_vulkan_hard_exit_available
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) {
W <- matrix(rnorm(2048 * 64), nrow = 2048)
X <- matrix(rnorm(4 * 64), nrow = 4)
Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2)
ggml_vulkan_shutdown() # clean up before the script exits
}
Create a Vulkan tensor-split buffer type
Description
Builds (or fetches from a cache) a ggml buffer type that row-splits a
weight tensor across n_devices GPUs for tensor parallelism. This is the
factory the graph allocator talks to (Stage E4): a weight placed in this buffer
type has its rows distributed across the devices, each holding a contiguous
band (see ggml_vulkan_split_row_ranges). Non-split tensors fall
back to main_device.
Usage
ggml_vulkan_split_buffer_type(
n_devices = ggml_vulkan_device_count(),
main_device = 0L,
weights = NULL,
device_ids = NULL,
transport = c("host-staging", "opaque-fd", "device-group"),
probe = NULL
)
Arguments
n_devices |
Number of GPUs to split across (default: all available).
Ignored if |
main_device |
Device index (0-based) holding non-split fallbacks (default 0). |
weights |
Optional numeric vector of length |
device_ids |
Optional integer vector of physical GPU indices (0-based) the
split slots map onto, e.g. |
transport |
Cross-device gather transport for the eventual compute:
|
probe |
A numeric vector |
Details
The buffer type is cached in C and lives for the rest of the session; the returned external pointer has no finalizer and must not be freed.
Value
A named list: ptr (external pointer to the cached buffer type),
name (its config-derived name) and alloc_size (sum of the
padded per-device slice sizes for probe, in bytes; 0 if no probe).
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) {
bt <- ggml_vulkan_split_buffer_type(n_devices = 2, probe = c(2048, 64))
bt$name
bt$alloc_size # total bytes for a 2048x64 f32 weight across 2 devices
}
Tensor-parallel matrix multiply across Vulkan devices
Description
Computes Y = X %*% t(W) with the rows of the weight matrix W
split across n_devices GPUs (true tensor parallelism), the activations
X broadcast to every device, and the resulting column slices gathered
back into a single matrix. This is Stage E3 of the Vulkan tensor-parallelism
work: it exercises the row-split + per-device compute + cross-device gather
path end to end on real hardware.
Usage
ggml_vulkan_split_mul_mat(
W,
X,
n_devices = ggml_vulkan_device_count(),
weights = NULL,
device_ids = NULL,
transport = c("host-staging", "opaque-fd", "device-group")
)
Arguments
W |
Weight matrix, |
X |
Activation matrix, |
n_devices |
Number of GPUs to split across (default: all available).
Ignored if |
weights |
Optional numeric vector of length |
device_ids |
Optional integer vector of physical GPU indices (0-based) to
split across, e.g. |
transport |
Cross-device gather transport: |
Details
Each device owns a contiguous band of W's rows (via the same row-split
math as ggml_vulkan_split_row_ranges) and produces the matching
band of Y's columns; the bands are gathered through the transport (host
staging by default). This is a validation / correctness entry point, not a hot
inference path — it stands up one backend per device per call and round-trips
through host memory.
Value
The M x N result matrix, equal to X %*% t(W) up to the
GPU's floating-point accumulation. Errors if the split compute fails; the
C-level diagnostic is attached as attribute "report".
Examples
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) {
W <- matrix(rnorm(8 * 4), nrow = 8) # 8 output features x 4 inputs
X <- matrix(rnorm(3 * 4), nrow = 3) # 3 samples x 4 inputs
Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2)
max(abs(Y - X %*% t(W))) # ~1e-3 (GPU f16 accumulation)
}
Compute tensor-parallel row-split ranges
Description
Pure arithmetic helper for the Vulkan tensor-parallel split buffer type: given
a number of tensor rows and a per-device weight vector, returns the half-open
row range [row_low, row_high) owned by each device. Row boundaries are
rounded down to a fixed granularity, and the last device always covers up to
nrows, so the ranges are contiguous, non-overlapping and cover every
row exactly once.
Usage
ggml_vulkan_split_row_ranges(nrows, n_devices, weights = NULL)
Arguments
nrows |
Number of rows in the tensor (integer). |
n_devices |
Number of devices to split across (integer, >= 1). |
weights |
Optional numeric vector of length |
Details
This touches no GPU and is exposed mainly to verify the split logic. It is the math behind row-split tensor parallelism (a weight matrix distributed across several GPUs), independent of any actual multi-GPU allocation.
Value
A named list with row_low and row_high: numeric vectors
of length n_devices holding 0-based, half-open row ranges.
Examples
if (ggml_vulkan_available()) {
# 4096 rows across 2 devices, evenly
ggml_vulkan_split_row_ranges(4096L, 2L)
# weighted 3:1
ggml_vulkan_split_row_ranges(4096L, 2L, weights = c(3, 1))
}
Print Vulkan status
Description
Prints information about Vulkan availability and devices, and returns the same information as a list so it can be used programmatically.
Usage
ggml_vulkan_status()
Value
Invisibly, a list with components:
- available
Logical, whether Vulkan support is compiled in and usable
- n_devices
Integer, number of Vulkan devices (0 if unavailable)
- devices
List of per-device property lists as returned by
ggml_vulkan_list_devices; an empty list if there are none
Status is also printed to the console.
Examples
ggml_vulkan_status()
st <- ggml_vulkan_status()
if (st$available && st$n_devices > 0) {
print(st$devices[[1]]$name)
}
Window Partition (Graph)
Description
Partitions a tensor into non-overlapping windows of size w.
Usage
ggml_win_part(ctx, a, w)
Arguments
ctx |
GGML context |
a |
Input tensor |
w |
Window size |
Value
Partitioned tensor
Window Un-partition (Graph)
Description
Reassembles windowed partitions produced by ggml_win_part.
Usage
ggml_win_unpart(ctx, a, w0, h0, w)
Arguments
ctx |
GGML context |
a |
Input tensor |
w0 |
Original width |
h0 |
Original height |
w |
Window size |
Value
Un-partitioned tensor
Execute with Temporary Context
Description
Creates a temporary context, executes code, and frees it automatically. Useful when you need to create large temporary tensors.
Usage
ggml_with_temp_ctx(mem_size, expr)
Arguments
mem_size |
Context memory size in bytes |
expr |
Expression to evaluate with the temporary context |
Value
Result of the expression
Examples
# Create tensors in temporary context
result <- ggml_with_temp_ctx(1024 * 1024, {
a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10)
ggml_set_f32(a, 1:10)
ggml_get_f32(a)
})
xIELU Activation (Graph)
Description
Creates a graph node for the xIELU activation:
a_p x^2 + \beta x for x > 0, and
a_n (e^{\min(x, \epsilon)} - 1 - x) + \beta x otherwise, with
a_p = \mathrm{softplus}(\alpha_p) and
a_n = \beta + \mathrm{softplus}(\alpha_n) (the raw parameters are
transformed when the node is built, as in ggml).
Usage
ggml_xielu(ctx, a, alpha_n, alpha_p, beta, eps)
Arguments
ctx |
GGML context |
a |
Input tensor |
alpha_n, alpha_p |
Raw (pre-softplus) negative and positive slopes |
beta, eps |
Linear coefficient and the clamp for the negative branch |
Value
Tensor representing the xielu operation
parsnip ggml engine: classification fit
Description
Internal fit wrapper called by parsnip when 'mlp()' is used with 'engine = "ggml"' in classification mode. Not intended for direct use.
Usage
ggmlr_parsnip_fit_classif(
x,
y,
hidden_layers = c(128L, 64L),
epochs = 10L,
dropout = 0.2,
activation = "relu",
learn_rate = NULL,
batch_size = 32L,
verbose = 0L,
validation_split = 0,
callbacks = list(),
optimizer = "adam",
backend = "auto",
seed = NULL,
...
)
Arguments
x |
Numeric matrix of predictors. |
y |
Factor of class labels. |
|
Integer vector of hidden layer widths. | |
epochs |
Number of training epochs. |
dropout |
Dropout rate in '[0, 1)'. |
activation |
Hidden activation, e.g. '"relu"'. |
learn_rate |
Optional learning rate (applied via callback). |
batch_size |
Minibatch size. |
verbose |
Verbosity level (0/1/2). |
validation_split |
Fraction in '[0, 1)' for validation. |
callbacks |
List of ggmlR callbacks. |
optimizer |
One of '"adam"', '"sgd"'. |
backend |
One of '"auto"', '"cpu"', '"vulkan"'. |
seed |
Optional integer RNG seed for reproducible weight init / training. |
... |
Unused. |
Value
A fitted 'ggmlr_parsnip_model' object.
parsnip ggml engine: regression fit
Description
Internal fit wrapper called by parsnip when 'mlp()' is used with 'engine = "ggml"' in regression mode. Not intended for direct use.
Usage
ggmlr_parsnip_fit_regr(
x,
y,
hidden_layers = c(128L, 64L),
epochs = 10L,
dropout = 0.2,
activation = "relu",
learn_rate = NULL,
batch_size = 32L,
verbose = 0L,
validation_split = 0,
callbacks = list(),
optimizer = "adam",
backend = "auto",
seed = NULL,
...
)
Arguments
x |
Numeric matrix of predictors. |
y |
Numeric response vector. |
|
Integer vector of hidden layer widths. | |
epochs |
Number of training epochs. |
dropout |
Dropout rate in '[0, 1)'. |
activation |
Hidden activation, e.g. '"relu"'. |
learn_rate |
Optional learning rate (applied via callback). |
batch_size |
Minibatch size. |
verbose |
Verbosity level (0/1/2). |
validation_split |
Fraction in '[0, 1)' for validation. |
callbacks |
List of ggmlR callbacks. |
optimizer |
One of '"adam"', '"sgd"'. |
backend |
One of '"auto"', '"cpu"', '"vulkan"'. |
seed |
Optional integer RNG seed for reproducible weight init / training. |
... |
Unused. |
Value
A fitted 'ggmlr_parsnip_model' object.
Free GGUF Resources
Description
Explicitly frees the internal GGUF context. Called automatically by the garbage collector, but can be called manually to release memory sooner.
Usage
gguf_free(x)
Arguments
x |
A |
Value
Called for its side effect (releases the GGUF context); invisibly returns NULL.
Load a GGUF File
Description
Opens a GGUF file and reads its metadata. By default also reads tensor data
into memory; with meta_only = TRUE only the header and key-value
metadata are read (no tensor data is allocated), which is cheap and enough
for inspecting architecture / type fields. Returns an S3 object of class
"gguf" wrapping the internal pointer.
Usage
gguf_load(path, meta_only = FALSE)
Arguments
path |
Path to a .gguf file. |
meta_only |
If |
Value
An object of class "gguf".
Load a GGUF Image from Memory
Description
Parses a GGUF file already read into an R raw vector (e.g. downloaded, or
extracted from an archive) without writing it to disk. Everything the
result needs is copied, so raw may be discarded afterwards. The
result is the same "gguf" object as from gguf_load,
with path = NA.
Usage
gguf_load_raw(raw, meta_only = FALSE)
Arguments
raw |
A raw vector holding the complete GGUF image. |
meta_only |
If |
Value
An object of class "gguf".
Get GGUF Metadata
Description
Returns all key-value metadata pairs from a GGUF file as a named list.
Usage
gguf_metadata(x)
Arguments
x |
A |
Value
A named list of metadata values.
Extract Tensor Data
Description
Dequantizes (if needed) and returns tensor weights as an R numeric array with dimensions matching the tensor shape.
Usage
gguf_tensor_data(x, name)
Arguments
x |
A |
name |
Tensor name (character). |
Value
A numeric array.
Get Tensor Info
Description
Returns name, shape, type, and size in bytes for a single tensor.
Usage
gguf_tensor_info(x, name)
Arguments
x |
A |
name |
Tensor name (character). |
Value
A list with elements name, shape, type,
size_bytes.
List Tensor Names in a GGUF File
Description
List Tensor Names in a GGUF File
Usage
gguf_tensor_names(x)
Arguments
x |
A |
Value
Character vector of tensor names.
Get Tensor Shape
Description
Returns the shape of a tensor as stored in the GGUF header, innermost
dimension first (ggml ne order), with trailing unit dimensions
dropped. Works with meta_only = TRUE objects as well.
Usage
gguf_tensor_ne(x, name)
Arguments
x |
A |
name |
Tensor name (character). |
Value
An integer vector.
One-row summary of a fitted ggml parsnip model
Description
Returns a single-row [tibble][tibble::tibble] summarising the fitted model, in broom 'glance()' style.
Usage
## S3 method for class 'ggmlr_parsnip_model'
glance(x, ...)
Arguments
x |
A fitted 'ggmlr_parsnip_model' (the engine object inside a parsnip fit; e.g. from 'extract_fit_engine()'). |
... |
Unused; for generic compatibility. |
Value
A one-row tibble with columns: 'mode', 'n_features', 'n_layers', 'total_params', 'optimizer', 'loss', 'backend', 'epochs', 'fit_time' (wall seconds) and 'final_loss' (last training loss, 'NA' if no history).
Examples
ggml_set_n_threads(1L) # deterministic, single OpenMP pool
spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |>
parsnip::set_engine("ggml", backend = "cpu") |>
parsnip::set_mode("regression")
fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars)
generics::glance(parsnip::extract_fit_engine(fit_obj))
Free IQ2 Quantization Tables
Description
Frees lookup tables for IQ2 quantization types.
Usage
iq2xs_free_impl(type)
Arguments
type |
GGML type constant |
Value
NULL invisibly
See Also
Initialize IQ2 Quantization Tables
Description
Initializes lookup tables for IQ2 quantization types. Must be called before using iq2_xxs, iq2_xs, or iq2_s quantization.
Usage
iq2xs_init_impl(type)
Arguments
type |
GGML type constant (e.g., GGML_TYPE_IQ2_XXS()) |
Value
NULL invisibly
See Also
Free IQ3 Quantization Tables
Description
Frees lookup tables for IQ3 quantization types.
Usage
iq3xs_free_impl(grid_size)
Arguments
grid_size |
Grid size for IQ3 |
Value
NULL invisibly
See Also
Initialize IQ3 Quantization Tables
Description
Initializes lookup tables for IQ3 quantization types. Must be called before using iq3_xxs or iq3_s quantization.
Usage
iq3xs_init_impl(grid_size)
Arguments
grid_size |
Grid size for IQ3 (typically 256) |
Value
NULL invisibly
See Also
Cosine-annealing learning rate scheduler
Description
Varies the learning rate following a cosine curve from lr_max down to
lr_min over T_max steps. Restarts (SGDR-style) if
restart = TRUE.
Usage
lr_scheduler_cosine(optimizer, T_max, lr_min = 0, restart = FALSE, T_mult = 1)
Arguments
optimizer |
Optimizer environment. |
T_max |
Number of steps for the first cosine cycle. |
lr_min |
Minimum learning rate (default 0). |
restart |
Logical; if |
T_mult |
Cycle-length multiplier applied after every restart (default
1, i.e. constant period). Only meaningful when |
Details
With restart = TRUE and T_mult > 1 this is the full
CosineAnnealingWarmRestarts (SGDR) schedule: the first cycle lasts
T_max steps and every subsequent cycle is T_mult times longer
than the previous one. T_mult = 1 (the default) keeps every cycle the
same length.
Value
An lr_scheduler_cosine environment
References
Loshchilov, I. & Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR.
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 0.1)
sch <- lr_scheduler_cosine(opt, T_max = 50L)
for (epoch in 1:50) sch$step()
# SGDR with doubling cycle length: 10, 20, 40, ...
opt2 <- optimizer_adam(list(w = w), lr = 0.1)
sch2 <- lr_scheduler_cosine(opt2, T_max = 10L, restart = TRUE, T_mult = 2)
for (epoch in 1:70) sch2$step()
Cyclical learning rate scheduler
Description
Cycles the learning rate linearly between base_lr and max_lr
over a triangular cycle of 2 * step_size_up steps (Smith, 2017).
Each call to $step() advances one step — call it per batch,
not per epoch.
Usage
lr_scheduler_cyclic(
optimizer,
base_lr,
max_lr,
step_size_up = 2000L,
step_size_down = NULL,
mode = c("triangular", "triangular2", "exp_range"),
gamma = 1
)
Arguments
optimizer |
Optimizer environment. |
base_lr |
Lower learning rate bound. |
max_lr |
Upper learning rate bound. |
step_size_up |
Steps in the increasing half of a cycle (default 2000). |
step_size_down |
Steps in the decreasing half. |
mode |
One of |
gamma |
Decay factor for |
Value
An lr_scheduler_cyclic environment with $step() and
$get_lr().
References
Smith, L. N. (2017). Cyclical Learning Rates for Training Neural Networks. WACV.
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_sgd(list(w = w), lr = 0.01)
sch <- lr_scheduler_cyclic(opt, base_lr = 0.001, max_lr = 0.01,
step_size_up = 10L)
for (i in 1:40) sch$step()
One-cycle learning rate scheduler
Description
Implements the 1cycle policy (Smith, 2018): the learning rate rises from
max_lr / div_factor to max_lr during the first
pct_start fraction of training, then anneals down to
max_lr / final_div_factor. Call $step() per batch;
total_steps is the total number of batches over the whole run
(epochs * batches_per_epoch).
Usage
lr_scheduler_onecycle(
optimizer,
max_lr,
total_steps,
pct_start = 0.3,
div_factor = 25,
final_div_factor = 10000,
anneal_strategy = c("cos", "linear"),
cycle_momentum = TRUE,
base_momentum = 0.85,
max_momentum = 0.95
)
Arguments
optimizer |
Optimizer environment. |
max_lr |
Peak learning rate. |
total_steps |
Total number of |
pct_start |
Fraction of |
div_factor |
Initial LR is |
final_div_factor |
Final LR is |
anneal_strategy |
|
cycle_momentum |
Logical; cycle momentum inversely to LR (default
|
base_momentum, max_momentum |
Momentum bounds used when
|
Details
When cycle_momentum = TRUE the momentum-like hyper-parameter moves
inversely to the learning rate: $momentum for SGD, $beta1 for
Adam. For other optimizers momentum cycling is skipped.
Value
An lr_scheduler_onecycle environment with $step() and
$get_lr().
References
Smith, L. N. (2018). A disciplined approach to neural network hyper-parameters. arXiv:1803.09820.
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_sgd(list(w = w), lr = 0.01, momentum = 0.9)
sch <- lr_scheduler_onecycle(opt, max_lr = 0.1, total_steps = 100L)
for (i in 1:100) sch$step()
Step-decay learning rate scheduler
Description
Multiplies the optimizer learning rate by gamma every
step_size calls to $step().
Usage
lr_scheduler_step(optimizer, step_size, gamma = 0.1)
Arguments
optimizer |
An |
step_size |
Decay every this many steps (epochs). |
gamma |
Multiplicative decay factor (default 0.1). |
Value
An lr_scheduler_step environment
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 0.1)
sch <- lr_scheduler_step(opt, step_size = 10L, gamma = 0.5)
for (epoch in 1:30) sch$step()
opt$lr # 0.1 * 0.5^3 = 0.0125
Linear warmup followed by cosine annealing
Description
Raises the learning rate linearly from warmup_start_lr to the
optimizer's initial lr over warmup_steps steps, then anneals it
to lr_min following a cosine curve over the remaining
total_steps - warmup_steps steps. This is the standard transformer
training schedule. Call $step() per batch.
Usage
lr_scheduler_warmup_cosine(
optimizer,
warmup_steps,
total_steps,
lr_min = 0,
warmup_start_lr = 0
)
Arguments
optimizer |
Optimizer environment. Its current |
warmup_steps |
Number of linear warmup steps. |
total_steps |
Total number of |
lr_min |
Final learning rate after annealing (default 0). |
warmup_start_lr |
Learning rate at step 0 (default 0). |
Value
An lr_scheduler_warmup_cosine environment with $step()
and $get_lr().
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 1e-3)
sch <- lr_scheduler_warmup_cosine(opt, warmup_steps = 10L, total_steps = 100L)
for (i in 1:100) sch$step()
Matrix Multiplication Hints
Description
Hint values for ggml_mul_mat_set_hint.
-
GGML_HINT_NONE: no hint (default) -
GGML_HINT_SRC0_IS_HADAMARD: the first operand is the normalized Walsh-Hadamard matrixH_n / \sqrt{n}
Usage
GGML_HINT_NONE
GGML_HINT_SRC0_IS_HADAMARD
Format
An object of class integer of length 1.
An object of class integer of length 1.
Topologically sort nodes reachable from output nodes
Description
Topologically sort nodes reachable from output nodes
Usage
nn_topo_sort(outputs)
Arguments
outputs |
List of output ggml_tensor_node objects |
Value
Named list: nodes in topological order (inputs first, outputs last)
ONNX model device/scheduler diagnostics
Description
Returns information about backend placement: which backends are available, how the scheduler splits the graph, and how many ops are supported by GPU vs CPU-only.
Usage
onnx_device_info(model)
Arguments
model |
An |
Value
A list with:
- backends
Character vector of backend names (e.g.
"Vulkan0","CPU")- n_backends
Number of backends
- n_splits
Number of scheduler splits (1 = all on one backend)
- n_nodes
Total graph nodes
- gpu_ops
Ops supported by GPU backend
- cpu_ops
Ops that can only run on CPU
- cpu_only_ops
Named integer vector: op type => count (empty if all on GPU)
ONNX graph structure
Description
The parsed graph as a table: one row per node, with its operator type and the names of its inputs and outputs. Use it to find which node produces a given edge, or what an edge feeds into, without re-parsing the file.
Usage
onnx_graph_info(model, producer_of = NULL, consumer_of = NULL)
Arguments
model |
An |
producer_of |
Optional edge name. When given, returns the rows whose node lists that name among its outputs. |
consumer_of |
Optional edge name. When given, returns the rows whose node lists that name among its inputs. |
Details
Reading the structure beats searching the .onnx binary for a name:
a name occurs both as a producer's output and as its consumers' input, and
telling those apart by hand means getting the protobuf field numbers right
every time.
Value
A data frame with columns index, op_type,
output (the first output) and list-columns outputs and
inputs.
Examples
if (file.exists("model.onnx")) {
m <- onnx_load("model.onnx")
g <- onnx_graph_info(m)
head(g)
onnx_graph_info(m, producer_of = "881")
}
List ONNX model inputs
Description
Returns the names and shapes of model inputs (excluding weight
initializers). Use this to know what to pass to onnx_run().
Usage
onnx_inputs(model)
Arguments
model |
An |
Value
A named list where names are input tensor names and values are integer vectors of dimension sizes (-1 for dynamic dimensions).
Load an ONNX model
Description
Parses an .onnx file, builds a ggml computation graph, and allocates tensors on the specified device. Weights are loaded via memory-mapped file (zero-copy where possible).
Usage
onnx_load(
path,
device = NULL,
input_shapes = NULL,
n_threads = NULL,
dtype = "f32"
)
Arguments
path |
Path to .onnx file. |
device |
Backend device: |
input_shapes |
Optional named list of integer vectors specifying
fixed shapes for inputs with dynamic dimensions. Names must match
input tensor names. Each shape must include all dimensions including
batch, e.g. |
n_threads |
Number of CPU threads. |
dtype |
Weight precision: |
Value
An opaque model object (external pointer) for use with
onnx_run(), onnx_summary(), and onnx_inputs().
Run ONNX model inference
Description
Run ONNX model inference
Usage
onnx_run(model, inputs)
Arguments
model |
An |
inputs |
A named list of numeric vectors/matrices.
Names must match the model's input tensor names.
Use |
Value
A named list of output tensors: numeric vectors carrying a
dim attribute when the output has more than one dimension.
The buffers are row-major, as ONNX stores them, so dim is given in
storage order – fastest-varying axis first – which is the reverse of the
shape the ONNX file declares. An output declared [nbox, 4] arrives
as dim = c(4, nbox), one detection per column. This keeps dim
an honest description of the memory, so ordinary R indexing is correct;
use predict.onnx_model to get arrays in the declared ONNX
order instead.
ONNX model summary
Description
Returns metadata about a loaded ONNX model.
Usage
onnx_summary(model)
Arguments
model |
An |
Value
A list with ir_version, opset_version,
producer, graph_name, n_nodes,
n_initializers, and ops.
Create an RMSprop optimizer
Description
Same algorithm and defaults as torch.optim.RMSprop. Per step, with
gradient g:
g <- g + weight_decay * w # L2, added to the gradient v <- alpha * v + (1 - alpha) * g^2 ga <- alpha * ga + (1 - alpha) * g # centered only den <- sqrt(v - ga^2) + eps # sqrt(v) + eps if not centered buf <- momentum * buf + g / den; w <- w - lr * buf # momentum > 0 w <- w - lr * g / den # momentum = 0
eps is added OUTSIDE the square root, as in PyTorch (Keras puts it
elsewhere). weight_decay is L2 regularisation, as in PyTorch's
RMSprop – not the decoupled decay of optimizer_adamw.
Usage
optimizer_rmsprop(
params,
lr = 0.01,
alpha = 0.99,
eps = 1e-08,
weight_decay = 0,
momentum = 0,
centered = FALSE,
accumulate_steps = 1L,
average = TRUE
)
Arguments
params |
List of ag_param tensors, named or not (names are used only for display). The same tensor listed twice is kept once, with a warning. |
lr |
Learning rate (default 0.01) |
alpha |
Smoothing constant of the squared-gradient average (default 0.99) |
eps |
Stability constant, outside the square root (default 1e-8) |
weight_decay |
L2 penalty added to the gradient (default 0) |
momentum |
Momentum factor (default 0) |
centered |
Normalise by the variance estimate instead of the raw second
moment (default |
accumulate_steps |
Number of |
average |
When accumulating, divide the accumulated gradient by
|
Details
State slots are created only when used: square_avg always,
momentum_buffer when momentum > 0, grad_avg when
centered. On the GPU the step runs in F32 on resident state, with an
F32 master for weights held in f16, as for optimizer_adam.
ag_capture_step does not support RMSprop.
Value
An optimizer environment with step(grads), zero_grad(),
state() and load_state(state); see optimizer_adam.
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_rmsprop(list(w = w), lr = 0.01, momentum = 0.9)
Autograd Optimizers
Description
Autograd Optimizers
Usage
optimizer_sgd(
params,
lr = 0.01,
momentum = 0,
accumulate_steps = 1L,
average = TRUE
)
optimizer_adam(
params,
lr = 0.001,
beta1 = 0.9,
beta2 = 0.999,
eps = 1e-08,
accumulate_steps = 1L,
average = TRUE
)
optimizer_adamw(
params,
lr = 0.001,
beta1 = 0.9,
beta2 = 0.999,
eps = 1e-08,
weight_decay = 0.01,
accumulate_steps = 1L,
average = TRUE
)
Arguments
params |
List of ag_param tensors, named or not (names are used only for display). The same tensor listed twice is kept once, with a warning. |
lr |
Learning rate (default 1e-3) |
momentum |
Momentum factor (default 0) |
accumulate_steps |
Number of |
average |
When accumulating, divide the accumulated gradient by
|
beta1 |
First moment decay (default 0.9) |
beta2 |
Second moment decay (default 0.999) |
eps |
Stability constant (default 1e-8) |
weight_decay |
Decoupled weight decay factor (default 0.01), |
Details
optimizer_sgd():
Returns: An optimizer environment with step(grads), zero_grad(), state() and load_state(state). state() returns a versioned list (moments as host double matrices, t, accum_count); load_state() checks the type, parameter count, names (when both sides have them) and shapes before writing anything, restores t and accum_count, and does NOT restore hyperparameters (it warns when they differ). Exact continuation holds on the same device and dtype; CPU <-> GPU in f32 continues within float tolerance; across f16 and f32 it cannot be exact. Only read saved states you trust (readRDS).
optimizer_adamw(): Adam with decoupled weight decay (Loshchilov & Hutter), as
torch.optim.AdamW: every step first shrinks the weight,
w <- w * (1 - lr * weight_decay), then applies the Adam update. The
decay is not added to the gradient, so it does not enter the moments (that
would be L2 regularisation). It scales with the current lr, so a
learning-rate schedule changes it too.
Everything else – device step, F32 master weights, accumulation,
state() / load_state(), ag_capture_step – is
as for optimizer_adam; the object inherits from
ag_optimizer_adam.
Returns: An optimizer environment, class c("ag_optimizer_adamw", "ag_optimizer_adam").
Value
An optimizer environment with step(grads), zero_grad(),
state() and load_state(state). state() returns a
versioned list (moments as host double matrices, t,
accum_count); load_state() checks the type, parameter
count, names (when both sides have them) and shapes before writing
anything, restores t and accum_count, and does NOT restore
hyperparameters (it warns when they differ). Exact continuation holds on
the same device and dtype; CPU <-> GPU in f32 continues within float
tolerance; across f16 and f32 it cannot be exact. Only read saved states
you trust (readRDS).
On the GPU the step always runs in F32, whatever ag_dtype
is: the moments are F32, the gradient is converted to F32 before it is
squared, and a weight held in f16 has an F32 master copy that the step
updates and then copies into the f16 weight, so updates smaller than the
f16 spacing still accumulate. state() includes these master
weights, and load_state() installs them as the parameter values;
load the optimizer state after the model's own weights.
No weight decay: for decoupled decay use optimizer_adamw.
Examples
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_sgd(list(w = w), lr = 0.01)
# accumulate 4 micro-batches per update
opt4 <- optimizer_sgd(list(w = w), lr = 0.01, accumulate_steps = 4L)
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 1e-3)
w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adamw(list(w = w), lr = 1e-3, weight_decay = 0.01)
Plot training history
Description
Plots loss and accuracy curves over epochs.
Usage
## S3 method for class 'ggml_history'
plot(x, ...)
Arguments
x |
A ggml_history object |
... |
Additional arguments (ignored) |
Value
The history object (invisibly).
Predict with a Trained Model
Description
Generates predictions from a trained model. Uses the standard R
predict generic for compatibility with keras3 and
the broader R ecosystem.
Usage
## S3 method for class 'ggml_sequential_model'
predict(object, x, batch_size = NULL, ...)
## S3 method for class 'ggml_functional_model'
predict(object, x, batch_size = NULL, ...)
Arguments
object |
A trained model object. |
x |
Input data (matrix, array, or list for multi-input models). |
batch_size |
Batch size for inference. |
... |
Additional arguments (ignored). |
Value
Matrix of predictions.
Predict with an ONNX Model
Description
Runs inference over x, batching it to the fixed input shape the
model was loaded with. This is the keras-compatible entry point: together
with the ggml_sequential_model and ggml_functional_model
methods it gives ggmlR and ONNX models a single predict() interface.
Usage
## S3 method for class 'onnx_model'
predict(object, x, batch_size = NULL, ...)
Arguments
object |
An |
x |
Input data: an array or matrix whose first dimension indexes samples, or a named list of such arrays for multi-input models. |
batch_size |
Ignored, with a warning if not |
... |
Ignored. |
Details
Unlike onnx_run, which takes a named list of inputs and always
returns a named list of outputs, this method takes an array (or a named list
for multi-input models) and unwraps a single output. The output keeps the
dimensions the model produced: a 2D output comes back as a matrix, a 4D
output stays 4D.
An ONNX graph is built for one fixed batch size at onnx_load
time and cannot be resized afterwards, so x is split into chunks of
that size. A trailing partial chunk is padded to a full batch and the
padding is dropped from the result.
Value
For a single-output model, an array with the model's output
dimensions and nrow(x) samples in the first one (a matrix when the
output is 2D). For a multi-output model, a named list of such arrays.
See Also
onnx_run for the lower-level named-list interface.
Print method for ag_tensor
Description
Print method for ag_tensor
Usage
## S3 method for class 'ag_tensor'
print(x, ...)
Arguments
x |
An |
... |
Ignored |
Value
The input x, returned invisibly (called for its side effect of printing).
Print method for ggml_functional_model
Description
Print method for ggml_functional_model
Usage
## S3 method for class 'ggml_functional_model'
print(x, ...)
Arguments
x |
A ggml_functional_model object |
... |
Additional arguments (ignored) |
Value
The model object (invisibly).
Print method for ggml_history
Description
Print method for ggml_history
Usage
## S3 method for class 'ggml_history'
print(x, ...)
Arguments
x |
A ggml_history object |
... |
Additional arguments (ignored) |
Value
The history object (invisibly).
Print method for ggml_sequential_model
Description
Prints a summary of the model architecture including layer types, output shapes, and parameter counts.
Usage
## S3 method for class 'ggml_sequential_model'
print(x, ...)
Arguments
x |
A ggml_sequential_model object |
... |
Additional arguments (ignored) |
Value
The model object (invisibly).
Print ONNX model summary
Description
Print ONNX model summary
Usage
## S3 method for class 'onnx_model'
print(x, ...)
Arguments
x |
An |
... |
Ignored. |
Value
Invisibly returns x.
The piece layout of a 4x4x4 cube
Description
Which stickers make up each piece, for a cube written as 96 stickers in face order. Pieces come in three kinds and the model is told which is which: 8 corners of three stickers, 24 wings of two, 24 centres of one. The short pieces are padded to three slots and the padding masked off.
Usage
pt_cube4_layout()
Details
Sticker numbers are 0-based, as the checkpoint's own layout table has them.
Value
A list with positions (a 56 x 3 integer matrix of
sticker indices), mask (56 x 3 logical, which slots are
real), types (length-56 integer, 0 corner / 1 wing / 2 centre),
state_size and num_classes.
See Also
Examples
lay <- pt_cube4_layout()
dim(lay$positions)
table(lay$types)
Run a piece transformer
Description
Reads a state, or a batch of them, and returns a value per action.
Usage
pt_forward(model, state)
Arguments
model |
A |
state |
Colours in |
Value
For one state, a numeric vector of n_actions values, one per
move. For a batch, a matrix with one row per state and one column per
action.
One state or many
A batch is not a convenience here, it is the way to get the work done quickly. Every layer but the attention treats a row on its own, so a batch of states is the same arithmetic on taller matrices — and a tall matmul is what BLAS is good at, where a short one spends its time in the call rather than the multiply. Scoring the ten states of a beam in one call is several times faster than scoring them one after another.
See Also
Examples
## Not run:
model <- pt_transformer_load("model.pth")
solved <- rep(0:5, each = 16)
pt_forward(model, solved) # 24 values
pt_forward(model, rbind(solved, solved)) # 2 x 24
## End(Not run)
Build a piece transformer
Description
Creates the model with its weights at their initial values. To run a trained
one, use pt_transformer_load, which builds this and then fills
it from a checkpoint.
Usage
pt_transformer(
layout,
d_model = 256L,
n_heads = 8L,
n_layers = 4L,
ff_dim = 1024L,
n_actions = 24L,
activation = "relu"
)
Arguments
layout |
A layout list, as |
d_model |
Width of the model (default 256). |
n_heads |
Attention heads (default 8). |
n_layers |
Encoder blocks (default 4). |
ff_dim |
Width of the feed-forward hidden layer (default 1024). |
n_actions |
Size of the output — one value per action (default 24). |
activation |
Feed-forward activation, |
Value
A pt_transformer object.
See Also
pt_transformer_load, pt_forward
Build a piece transformer and load a checkpoint into it
Description
Reads a .pth saved by torch.save() and returns the model ready
to run. Everything that differs between torch's conventions and this
package's is resolved here rather than at inference time.
Usage
pt_transformer_load(path, layout = pt_cube4_layout(), ...)
Arguments
path |
Path to the |
layout |
A layout list; defaults to |
... |
Passed to |
Value
A pt_transformer with the checkpoint's weights.
See Also
pth_catalogue to see what a checkpoint holds.
Examples
## Not run:
model <- pt_transformer_load("model.pth")
q <- pt_forward(model, rep(0:5, each = 16))
which.max(q)
## End(Not run)
List the tensors in a PyTorch checkpoint
Description
Reads the metadata of a .pth file — the tensor names, shapes and
dtypes — without reading the tensors. A checkpoint is a zip holding one
pickle of metadata and one entry per tensor, so the catalogue is cheap
whatever the file weighs.
Usage
pth_catalogue(path)
Arguments
path |
Path to a |
Details
What comes back is the checkpoint's own state_dict order, which is
the order the model's modules were built in.
Value
A data.frame with one row per tensor and the columns name,
shape (as "a x b"), dtype, n (element count),
bytes, and entry (the zip entry holding the data).
The attribute "file" carries the path.
Examples
## Not run:
cat <- pth_catalogue("model.pth")
head(cat)
sum(cat$n) # total parameters
## End(Not run)
Read the tensors of a PyTorch checkpoint
Description
Reads a .pth file saved by torch.save() into plain R arrays.
No Python and no torch: a checkpoint is a zip of one metadata pickle and one
entry per tensor, and both are read here directly.
Usage
pth_load(path, names = NULL)
Arguments
path |
Path to a |
names |
Optional character vector of tensor names to read. The default
reads all of them. Use |
Value
A named list of numeric arrays, in the checkpoint's own order.
Scalars come back as length-1 vectors; everything else carries a
dim. The attribute "file" carries the path.
Row-major, and what that means for shapes
PyTorch stores a tensor row-major and R stores an array column-major, so the
same bytes describe transposed arrays in the two languages. The tensors come
back with their dimensions reversed — a PyTorch [768, 256]
weight arrives as a 256 x 768 R matrix — which keeps the elements
where they belong without moving a single one. For a linear layer that is
the happy case: x %*% w is then the layer's own arithmetic, no
transpose needed.
See Also
pth_catalogue to list a checkpoint without reading it.
Examples
## Not run:
w <- pth_load("model.pth")
dim(w$output_layer.weight)
# just the one tensor
pth_load("model.pth", names = "output_layer.bias")
## End(Not run)
Quantize data
Description
Quantize float data, row by row, with an optional importance matrix. One function per quantization type:
- Basic
q4_0,q4_1,q5_0,q5_1,q8_0.- K-quants
q2_Kthroughq6_K; better quality/size tradeoffs than basic quants.- Ternary
tq1_0,tq2_0.- IQ
iq2_xxs,iq2_xs,iq2_s,iq3_xxs,iq3_s,iq1_s,iq1_m,iq4_nl,iq4_xs. IQ formats require lookup table initialization before use (seeiq2xs_init_impl,iq3xs_init_impl).- MXFP4
Microscaling FP4.
- Q1_0
1-bit-per-weight sign quantization;
n_per_rowmust be a multiple of 128;imatrixis ignored.- Q2_0
2 bits per weight, blocks of 64 with one FP16 scale
d(the block's largest absolute value). The 2-bit codes decode to-d, 0, d, 2d; sincedis the block maximum, this quantizer only produces-d, 0andd.n_per_rowmust be a multiple of 64;imatrixis ignored.- NVFP4
NVIDIA FP4 with UE4M3 per-sub-block scale;
n_per_rowmust be a multiple of 64;imatrixis ignored.
Usage
quantize_q4_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q4_1(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q5_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q5_1(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q8_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q2_K(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q3_K(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q4_K(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q5_K(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q6_K(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_tq1_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_tq2_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq2_xxs(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq2_xs(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq2_s(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq3_xxs(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq3_s(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq1_s(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq1_m(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq4_nl(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_iq4_xs(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_mxfp4(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q1_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_q2_0(src_data, n_rows, n_per_row, imatrix = NULL)
quantize_nvfp4(src_data, n_rows, n_per_row, imatrix = NULL)
Arguments
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row |
imatrix |
Optional importance matrix (numeric vector or NULL) |
Value
Raw vector of quantized data
See Also
dequantize_row, quantize_row_ref,
ggmlR-quantization
Quantize row (reference implementations)
Description
Basic row-level quantization without importance matrix. These are the
reference implementations: q4_0, q4_1, q5_0,
q5_1, q8_0, q8_1; K-quants q2_K through
q8_K; ternary tq1_0, tq2_0; IQ iq3_xxs,
iq4_nl, iq4_xs, iq3_s, iq2_s; mxfp4,
q1_0, q2_0 (n_elements a multiple of 64) and
nvfp4.
Usage
quantize_row_q4_0_ref(src_data, n_elements)
quantize_row_q4_1_ref(src_data, n_elements)
quantize_row_q5_0_ref(src_data, n_elements)
quantize_row_q5_1_ref(src_data, n_elements)
quantize_row_q8_0_ref(src_data, n_elements)
quantize_row_q8_1_ref(src_data, n_elements)
quantize_row_q2_K_ref(src_data, n_elements)
quantize_row_q3_K_ref(src_data, n_elements)
quantize_row_q4_K_ref(src_data, n_elements)
quantize_row_q5_K_ref(src_data, n_elements)
quantize_row_q6_K_ref(src_data, n_elements)
quantize_row_q8_K_ref(src_data, n_elements)
quantize_row_tq1_0_ref(src_data, n_elements)
quantize_row_tq2_0_ref(src_data, n_elements)
quantize_row_iq3_xxs_ref(src_data, n_elements)
quantize_row_iq4_nl_ref(src_data, n_elements)
quantize_row_iq4_xs_ref(src_data, n_elements)
quantize_row_iq3_s_ref(src_data, n_elements)
quantize_row_iq2_s_ref(src_data, n_elements)
quantize_row_mxfp4_ref(src_data, n_elements)
quantize_row_q1_0_ref(src_data, n_elements)
quantize_row_q2_0_ref(src_data, n_elements)
quantize_row_nvfp4_ref(src_data, n_elements)
Arguments
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Value
Raw vector of quantized data
See Also
quantize_data, ggmlR-quantization
Objects exported from other packages
Description
These objects are imported from other packages. Follow the links below to see their documentation.
RoPE Mode Constants
Description
RoPE (Rotary Position Embedding) Type Constants
Usage
GGML_ROPE_TYPE_NORM
GGML_ROPE_TYPE_NEOX
GGML_ROPE_TYPE_MROPE
GGML_ROPE_TYPE_VISION
Format
Integer constants
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
Details
Constants for RoPE (Rotary Position Embedding) modes used in transformer models. Different models use different RoPE implementations.
-
GGML_ROPE_TYPE_NORM(0): Standard RoPE as in original paper (LLaMA, Mistral) -
GGML_ROPE_TYPE_NEOX(2): GPT-NeoX style RoPE with different interleaving -
GGML_ROPE_TYPE_MROPE(8): Multi-RoPE for multimodal models (Qwen2-VL) -
GGML_ROPE_TYPE_VISION(24): Vision model RoPE variant
Value
An integer constant representing a RoPE type
Examples
GGML_ROPE_TYPE_NORM # 0 - Standard RoPE (LLaMA, Mistral)
GGML_ROPE_TYPE_NEOX # 2 - GPT-NeoX style
GGML_ROPE_TYPE_MROPE # 8 - Multi-RoPE (Qwen2-VL)
GGML_ROPE_TYPE_VISION # 24 - Vision models
Summary method for ggml_sequential_model
Description
Prints a detailed summary including input shape, layer details, trainable/non-trainable parameter counts, and memory estimate.
Usage
## S3 method for class 'ggml_sequential_model'
summary(object, ...)
Arguments
object |
A ggml_sequential_model object |
... |
Additional arguments (ignored) |
Value
The model object (invisibly).
Tidy a fitted ggml parsnip model into a per-layer table
Description
Returns one row per layer of the underlying sequential network, in broom style. Useful for comparing architectures across experiments in a R Markdown / Quarto report.
Usage
## S3 method for class 'ggmlr_parsnip_model'
tidy(x, ...)
Arguments
x |
A fitted 'ggmlr_parsnip_model' (the engine object inside a parsnip fit; e.g. from 'extract_fit_engine()'). |
... |
Unused; for generic compatibility. |
Value
A [tibble][tibble::tibble] with columns: 'layer' (name), 'type', 'units' (output units, 'NA' if not applicable), 'activation', 'output_shape' (character), 'params' (trainable parameter count) and 'trainable' (logical).
Examples
ggml_set_n_threads(1L) # deterministic, single OpenMP pool
spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |>
parsnip::set_engine("ggml", backend = "cpu") |>
parsnip::set_mode("regression")
fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars)
generics::tidy(parsnip::extract_fit_engine(fit_obj))
Run code with gradient tape enabled
Description
Records all ag_* operations inside expr for later backward().
When the default device is "gpu", the ggml context is reset at the
start of each tape.
Usage
with_grad_tape(expr)
Arguments
expr |
Expression to evaluate under gradient tape |
Value
Value of last expression in expr (invisibly)
Inputs computed on the device
The reset frees every intermediate result of earlier ag_* calls; only
ag_param weights (and tensors made from host matrices, which
keep a host copy) survive it. An input built from ag_* operations on
the GPU – an observation normalised with ag_sub/ag_mul, say –
must therefore be built inside expr: built before the tape, it
is destroyed on entry and the forward pass fails with "buffer was freed by a
tape reset".
Examples
w <- ag_param(matrix(c(1, 0, 0, 1), 2, 2))
x <- ag_tensor(matrix(c(1, 2), 2, 1))
y <- ag_tensor(matrix(c(1, 2), 2, 1))
with_grad_tape({
out <- ag_matmul(w, x)
loss <- ag_mse_loss(out, y)
})
backward(loss)