Hosted models answer questions; they rarely show how they arrived at
an answer. A model that runs on one’s own computer can show more: the
probability it assigns to every possible next token, how it continues a
reply whose beginning someone else wrote, and how probable it finds a
sentence it did not write. LLMR runs such models through llama.cpp, a widely
used program for running language models on ordinary hardware, and makes
them available through the same llm_config() used for
hosted providers.
The models involved are small, under one gigabyte, and much weaker than current hosted models. They are tools for studying how language models behave, not substitutes for the models a study is about.
One call downloads what is needed: the llama.cpp server program for this computer and the model file.
Before downloading, the function lists each file with its size,
source, and license, and asks for agreement; a script passes
accept_license = TRUE instead. Every file is pinned to an
exact version and checked against its sha256, and a second call makes no
network request. The files live in llm_local_dir();
llm_local_uninstall() removes them, and
llm_local_models() lists the models available by name.
An installed model is a provider like any other. The server starts on the first call and stops when R exits.
cfg <- llm_config("llamacpp", "qwen3-0.6b", temperature = 0, thinking = FALSE)
call_llm(cfg, "What is the capital of France? Answer in one sentence.")thinking = FALSE matters for Qwen3, which otherwise
reasons at length before answering. Structured output, tools, streaming,
log-probabilities, and the tidy verbs work as they do for hosted
models.
llm_next_token() returns the model’s probability
distribution over the next token after a context. The context is either
a chat, rendered with the model’s own template, or raw text given as
prompt.
llm_next_token(cfg, messages = "What is the capital of France? Answer in one sentence.",
prefill = "The capital of France is", top_n = 5)These are the model’s probabilities before any sampling setting acts,
so they do not depend on temperature or top-p. full = TRUE
returns the whole vocabulary, about 150,000 rows for Qwen3.
prefill writes the beginning of the assistant’s reply.
llm_continue() then shows how the model goes on from there;
with n above one it returns a sample of continuations from
the same point.
llm_continue(cfg, messages = "Is coffee good for you?",
prefill = "Honestly, I am not sure, but", n = 3,
temperature = 1, max_tokens = 30, seed = 110)[, c("sample", "continuation")]Unless the configuration or the call sets them, the continuations are
drawn from the model’s full distribution: temperature 1, with none of
the top-k, top-p, or min-p truncation that llama.cpp otherwise applies.
The call above sets temperature = 1 because
cfg fixed it at 0, which gives the single most probable
continuation. A seed makes a sample reproducible on the same llama.cpp
build and device; continuation i uses
seed + i - 1.
Earlier turns can be faked as well, by passing them in
messages with role "assistant".
llm_score() gives the log-probability of each token of a
text after a context, whether or not the model would have produced
it.
s <- llm_score(cfg, " Lyon.", prompt = "The capital of France is")
s[, c("token", "logprob", "rank")]
attr(s, "total_logprob")llm_choice_probs() compares several continuations of the
same context. Longer options have lower total probability by
construction, so options of similar length compare most directly. Each
option is written as the model would produce it: at the start of a chat
reply, without a leading space.
llm_tokenize() and llm_detokenize() use the
model’s tokenizer, and llm_render_template() shows the
exact text the model reads.
Four multilingual embedding models are available by name, from 132 MB
to 635 MB. multilingual-e5-small is the smallest and takes
the prefix "query: " by default
(task = "passage" for documents in retrieval);
granite-embedding-311m and bge-m3 need no
prefix, and bge-m3 has the strongest published results for
Persian and Arabic; embeddinggemma-2 was trained with task
prefixes, which LLMR adds from task = (default
"similarity").
llm_local_install("multilingual-e5-small")
texts <- c("The government raised the national budget for schools.",
"دولت بودجهٔ ملی مدارس را افزایش داد.")
X <- get_batched_embeddings(texts, llm_config("llamacpp", "multilingual-e5-small"))
sum(X[1, ] * X[2, ]) # vectors are unit length, so this is the cosineWith embeddinggemma-2,
task = "search_query" and task = "document"
give the asymmetric prefixes for retrieval, task = "none"
sends the text as written, and dimensions = 256 truncates
to one of the sizes the model was trained for.
Each result of the token-level functions carries an attribute
llmr_local that records the model, the sha256 of its file,
the llama.cpp build, the device, and the exact context.
The device matters. On the same model, log-probabilities computed on a Mac’s GPU and on its CPU differ in the first decimal place (by up to about 0.2 in tests with Qwen3 0.6B), and on the CPU they also depend on settings such as flash attention, which LLMR turns off. Repeated requests on one device and build give identical values, so numbers meant to be compared should come from the same device, and on the CPU from the same type of machine. The device can be chosen when the server starts:
llm_score() processes each prefix afresh, while the
log-probabilities reported during generation come from processing tokens
one at a time; the two differ by floating-point effects (about 1e-3 on
the GPU, up to about 0.1 on the CPU), and cache = TRUE
reproduces the generation values. Embeddings of the same text can differ
by about 1e-3 between calls that batch it with different texts; starting
an embedding server with parallel = 1 and embedding texts
one at a time makes them repeat exactly. Results that carry a claim are
best checked against a full-precision reference implementation of the
same model.
llm_local_model() describes any GGUF file on Hugging
Face; its version and checksum are read from Hugging Face at install
time and recorded.
The official llama.cpp builds run on macOS 13.3 or later, Windows
with the Microsoft Visual C++ runtime, and Linux with glibc 2.34 or
later (Ubuntu 22.04, Debian 12, RHEL 9, or newer). On older Linux
systems, such as many computing clusters, a llama.cpp built on that
system can be used through llm_local_start(server = ).