Active Learning with conflibertR

Example data

The package bundles a small demo dataset: a 20-text labeled seed, a 61-text unlabeled pool, and a dev set. It also includes oracle labels for the pool so you can simulate a full loop without a human in the loop (for testing only; don’t use the oracle in real workflows).

demo <- conflibert_example("active")

nrow(demo$seed)      # 20 labeled seed texts
length(demo$pool)    # 61 unlabeled pool texts
nrow(demo$dev)       # 20 dev texts
length(demo$pool_labels)  # oracle labels (for simulation)

Starting a session

conflibert_active_start() trains a classifier on the seed and returns a session object containing the first uncertain batch. Each round, pass the session to conflibert_active_next() along with your labels.

session <- conflibert_active_start(
  seed       = demo$seed,
  pool       = demo$pool,
  dev        = demo$dev,
  model      = "ConfliBERT",
  task       = "binary",
  strategy   = "entropy",   # or "margin", "least_confidence"
  query_size = 10,
  epochs     = 1
)

session

The session’s $query is a tibble of texts to label next, with an uncertainty column showing how unsure the model is. $metrics tracks scores across rounds; $labeled_n / $pool_n track progress.

Labeling and iterating

For real labeling, the easiest route is the built-in Shiny gadget. It opens a modal dialog (or browser tab) showing every row of the current query with radio buttons for each class: click, submit, done.

labels  <- conflibert_active_label(session)
session <- conflibert_active_next(session, labels = labels)

You can also provide the labels by hand (useful for scripting, or if you prefer a console-only workflow):

# labels in the same order as session$query
labels  <- c(1, 0, 1, 0, 0, 1, 0, 1, 0, 1)
session <- conflibert_active_next(session, labels = labels)

For this vignette we use the bundled oracle to simulate labeling:

labels  <- unname(demo$pool_labels[session$query$text])
session <- conflibert_active_next(session, labels = labels)
session

Repeat until the pool is exhausted or the learning curve flattens. Here’s a short simulation loop:

for (round in 2:5) {
  if (session$done) break
  labels  <- unname(demo$pool_labels[session$query$text])
  session <- conflibert_active_next(session, labels = labels)
}

session$metrics

Visualizing progress

plot() produces a two-panel diagnostic: the learning curve on top (metrics vs labeled-set size) and the query uncertainty trend on the bottom. When mean uncertainty flattens, the model is no longer finding informative samples, a good signal to stop.

plot(session)

# or a single panel:
plot(session, which = "metrics")
plot(session, which = "uncertainty")

Query strategies

Three uncertainty strategies are available. Pass one via strategy:

• "entropy" (default): highest Shannon entropy of the predicted class distribution. Works well for both binary and multiclass.
• "margin": smallest gap between the top two class probabilities. Targets ambiguous samples on the decision boundary.
• "least_confidence": lowest maximum class probability. Simplest strategy; a good baseline.

Diversity-aware batches

Pure uncertainty sampling can pick several near-duplicates in one batch, a problem when your pool has many similar texts. Pass diverse = TRUE to cluster the top-scoring candidates in the model’s embedding space and pick the highest-scoring sample from each cluster:

session <- conflibert_active_start(
  seed = demo$seed, pool = demo$pool, dev = demo$dev,
  strategy = "entropy",
  diverse  = TRUE,
  diversity_candidates = 30    # defaults to 3 * query_size
)

LoRA fine-tuning

For bigger base models or tighter GPU budgets, train only a low-rank adapter each round. The adapter is merged into the base model before every round ends, so scoring, saving, and reloading behave exactly like full fine-tuning:

session <- conflibert_active_start(
  seed = demo$seed, pool = demo$pool, dev = demo$dev,
  model      = "DeBERTa v3 Base",
  use_lora   = TRUE,
  lora_rank  = 8,
  lora_alpha = 16
)

Saving the model

Persist the final model as a standard HuggingFace checkpoint:

conflibert_active_save(session, "my_al_model")

You can reload it with any transformers tool, or point AutoModelForSequenceClassification.from_pretrained() at the directory from Python.

Tips

• Start small. A seed of 10–50 texts is often enough; active learning shines when the pool is much larger than the labeled set.
• Watch uncertainty trends, not just metrics. If dev metrics are noisy on small sets, the uncertainty panel often tells a clearer story about whether the model is still learning.
• Parameters carry across rounds. The model, task, strategy, query size, and training hyperparameters are fixed when you call conflibert_active_start() and reused for every subsequent round.
• Sessions are in-memory. The trained model lives inside the session object as a Python handle. Calling saveRDS() on the session won’t serialize the model; use conflibert_active_save() to persist it, and re-run rounds from a fresh session if needed.