The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Data Schemas and Prompt Management

This article follows data from source samples to modeling inputs and shows how prompt templates are managed. All examples are deterministic and run without provider credentials.

Samples and pairs

read_samples_df() normalizes a data frame and read_samples_dir() reads one text file per item. Both return an ID/text table suitable for the fixed-pair workflow. Existing data frames can be used directly when they already have those columns.

library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")
samples <- read_samples_df(example_writing_samples[, c("ID", "text")])

pairs <- samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 4L, seed = 10L) |>
  randomize_pair_order(seed = 11L)

pairs[, c("ID1", "text1", "ID2", "text2")]
#> # A tibble: 4 × 4
#>   ID1   text1                                                        ID2   text2
#>   <chr> <chr>                                                        <chr> <chr>
#> 1 S12   "Evaluating writing is challenging because no rubric can fu… S10   "The…
#> 2 S09   "Assessing writing is difficult because the construct is mu… S05   "Wri…
#> 3 S07   "Writing assessment is difficult because writing is a compl… S20   "Wri…
#> 4 S19   "Writing assessment is difficult because it asks us to quan… S16   "Ass…

The fixed-pair schema uses ID1, text1, ID2, and text2. Reversal helpers preserve the unordered pair while changing presentation order. Keep the identifiers: they are needed to map a provider’s SAMPLE_1 or SAMPLE_2 decision back to better_id.

Traits and prompt templates

Templates require all four placeholders: {TRAIT_NAME}, {TRAIT_DESCRIPTION}, {SAMPLE_1}, and {SAMPLE_2}. build_prompt() substitutes them but does not submit anything to a provider.

trait <- trait_description("overall_quality")
template <- set_prompt_template()

prompt <- build_prompt(
  template = template,
  trait_name = trait$name,
  trait_desc = trait$description,
  text1 = pairs$text1[[1]],
  text2 = pairs$text2[[1]]
)

substr(prompt, 1L, 180L)
#> [1] "You are a debate adjudicator. Your task is to weigh the comparative strengths of two writing samples regarding a specific trait.\n\nTRAIT: Overall Quality\nDEFINITION: Overall quality"

The named registry lasts only for the current R session. Registration validates placeholders; overwrite = FALSE protects an existing name. Persist a custom template by storing its text in your project and registering it from a startup script, not by modifying package files.

example_name <- "task07_example"
register_prompt_template(example_name, template = template)
example_name %in% list_prompt_templates()
#> [1] TRUE
identical(get_prompt_template(example_name), template)
#> [1] TRUE

register_prompt_template(example_name, template = template, overwrite = TRUE)
remove_prompt_template(example_name)

set_prompt_template() returns the built-in default or validates an inline/file template. get_prompt_template() resolves a user registration before a built-in template of the same name. remove_prompt_template() removes only session registrations; it cannot delete built-in files.

Provider and result schemas

The live wrappers accept the fixed-pair rows directly. Batch request builders convert the same rows to provider request records, and their matching parsers convert downloaded output back to package results. Do not send a request table built for one provider to another provider’s submission function.

The normalized result bundle retains pair identifiers and adds provider metadata, visible content, optional thoughts, better_sample, better_id, and token counts. Row-wise live calls return a list whose valid results, unresolved failed_pairs, and attempt-level failed_attempts components must be inspected separately. Batch parsers may omit unsuccessful rows from their successful result table, so preserve the provider output/error file and job registry for audit and recovery.

samples (ID, text)
  -> pairs (ID1, text1, ID2, text2)
     -> live call ---------------------> normalized results
     -> provider request + batch output -> provider parser -> normalized results

Modeling and adaptive branches

For frequentist Bradley–Terry or Elo models, start from rows with ID1, ID2, and a valid better_id and call build_bt_data() or build_elo_data(). Invalid or missing winners are not valid outcomes; inspect failures before modeling rather than silently treating them as ties.

data("example_writing_pairs", package = "pairwiseLLM")

bt_data <- build_bt_data(example_writing_pairs)
elo_data <- build_elo_data(example_writing_pairs)
bayes_data <- build_btl_results_data(example_writing_pairs)

names(bt_data)
#> [1] "object1" "object2" "result"
names(elo_data)
#> [1] "winner" "loser"
names(bayes_data)
#>  [1] "pair_uid"      "unordered_key" "ordered_key"   "A_id"         
#>  [5] "B_id"          "better_id"     "winner_pos"    "phase"        
#>  [9] "iter"          "received_at"   "backend"       "model"

Standalone Bayesian BTL uses the stricter canonical schema returned by build_btl_results_data(); see Standalone Bayesian BTL with CmdStan.

Adaptive ranking is a separate branch. adaptive_rank() starts from raw items and records each attempt in an adaptive state with canonical step, round, link-stage, and item logs. A normalized fixed-pair result table is not a drop-in replacement for an adaptive session. Use adaptive_results_history() only when you need the committed adaptive outcomes in build_bt_data() format.

Failure checks at schema boundaries

Citation

Mercer, S. H. (2026). Data schemas and prompt management [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.