The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

pairwiseLLM 1.3.1

New Features

Adaptive linking

• Added cross-set adaptive linking through adaptive_rank(), with one-spoke and multi-spoke hub-and-spoke run modes for placing separately ranked sets on a shared scale.
• Phase A can run canonical within-set ranking, import saved Phase A artifacts, or combine imported and newly run sets. Phase B uses anchored-joint estimation with hard-locked hub parameters, globally shared judge parameters, and concurrent spoke processing.
• Added D-optimal cross-set candidate selection, spoke-aware comparison budgets, and fixed per-refit held-out probes with explicit coverage, calibration, and stopping gates.
• Added persistent Phase A manifests and linking sessions, resume support, link_stage_log, and explicit blocker and invalid-judgment reporting for auditable long-running workflows.

Model support & live API improvements

• Added a separate Vertex AI Gemini live backend via backend = "vertex" with VERTEX_API_KEY.
  • Vertex remains live-only in this series; generic batch wrappers reject backend = "vertex" explicitly rather than routing it through Gemini batch helpers.
• Clarified provider-specific service_tier behavior for Gemini Developer API and Vertex AI Gemini API in the package docs, README, and vignettes.
• Added request-shape handling for the OpenAI GPT-5.x and GPT-5.6 model families.

Bayesian BTL usability

• Added exported build_btl_results_data() to convert conventional ID1/ID2/better_id outcomes into the canonical input required by fit_bayes_btl_mcmc().
• Standalone Bayesian BTL fits now report reliability and sampler diagnostics through the same result and log surfaces used by adaptive workflows.

Improvements

• Reduced adaptive-linking refit, candidate-generation, logging, and checkpoint overhead for large runs, including bounded candidate construction and analytic anchored-joint derivatives.
• Hardened adaptive-linking resume behavior, Phase A stopping scope, spoke routing, probe scheduling, and Phase B pair accounting.

Documentation

• Updated the backend support matrix to distinguish Gemini Developer API from Vertex AI Gemini API and to document the live-only Vertex scope.
• Replaced the undated supported-model list with a dated, machine-readable compatibility registry that separates backend implementation, live/batch testing, and current provider availability.
• Corrected stale provider identifiers, API-key and CmdStan setup guidance, OpenAI Flex wording, cost-estimator summaries, and broken public examples.

Bug fixes

• Fixed Anthropic extended-thinking live requests when users explicitly set the required temperature = 1.
• OpenAI Responses live requests now forward and enforce an explicitly supplied max_output_tokens limit.
• Changed omitted temperature and top_p controls to use model/provider defaults instead of injecting deterministic sampling values. Provider-required reasoning constraints are still enforced.
• Provider smoke runs now exit unsuccessfully when a selected API key is absent, unless missing-key skips were explicitly allowed.

pairwiseLLM 1.3.0

New Features

Adaptive pairing & ranking framework

• Introduced an adaptive pairing / adaptive ranking framework designed to reduce unnecessary comparisons through uncertainty-aware pair selection and Bayesian inference.

• Added adaptive_rank(), the primary user-facing wrapper that runs the complete adaptive workflow end-to-end, including warm start, adaptive pairing rounds, Bayesian BTL refits, diagnostics, and stopping.

• Advanced control is available via:

  • adaptive_rank_start() — initialize an adaptive run and state
  • adaptive_rank_run_live() — execute live adaptive comparisons
  • adaptive_rank_resume() — resume interrupted or long-running runs
    These functions are intended for custom orchestration and fault-tolerant execution.

• Adaptive pairing is organized into rounds that balance global scale identification and local refinement using a mixture of anchor, long-range, mid-range, and local comparisons.

• The adaptive controller tracks a global identifiability state based on Bayesian diagnostics and agreement between online (TrueSkill) and global (BTL) rankings. Once the global scale is identified:

  • long-range comparisons are automatically tapered,
  • comparison budget is reallocated toward local and boundary-refining pairs,
  • exploration rates are reduced to focus on decision-relevant uncertainty.

• Long-range comparisons are additionally posterior-gated in later stages, preventing wasted comparisons on pairs that are already decisively ordered.

• Late-stage local pairing prioritizes near-tie pairs, with limited, auditable overrides to degree caps when especially informative comparisons are blocked.

• Adaptive runs produce fully auditable step-, round-, and refit-level logs, recording candidate generation, fallbacks, gating decisions, quota reallocations, and stopping criteria.

• All adaptive workflows use standardized configuration, state, and logging contracts to ensure reproducibility and future extensibility.

Bayesian Bradley–Terry–Luce (BTL) modeling

• Added a fully Bayesian Bradley–Terry–Luce (BTL) model implemented via CmdStan, providing posterior uncertainty estimates for item skill parameters.
• New entrypoint fit_bayes_btl_mcmc() enables direct posterior inference from pairwise comparison data, independent of or integrated with adaptive workflows.
• Supports multiple model variants (including error and positional bias extensions) and optional refitting on increasing subsets of comparisons.
• Bayesian BTL outputs integrate seamlessly with adaptive ranking utilities (summarize_items(), summarize_refits()), serving as the statistical backbone for adaptive pairing decisions.

Model support & live API improvements

• Added support for the Gemini Flash model gemini-3-flash-preview for live pairwise comparisons.
• Added support for OpenAI service tiers / priority routing via service_tier for applicable live models.
  • Enables tiers such as "flex" and "priority" when supported by the selected model.
  • Integrated into the live submission path without requiring changes to calling code.

Documentation

• Expanded and clarified documentation for adaptive ranking, Bayesian BTL, and live model configuration.
• Updated examples to reflect new adaptive and Bayesian APIs.

pairwiseLLM 1.2.0

New Features

• Parallel Processing:
  • submit_llm_pairs() and backend-specific live functions (OpenAI, Anthropic, Gemini, Together, Ollama) now support parallel execution via parallel = TRUE and workers = n (requires the {future} package).
• Incremental Saving & Resume:
  • Added save_path argument to live submission functions. Results are saved to CSV incrementally, allowing interrupted jobs to resume automatically by skipping previously processed pairs.
• Robust Error Handling:
  • Failed API calls no longer stop the entire process. Failures are captured and returned separately, allowing for easier inspection and re-submission.
• Added estimate_llm_pairs_cost() to estimate costs in live and batch mode.
• Introduced llm_submit_pairs_multi_batch() and llm_resume_multi_batches() to split large comparison sets across multiple batches and resume polling later. These helpers support writing per‑batch and combined results, along with an optional jobs registry.

Bug fixes

• The prompt format for anthropic batch comparisons now match the anthropic live format.
• Reverse consistency functions can now handle duplicate pairs.

Breaking Changes

• submit_llm_pairs() and its backend-specific counterparts now return a list containing two elements: $results (a tibble of successful comparisons) and $failed_pairs (a tibble of inputs that failed). Previous versions returned a single tibble.

pairwiseLLM 1.1.0

Models

• Added GPT-5.2
• Ensured models can be called with date format, e.g. gpt-5.2-2025-12-11
• Default temperature setting is set to 0 for non-reasoning models, provider default for reasoning models (typically 1)

Tests

• Tests added to improve coverage

Documentation

• Changed pkgdown site layout
• Added codemeta.json
• Added repo logo
• Updated function examples
• Add references to Description

Miscellaneous

• No longer set global variables, now done in individual functions
• Added verbose option in fit_bt_model() and summarize_bt_fit()
• Moved null coalescing helper to separate R file
• Changed validation of API keys in multiple functions

pairwiseLLM 1.0.0

• Initial release.
• Unified live and batch LLM comparison framework (OpenAI / Anthropic / Gemini).
• Live support for Together.ai and local Ollama backends.
• Tools for Bradley–Terry and Elo models, positional bias checks

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.