The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Research-scale validation execution

Purpose

eyeprocess separates an executable model from evidence that the model is scientifically dependable. The validation execution engine converts a declared Monte Carlo design into deterministic jobs, atomic checkpoints, resumable runs, auditable failures, recovery summaries, calibration diagnostics, and promotion decisions.

Deterministic plans

library(eyeprocess)

plan <- validation_job_plan(
  grid = list(
    n_person = c(50L, 150L, 500L),
    n_item = c(10L, 30L),
    process_effect = c(0, 0.25, 0.50),
    feature_reliability = c(0.50, 0.80),
    missingness = c(0, 0.15)
  ),
  replications = 500L,
  base_seed = 20260805L,
  model_family = "dynamic_irtree",
  chunk_size = 25L
)

write_validation_job_manifest(plan, "validation/dynamic-irtree")

A job seed is determined by the complete design cell, replication, and base seed. Reordering a plan therefore does not alter the simulated study.

Atomic execution and resumption

run_validation_jobs(
  plan,
  simulator = simulate_one_study,
  fitter = fit_one_model,
  extractor = extract_estimates,
  truth_extractor = extract_truth,
  diagnostics_extractor = extract_diagnostics,
  draws_extractor = extract_draws,
  output_dir = "validation/dynamic-irtree",
  workers = 8L,
  backend = "future",
  isolation = "callr",
  timeout_seconds = 3600,
  memory_limit_mb = 8192
)

resume_validation_jobs(
  plan,
  "validation/dynamic-irtree",
  retry = c("missing", "failed", "nonconverged")
)

Every checkpoint preserves the job specification, seed, warnings, messages, errors, runtime, estimates, diagnostics, optional posterior draws, predictions, and session metadata. Failed jobs are evidence and are never silently removed.

Collection and evidence

result <- collect_validation_jobs("validation/dynamic-irtree", plan)
validation_recovery_summary(result)
validation_failure_summary(result)
validation_runtime_summary(result)
validation_calibration_summary(result)
validation_sbc_summary(result)

audit <- audit_validation_completion(result)
plot_parameter_recovery(result)
plot_interval_coverage(result)
plot_sbc_rank(result)
plot_validation_failures(result)
plot_validation_runtime(result)
write_validation_release_report(result, "validation-report.md")

Promotion remains gated

evidence <- list(
  dynamic_irtree = list(
    completion = audit,
    sbc = sbc_audit,
    misspecification = misspecification_audit,
    grouped_validation = grouped_result,
    engine_equivalence = equivalence_result,
    empirical_reproduction = reproduction_result,
    preprocessing_sensitivity = aoi_sensitivity
  )
)

audit_model_promotion(evidence)

The audit reports experimental whenever any required gate is absent or fails. Code execution alone is not a promotion criterion.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.