The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

aiEvalR 0.1.0

First release.

aiEvalR provides a statistical and psychometric framework for evaluating large language models and other AI systems, organized around seven modules.

Modules

• Psychometric quality (methodological core): multi-facet generalizability theory for AI evaluation. ai_generalizability() fits a crossed G-study (case x prompt x model x run) via lme4; ai_dstudy() projects the generalizability coefficient and dependability index (Phi) for any measurement design without refitting. Also ai_conditional_sem() and ai_decision_consistency().
• Reliability: ai_test_retest() (ICC), ai_internal_consistency() (Cronbach’s alpha), ai_bootstrap_reliability(), and the ai_reliability() wrapper.
• Robustness: stress_test(), prompt_sensitivity(), adversarial_stability(), and the ai_robustness() wrapper.
• Fairness: ai_group_disparity() (demographic parity, equalized odds), ai_dif_mh() (Mantel-Haenszel screening), and ai_fairness(), which integrates with the companion aiDIF package for item-level IRT-based AI-scoring bias when item MLEs are supplied.
• Calibration: ece(), brier_score(), calibration_curve(), empirical_output_interval(), and the ai_calibration() wrapper, with input validation throughout.
• Hallucination: hallucination_rate() (aggregation of adjudicated verdicts), lexical_overlap() (a lexical baseline, deliberately named to avoid implying factual-consistency detection), and citation_accuracy().
• Dashboard: ai_dashboard() combines rescaled module scores into a descriptive overall profile.

Testing

Comprehensive test suite (~93% line coverage), verifying statistical behavior against known-value cases rather than merely checking for non-null output.

Vignettes

• getting-started: a module-by-module tour with runnable examples.
• psychometric-core: a deeper treatment of the generalizability-theory workflow (G-studies and D-studies).

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.