This report documents the results of a simulation based calibration
(SBC) run for OncoBayes2. TODO
The calibration data presented here has been generated at and with
the OncoBayes git version as:
## Created: 2025-02-10 08:32:59 UTC
## git hash: a55a3b0bf8ca9122cc2cb8985c1b99e5143d38bb
## MD5: fcc463ea71d55ebc403d6c04916ed755The MD5 hash of the calibration data file presented here must match the above listed MD5:
## /home/rdocker/rwork/OncoBayes2/inst/sbc/calibration.rds
## "fcc463ea71d55ebc403d6c04916ed755"Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
The fake data simulation function returns … TODO. Please refer to the
sbc_tools.R and make_reference_rankhist.R R
programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
| data_scenario | N | total_divergent | min_ess_bulk | min_ess_tail | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail |
|---|---|---|---|---|---|---|---|---|
| combo2_EX | 10000 | 0 | 230.412 | 67.083 | 1.027 | 0 | 394.198 | 552.359 |
| combo2_EXNEX | 10000 | 0 | 22.869 | 32.308 | 1.081 | 0 | 396.724 | 519.645 |
| combo3_EXNEX | 10000 | 0 | 29.220 | 49.527 | 1.065 | 0 | 286.585 | 423.331 |
| log2bayes_EX | 10000 | 0 | 895.588 | 598.414 | 1.015 | 0 | 330.011 | 447.230 |
| log2bayes_EXNEX | 10000 | 0 | 412.173 | 268.109 | 1.015 | 0 | 408.568 | 520.341 |
Large Rhat is defined as exceeding \(1.1\).
ESS speed is in units of ESS per second.
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 31.795 | 31 | 0.427 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 39.891 | 31 | 0.132 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 40.320 | 31 | 0.122 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 19.213 | 31 | 0.951 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 27.789 | 31 | 0.632 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 42.022 | 31 | 0.089 |
| beta_group\[D,I(log(drug_A/1)),intercept\] | 23.130 | 31 | 0.844 |
| beta_group\[D,I(log(drug_A/1)),log_slope\] | 37.901 | 31 | 0.184 |
| beta_group\[E,I(log(drug_A/1)),intercept\] | 35.712 | 31 | 0.256 |
| beta_group\[E,I(log(drug_A/1)),log_slope\] | 31.571 | 31 | 0.438 |
| beta_group\[F,I(log(drug_A/1)),intercept\] | 27.648 | 31 | 0.639 |
| beta_group\[F,I(log(drug_A/1)),log_slope\] | 38.304 | 31 | 0.172 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 46.202 | 31 | 0.039 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 32.378 | 31 | 0.399 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_intercept\] | 40.672 | 31 | 0.115 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_log_slope\] | 25.715 | 31 | 0.735 |
| tau_log_beta\[stratum_2,I(log(drug_A/1)),tau_intercept\] | 30.573 | 31 | 0.488 |
| tau_log_beta\[stratum_2,I(log(drug_A/1)),tau_log_slope\] | 25.440 | 31 | 0.748 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 25.363 | 31 | 0.751 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 31.930 | 31 | 0.420 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 25.709 | 31 | 0.735 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 35.507 | 31 | 0.264 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 27.994 | 31 | 0.621 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 32.288 | 31 | 0.403 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 34.547 | 31 | 0.302 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 39.277 | 31 | 0.146 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_intercept\] | 40.973 | 31 | 0.109 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_log_slope\] | 42.042 | 31 | 0.089 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 28.800 | 31 | 0.580 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 38.246 | 31 | 0.174 |
| beta_group\[A,I(log(drug_B/2)),intercept\] | 35.328 | 31 | 0.271 |
| beta_group\[A,I(log(drug_B/2)),log_slope\] | 27.840 | 31 | 0.629 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 37.658 | 31 | 0.191 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 28.160 | 31 | 0.613 |
| beta_group\[B,I(log(drug_B/2)),intercept\] | 48.973 | 31 | 0.021 |
| beta_group\[B,I(log(drug_B/2)),log_slope\] | 40.710 | 31 | 0.114 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 30.842 | 31 | 0.474 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 26.784 | 31 | 0.683 |
| beta_group\[C,I(log(drug_B/2)),intercept\] | 53.485 | 31 | 0.007 |
| beta_group\[C,I(log(drug_B/2)),log_slope\] | 42.822 | 31 | 0.077 |
| beta_group\[D,I(log(drug_A/1)),intercept\] | 19.853 | 31 | 0.939 |
| beta_group\[D,I(log(drug_A/1)),log_slope\] | 31.501 | 31 | 0.441 |
| beta_group\[D,I(log(drug_B/2)),intercept\] | 47.514 | 31 | 0.029 |
| beta_group\[D,I(log(drug_B/2)),log_slope\] | 23.533 | 31 | 0.829 |
| eta_group\[A,I(drug_A/1 \* drug_B/2)\] | 39.552 | 31 | 0.139 |
| eta_group\[B,I(drug_A/1 \* drug_B/2)\] | 34.829 | 31 | 0.291 |
| eta_group\[C,I(drug_A/1 \* drug_B/2)\] | 34.067 | 31 | 0.322 |
| eta_group\[D,I(drug_A/1 \* drug_B/2)\] | 21.882 | 31 | 0.887 |
| mu_eta\[m\[1\]] | 45.798 | 31 | 0.042 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 21.850 | 31 | 0.888 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 19.846 | 31 | 0.939 |
| mu_log_beta\[I(log(drug_B/2)),intercept\] | 47.309 | 31 | 0.031 |
| mu_log_beta\[I(log(drug_B/2)),log_slope\] | 36.461 | 31 | 0.230 |
| tau_eta\[stratum_1,m\[1\]] | 36.704 | 31 | 0.221 |
| tau_eta\[stratum_2,m\[1\]] | 20.608 | 31 | 0.922 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_intercept\] | 27.853 | 31 | 0.629 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_log_slope\] | 42.579 | 31 | 0.081 |
| tau_log_beta\[stratum_1,I(log(drug_B/2)),tau_intercept\] | 23.706 | 31 | 0.822 |
| tau_log_beta\[stratum_1,I(log(drug_B/2)),tau_log_slope\] | 45.184 | 31 | 0.048 |
| tau_log_beta\[stratum_2,I(log(drug_A/1)),tau_intercept\] | 23.475 | 31 | 0.831 |
| tau_log_beta\[stratum_2,I(log(drug_A/1)),tau_log_slope\] | 50.656 | 31 | 0.014 |
| tau_log_beta\[stratum_2,I(log(drug_B/2)),tau_intercept\] | 23.558 | 31 | 0.828 |
| tau_log_beta\[stratum_2,I(log(drug_B/2)),tau_log_slope\] | 41.504 | 31 | 0.098 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 26.003 | 31 | 0.721 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 28.582 | 31 | 0.591 |
| beta_group\[A,I(log(drug_B/2)),intercept\] | 51.424 | 31 | 0.012 |
| beta_group\[A,I(log(drug_B/2)),log_slope\] | 31.994 | 31 | 0.417 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 19.750 | 31 | 0.941 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 24.365 | 31 | 0.795 |
| beta_group\[B,I(log(drug_B/2)),intercept\] | 37.594 | 31 | 0.193 |
| beta_group\[B,I(log(drug_B/2)),log_slope\] | 20.672 | 31 | 0.920 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 29.357 | 31 | 0.551 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 27.558 | 31 | 0.644 |
| beta_group\[C,I(log(drug_B/2)),intercept\] | 39.757 | 31 | 0.135 |
| beta_group\[C,I(log(drug_B/2)),log_slope\] | 30.432 | 31 | 0.495 |
| eta_group\[A,I(drug_A/1 \* drug_B/2)\] | 39.283 | 31 | 0.146 |
| eta_group\[B,I(drug_A/1 \* drug_B/2)\] | 22.848 | 31 | 0.855 |
| eta_group\[C,I(drug_A/1 \* drug_B/2)\] | 27.322 | 31 | 0.656 |
| mu_eta\[m\[1\]] | 30.541 | 31 | 0.490 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 29.440 | 31 | 0.546 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 24.371 | 31 | 0.795 |
| mu_log_beta\[I(log(drug_B/2)),intercept\] | 35.053 | 31 | 0.282 |
| mu_log_beta\[I(log(drug_B/2)),log_slope\] | 18.221 | 31 | 0.967 |
| tau_eta\[stratum_1,m\[1\]] | 24.800 | 31 | 0.777 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_intercept\] | 28.762 | 31 | 0.582 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_log_slope\] | 27.974 | 31 | 0.622 |
| tau_log_beta\[stratum_1,I(log(drug_B/2)),tau_intercept\] | 33.683 | 31 | 0.339 |
| tau_log_beta\[stratum_1,I(log(drug_B/2)),tau_log_slope\] | 25.466 | 31 | 0.747 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 34.387 | 31 | 0.309 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 32.134 | 31 | 0.410 |
| beta_group\[A,I(log(drug_B/2)),intercept\] | 52.883 | 31 | 0.008 |
| beta_group\[A,I(log(drug_B/2)),log_slope\] | 30.624 | 31 | 0.485 |
| beta_group\[A,I(log(drug_C/4)),intercept\] | 33.952 | 31 | 0.327 |
| beta_group\[A,I(log(drug_C/4)),log_slope\] | 31.046 | 31 | 0.464 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 32.666 | 31 | 0.385 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 47.859 | 31 | 0.027 |
| beta_group\[B,I(log(drug_B/2)),intercept\] | 44.954 | 31 | 0.050 |
| beta_group\[B,I(log(drug_B/2)),log_slope\] | 29.888 | 31 | 0.523 |
| beta_group\[B,I(log(drug_C/4)),intercept\] | 23.149 | 31 | 0.844 |
| beta_group\[B,I(log(drug_C/4)),log_slope\] | 26.739 | 31 | 0.685 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 40.333 | 31 | 0.122 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 35.014 | 31 | 0.283 |
| beta_group\[C,I(log(drug_B/2)),intercept\] | 37.024 | 31 | 0.211 |
| beta_group\[C,I(log(drug_B/2)),log_slope\] | 22.118 | 31 | 0.879 |
| beta_group\[C,I(log(drug_C/4)),intercept\] | 41.875 | 31 | 0.092 |
| beta_group\[C,I(log(drug_C/4)),log_slope\] | 26.963 | 31 | 0.674 |
| eta_group\[A,I(drug_A/1 \* drug_B/2 \* drug_C/4)\] | 23.008 | 31 | 0.849 |
| eta_group\[A,I(drug_A/1 \* drug_B/2)\] | 30.227 | 31 | 0.506 |
| eta_group\[A,I(drug_A/1 \* drug_C/4)\] | 18.221 | 31 | 0.967 |
| eta_group\[A,I(drug_B/2 \* drug_C/4)\] | 26.784 | 31 | 0.683 |
| eta_group\[B,I(drug_A/1 \* drug_B/2 \* drug_C/4)\] | 18.810 | 31 | 0.958 |
| eta_group\[B,I(drug_A/1 \* drug_B/2)\] | 26.803 | 31 | 0.682 |
| eta_group\[B,I(drug_A/1 \* drug_C/4)\] | 22.906 | 31 | 0.853 |
| eta_group\[B,I(drug_B/2 \* drug_C/4)\] | 33.210 | 31 | 0.360 |
| eta_group\[C,I(drug_A/1 \* drug_B/2 \* drug_C/4)\] | 20.954 | 31 | 0.913 |
| eta_group\[C,I(drug_A/1 \* drug_B/2)\] | 29.254 | 31 | 0.556 |
| eta_group\[C,I(drug_A/1 \* drug_C/4)\] | 21.530 | 31 | 0.897 |
| eta_group\[C,I(drug_B/2 \* drug_C/4)\] | 32.531 | 31 | 0.391 |
| mu_eta\[m\[1\]] | 32.314 | 31 | 0.402 |
| mu_eta\[m\[2\]] | 21.837 | 31 | 0.888 |
| mu_eta\[m\[3\]] | 19.789 | 31 | 0.940 |
| mu_eta\[m\[4\]] | 40.192 | 31 | 0.125 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 26.739 | 31 | 0.685 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 43.648 | 31 | 0.065 |
| mu_log_beta\[I(log(drug_B/2)),intercept\] | 30.304 | 31 | 0.502 |
| mu_log_beta\[I(log(drug_B/2)),log_slope\] | 33.645 | 31 | 0.341 |
| mu_log_beta\[I(log(drug_C/4)),intercept\] | 35.987 | 31 | 0.246 |
| mu_log_beta\[I(log(drug_C/4)),log_slope\] | 32.486 | 31 | 0.393 |
| tau_eta\[stratum_1,m\[1\]] | 38.368 | 31 | 0.170 |
| tau_eta\[stratum_1,m\[2\]] | 26.125 | 31 | 0.715 |
| tau_eta\[stratum_1,m\[3\]] | 19.008 | 31 | 0.955 |
| tau_eta\[stratum_1,m\[4\]] | 27.840 | 31 | 0.629 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_intercept\] | 27.827 | 31 | 0.630 |
| tau_log_beta\[stratum_1,I(log(drug_A/1)),tau_log_slope\] | 36.806 | 31 | 0.218 |
| tau_log_beta\[stratum_1,I(log(drug_B/2)),tau_intercept\] | 33.606 | 31 | 0.342 |
| tau_log_beta\[stratum_1,I(log(drug_B/2)),tau_log_slope\] | 25.229 | 31 | 0.757 |
| tau_log_beta\[stratum_1,I(log(drug_C/4)),tau_intercept\] | 25.958 | 31 | 0.723 |
| tau_log_beta\[stratum_1,I(log(drug_C/4)),tau_log_slope\] | 46.144 | 31 | 0.039 |
## R version 4.3.3 (2024-02-29)
## Platform: aarch64-unknown-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/aarch64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/aarch64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
##
## locale:
## [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C
## [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8
## [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8
## [7] LC_PAPER=en_US.utf8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C
##
## time zone: Etc/UTC
## tzcode source: system (glibc)
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] ggplot2_3.5.1 broom_1.0.6 tidyr_1.3.1 dplyr_1.1.4
## [5] assertthat_0.2.1 knitr_1.47 here_1.0.1 parallelly_1.37.1
##
## loaded via a namespace (and not attached):
## [1] vctrs_0.6.5 cli_3.6.2 rlang_1.1.4 xfun_0.45
## [5] purrr_1.0.2 generics_0.1.3 glue_1.7.0 colorspace_2.1-0
## [9] backports_1.5.0 rprojroot_2.0.4 htmltools_0.5.8.1 scales_1.3.0
## [13] fansi_1.0.6 rmarkdown_2.27 grid_4.3.3 munsell_0.5.1
## [17] evaluate_0.24.0 tibble_3.2.1 fastmap_1.2.0 yaml_2.3.8
## [21] lifecycle_1.0.4 compiler_4.3.3 pkgconfig_2.0.3 digest_0.6.35
## [25] R6_2.5.1 tidyselect_1.2.1 utf8_1.2.4 pillar_1.9.0
## [29] parallel_4.3.3 magrittr_2.0.3 withr_3.0.0 gtable_0.3.5