This report documents the results of a simulation based calibration (SBC) run for OncoBayes2
. TODO
The calibration data presented here has been generated at and with the OncoBayes
git version as:
## Created: 2021-08-16 09:39:32 UTC
## git hash: 9b7dc4d107dea0844862f274850b462d947e7726
## MD5: 41d58fb91ec1652d3a1b33bd9cb11b76
The MD5 hash of the calibration data file presented here must match the above listed MD5:
## calibration.rds
## "41d58fb91ec1652d3a1b33bd9cb11b76"
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
The fake data simulation function returns … TODO. Please refer to the sbc_tools.R
and make_reference_rankhist.R
R programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
model | problem | N | total_divergent | min_ess | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail |
---|---|---|---|---|---|---|---|---|
combo2_EX | base | 9950 | 0 | 337 | 1.011 | 0 | 160 | 214 |
combo2_EX | warmup_base | 50 | 0 | 775 | 1.003 | 0 | 222 | 388 |
combo2_EXNEX | base | 9950 | 0 | 37 | 1.071 | 0 | 64 | 177 |
combo2_EXNEX | warmup_base | 50 | 0 | 209 | 1.014 | 0 | 252 | 390 |
combo3_EXNEX | base | 9950 | 0 | 51 | 1.061 | 0 | 118 | 229 |
combo3_EXNEX | warmup_base | 50 | 0 | 355 | 1.005 | 0 | 282 | 247 |
log2bayes_EXNEX | base | 9950 | 0 | 44 | 1.022 | 0 | 23 | 128 |
log2bayes_EXNEX | warmup_base | 50 | 0 | 1084 | 1.008 | 0 | 320 | 328 |
Large Rhat is defined as exceeding \(1.2\).
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 35.917 | 31 | 0.249 |
beta_group[A,I(log(drug_A/1)),log_slope] | 34.048 | 31 | 0.323 |
beta_group[B,I(log(drug_A/1)),intercept] | 43.955 | 31 | 0.062 |
beta_group[B,I(log(drug_A/1)),log_slope] | 26.541 | 31 | 0.695 |
beta_group[C,I(log(drug_A/1)),intercept] | 25.722 | 31 | 0.735 |
beta_group[C,I(log(drug_A/1)),log_slope] | 26.214 | 31 | 0.711 |
mu_log_beta[I(log(drug_A/1)),intercept] | 33.965 | 31 | 0.327 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 31.654 | 31 | 0.434 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 26.483 | 31 | 0.698 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 47.706 | 31 | 0.028 |
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 31.219 | 31 | 0.455 |
beta_group[A,I(log(drug_A/1)),log_slope] | 37.530 | 31 | 0.195 |
beta_group[A,I(log(drug_B/1)),intercept] | 27.667 | 31 | 0.638 |
beta_group[A,I(log(drug_B/1)),log_slope] | 35.827 | 31 | 0.252 |
beta_group[B,I(log(drug_A/1)),intercept] | 23.910 | 31 | 0.814 |
beta_group[B,I(log(drug_A/1)),log_slope] | 18.266 | 31 | 0.966 |
beta_group[B,I(log(drug_B/1)),intercept] | 37.408 | 31 | 0.198 |
beta_group[B,I(log(drug_B/1)),log_slope] | 35.411 | 31 | 0.268 |
beta_group[C,I(log(drug_A/1)),intercept] | 33.338 | 31 | 0.354 |
beta_group[C,I(log(drug_A/1)),log_slope] | 32.525 | 31 | 0.392 |
beta_group[C,I(log(drug_B/1)),intercept] | 29.203 | 31 | 0.559 |
beta_group[C,I(log(drug_B/1)),log_slope] | 45.958 | 31 | 0.041 |
eta_group[A,I(drug_A/1 * drug_B/1)] | 35.206 | 31 | 0.276 |
eta_group[B,I(drug_A/1 * drug_B/1)] | 44.800 | 31 | 0.052 |
eta_group[C,I(drug_A/1 * drug_B/1)] | 41.005 | 31 | 0.108 |
mu_eta[I(drug_A/1 * drug_B/1)] | 30.336 | 31 | 0.500 |
mu_log_beta[I(log(drug_A/1)),intercept] | 35.123 | 31 | 0.279 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 30.342 | 31 | 0.500 |
mu_log_beta[I(log(drug_B/1)),intercept] | 34.835 | 31 | 0.290 |
mu_log_beta[I(log(drug_B/1)),log_slope] | 35.629 | 31 | 0.260 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] | 32.019 | 31 | 0.416 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 38.432 | 31 | 0.168 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 30.944 | 31 | 0.469 |
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] | 32.397 | 31 | 0.398 |
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] | 31.782 | 31 | 0.427 |
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 31.661 | 31 | 0.433 |
beta_group[A,I(log(drug_A/1)),log_slope] | 24.518 | 31 | 0.789 |
beta_group[A,I(log(drug_B/1)),intercept] | 27.770 | 31 | 0.633 |
beta_group[A,I(log(drug_B/1)),log_slope] | 41.786 | 31 | 0.093 |
beta_group[B,I(log(drug_A/1)),intercept] | 42.701 | 31 | 0.079 |
beta_group[B,I(log(drug_A/1)),log_slope] | 16.621 | 31 | 0.984 |
beta_group[B,I(log(drug_B/1)),intercept] | 20.742 | 31 | 0.919 |
beta_group[B,I(log(drug_B/1)),log_slope] | 42.496 | 31 | 0.082 |
beta_group[C,I(log(drug_A/1)),intercept] | 47.667 | 31 | 0.028 |
beta_group[C,I(log(drug_A/1)),log_slope] | 34.426 | 31 | 0.307 |
beta_group[C,I(log(drug_B/1)),intercept] | 29.715 | 31 | 0.532 |
beta_group[C,I(log(drug_B/1)),log_slope] | 17.517 | 31 | 0.975 |
eta_group[A,I(drug_A/1 * drug_B/1)] | 27.258 | 31 | 0.659 |
eta_group[B,I(drug_A/1 * drug_B/1)] | 29.830 | 31 | 0.526 |
eta_group[C,I(drug_A/1 * drug_B/1)] | 25.357 | 31 | 0.752 |
mu_eta[I(drug_A/1 * drug_B/1)] | 24.435 | 31 | 0.792 |
mu_log_beta[I(log(drug_A/1)),intercept] | 32.787 | 31 | 0.379 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 22.016 | 31 | 0.883 |
mu_log_beta[I(log(drug_B/1)),intercept] | 25.120 | 31 | 0.762 |
mu_log_beta[I(log(drug_B/1)),log_slope] | 34.682 | 31 | 0.297 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] | 35.597 | 31 | 0.261 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 26.886 | 31 | 0.678 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 23.398 | 31 | 0.834 |
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] | 36.723 | 31 | 0.221 |
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] | 14.682 | 31 | 0.994 |
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 30.918 | 31 | 0.470 |
beta_group[A,I(log(drug_A/1)),log_slope] | 27.078 | 31 | 0.668 |
beta_group[A,I(log(drug_B/1)),intercept] | 33.702 | 31 | 0.338 |
beta_group[A,I(log(drug_B/1)),log_slope] | 40.307 | 31 | 0.122 |
beta_group[A,I(log(drug_C/1)),intercept] | 33.965 | 31 | 0.327 |
beta_group[A,I(log(drug_C/1)),log_slope] | 33.792 | 31 | 0.334 |
beta_group[B,I(log(drug_A/1)),intercept] | 43.898 | 31 | 0.062 |
beta_group[B,I(log(drug_A/1)),log_slope] | 33.587 | 31 | 0.343 |
beta_group[B,I(log(drug_B/1)),intercept] | 29.613 | 31 | 0.537 |
beta_group[B,I(log(drug_B/1)),log_slope] | 21.894 | 31 | 0.886 |
beta_group[B,I(log(drug_C/1)),intercept] | 37.946 | 31 | 0.182 |
beta_group[B,I(log(drug_C/1)),log_slope] | 31.430 | 31 | 0.445 |
beta_group[C,I(log(drug_A/1)),intercept] | 26.842 | 31 | 0.680 |
beta_group[C,I(log(drug_A/1)),log_slope] | 36.326 | 31 | 0.234 |
beta_group[C,I(log(drug_B/1)),intercept] | 41.862 | 31 | 0.092 |
beta_group[C,I(log(drug_B/1)),log_slope] | 41.677 | 31 | 0.095 |
beta_group[C,I(log(drug_C/1)),intercept] | 32.077 | 31 | 0.413 |
beta_group[C,I(log(drug_C/1)),log_slope] | 38.048 | 31 | 0.179 |
eta_group[A,I(drug_A/1 * drug_B/1 * drug_C/1)] | 19.930 | 31 | 0.937 |
eta_group[A,I(drug_A/1 * drug_B/1)] | 22.221 | 31 | 0.876 |
eta_group[A,I(drug_A/1 * drug_C/1)] | 34.272 | 31 | 0.314 |
eta_group[A,I(drug_B/1 * drug_C/1)] | 45.075 | 31 | 0.049 |
eta_group[B,I(drug_A/1 * drug_B/1 * drug_C/1)] | 35.808 | 31 | 0.253 |
eta_group[B,I(drug_A/1 * drug_B/1)] | 41.331 | 31 | 0.102 |
eta_group[B,I(drug_A/1 * drug_C/1)] | 44.902 | 31 | 0.051 |
eta_group[B,I(drug_B/1 * drug_C/1)] | 25.338 | 31 | 0.752 |
eta_group[C,I(drug_A/1 * drug_B/1 * drug_C/1)] | 29.728 | 31 | 0.531 |
eta_group[C,I(drug_A/1 * drug_B/1)] | 33.568 | 31 | 0.344 |
eta_group[C,I(drug_A/1 * drug_C/1)] | 21.018 | 31 | 0.911 |
eta_group[C,I(drug_B/1 * drug_C/1)] | 21.530 | 31 | 0.897 |
mu_eta[I(drug_A/1 * drug_B/1 * drug_C/1)] | 30.349 | 31 | 0.499 |
mu_eta[I(drug_A/1 * drug_B/1)] | 30.522 | 31 | 0.490 |
mu_eta[I(drug_A/1 * drug_C/1)] | 42.867 | 31 | 0.076 |
mu_eta[I(drug_B/1 * drug_C/1)] | 22.637 | 31 | 0.862 |
mu_log_beta[I(log(drug_A/1)),intercept] | 32.378 | 31 | 0.399 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 43.642 | 31 | 0.066 |
mu_log_beta[I(log(drug_B/1)),intercept] | 38.118 | 31 | 0.177 |
mu_log_beta[I(log(drug_B/1)),log_slope] | 31.674 | 31 | 0.433 |
mu_log_beta[I(log(drug_C/1)),intercept] | 17.843 | 31 | 0.972 |
mu_log_beta[I(log(drug_C/1)),log_slope] | 35.930 | 31 | 0.248 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1 * drug_C/1)] | 23.264 | 31 | 0.839 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] | 24.346 | 31 | 0.796 |
tau_eta[STRAT,I(drug_A/1 * drug_C/1)] | 27.584 | 31 | 0.643 |
tau_eta[STRAT,I(drug_B/1 * drug_C/1)] | 24.525 | 31 | 0.789 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 20.576 | 31 | 0.923 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 19.795 | 31 | 0.940 |
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] | 28.608 | 31 | 0.590 |
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] | 28.800 | 31 | 0.580 |
tau_log_beta[STRAT,I(log(drug_C/1)),intercept] | 14.541 | 31 | 0.995 |
tau_log_beta[STRAT,I(log(drug_C/1)),log_slope] | 27.795 | 31 | 0.632 |
## R version 3.6.3 (2020-02-29)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] mvtnorm_1.1-0 tibble_3.0.0 rstan_2.19.3 StanHeaders_2.19.2
## [5] abind_1.4-5 Formula_1.2-3 checkmate_2.0.0 OncoBayes2_0.8-1
## [9] testthat_3.0.3 devtools_2.3.0 usethis_1.6.0 ggplot2_3.3.0
## [13] broom_0.5.5 tidyr_1.0.2 dplyr_0.8.5 assertthat_0.2.1
## [17] knitr_1.28 rmarkdown_2.1
##
## loaded via a namespace (and not attached):
## [1] Rcpp_1.0.4.6 lattice_0.20-44 prettyunits_1.1.1 ps_1.6.0
## [5] rprojroot_1.3-2 digest_0.6.25 plyr_1.8.6 R6_2.4.1
## [9] ggridges_0.5.2 backports_1.1.6 stats4_3.6.3 evaluate_0.14
## [13] highr_0.8 pillar_1.4.3 rlang_0.4.11 rstudioapi_0.11
## [17] callr_3.7.0 desc_1.2.0 stringr_1.4.0 loo_2.2.0
## [21] munsell_0.5.0 compiler_3.6.3 xfun_0.13 pkgconfig_2.0.3
## [25] pkgbuild_1.0.6 rstantools_2.1.1 htmltools_0.4.0 tidyselect_1.0.0
## [29] gridExtra_2.3 codetools_0.2-18 matrixStats_0.56.0 crayon_1.3.4
## [33] withr_2.4.2 grid_3.6.3 nlme_3.1-152 gtable_0.3.0
## [37] lifecycle_0.2.0 magrittr_1.5 scales_1.1.0 RcppParallel_5.1.4
## [41] cli_2.5.0 stringi_1.4.6 fs_1.4.1 remotes_2.1.1
## [45] ellipsis_0.3.0 generics_0.0.2 vctrs_0.2.4 glue_1.4.0
## [49] purrr_0.3.4 processx_3.5.2 pkgload_1.0.2 parallel_3.6.3
## [53] yaml_2.2.1 inline_0.3.15 colorspace_1.4-1 sessioninfo_1.1.1
## [57] bayesplot_1.7.1 memoise_1.1.0