Simulation based calibration for OncoBayes2

Wed Sep 1 11:43:01 2021

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2021-08-16 09:39:32 UTC
## git hash: 9b7dc4d107dea0844862f274850b462d947e7726
## MD5:      41d58fb91ec1652d3a1b33bd9cb11b76

The MD5 hash of the calibration data file presented here must match the above listed MD5:

##                    calibration.rds 
## "41d58fb91ec1652d3a1b33bd9cb11b76"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

model problem N total_divergent min_ess max_Rhat total_large_Rhat min_lp_ess_bulk min_lp_ess_tail
combo2_EX base 9950 0 337 1.011 0 160 214
combo2_EX warmup_base 50 0 775 1.003 0 222 388
combo2_EXNEX base 9950 0 37 1.071 0 64 177
combo2_EXNEX warmup_base 50 0 209 1.014 0 252 390
combo3_EXNEX base 9950 0 51 1.061 0 118 229
combo3_EXNEX warmup_base 50 0 355 1.005 0 282 247
log2bayes_EXNEX base 9950 0 44 1.022 0 23 128
log2bayes_EXNEX warmup_base 50 0 1084 1.008 0 320 328

Large Rhat is defined as exceeding \(1.2\).

\(\chi^2\) Statistic, Model 1: Single-agent logistic regression

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 35.917 31 0.249
beta_group[A,I(log(drug_A/1)),log_slope] 34.048 31 0.323
beta_group[B,I(log(drug_A/1)),intercept] 43.955 31 0.062
beta_group[B,I(log(drug_A/1)),log_slope] 26.541 31 0.695
beta_group[C,I(log(drug_A/1)),intercept] 25.722 31 0.735
beta_group[C,I(log(drug_A/1)),log_slope] 26.214 31 0.711
mu_log_beta[I(log(drug_A/1)),intercept] 33.965 31 0.327
mu_log_beta[I(log(drug_A/1)),log_slope] 31.654 31 0.434
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 26.483 31 0.698
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 47.706 31 0.028

\(\chi^2\) Statistic, Model 2: Double combination, fully exchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 31.219 31 0.455
beta_group[A,I(log(drug_A/1)),log_slope] 37.530 31 0.195
beta_group[A,I(log(drug_B/1)),intercept] 27.667 31 0.638
beta_group[A,I(log(drug_B/1)),log_slope] 35.827 31 0.252
beta_group[B,I(log(drug_A/1)),intercept] 23.910 31 0.814
beta_group[B,I(log(drug_A/1)),log_slope] 18.266 31 0.966
beta_group[B,I(log(drug_B/1)),intercept] 37.408 31 0.198
beta_group[B,I(log(drug_B/1)),log_slope] 35.411 31 0.268
beta_group[C,I(log(drug_A/1)),intercept] 33.338 31 0.354
beta_group[C,I(log(drug_A/1)),log_slope] 32.525 31 0.392
beta_group[C,I(log(drug_B/1)),intercept] 29.203 31 0.559
beta_group[C,I(log(drug_B/1)),log_slope] 45.958 31 0.041
eta_group[A,I(drug_A/1 * drug_B/1)] 35.206 31 0.276
eta_group[B,I(drug_A/1 * drug_B/1)] 44.800 31 0.052
eta_group[C,I(drug_A/1 * drug_B/1)] 41.005 31 0.108
mu_eta[I(drug_A/1 * drug_B/1)] 30.336 31 0.500
mu_log_beta[I(log(drug_A/1)),intercept] 35.123 31 0.279
mu_log_beta[I(log(drug_A/1)),log_slope] 30.342 31 0.500
mu_log_beta[I(log(drug_B/1)),intercept] 34.835 31 0.290
mu_log_beta[I(log(drug_B/1)),log_slope] 35.629 31 0.260
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 32.019 31 0.416
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 38.432 31 0.168
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 30.944 31 0.469
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 32.397 31 0.398
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 31.782 31 0.427

\(\chi^2\) Statistic, Model 3: Double combination, EXchangeable/NonEXchangeable model

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 31.661 31 0.433
beta_group[A,I(log(drug_A/1)),log_slope] 24.518 31 0.789
beta_group[A,I(log(drug_B/1)),intercept] 27.770 31 0.633
beta_group[A,I(log(drug_B/1)),log_slope] 41.786 31 0.093
beta_group[B,I(log(drug_A/1)),intercept] 42.701 31 0.079
beta_group[B,I(log(drug_A/1)),log_slope] 16.621 31 0.984
beta_group[B,I(log(drug_B/1)),intercept] 20.742 31 0.919
beta_group[B,I(log(drug_B/1)),log_slope] 42.496 31 0.082
beta_group[C,I(log(drug_A/1)),intercept] 47.667 31 0.028
beta_group[C,I(log(drug_A/1)),log_slope] 34.426 31 0.307
beta_group[C,I(log(drug_B/1)),intercept] 29.715 31 0.532
beta_group[C,I(log(drug_B/1)),log_slope] 17.517 31 0.975
eta_group[A,I(drug_A/1 * drug_B/1)] 27.258 31 0.659
eta_group[B,I(drug_A/1 * drug_B/1)] 29.830 31 0.526
eta_group[C,I(drug_A/1 * drug_B/1)] 25.357 31 0.752
mu_eta[I(drug_A/1 * drug_B/1)] 24.435 31 0.792
mu_log_beta[I(log(drug_A/1)),intercept] 32.787 31 0.379
mu_log_beta[I(log(drug_A/1)),log_slope] 22.016 31 0.883
mu_log_beta[I(log(drug_B/1)),intercept] 25.120 31 0.762
mu_log_beta[I(log(drug_B/1)),log_slope] 34.682 31 0.297
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 35.597 31 0.261
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 26.886 31 0.678
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 23.398 31 0.834
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 36.723 31 0.221
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 14.682 31 0.994

\(\chi^2\) Statistic, Model 4: Triple combination, EX/NEX model

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 30.918 31 0.470
beta_group[A,I(log(drug_A/1)),log_slope] 27.078 31 0.668
beta_group[A,I(log(drug_B/1)),intercept] 33.702 31 0.338
beta_group[A,I(log(drug_B/1)),log_slope] 40.307 31 0.122
beta_group[A,I(log(drug_C/1)),intercept] 33.965 31 0.327
beta_group[A,I(log(drug_C/1)),log_slope] 33.792 31 0.334
beta_group[B,I(log(drug_A/1)),intercept] 43.898 31 0.062
beta_group[B,I(log(drug_A/1)),log_slope] 33.587 31 0.343
beta_group[B,I(log(drug_B/1)),intercept] 29.613 31 0.537
beta_group[B,I(log(drug_B/1)),log_slope] 21.894 31 0.886
beta_group[B,I(log(drug_C/1)),intercept] 37.946 31 0.182
beta_group[B,I(log(drug_C/1)),log_slope] 31.430 31 0.445
beta_group[C,I(log(drug_A/1)),intercept] 26.842 31 0.680
beta_group[C,I(log(drug_A/1)),log_slope] 36.326 31 0.234
beta_group[C,I(log(drug_B/1)),intercept] 41.862 31 0.092
beta_group[C,I(log(drug_B/1)),log_slope] 41.677 31 0.095
beta_group[C,I(log(drug_C/1)),intercept] 32.077 31 0.413
beta_group[C,I(log(drug_C/1)),log_slope] 38.048 31 0.179
eta_group[A,I(drug_A/1 * drug_B/1 * drug_C/1)] 19.930 31 0.937
eta_group[A,I(drug_A/1 * drug_B/1)] 22.221 31 0.876
eta_group[A,I(drug_A/1 * drug_C/1)] 34.272 31 0.314
eta_group[A,I(drug_B/1 * drug_C/1)] 45.075 31 0.049
eta_group[B,I(drug_A/1 * drug_B/1 * drug_C/1)] 35.808 31 0.253
eta_group[B,I(drug_A/1 * drug_B/1)] 41.331 31 0.102
eta_group[B,I(drug_A/1 * drug_C/1)] 44.902 31 0.051
eta_group[B,I(drug_B/1 * drug_C/1)] 25.338 31 0.752
eta_group[C,I(drug_A/1 * drug_B/1 * drug_C/1)] 29.728 31 0.531
eta_group[C,I(drug_A/1 * drug_B/1)] 33.568 31 0.344
eta_group[C,I(drug_A/1 * drug_C/1)] 21.018 31 0.911
eta_group[C,I(drug_B/1 * drug_C/1)] 21.530 31 0.897
mu_eta[I(drug_A/1 * drug_B/1 * drug_C/1)] 30.349 31 0.499
mu_eta[I(drug_A/1 * drug_B/1)] 30.522 31 0.490
mu_eta[I(drug_A/1 * drug_C/1)] 42.867 31 0.076
mu_eta[I(drug_B/1 * drug_C/1)] 22.637 31 0.862
mu_log_beta[I(log(drug_A/1)),intercept] 32.378 31 0.399
mu_log_beta[I(log(drug_A/1)),log_slope] 43.642 31 0.066
mu_log_beta[I(log(drug_B/1)),intercept] 38.118 31 0.177
mu_log_beta[I(log(drug_B/1)),log_slope] 31.674 31 0.433
mu_log_beta[I(log(drug_C/1)),intercept] 17.843 31 0.972
mu_log_beta[I(log(drug_C/1)),log_slope] 35.930 31 0.248
tau_eta[STRAT,I(drug_A/1 * drug_B/1 * drug_C/1)] 23.264 31 0.839
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 24.346 31 0.796
tau_eta[STRAT,I(drug_A/1 * drug_C/1)] 27.584 31 0.643
tau_eta[STRAT,I(drug_B/1 * drug_C/1)] 24.525 31 0.789
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 20.576 31 0.923
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 19.795 31 0.940
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 28.608 31 0.590
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 28.800 31 0.580
tau_log_beta[STRAT,I(log(drug_C/1)),intercept] 14.541 31 0.995
tau_log_beta[STRAT,I(log(drug_C/1)),log_slope] 27.795 31 0.632

Session Info

## R version 3.6.3 (2020-02-29)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.5 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] mvtnorm_1.1-0      tibble_3.0.0       rstan_2.19.3       StanHeaders_2.19.2
##  [5] abind_1.4-5        Formula_1.2-3      checkmate_2.0.0    OncoBayes2_0.8-1  
##  [9] testthat_3.0.3     devtools_2.3.0     usethis_1.6.0      ggplot2_3.3.0     
## [13] broom_0.5.5        tidyr_1.0.2        dplyr_0.8.5        assertthat_0.2.1  
## [17] knitr_1.28         rmarkdown_2.1     
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.4.6       lattice_0.20-44    prettyunits_1.1.1  ps_1.6.0          
##  [5] rprojroot_1.3-2    digest_0.6.25      plyr_1.8.6         R6_2.4.1          
##  [9] ggridges_0.5.2     backports_1.1.6    stats4_3.6.3       evaluate_0.14     
## [13] highr_0.8          pillar_1.4.3       rlang_0.4.11       rstudioapi_0.11   
## [17] callr_3.7.0        desc_1.2.0         stringr_1.4.0      loo_2.2.0         
## [21] munsell_0.5.0      compiler_3.6.3     xfun_0.13          pkgconfig_2.0.3   
## [25] pkgbuild_1.0.6     rstantools_2.1.1   htmltools_0.4.0    tidyselect_1.0.0  
## [29] gridExtra_2.3      codetools_0.2-18   matrixStats_0.56.0 crayon_1.3.4      
## [33] withr_2.4.2        grid_3.6.3         nlme_3.1-152       gtable_0.3.0      
## [37] lifecycle_0.2.0    magrittr_1.5       scales_1.1.0       RcppParallel_5.1.4
## [41] cli_2.5.0          stringi_1.4.6      fs_1.4.1           remotes_2.1.1     
## [45] ellipsis_0.3.0     generics_0.0.2     vctrs_0.2.4        glue_1.4.0        
## [49] purrr_0.3.4        processx_3.5.2     pkgload_1.0.2      parallel_3.6.3    
## [53] yaml_2.2.1         inline_0.3.15      colorspace_1.4-1   sessioninfo_1.1.1 
## [57] bayesplot_1.7.1    memoise_1.1.0