Simulation based calibration for OncoBayes2

Thu Dec 12 14:12:10 2019

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2019-12-11 16:27:14 UTC
## git hash: 7feda1da194006ab2eed1e533685f94ea6278e7f
## MD5:      879cb1025ef56fc5cb4dbe9944bbe84e

The MD5 hash of the calibration data file presented here must match the above listed MD5:

##                    calibration.rds 
## "879cb1025ef56fc5cb4dbe9944bbe84e"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Model 1: Single-agent logistic regression

Component intercept/slopes

Means

Standard deviations (tau’s)

Model 2: Double combination, fully exchangeable

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation

Model 3: Double combination, EXchangeable/NonEXchangeable model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation (tau)

Model 4: Triple combination, EX/NEX model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (means from exchangeable part)

Mean

Standard deviation (tau)

\(\chi^2\) Statistic, Model 1: Single-agent logistic regression

param statistic df p.value
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 43.091 31 0.073
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 29.005 31 0.569
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 35.974 31 0.247
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 51.072 31 0.013

\(\chi^2\) Statistic, Model 2: Double combination, fully exchangeable

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 27.610 31 0.641
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 29.754 31 0.530
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 27.904 31 0.626
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 23.987 31 0.811
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 45.146 31 0.048
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 21.338 31 0.903
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 27.802 31 0.631
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 24.531 31 0.788
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 27.629 31 0.640
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 19.629 31 0.943

\(\chi^2\) Statistic, Model 3: Double combination, EXchangeable/NonEXchangeable model

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 35.059 31 0.281
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 34.035 31 0.324
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 26.451 31 0.699
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 21.459 31 0.899
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 35.059 31 0.281
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 26.995 31 0.672
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 31.328 31 0.450
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 25.107 31 0.763
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 19.322 31 0.949
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 23.296 31 0.838

\(\chi^2\) Statistic, Model 4: Triple combination, EX/NEX model

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] 37.229 31 0.204
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 48.006 31 0.026
mu_eta[I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] 20.211 31 0.931
mu_eta[I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] 29.146 31 0.562
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 24.954 31 0.770
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 26.515 31 0.696
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 31.443 31 0.444
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 43.840 31 0.063
mu_log_beta[I(log(DosesAdm3/dref[3])),intercept] 18.445 31 0.964
mu_log_beta[I(log(DosesAdm3/dref[3])),log_slope] 32.269 31 0.404
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] 20.390 31 0.927
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 19.328 31 0.949
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] 27.053 31 0.670
tau_eta[1,I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] 34.323 31 0.311
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 27.802 31 0.631
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 33.414 31 0.351
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 21.754 31 0.891
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 20.294 31 0.929
tau_log_beta[1,I(log(DosesAdm3/dref[3])),intercept] 21.318 31 0.903
tau_log_beta[1,I(log(DosesAdm3/dref[3])),log_slope] 33.926 31 0.328

Session Info

## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
## 
## locale:
## [1] C
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] mvtnorm_1.0-11     RBesT_1.4-0        tibble_2.1.3      
##  [4] rstan_2.19.2       StanHeaders_2.19.0 abind_1.4-5       
##  [7] Formula_1.2-3      checkmate_1.9.4    OncoBayes2_0.5-8  
## [10] testthat_2.2.1     Rcpp_1.0.2         devtools_2.2.1    
## [13] usethis_1.5.1      ggplot2_3.2.1      broom_0.5.2       
## [16] tidyr_1.0.0        dplyr_0.8.3        assertthat_0.2.1  
## [19] knitr_1.25         rmarkdown_1.16    
## 
## loaded via a namespace (and not attached):
##  [1] lattice_0.20-38    prettyunits_1.0.2  ps_1.3.0          
##  [4] zeallot_0.1.0      rprojroot_1.3-2    digest_0.6.21     
##  [7] plyr_1.8.4         R6_2.4.0           ggridges_0.5.1    
## [10] backports_1.1.5    stats4_3.6.1       evaluate_0.14     
## [13] highr_0.8          pillar_1.4.2       rlang_0.4.0       
## [16] lazyeval_0.2.2     rstudioapi_0.10    callr_3.3.2       
## [19] labeling_0.3       desc_1.2.0         stringr_1.4.0     
## [22] loo_2.1.0          munsell_0.5.0      compiler_3.6.1    
## [25] xfun_0.10          pkgconfig_2.0.3    pkgbuild_1.0.6    
## [28] rstantools_2.0.0   htmltools_0.4.0    tidyselect_0.2.5  
## [31] gridExtra_2.3      codetools_0.2-16   matrixStats_0.55.0
## [34] crayon_1.3.4       withr_2.1.2        grid_3.6.1        
## [37] nlme_3.1-141       gtable_0.3.0       lifecycle_0.1.0   
## [40] magrittr_1.5       scales_1.0.0       cli_1.1.0         
## [43] stringi_1.4.3      fs_1.3.1           remotes_2.1.0     
## [46] ellipsis_0.3.0     generics_0.0.2     vctrs_0.2.0       
## [49] glue_1.3.1         purrr_0.3.3        processx_3.4.1    
## [52] pkgload_1.0.2      parallel_3.6.1     yaml_2.2.0        
## [55] inline_0.3.15      colorspace_1.4-1   sessioninfo_1.1.1 
## [58] bayesplot_1.7.0    memoise_1.1.0