Compare MPT Analysis Approaches

Marius Barth

2018-11-09

Prerequisites

First of all, make sure that you have installed the latest version of all necessary R packages and of JAGS. To install JAGS, go to mcmc-jags.sourceforge.net and follow installation instructions. After that, install or update the required R packages:

install.packages("MPTmultiverse")
update.packages(ask = FALSE)

Example 1: Bayen & Kuhlmann (2011)

# load packages
library("MPTmultiverse")

# If you're running the analysis from an .rmd file, you only need to ensure that
# the .rmd, .eqn, and .csv files are all in the same directory.

# ------------------------------------------------------------------------------
# MPT model definition & data

EQN_FILE <- "2HTSM_Submodel4.eqn"
DATA_FILE <- "Kuhlmann_dl7.csv"


# if .csv format uses semicolons ";" (German format):
data <- read.csv2(DATA_FILE, fileEncoding = "UTF-8-BOM")

# if .csv format uses commata "," (international format):
# data <- read.csv(DATA_FILE, fileEncoding = "UTF-8-BOM")

# take a glimpse at the data
head(data)
#>   Subject ExpCond Yee Yeu Yen Yue Yuu Yun Yne Ynu Ynn
#> 1     110       1   6  12   5   2  18   5   0   2  14
#> 2     138       1  11   8   6   3  16   4   2   2  12
#> 3     141       1   9  10   7   1  16   5   2   1  13
#> 4     149       1  10   9   4   0  19   6   0   0  16
#> 5     102       1  12   4   8   5  11   8   0   0  16
#> 6     105       1  14   2   5   1  19   7   2   0  14
TreeBUGS::plotFreq(data, boxplot = FALSE, eqn = EQN_FILE)


COL_ID <- "Subject"         # name of the variable encoding subject ID
COL_CONDITION <- "ExpCond"  # name of the variable encoding group membership


# Experimental conditions should be labeled in a meaningful way. To accomplish
# this, you may want to use the `factor` function:
unique(data[, COL_CONDITION])
#> [1] 1 2

data[[COL_CONDITION]] <- factor(
  data[[COL_CONDITION]]
  , levels = c(1:2)
  , labels = c("no_load", "load")
)

### check input data frame
head(data)
#>   Subject ExpCond Yee Yeu Yen Yue Yuu Yun Yne Ynu Ynn
#> 1     110 no_load   6  12   5   2  18   5   0   2  14
#> 2     138 no_load  11   8   6   3  16   4   2   2  12
#> 3     141 no_load   9  10   7   1  16   5   2   1  13
#> 4     149 no_load  10   9   4   0  19   6   0   0  16
#> 5     102 no_load  12   4   8   5  11   8   0   0  16
#> 6     105 no_load  14   2   5   1  19   7   2   0  14

Options

Every time the package MPTmultiverse is loaded, it automatically sets some more or less useful defaults for model estimation, usage of multiple processor cores, number of posterior predictive samples, etc. By calling mpt_options() without any parameters, you can inspect these default values. If you want to change them, call mpt_options with the respective parameter specified, i.e. mpt_options(n.iter = 1000). For testing purposes, you can also specify mpt_options("test"), which is a shorthand for setting fast, but highly unreliable settings. You can set options to defaults, again, by typing the shorthand mpt_options("default").

# How to change a single option:
mpt_options(n.iter = 1e3)

# For testing purposes, you can use this shorthand to set fast, but unreliable options:
mpt_options("test")

# List all options that were set for the different analysis approaches:
mpt_options()

Estimation

In the next chunk, the main computations are done. Type ?fit_mpt in the R console if you want to find out more about the parameters of the function.

results <- fit_mpt(
  model = EQN_FILE
  , dataset = DATA_FILE
  , data = data
  , id = COL_ID
  , condition = COL_CONDITION
  , core = c("D", "d")
)

After computations finished, which may take a couple of days, check if model estimation worked by using the function check_results.

check_results(results)
#> ## MPTinR: no pooling
#> Convergence checks failed for unkown reason.
#> 
#> 
#> ## MPTinR: complete pooling
#> No convergence problems.
#> 
#> 
#> ## TreeBUGS, no, simple:
#> All Rhat < 1.05 .
#> All effect sample sizes > 2000 .
#> 
#> 
#> ## TreeBUGS, complete, simple:
#> All Rhat < 1.05 .
#> All effect sample sizes > 2000 .
#> 
#> 
#> ## TreeBUGS, partial, trait:
#> All Rhat < 1.05 .
#> All effect sample sizes > 2000 .
#> 
#> 
#> ## TreeBUGS, partial, trait_uncorrelated:
#> All Rhat < 1.05 .
#> All effect sample sizes > 2000 .
#> 
#> 
#> ## TreeBUGS, partial, beta:
#> All Rhat < 1.05 .
#> All effect sample sizes > 2000 .
#> 
#> 
#> ## TreeBUGS, partial, betacpp:
#> All Rhat < 1.05 .
#> 28 parameters with effect sample size n.eff < 2000 :
#> sd[2], sd[3], alph[2], alph[3], bet[2], bet[3], mean[2], sd[2], sd[3], alph[2], alph[3], bet[2], bet[3], theta[2,4], theta[2,5], theta[2,6], theta[2,8], theta[2,9], theta[2,10], theta[2,11], theta[2,13], theta[2,14], theta[2,15], theta[2,17], theta[2,18], theta[2,19], theta[2,21], theta[2,22]

In this example, convergence failed for the no-pooling maximum-likelihood approach. A possible means to resolve this issue could be increasing the number of optimization runs, which can be done by typing mpt_options(n.iter = 40) and re-run. For the Bayesian approaches, the betacpp did not reach an effective sample size \(\mathit{ESS} > 2{,}000\). Increasing the number of iterations by typing mpt_options(n.iter = 2e5), and re-running, should solve this problem.

# store results
save(
  results
  , file = "results_bayen_kuhlmann.RData"
)

Post-processing of Results

The analysis output results is an object of class multiverseMPT, that has its own plot() method. Type ?plot.multiverseMPT to see the documentation of possible arguments to this method.

To plot group-level parameter estimates, type:

plot(results, save = FALSE, "est")

To plot between-subjects comparisons, type:

plot(results, save = FALSE, "test_between")

To plot overall goodness-of-fit, type:

plot(results, save = FALSE, "gof1")

To plot group-wise goodness-of-fit, type:

plot(results, save = FALSE, "gof2")