Power with simglm

2017-05-25

Power Analysis with simglm

The simglm package allows the ability to conduct a power analysis through simulation. This will be particularly helpful with multilevel models and generalized linear models. To show the process, we will start with basic regression models.

Single Level Power Analysis

Let’s look at a simple single level regression example to get started:

fixed <- ~ 1 + act + diff + numCourse + act:numCourse
fixed_param <- c(0.5, 1.1, 0.6, 0.9, 1.1)
cov_param <- list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                  var_type = c("single", "single", "single"),
                  opts = list(list(mean = 0, sd = 2),
                              list(mean = 0, sd = 2),
                              list(mean = 0, sd = 1)))
n <- 150
error_var <- 20
with_err_gen <- 'rnorm'
pow_param <- c('(Intercept)', 'act', 'diff', 'numCourse')
alpha <- .01
pow_dist <- "t"
pow_tail <- 2
replicates <- 100
power_out <- sim_pow(fixed = fixed, fixed_param = fixed_param, 
                     cov_param = cov_param, n = n, error_var = error_var,
                     with_err_gen = with_err_gen, data_str = "single",
                     pow_param = pow_param, alpha = alpha,
                     pow_dist = pow_dist, pow_tail = pow_tail, 
                     replicates = replicates)

Much of the output here is the same from the sim_reg function. The additional arguments, pow_param represents the terms to conduct a power analysis for and must be a subset of the fixed argument, alpha represents the per term level of significance, pow_dist represents the sampling distribution to refer to, either ‘z’ or ‘t’, pow_tail represents whether a one or two tailed hypothesis is being tested, and replicates represents the number of simulations to conduct. Note, to do a power analysis for the intercept, ‘(Intercept)’ must be used. By default, if pow_param is not specified power is conducted for all terms.

Finally, looking at the output from the above call:

power_out
var avg_test_stat sd_test_stat power num_reject num_repl data
(Intercept) 1.528405 0.9137595 0.13 13 100 1.00000000, 2.00000000, 3.00000000, 4.00000000, 5.00000000, 6.00000000, 7.00000000, 8.00000000, 9.00000000, 10.00000000, 11.00000000, 12.00000000, 13.00000000, 14.00000000, 15.00000000, 16.00000000, 17.00000000, 18.00000000, 19.00000000, 20.00000000, 21.00000000, 22.00000000, 23.00000000, 24.00000000, 25.00000000, 26.00000000, 27.00000000, 28.00000000, 29.00000000, 30.00000000, 31.00000000, 32.00000000, 33.00000000, 34.00000000, 35.00000000, 36.00000000, 37.00000000, 38.00000000, 39.00000000, 40.00000000, 41.00000000, 42.00000000, 43.00000000, 44.00000000, 45.00000000, 46.00000000, 47.00000000, 48.00000000, 49.00000000, 50.00000000, 51.00000000, 52.00000000, 53.00000000, 54.00000000, 55.00000000, 56.00000000, 57.00000000, 58.00000000, 59.00000000, 60.00000000, 61.00000000, 62.00000000, 63.00000000, 64.00000000, 65.00000000, 66.00000000, 67.00000000, 68.00000000, 69.00000000, 70.00000000, 71.00000000, 72.00000000, 73.00000000, 74.00000000, 75.00000000, 76.00000000, 77.00000000, 78.00000000, 79.00000000, 80.00000000, 81.00000000, 82.00000000, 83.00000000, 84.00000000, 85.00000000, 86.00000000, 87.00000000, 88.00000000, 89.00000000, 90.00000000, 91.00000000, 92.00000000, 93.00000000, 94.00000000, 95.00000000, 96.00000000, 97.00000000, 98.00000000, 99.00000000, 100.00000000, 1.10147928, 1.41282032, 0.29306936, 2.80871215, 0.16729402, 0.77472507, 0.26334561, 2.73634014, 1.38934617, 1.56159945, 2.11781324, 2.06834598, 1.17266006, 1.03569730, 1.74324842, 1.87946068, 0.04655373, 0.88984302, 2.51798805, 1.08067289, 0.77688087, 3.42177438, 1.95399290, 2.72900495, 2.39000973, 0.53524957, 2.24575201, 0.11500451, 0.28070141, 0.43908385, 1.37409657, 2.01442763, 3.01963103, 3.90336644, 1.31046986, 1.80016573, 1.86071740, 0.94132483, 2.01643650, 0.75625597, 2.12102739, 2.80359885, 1.51441097, 1.60450083, 0.13553758, 1.47848262, 1.01609209, 1.29122989, 1.53138633, 0.14686631, 2.04613862, 0.99368759, 1.75277605, 0.68272956, 0.46172671, 2.23997020, 0.42715414, 1.67975398, 2.04436506, 0.57469199, 0.84941652, 0.54980287, 0.98240344, 3.61024076, 3.29639637, 1.20089729, 0.05629247, 2.20125101, 1.43893523, 1.77428305, 0.52131674, 2.09698353, 0.51003910, 1.07228993, 0.80603542, 0.84662243, 1.18256126, 2.33250480, 2.61888834, 1.98492502, 1.26652183, 2.92221050, 0.61454767, 2.12143224, 2.36279568, 1.69232775, 3.88422232, 1.91123453, 2.26590220, 1.03255220, 1.86330777, 0.57231392, 2.47460502, 2.00057360, 0.51421483, 0.11850230, 2.66519385, 1.59845056, 2.17588734, 1.36011506, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000
act 5.938459 1.1186396 1.00 100 100 1.000000, 2.000000, 3.000000, 4.000000, 5.000000, 6.000000, 7.000000, 8.000000, 9.000000, 10.000000, 11.000000, 12.000000, 13.000000, 14.000000, 15.000000, 16.000000, 17.000000, 18.000000, 19.000000, 20.000000, 21.000000, 22.000000, 23.000000, 24.000000, 25.000000, 26.000000, 27.000000, 28.000000, 29.000000, 30.000000, 31.000000, 32.000000, 33.000000, 34.000000, 35.000000, 36.000000, 37.000000, 38.000000, 39.000000, 40.000000, 41.000000, 42.000000, 43.000000, 44.000000, 45.000000, 46.000000, 47.000000, 48.000000, 49.000000, 50.000000, 51.000000, 52.000000, 53.000000, 54.000000, 55.000000, 56.000000, 57.000000, 58.000000, 59.000000, 60.000000, 61.000000, 62.000000, 63.000000, 64.000000, 65.000000, 66.000000, 67.000000, 68.000000, 69.000000, 70.000000, 71.000000, 72.000000, 73.000000, 74.000000, 75.000000, 76.000000, 77.000000, 78.000000, 79.000000, 80.000000, 81.000000, 82.000000, 83.000000, 84.000000, 85.000000, 86.000000, 87.000000, 88.000000, 89.000000, 90.000000, 91.000000, 92.000000, 93.000000, 94.000000, 95.000000, 96.000000, 97.000000, 98.000000, 99.000000, 100.000000, 6.409506, 4.818588, 5.575636, 3.165420, 6.513996, 5.823233, 6.152361, 8.375661, 6.405437, 6.884019, 5.281911, 6.346295, 7.015353, 6.747475, 3.973231, 6.350540, 5.377681, 6.213939, 5.461305, 8.130971, 5.516355, 5.362565, 5.645011, 6.175820, 8.393988, 6.199767, 7.564711, 6.202021, 4.008091, 6.915596, 3.959105, 7.089467, 4.375978, 6.958398, 3.232800, 5.499597, 5.682899, 4.176759, 7.404979, 6.240151, 6.047598, 6.528366, 6.463319, 6.568765, 6.600070, 6.461411, 4.175309, 5.375334, 6.263758, 5.284533, 5.799264, 5.834973, 8.085309, 5.688395, 6.630870, 5.026313, 5.504188, 7.229091, 6.561605, 5.599119, 6.219017, 6.041239, 8.182182, 6.337747, 3.578453, 5.814397, 7.894672, 5.526020, 6.395878, 3.942780, 5.226560, 6.575985, 5.272636, 6.630969, 5.385810, 5.522348, 4.497971, 6.528522, 4.112246, 6.584186, 4.391620, 6.515945, 4.575136, 5.879205, 5.417969, 5.786870, 7.376332, 6.570647, 4.706134, 4.091016, 7.361778, 5.955001, 6.724656, 6.253115, 6.103439, 5.159121, 5.615334, 6.429646, 6.009807, 7.299276, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000
diff 3.354537 1.0598143 0.80 80 100 1.0000000, 2.0000000, 3.0000000, 4.0000000, 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000, 10.0000000, 11.0000000, 12.0000000, 13.0000000, 14.0000000, 15.0000000, 16.0000000, 17.0000000, 18.0000000, 19.0000000, 20.0000000, 21.0000000, 22.0000000, 23.0000000, 24.0000000, 25.0000000, 26.0000000, 27.0000000, 28.0000000, 29.0000000, 30.0000000, 31.0000000, 32.0000000, 33.0000000, 34.0000000, 35.0000000, 36.0000000, 37.0000000, 38.0000000, 39.0000000, 40.0000000, 41.0000000, 42.0000000, 43.0000000, 44.0000000, 45.0000000, 46.0000000, 47.0000000, 48.0000000, 49.0000000, 50.0000000, 51.0000000, 52.0000000, 53.0000000, 54.0000000, 55.0000000, 56.0000000, 57.0000000, 58.0000000, 59.0000000, 60.0000000, 61.0000000, 62.0000000, 63.0000000, 64.0000000, 65.0000000, 66.0000000, 67.0000000, 68.0000000, 69.0000000, 70.0000000, 71.0000000, 72.0000000, 73.0000000, 74.0000000, 75.0000000, 76.0000000, 77.0000000, 78.0000000, 79.0000000, 80.0000000, 81.0000000, 82.0000000, 83.0000000, 84.0000000, 85.0000000, 86.0000000, 87.0000000, 88.0000000, 89.0000000, 90.0000000, 91.0000000, 92.0000000, 93.0000000, 94.0000000, 95.0000000, 96.0000000, 97.0000000, 98.0000000, 99.0000000, 100.0000000, 4.2042622, 0.9789173, 4.4008038, 3.6542935, 1.8618072, 3.4390753, 4.6042326, 1.5629083, 2.7691079, 3.2331152, 3.0981985, 2.5882571, 4.2667673, 4.1641133, 5.1109399, 4.1875181, 3.1252685, 3.4821991, 2.9283731, 3.2744338, 4.8281937, 2.1586464, 3.3510310, 4.2219693, 4.5145543, 4.4741806, 4.5844162, 3.2863867, 3.8505061, 4.4639270, 3.7696399, 3.0509903, 0.9508020, 6.8215442, 3.1206079, 3.2408109, 3.0943764, 4.4348844, 3.5086001, 3.4457905, 4.3511466, 2.7806244, 4.0522843, 4.2414238, 3.3563355, 3.4214961, 3.2906587, 4.0698956, 1.4458030, 0.8565017, 2.7808459, 3.6095596, 3.3840563, 2.9151102, 2.1132757, 2.8867503, 1.7463189, 2.1107657, 3.3635094, 2.9082345, 2.7436699, 3.8777837, 5.0144008, 3.4882772, 2.5017543, 2.9269480, 4.0480927, 2.4573834, 2.9568776, 4.2990787, 1.6136006, 4.1347685, 3.3047833, 4.4401342, 2.8841412, 2.0199475, 3.1038864, 3.6050764, 1.7974758, 4.4045785, 3.5424223, 3.8263064, 3.3580639, 2.9858116, 4.5475266, 2.1517516, 4.8702072, 3.5839240, 3.4649268, 5.3869180, 3.9642625, 1.9828281, 3.7879170, 4.4104838, 0.7321225, 1.7133222, 4.0263574, 2.6228936, 3.5594342, 3.5225406, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000
numCourse 2.455904 1.1109747 0.39 39 100 1.0000000, 2.0000000, 3.0000000, 4.0000000, 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000, 10.0000000, 11.0000000, 12.0000000, 13.0000000, 14.0000000, 15.0000000, 16.0000000, 17.0000000, 18.0000000, 19.0000000, 20.0000000, 21.0000000, 22.0000000, 23.0000000, 24.0000000, 25.0000000, 26.0000000, 27.0000000, 28.0000000, 29.0000000, 30.0000000, 31.0000000, 32.0000000, 33.0000000, 34.0000000, 35.0000000, 36.0000000, 37.0000000, 38.0000000, 39.0000000, 40.0000000, 41.0000000, 42.0000000, 43.0000000, 44.0000000, 45.0000000, 46.0000000, 47.0000000, 48.0000000, 49.0000000, 50.0000000, 51.0000000, 52.0000000, 53.0000000, 54.0000000, 55.0000000, 56.0000000, 57.0000000, 58.0000000, 59.0000000, 60.0000000, 61.0000000, 62.0000000, 63.0000000, 64.0000000, 65.0000000, 66.0000000, 67.0000000, 68.0000000, 69.0000000, 70.0000000, 71.0000000, 72.0000000, 73.0000000, 74.0000000, 75.0000000, 76.0000000, 77.0000000, 78.0000000, 79.0000000, 80.0000000, 81.0000000, 82.0000000, 83.0000000, 84.0000000, 85.0000000, 86.0000000, 87.0000000, 88.0000000, 89.0000000, 90.0000000, 91.0000000, 92.0000000, 93.0000000, 94.0000000, 95.0000000, 96.0000000, 97.0000000, 98.0000000, 99.0000000, 100.0000000, 3.2751760, 3.4009965, 2.7748830, 2.3082406, 1.9956370, 3.0461759, 1.6869767, 4.4334153, 2.2954593, 3.4362029, 3.6065036, 2.9655579, 2.1797393, 2.2867398, 0.7580637, 2.5231119, 0.6458258, 2.2843246, 0.2398337, 3.1492807, 2.5096761, 3.7372126, 1.8937458, 3.4071330, 2.5852875, 2.9098445, 3.3429509, 2.4123478, 4.1847331, 3.9477607, 0.9755560, 1.8075271, 4.6300747, 3.3632872, 2.0495852, 2.4201229, 0.9185081, 2.4093852, 2.8203588, 1.5664756, 2.8346846, 1.3292648, 2.0250627, 1.8833959, 2.9389081, 2.1106898, 1.5002042, 4.1633527, 0.2003610, 1.1322771, 1.0410759, 0.3517875, 1.7235113, 1.8707892, 1.3853664, 1.8369246, 3.0150926, 2.1029281, 2.2036206, 1.1697847, 3.5332958, 1.4604487, 4.7495308, 3.7021407, 3.9019391, 2.4890278, 3.6655911, 3.1797168, 1.2481895, 0.4835791, 3.1871457, 2.1722876, 1.4471848, 2.5719016, 4.5516921, 3.3201809, 2.2933499, 3.5453716, 1.7477251, 2.4078056, 3.2284155, 2.3976975, 2.3584365, 1.2041354, 4.0047682, 3.0716305, 0.9102156, 3.3075102, 4.6878431, 0.5210591, 2.0933522, 1.6092507, 2.5619951, 4.2514415, 4.6966788, 2.0935832, 2.1974528, 1.2296616, 1.0288434, 2.4754934, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000

The output contains the variable name, the average test statistic, the standard deviation of the test statistic, the power rate, the number of null hypotheses rejects, and the total number of replications. Increasing the number of replications would increase the precision of the power analysis, however may significantly increase the computational time.

Standardized Coefficients

By default, the simglm package uses unstandardized regression coefficients when doing the simulation. A way to use standardized coefficients however, would be to generate standardized variables. For example:

fixed <- ~ 1 + act + diff + numCourse + act:numCourse
fixed_param <- c(0.2, 0.4, 0.25, 0.7, 0.1)
cov_param <- list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                  var_type = c("single", "single", "single"),
                  opts = list(list(mean = 0, sd = 1),
                              list(mean = 0, sd = 1),
                              list(mean = 0, sd = 1)))
n <- 150
error_var <- 1
with_err_gen <- 'rnorm'
pow_param <- c('(Intercept)', 'act', 'diff', 'numCourse')
alpha <- .01
pow_dist <- "t"
pow_tail <- 2
replicates <- 100
power_out <- sim_pow(fixed = fixed, fixed_param = fixed_param, 
                     cov_param = cov_param,n = n, error_var = error_var,
                     with_err_gen = with_err_gen, data_str = "single",
                     pow_param = pow_param, alpha = alpha,
                     pow_dist = pow_dist, pow_tail = pow_tail, 
                     replicates = replicates)
power_out
var avg_test_stat sd_test_stat power num_reject num_repl data
(Intercept) 2.398595 1.041257 0.40 40 100 1.0000000, 2.0000000, 3.0000000, 4.0000000, 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000, 10.0000000, 11.0000000, 12.0000000, 13.0000000, 14.0000000, 15.0000000, 16.0000000, 17.0000000, 18.0000000, 19.0000000, 20.0000000, 21.0000000, 22.0000000, 23.0000000, 24.0000000, 25.0000000, 26.0000000, 27.0000000, 28.0000000, 29.0000000, 30.0000000, 31.0000000, 32.0000000, 33.0000000, 34.0000000, 35.0000000, 36.0000000, 37.0000000, 38.0000000, 39.0000000, 40.0000000, 41.0000000, 42.0000000, 43.0000000, 44.0000000, 45.0000000, 46.0000000, 47.0000000, 48.0000000, 49.0000000, 50.0000000, 51.0000000, 52.0000000, 53.0000000, 54.0000000, 55.0000000, 56.0000000, 57.0000000, 58.0000000, 59.0000000, 60.0000000, 61.0000000, 62.0000000, 63.0000000, 64.0000000, 65.0000000, 66.0000000, 67.0000000, 68.0000000, 69.0000000, 70.0000000, 71.0000000, 72.0000000, 73.0000000, 74.0000000, 75.0000000, 76.0000000, 77.0000000, 78.0000000, 79.0000000, 80.0000000, 81.0000000, 82.0000000, 83.0000000, 84.0000000, 85.0000000, 86.0000000, 87.0000000, 88.0000000, 89.0000000, 90.0000000, 91.0000000, 92.0000000, 93.0000000, 94.0000000, 95.0000000, 96.0000000, 97.0000000, 98.0000000, 99.0000000, 100.0000000, 2.4808481, 2.5785573, 2.7823954, 0.7142574, 3.8262641, 0.2741152, 2.3215778, 1.7260406, 4.1773672, 1.8378865, 2.2885833, 1.0759879, 1.6021223, 1.0077462, 2.5505894, 1.8685551, 2.2768681, 4.0015338, 2.3570325, 1.7744121, 2.7155617, 3.2183667, 3.1712334, 2.4751493, 2.1666373, 3.3331839, 1.8328447, 3.5193888, 0.8496792, 2.4143260, 2.9429605, 3.1778945, 0.4208843, 1.8473141, 4.6583083, 3.4063577, 2.7563293, 2.5788445, 2.0142979, 2.0396512, 4.4321121, 2.7777622, 1.2571875, 0.1990811, 3.3470684, 0.4000809, 1.6948301, 3.5607141, 1.7121139, 3.8052183, 2.7757784, 3.6356993, 1.2649334, 2.7195079, 4.2225899, 1.5478608, 1.3291814, 1.9508269, 1.3770706, 2.1147779, 2.5863474, 1.9572944, 5.2836551, 2.6234792, 1.5869664, 1.8039646, 2.0636507, 3.6734904, 5.0640888, 2.4927641, 1.6757923, 0.2039761, 3.0286968, 2.4395839, 2.3592396, 2.8138900, 2.7967254, 3.4352621, 2.0652513, 1.0797729, 2.6739948, 2.5745603, 3.0476724, 3.1753574, 2.3904628, 2.5076436, 0.4512576, 1.1186191, 2.6987468, 2.0620353, 3.3969026, 3.0046125, 1.2883931, 2.2283772, 2.7081746, 3.1652951, 1.4240109, 2.4107763, 2.1875415, 3.1247974, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000
act 4.734279 1.006563 0.98 98 100 1.000000, 2.000000, 3.000000, 4.000000, 5.000000, 6.000000, 7.000000, 8.000000, 9.000000, 10.000000, 11.000000, 12.000000, 13.000000, 14.000000, 15.000000, 16.000000, 17.000000, 18.000000, 19.000000, 20.000000, 21.000000, 22.000000, 23.000000, 24.000000, 25.000000, 26.000000, 27.000000, 28.000000, 29.000000, 30.000000, 31.000000, 32.000000, 33.000000, 34.000000, 35.000000, 36.000000, 37.000000, 38.000000, 39.000000, 40.000000, 41.000000, 42.000000, 43.000000, 44.000000, 45.000000, 46.000000, 47.000000, 48.000000, 49.000000, 50.000000, 51.000000, 52.000000, 53.000000, 54.000000, 55.000000, 56.000000, 57.000000, 58.000000, 59.000000, 60.000000, 61.000000, 62.000000, 63.000000, 64.000000, 65.000000, 66.000000, 67.000000, 68.000000, 69.000000, 70.000000, 71.000000, 72.000000, 73.000000, 74.000000, 75.000000, 76.000000, 77.000000, 78.000000, 79.000000, 80.000000, 81.000000, 82.000000, 83.000000, 84.000000, 85.000000, 86.000000, 87.000000, 88.000000, 89.000000, 90.000000, 91.000000, 92.000000, 93.000000, 94.000000, 95.000000, 96.000000, 97.000000, 98.000000, 99.000000, 100.000000, 5.006892, 6.706992, 4.854214, 5.152052, 4.262358, 4.841091, 2.700973, 5.303073, 5.651683, 3.719292, 5.391532, 5.692714, 6.171878, 3.766363, 5.357891, 4.557133, 5.167590, 5.510545, 4.045020, 4.321456, 3.397165, 5.236155, 4.477353, 5.217156, 2.453321, 5.308229, 4.652909, 4.966611, 3.711655, 4.050181, 3.165444, 5.064765, 4.764941, 4.358269, 3.803452, 4.731465, 4.762985, 5.081641, 5.470271, 4.741156, 4.301207, 5.109899, 5.978024, 5.942350, 2.819677, 6.159302, 2.073059, 5.726956, 4.297071, 6.505645, 5.140989, 6.212310, 5.125966, 4.495297, 3.189890, 3.549170, 4.573786, 3.945371, 3.892767, 5.396208, 3.758086, 3.848438, 4.571912, 4.095452, 5.159955, 4.379180, 3.872307, 5.180324, 4.988846, 7.130971, 5.857967, 4.715697, 3.228205, 5.466048, 5.103293, 5.368153, 6.302320, 5.699648, 4.805577, 4.384507, 6.241609, 5.172178, 4.336232, 5.161116, 3.960246, 4.235223, 3.195700, 5.423790, 4.227442, 5.970748, 4.160221, 3.428883, 3.530014, 5.497654, 3.741401, 3.805418, 6.225283, 3.447917, 7.115508, 4.603582, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000
diff 3.090354 1.006858 0.69 69 100 1.0000000, 2.0000000, 3.0000000, 4.0000000, 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000, 10.0000000, 11.0000000, 12.0000000, 13.0000000, 14.0000000, 15.0000000, 16.0000000, 17.0000000, 18.0000000, 19.0000000, 20.0000000, 21.0000000, 22.0000000, 23.0000000, 24.0000000, 25.0000000, 26.0000000, 27.0000000, 28.0000000, 29.0000000, 30.0000000, 31.0000000, 32.0000000, 33.0000000, 34.0000000, 35.0000000, 36.0000000, 37.0000000, 38.0000000, 39.0000000, 40.0000000, 41.0000000, 42.0000000, 43.0000000, 44.0000000, 45.0000000, 46.0000000, 47.0000000, 48.0000000, 49.0000000, 50.0000000, 51.0000000, 52.0000000, 53.0000000, 54.0000000, 55.0000000, 56.0000000, 57.0000000, 58.0000000, 59.0000000, 60.0000000, 61.0000000, 62.0000000, 63.0000000, 64.0000000, 65.0000000, 66.0000000, 67.0000000, 68.0000000, 69.0000000, 70.0000000, 71.0000000, 72.0000000, 73.0000000, 74.0000000, 75.0000000, 76.0000000, 77.0000000, 78.0000000, 79.0000000, 80.0000000, 81.0000000, 82.0000000, 83.0000000, 84.0000000, 85.0000000, 86.0000000, 87.0000000, 88.0000000, 89.0000000, 90.0000000, 91.0000000, 92.0000000, 93.0000000, 94.0000000, 95.0000000, 96.0000000, 97.0000000, 98.0000000, 99.0000000, 100.0000000, 3.4846229, 2.9581237, 2.1116722, 4.0654622, 3.4487390, 3.1518931, 2.0764831, 3.8534571, 2.8274472, 2.9801243, 2.2384884, 3.4141292, 1.1253211, 3.5762912, 3.1208757, 1.5409429, 4.2091619, 2.0736330, 3.3071353, 3.5056910, 2.6753625, 1.4376321, 2.4490770, 3.3045056, 3.5833528, 4.4142033, 0.7599205, 5.4873759, 3.0177130, 0.7837205, 2.6146571, 1.5733285, 2.8051888, 5.5306175, 4.3116302, 1.8946900, 3.1079783, 4.0535677, 2.3002373, 3.5179967, 2.4643251, 4.7237491, 3.7026730, 4.4300247, 3.0207578, 4.8575687, 3.7536162, 4.4740315, 3.1081831, 2.9592401, 2.0589678, 4.3466583, 0.3702495, 2.4089316, 2.5463925, 3.7869909, 3.4003472, 2.7188153, 2.9174944, 3.6596567, 2.1281058, 3.3831568, 3.4480305, 2.9274011, 3.9516845, 2.0556258, 3.1370275, 2.7616090, 5.0387700, 5.0097677, 3.4035724, 2.8002646, 2.3990623, 4.6790688, 1.8322174, 2.2085787, 3.3094012, 1.9886544, 3.0021828, 2.7376663, 2.4771490, 4.3878591, 3.2134756, 3.0508912, 2.0990073, 3.8533299, 4.0632380, 3.3604909, 2.3992761, 2.2231252, 3.7514630, 3.3997732, 3.7823181, 2.4156163, 2.6902733, 4.0880207, 2.1655460, 3.6731282, 1.8417734, 3.4967152, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000
numCourse 8.456136 1.236331 1.00 100 100 1.000000, 2.000000, 3.000000, 4.000000, 5.000000, 6.000000, 7.000000, 8.000000, 9.000000, 10.000000, 11.000000, 12.000000, 13.000000, 14.000000, 15.000000, 16.000000, 17.000000, 18.000000, 19.000000, 20.000000, 21.000000, 22.000000, 23.000000, 24.000000, 25.000000, 26.000000, 27.000000, 28.000000, 29.000000, 30.000000, 31.000000, 32.000000, 33.000000, 34.000000, 35.000000, 36.000000, 37.000000, 38.000000, 39.000000, 40.000000, 41.000000, 42.000000, 43.000000, 44.000000, 45.000000, 46.000000, 47.000000, 48.000000, 49.000000, 50.000000, 51.000000, 52.000000, 53.000000, 54.000000, 55.000000, 56.000000, 57.000000, 58.000000, 59.000000, 60.000000, 61.000000, 62.000000, 63.000000, 64.000000, 65.000000, 66.000000, 67.000000, 68.000000, 69.000000, 70.000000, 71.000000, 72.000000, 73.000000, 74.000000, 75.000000, 76.000000, 77.000000, 78.000000, 79.000000, 80.000000, 81.000000, 82.000000, 83.000000, 84.000000, 85.000000, 86.000000, 87.000000, 88.000000, 89.000000, 90.000000, 91.000000, 92.000000, 93.000000, 94.000000, 95.000000, 96.000000, 97.000000, 98.000000, 99.000000, 100.000000, 9.093260, 8.079784, 8.939363, 6.950533, 6.977439, 9.018637, 8.769543, 7.126925, 8.244640, 5.886155, 10.715221, 9.115648, 7.295412, 6.855297, 8.367392, 7.651711, 7.520398, 8.073199, 9.770331, 7.533065, 6.898306, 8.540596, 9.274431, 6.732386, 7.977514, 9.152312, 8.282816, 9.946607, 10.264643, 7.957263, 6.940811, 8.132815, 8.310852, 10.351927, 10.003102, 10.139714, 6.214281, 6.801374, 8.353700, 8.890604, 8.122419, 8.534917, 6.718729, 8.450474, 8.130831, 10.382103, 9.976021, 7.790496, 9.392565, 10.925062, 8.136876, 9.923295, 7.499586, 9.319287, 8.372712, 8.713890, 7.512197, 9.269470, 9.331496, 9.408726, 9.940852, 8.731688, 8.135876, 9.312822, 8.747983, 9.673948, 7.852983, 6.730415, 7.208273, 8.695059, 8.280150, 7.604194, 8.222256, 8.470217, 9.282984, 9.677853, 7.973163, 11.322868, 7.984666, 8.043150, 8.708347, 7.426858, 6.758275, 8.096828, 9.671546, 8.887221, 7.461361, 5.720672, 6.733934, 11.011748, 6.170877, 6.957890, 10.513553, 8.772181, 7.135289, 10.776000, 10.227788, 8.447146, 8.066308, 9.117224, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000

Varying Arguments

fixed <- ~ 1 + act + diff + numCourse + act:numCourse
fixed_param <- c(0.5, 1.1, 0.6, 0.9, 1.1)
cov_param <- list(dist_fun = c('rnorm', 'rnorm', 'rnorm'), 
                  var_type = c("single", "single", "single"),
                  opts = list(list(mean = 0, sd = 2),
                              list(mean = 0, sd = 2),
                              list(mean = 0, sd = 1)))
n <- NULL
error_var <- NULL
with_err_gen <- 'rnorm'
pow_param <- c('(Intercept)', 'act', 'diff', 'numCourse')
alpha <- .01
pow_dist <- "t"
pow_tail <- 2
replicates <- 10
terms_vary <- list(n = c(20, 40, 60, 80, 100), error_var = c(5, 10, 20),
                   fixed_param = list(c(0.5, 1.1, 0.6, 0.9, 1.1), 
                                      c(0.6, 1.1, 0.6, 0.9, 1.1)),
                cov_param = list(list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                                       mean = c(0, 0, 0), sd = c(2, 2, 1), 
                                  var_type = c("single", "single", "single")),
                                  list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                                       mean = c(0.5, 0, 0), sd = c(2, 2, 1), 
                                  var_type = c("single", "single", "single"))
                                  )
                   )
power_out <- sim_pow(fixed = fixed, fixed_param = fixed_param, 
                     cov_param = cov_param,
                     n = n, error_var = error_var, with_err_gen = with_err_gen, 
                     data_str = "single", pow_param = pow_param, alpha = alpha,
                     pow_dist = pow_dist, pow_tail = pow_tail, 
                     replicates = replicates, terms_vary = terms_vary)

Model Misspecification

It is also possible to specify a model different than the generating model for evaluation of power. This may be useful if it is thought another variable is important in explaining variation, however due to design issues is not possible to collect this variable. As a result, there would likely be additional variation due to this variable that will not be able to be explained. Building this into the generating model can help provide additional information about the impact on power.

An example using single level power analysis is shown below.

fixed <- ~ 1 + act + diff + numCourse + act:numCourse
fixed_param <- c(0.5, 1.1, 0.6, 0.9, 1.1)
cov_param <- list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                  var_type = c("single", "single", "single"),
                  opts = list(list(mean = 0, sd = 2),
                              list(mean = 0, sd = 2),
                              list(mean = 0, sd = 1)))
n <- 150
error_var <- 20
with_err_gen <- 'rnorm'
pow_param <- c('(Intercept)', 'act', 'diff', 'numCourse')
alpha <- .01
pow_dist <- "t"
pow_tail <- 2
replicates <- 100

lm_fit_mod <- sim_data ~ 1 + act + diff

power_out <- sim_pow(fixed = fixed, fixed_param = fixed_param, 
                     cov_param = cov_param, n = n, error_var = error_var,
                     with_err_gen = with_err_gen, data_str = "single",
                     pow_param = pow_param, alpha = alpha,
                     pow_dist = pow_dist, pow_tail = pow_tail, 
                     replicates = replicates, lm_fit_mod = lm_fit_mod)
power_out
var avg_test_stat sd_test_stat power num_reject num_repl data
(Intercept) 1.197679 0.808495 0.08 8 100 1.00000000, 2.00000000, 3.00000000, 4.00000000, 5.00000000, 6.00000000, 7.00000000, 8.00000000, 9.00000000, 10.00000000, 11.00000000, 12.00000000, 13.00000000, 14.00000000, 15.00000000, 16.00000000, 17.00000000, 18.00000000, 19.00000000, 20.00000000, 21.00000000, 22.00000000, 23.00000000, 24.00000000, 25.00000000, 26.00000000, 27.00000000, 28.00000000, 29.00000000, 30.00000000, 31.00000000, 32.00000000, 33.00000000, 34.00000000, 35.00000000, 36.00000000, 37.00000000, 38.00000000, 39.00000000, 40.00000000, 41.00000000, 42.00000000, 43.00000000, 44.00000000, 45.00000000, 46.00000000, 47.00000000, 48.00000000, 49.00000000, 50.00000000, 51.00000000, 52.00000000, 53.00000000, 54.00000000, 55.00000000, 56.00000000, 57.00000000, 58.00000000, 59.00000000, 60.00000000, 61.00000000, 62.00000000, 63.00000000, 64.00000000, 65.00000000, 66.00000000, 67.00000000, 68.00000000, 69.00000000, 70.00000000, 71.00000000, 72.00000000, 73.00000000, 74.00000000, 75.00000000, 76.00000000, 77.00000000, 78.00000000, 79.00000000, 80.00000000, 81.00000000, 82.00000000, 83.00000000, 84.00000000, 85.00000000, 86.00000000, 87.00000000, 88.00000000, 89.00000000, 90.00000000, 91.00000000, 92.00000000, 93.00000000, 94.00000000, 95.00000000, 96.00000000, 97.00000000, 98.00000000, 99.00000000, 100.00000000, 0.77905556, 1.60289923, 0.45870822, 2.46555799, 1.07583628, 1.30474456, 0.24849595, 2.45559024, 1.39711471, 2.02657770, 0.75789928, 2.07733835, 1.12875492, 0.37572008, 1.04236554, 0.78229764, 1.05896813, 0.03637156, 2.29142641, 1.67958714, 0.50135571, 0.16675236, 1.52515869, 1.26240230, 0.85057334, 0.76501042, 1.38483655, 1.11145473, 1.26765530, 0.58391355, 1.39882313, 1.17743211, 1.98326553, 0.45828347, 0.24346883, 1.78361896, 1.83320281, 0.01856525, 2.70955631, 0.54222400, 0.13398929, 0.07528563, 3.18932227, 1.23894482, 2.33028161, 0.51721891, 0.53220038, 0.61731614, 1.60442907, 0.35748245, 1.66589022, 0.53525340, 1.65506570, 1.68361445, 1.34559909, 0.13318836, 0.32711495, 2.65209923, 0.89999639, 0.39709025, 1.26054590, 2.63794874, 1.46079651, 3.33782128, 0.77119833, 0.64199218, 2.89641315, 1.10345621, 1.02189274, 1.42144576, 0.25255013, 0.94421150, 1.11375432, 1.16387521, 0.34949314, 0.53168767, 1.88419879, 0.55756858, 0.45418494, 0.37822431, 0.95106582, 0.59712296, 1.11069809, 1.25533393, 1.50537903, 1.82384078, 0.07738392, 0.34606618, 3.42824658, 1.93344483, 1.46580108, 1.16360389, 1.10652781, 1.35545150, 2.06773452, 0.41346729, 0.13120970, 2.83017122, 2.31172982, 1.20614809, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000
act 5.289436 1.272097 0.98 98 100 1.000000, 2.000000, 3.000000, 4.000000, 5.000000, 6.000000, 7.000000, 8.000000, 9.000000, 10.000000, 11.000000, 12.000000, 13.000000, 14.000000, 15.000000, 16.000000, 17.000000, 18.000000, 19.000000, 20.000000, 21.000000, 22.000000, 23.000000, 24.000000, 25.000000, 26.000000, 27.000000, 28.000000, 29.000000, 30.000000, 31.000000, 32.000000, 33.000000, 34.000000, 35.000000, 36.000000, 37.000000, 38.000000, 39.000000, 40.000000, 41.000000, 42.000000, 43.000000, 44.000000, 45.000000, 46.000000, 47.000000, 48.000000, 49.000000, 50.000000, 51.000000, 52.000000, 53.000000, 54.000000, 55.000000, 56.000000, 57.000000, 58.000000, 59.000000, 60.000000, 61.000000, 62.000000, 63.000000, 64.000000, 65.000000, 66.000000, 67.000000, 68.000000, 69.000000, 70.000000, 71.000000, 72.000000, 73.000000, 74.000000, 75.000000, 76.000000, 77.000000, 78.000000, 79.000000, 80.000000, 81.000000, 82.000000, 83.000000, 84.000000, 85.000000, 86.000000, 87.000000, 88.000000, 89.000000, 90.000000, 91.000000, 92.000000, 93.000000, 94.000000, 95.000000, 96.000000, 97.000000, 98.000000, 99.000000, 100.000000, 6.283532, 5.037249, 6.355029, 6.286384, 5.928828, 3.786360, 5.049874, 7.908124, 6.380716, 5.079286, 5.952551, 6.502268, 5.899357, 5.062256, 4.192864, 6.794908, 5.183571, 6.281530, 3.994047, 5.214139, 5.948478, 3.502997, 6.567663, 5.678171, 5.670135, 5.975060, 6.319631, 4.517460, 3.038061, 4.608153, 6.626207, 3.596749, 3.298509, 5.387894, 3.161845, 5.555233, 3.927894, 3.832579, 6.712608, 7.317964, 5.446578, 4.194238, 5.890593, 2.489958, 6.049644, 5.333035, 6.993083, 4.076358, 6.028510, 6.372231, 3.075481, 6.372119, 4.602915, 5.016383, 6.282829, 4.349852, 4.176210, 3.896120, 4.750729, 4.383927, 4.879417, 6.520178, 5.338607, 5.728545, 5.377298, 3.939188, 6.152585, 4.127610, 5.067685, 6.207620, 3.838725, 4.087135, 3.867565, 5.767773, 4.104016, 4.846374, 4.704557, 5.992559, 5.896424, 6.415497, 4.213941, 7.828062, 3.471393, 3.526151, 8.550426, 4.895826, 6.874564, 3.372910, 5.542224, 2.405953, 6.408792, 5.410036, 6.008823, 5.490316, 5.943677, 5.190884, 7.197060, 7.487302, 4.532831, 7.536122, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000
diff 2.848462 1.078851 0.59 59 100 1.0000000, 2.0000000, 3.0000000, 4.0000000, 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000, 10.0000000, 11.0000000, 12.0000000, 13.0000000, 14.0000000, 15.0000000, 16.0000000, 17.0000000, 18.0000000, 19.0000000, 20.0000000, 21.0000000, 22.0000000, 23.0000000, 24.0000000, 25.0000000, 26.0000000, 27.0000000, 28.0000000, 29.0000000, 30.0000000, 31.0000000, 32.0000000, 33.0000000, 34.0000000, 35.0000000, 36.0000000, 37.0000000, 38.0000000, 39.0000000, 40.0000000, 41.0000000, 42.0000000, 43.0000000, 44.0000000, 45.0000000, 46.0000000, 47.0000000, 48.0000000, 49.0000000, 50.0000000, 51.0000000, 52.0000000, 53.0000000, 54.0000000, 55.0000000, 56.0000000, 57.0000000, 58.0000000, 59.0000000, 60.0000000, 61.0000000, 62.0000000, 63.0000000, 64.0000000, 65.0000000, 66.0000000, 67.0000000, 68.0000000, 69.0000000, 70.0000000, 71.0000000, 72.0000000, 73.0000000, 74.0000000, 75.0000000, 76.0000000, 77.0000000, 78.0000000, 79.0000000, 80.0000000, 81.0000000, 82.0000000, 83.0000000, 84.0000000, 85.0000000, 86.0000000, 87.0000000, 88.0000000, 89.0000000, 90.0000000, 91.0000000, 92.0000000, 93.0000000, 94.0000000, 95.0000000, 96.0000000, 97.0000000, 98.0000000, 99.0000000, 100.0000000, 4.1825192, 3.2359515, 3.1608719, 2.6464339, 2.8695526, 2.8455206, 3.9321218, 3.2392687, 3.8635536, 1.7745429, 3.7508433, 2.9365496, 5.9695848, 3.8265766, 0.9381103, 2.4020388, 3.4339521, 2.5833650, 2.0115524, 2.2369556, 1.6487223, 4.3132159, 3.6921481, 2.1909603, 2.2106343, 4.4969452, 2.4471038, 3.9722267, 2.3818349, 2.6739935, 2.5786072, 3.9645166, 2.6657225, 3.1803938, 2.4315527, 3.5656397, 1.6195637, 3.9004417, 3.9309483, 2.1311830, 3.1286531, 2.7389550, 2.9154382, 5.0227322, 1.8705837, 3.5927564, 3.0505244, 2.7652757, 3.9026321, 3.2453761, 3.4793634, 2.5978219, 3.3034946, 3.2432605, 0.9963222, 3.6996754, 1.9218159, 3.2483244, 1.3466492, 3.7971277, 1.5209230, 2.3760695, 2.7527053, 1.6697456, 2.0327969, 3.1437863, 1.0459321, 2.9109784, 2.7994801, 1.6549526, 3.3198743, 1.1057727, 3.0511062, 2.3088477, 2.7269277, 4.2594924, 2.4358238, 1.2205416, 2.9288670, 2.5665226, 0.7913365, 5.8934464, 4.1772719, 0.5327952, 3.4699545, 3.5110704, 5.3700960, 2.1400662, 2.8372083, 2.2135971, 0.7620036, 3.2680781, 2.1322230, 2.9077237, 2.5819995, 3.8558828, 4.1624258, 2.0884528, 0.6796728, 1.9367293, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 0.0000000, 1.0000000, 1.0000000, 0.0000000, 0.0000000, 0.0000000
numCourse NA NaN NA NA 100 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

You’ll notice, compared to above, the power is somewhat reduced for the other three fixed effects when not modeling the numCourse variable. You can also build in the lm_fit_mod argument into the design via the terms_vary argument.

fixed <- ~ 1 + act + diff + numCourse + act:numCourse
fixed_param <- c(0.5, 1.1, 0.6, 0.9, 1.1)
cov_param <- list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                  var_type = c("single", "single", "single"),
                  opts = list(list(mean = 0, sd = 2),
                              list(mean = 0, sd = 2),
                              list(mean = 0, sd = 1)))
n <- 150
error_var <- 20
with_err_gen <- 'rnorm'
pow_param <- c('(Intercept)', 'act', 'diff', 'numCourse')
alpha <- .01
pow_dist <- "t"
pow_tail <- 2
replicates <- 100

terms_vary <- list(n = c(20, 40, 60, 80, 100), error_var = c(5, 10, 20),
                   fixed_param = list(c(0.5, 1.1, 0.6, 0.9, 1.1), 
                                      c(0.6, 1.1, 0.6, 0.9, 1.1)),
                lm_fit_mod = list(sim_data ~ 1 + act + diff, 
                                  sim_data ~ 1 + act)
)

power_out <- sim_pow(fixed = fixed, fixed_param = fixed_param, 
                     cov_param = cov_param, n = n, error_var = error_var,
                     with_err_gen = with_err_gen, data_str = "single",
                     pow_param = pow_param, alpha = alpha,
                     pow_dist = pow_dist, pow_tail = pow_tail, 
                     replicates = replicates, terms_vary = terms_vary)
power_out
## Source: local data frame [240 x 11]
## Groups: var, n, error_var, fixed_param [?]
## 
##            var     n error_var         fixed_param
##         <fctr> <dbl>     <dbl>              <fctr>
## 1  (Intercept)    20         5 0.5,1.1,0.6,0.9,1.1
## 2  (Intercept)    20         5 0.5,1.1,0.6,0.9,1.1
## 3  (Intercept)    20         5 0.6,1.1,0.6,0.9,1.1
## 4  (Intercept)    20         5 0.6,1.1,0.6,0.9,1.1
## 5  (Intercept)    20        10 0.5,1.1,0.6,0.9,1.1
## 6  (Intercept)    20        10 0.5,1.1,0.6,0.9,1.1
## 7  (Intercept)    20        10 0.6,1.1,0.6,0.9,1.1
## 8  (Intercept)    20        10 0.6,1.1,0.6,0.9,1.1
## 9  (Intercept)    20        20 0.5,1.1,0.6,0.9,1.1
## 10 (Intercept)    20        20 0.5,1.1,0.6,0.9,1.1
## # ... with 230 more rows, and 7 more variables: lm_fit_mod <fctr>,
## #   avg_test_stat <dbl>, sd_test_stat <dbl>, power <dbl>,
## #   num_reject <dbl>, num_repl <dbl>, data <list>

Nested Data

Extending the power analysis to two level models is a straightforward addition.

fixed <- ~1 + time + diff + act + time:act
random <- ~1 + time
fixed_param <- c(0, 0.2, 0.1, 0.3, 0.05)
random_param <- list(random_var = c(7, 4), rand_gen = "rnorm")
cov_param <- list(dist_fun = c('rnorm', 'rnorm'),
                  var_type = c("level1", "level2"),
                  opts = list(list(mean = 0, sd = 1),
                              list(mean = 0, sd = 1)))
n <- 150
p <- 30
error_var <- 1
data_str <- "long"
pow_param <- c('time', 'diff', 'act')
alpha <- .01
pow_dist <- "z"
pow_tail <- 2
replicates <- 10
power_out <- sim_pow(fixed = fixed, random = random, 
                     fixed_param = fixed_param, 
                     random_param = random_param, cov_param = cov_param, 
                     k = NULL, n = n, p = p,
                     error_var = error_var, with_err_gen = "rnorm",
                     data_str = data_str, unbal = list(level2 = FALSE, level3 = FALSE),
                     pow_param = pow_param, 
                     alpha = alpha, pow_dist = pow_dist, pow_tail = pow_tail,
                     replicates = replicates)

A few notes about the sim_pow function in relation to nested data. First, the lmer function from the lme4 package is used to fit the models. When arima = TRUE, then the nlme package is used, but this is currently not supported. One note, the power simulation takes more computational time compared to the single level example.

The power output is identical to the single level model above:

power_out
var avg_test_stat sd_test_stat power num_reject num_repl data
act 1.103334 0.6105125 0.0 0 10 0.4771322, 0.3314951, 1.5090936, 0.8202437, 1.1570577, 1.4687818, 2.1694260, 1.7259550, 0.9265550, 0.4476004, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000
diff 5.879843 0.5384399 1.0 10 10 5.302742, 5.611201, 5.649725, 5.215739, 5.421993, 6.519927, 6.852179, 5.939845, 6.185340, 6.099742, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000
time 1.088671 1.0268648 0.1 1 10 0.3941660, 1.5578184, 0.3724755, 2.1906232, 0.6070795, 0.2172024, 1.3772243, 3.3330537, 0.5771376, 0.2599258, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 0.0000000, 1.0000000, 0.0000000, 0.0000000

Three Level Designs

fixed <- ~1 + time + diff + act + actClust + time:act
random <- ~1 + time
random3 <- ~ 1 + time
fixed_param <- c(4, 2, 6, 2.3, 7, 0)
random_param <- list(random_var = c(7, 4), rand_gen = 'rnorm')
random_param3 <- list(random_var = c(4, 2), rand_gen = 'rnorm')
cov_param <- list(dist_fun = c('rnorm', 'rnorm', 'rnorm'),
                  var_type = c("level1", "level2", "level3"),
                  opts = list(list(mean = 0, sd = 1.5),
                              list(mean = 0, sd = 4),
                              list(mean = 0, sd = 2)))
k <- 10
n <- 45
p <- 8
error_var <- 4
with_err_gen <- 'rnorm'
data_str <- "long"
pow_param <- c('time', 'diff', 'act', 'actClust')
alpha <- .01
pow_dist <- "z"
pow_tail <- 2
replicates <- 3
power_out <- sim_pow(fixed = fixed, random = random, random3 = random3,
                     fixed_param = fixed_param,
                     random_param = random_param, random_param3 = random_param3,
                     cov_param = cov_param,
                     k = k, n = n, p = p,
                     error_var = error_var, with_err_gen = "rnorm",
                     data_str = data_str, unbal = list(level2 = FALSE, level3 = FALSE),
                     pow_param = pow_param, alpha = alpha,
                     pow_dist = pow_dist, pow_tail = pow_tail, 
                     replicates = replicates)
power_out
var avg_test_stat sd_test_stat power num_reject num_repl data
act 64.686677 2.1217617 1 3 3 63.85094, 67.09904, 63.11004, 1.00000, 1.00000, 1.00000
actClust 24.169981 11.3124589 1 3 3 31.53194, 29.83368, 11.14433, 1.00000, 1.00000, 1.00000
diff 237.475877 1.5551763 1 3 3 239.2488, 236.8366, 236.3422, 1.0000, 1.0000, 1.0000
time 3.970478 0.8059805 1 3 3 3.466277, 3.545127, 4.900030, 1.000000, 1.000000, 1.000000

Generalized Power Analysis

fixed <- ~ 1 + act + diff
fixed_param <- c(0.1, 0.5, 0.3)
cov_param <- list(dist_fun = c('rnorm', 'rnorm'),
                  var_type = c("single", "single"),
                  opts = list(list(mean = 0, sd = 2),
                              list(mean = 0, sd = 4)))
n <- 50
pow_param <- c('(Intercept)', 'act', 'diff')
alpha <- .01
pow_dist <- "z"
pow_tail <- 2
replicates <- 10

power_out <- sim_pow_glm(fixed = fixed, fixed_param = fixed_param, 
                         cov_param = cov_param, 
                         n = n, data_str = "single", 
                         pow_param = pow_param, alpha = alpha,
                         pow_dist = pow_dist, pow_tail = pow_tail, 
                         replicates = replicates)
power_out
var avg_test_stat sd_test_stat power num_reject num_repl data
(Intercept) 0.7810206 0.5113429 0.0 0 10 0.52206437, 0.84307587, 0.04607766, 1.23234087, 0.47978691, 1.00539964, 1.02782242, 0.21077762, 0.66312698, 1.77973406, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000
act 2.5471918 0.6574085 0.4 4 10 3.371753, 1.674532, 1.999520, 2.388516, 2.448675, 2.844254, 1.628352, 3.273739, 3.353219, 2.489358, 1.000000, 0.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 1.000000, 1.000000, 0.000000
diff 2.2870190 0.7947209 0.4 4 10 1.572070, 2.442469, 1.535645, 1.734473, 3.680508, 2.836920, 1.875525, 2.950037, 1.293329, 2.949214, 0.000000, 0.000000, 0.000000, 0.000000, 1.000000, 1.000000, 0.000000, 1.000000, 0.000000, 1.000000

Vary Arguments

fixed <- ~ 1 + act + diff
fixed_param <- c(0.1, 0.5, 0.3)
cov_param <- list(dist_fun = c('rnorm', 'rnorm'),
                  var_type = c("single", "single"),
                  opts = list(list(mean = 0, sd = 5),
                              list(mean = 0, sd = 8)))
n <- NULL
pow_param <- c('(Intercept)', 'act', 'diff')
alpha <- .01
pow_dist <- "z"
pow_tail <- 2
replicates <- 10
terms_vary <- list(n = c(20, 40, 60, 80, 100),
                   fixed_param = list(c(0.5, 0.1, 0.2), 
                                      c(0.6, 0.1, 0.2)))

power_out <- sim_pow_glm(fixed = fixed, fixed_param = fixed_param, 
                         cov_param = cov_param, 
                         n = n, data_str = "single", 
                         pow_param = pow_param, alpha = alpha,
                         pow_dist = pow_dist, pow_tail = pow_tail, 
                         replicates = replicates, terms_vary = terms_vary)
power_out
## Source: local data frame [30 x 9]
## Groups: var, n [?]
## 
##            var     n fixed_param avg_test_stat sd_test_stat power
##         <fctr> <dbl>      <fctr>         <dbl>        <dbl> <dbl>
## 1  (Intercept)    20 0.5,0.1,0.2     0.8361101    0.4773758   0.0
## 2  (Intercept)    20 0.6,0.1,0.2     1.0683651    0.5250532   0.0
## 3  (Intercept)    40 0.5,0.1,0.2     1.4235620    0.5282677   0.0
## 4  (Intercept)    40 0.6,0.1,0.2     1.5021414    0.4385781   0.0
## 5  (Intercept)    60 0.5,0.1,0.2     1.4676779    0.9857969   0.2
## 6  (Intercept)    60 0.6,0.1,0.2     1.6735041    0.5129815   0.0
## 7  (Intercept)    80 0.5,0.1,0.2     1.4298346    0.7336385   0.1
## 8  (Intercept)    80 0.6,0.1,0.2     1.7570838    0.7910724   0.2
## 9  (Intercept)   100 0.5,0.1,0.2     1.7931930    0.7118306   0.2
## 10 (Intercept)   100 0.6,0.1,0.2     2.5996231    0.8413887   0.5
## # ... with 20 more rows, and 3 more variables: num_reject <dbl>,
## #   num_repl <dbl>, data <list>