This section elaborates on the data definition process to provide
more details on how to create data sets.
Formulas
The data definition table for a new data set is constructed
sequentially. As each new row or variable is added, the formula (and in
some cases the variance) can refer back to a previously defined
variable. The first row by necessity cannot refer to another variable,
so the formula must be a specific value (i.e. not a string formula).
Starting with the second row, the formula can either be a value or any
valid R equation with quotes and can include any variables
previously defined.
In the definition we created above, the probability being
female is a function of age, which was
previously defined. Likewise, the number of visits is a
function of both age and female. Since
age is the first row in the table, we had to use a
scalar to define the mean.
def <- defData(varname = "age", dist = "normal", formula = 10,
variance = 2)
def <- defData(def, varname = "female", dist = "binary",
formula = "-2 + age * 0.1", link = "logit")
def <- defData(def, varname = "visits", dist = "poisson",
formula = "1.5 - 0.2 * age + 0.5 * female", link = "log")
Formulas can include R functions or user-defined
functions. Here is an example with a user-defined function
myinv and the log function from base
R:
myinv <- function(x) {
1/x
}
def <- defData(varname = "age", formula = 10, variance = 2,
dist = "normal")
def <- defData(def, varname = "loginvage", formula = "log(myinv(age))",
variance = 0.1, dist = "normal")
genData(5, def)
## Key: <id>
## id age loginvage
## <int> <num> <num>
## 1: 1 10.31 -2.58
## 2: 2 7.90 -1.94
## 3: 3 9.83 -1.93
## 4: 4 9.10 -2.42
## 5: 5 10.18 -2.21
Replication is an important aspect of data simulation - it is often
very useful to generate data under different sets of assumptions.
simstudy facilitates this in at least two different ways.
There is function updateDef which allows row by row changes
of a data definition table. In this case, we are changing the formula of
loginvage in def so that it uses the
function log10 instead of log:
def10 <- updateDef(def, changevar = "loginvage", newformula = "log10(myinv(age))")
def10
## varname formula variance dist link
## <char> <char> <char> <char> <char>
## 1: age 10 2 normal identity
## 2: loginvage log10(myinv(age)) 0.1 normal identity
## Key: <id>
## id age loginvage
## <int> <num> <num>
## 1: 1 9.82 -0.338
## 2: 2 10.97 -0.633
## 3: 3 11.79 -1.267
## 4: 4 9.74 -0.882
## 5: 5 10.11 -1.519
A more powerful feature of simstudy that allows for
dynamic definition tables is the external reference capability through
the double-dot notation. Any variable reference in a formula
that is preceded by “..” refers to an externally defined variable:
age_effect <- 3
def <- defData(varname = "age", formula = 10, variance = 2,
dist = "normal")
def <- defData(def, varname = "agemult", formula = "age * ..age_effect",
dist = "nonrandom")
def
## varname formula variance dist link
## <char> <char> <num> <char> <char>
## 1: age 10 2 normal identity
## 2: agemult age * ..age_effect 0 nonrandom identity
## Key: <id>
## id age agemult
## <int> <num> <num>
## 1: 1 9.69 29.1
## 2: 2 9.63 28.9
But the real power of dynamic definition is seen in the context of
iteration over multiple values:
age_effects <- c(0, 5, 10)
list_of_data <- list()
for (i in seq_along(age_effects)) {
age_effect <- age_effects[i]
list_of_data[[i]] <- genData(2, def)
}
list_of_data
## [[1]]
## Key: <id>
## id age agemult
## <int> <num> <num>
## 1: 1 11.4 0
## 2: 2 10.7 0
##
## [[2]]
## Key: <id>
## id age agemult
## <int> <num> <num>
## 1: 1 11.3 56.6
## 2: 2 11.2 56.1
##
## [[3]]
## Key: <id>
## id age agemult
## <int> <num> <num>
## 1: 1 9.32 93.2
## 2: 2 10.62 106.2
Distributions
The foundation of generating data is the assumptions we make about
the distribution of each variable. simstudy provides 15
types of distributions, which are listed in the following table:
| beta |
mean |
both |
- |
dispersion |
X |
- |
X |
| binary |
probability |
both |
- |
- |
X |
X |
X |
| binomial |
probability |
both |
- |
# of trials |
X |
X |
X |
| categorical |
probability |
string |
p_1;p_2;…;p_n |
a;b;c |
X |
- |
X |
| clusterSize |
total N |
both |
- |
dispersion |
X |
- |
- |
| custom |
function |
string |
- |
arguments |
X |
- |
- |
| exponential |
mean |
both |
- |
- |
X |
X |
- |
| gamma |
mean |
both |
- |
dispersion |
X |
X |
- |
| mixture |
formula |
string |
x_1 | p_1 + … + x_n | p_n |
- |
X |
- |
- |
| negBinomial |
mean |
both |
- |
dispersion |
X |
X |
- |
| nonrandom |
formula |
both |
- |
- |
X |
- |
- |
| normal |
mean |
both |
- |
variance |
X |
- |
- |
| noZeroPoisson |
mean |
both |
- |
- |
X |
X |
- |
| poisson |
mean |
both |
- |
- |
X |
X |
- |
| trtAssign |
ratio |
string |
r_1;r_2;…;r_n |
stratification |
X |
X |
- |
| uniform |
range |
string |
from ; to |
- |
X |
- |
- |
| uniformInt |
range |
string |
from ; to |
- |
X |
- |
- |
beta
A beta distribution is a continuous data distribution that
takes on values between \(0\) and \(1\). The formula specifies the
mean \(\mu\) (with the ‘identity’ link)
or the log-odds of the mean (with the ‘logit’ link). The scalar value in
the ‘variance’ represents the dispersion value \(d\). The variance \(\sigma^2\) for a beta distributed variable
will be \(\mu (1- \mu)/(1 + d)\).
Typically, the beta distribution is specified using two shape parameters
\(\alpha\) and \(\beta\), where \(\mu = \alpha/(\alpha + \beta)\) and \(\sigma^2 = \alpha\beta/[(\alpha + \beta)^2 (\alpha
+ \beta + 1)]\).
binary
A binary distribution is a discrete data distribution that
takes values \(0\) or \(1\). (It is more conventionally called a
Bernoulli distribution, or is a binomial distribution
with a single trial \(n=1\).) The
formula represents the probability (with the ‘identity’
link), the relative risk (with the ‘log’ link), or the log odds (with
the ‘logit’ link) that the variable takes the value of 1. The mean of
this distribution is \(p\), and
variance \(\sigma^2\) is \(p(1-p)\).
binomial
A binomial distribution is a discrete data distribution that
represents the count of the number of successes given a number of
trials. The formula specifies the probability of success (with the
‘identity’ link), the relative risk (with the ‘log’ link), or the log
odds (with the ‘logit’ link) that the variable takes the value of 1. and
the variance field is used to specify the number of trials \(n\). Given a value of \(p\), the mean \(\mu\) of this distribution is \(n*p\), and the variance \(\sigma^2\) is \(np(1-p)\).
categorical
A categorical distribution is a discrete data distribution
taking on values from \(1\) to \(K\), with each value representing a
specific category, and there are \(K\)
categories. The categories may or may not be ordered. For a categorical
variable with \(k\) categories, the
formula is a string of probabilities that sum to 1, each
separated by a semi-colon: \((p_1 ; p_2 ; ...
; p_k)\). \(p_1\) is the
probability of the random variable falling in category \(1\), \(p_2\) is the probability of category \(2\), etc. The probabilities can be
specified as functions of other variables previously defined. The helper
function genCatFormula is an easy way to create different
probability strings. The link options are identity
or logit. The variance field is optional an allows
to provide categories other than the default 1...n in the
same format as formula: “a;b;c”. Numeric variance Strings
(e.g. “50;100;200”) will be converted to numeric when possible. All
probabilities will be rounded to 1e12 decimal points to prevent possible
rounding errors.
clusterSize
The clusterSize distribution allocates a total sample size
N (specified in the formula argument) across the
k rows of the data.table such that the sum of the rows equals
N. If the dispersion argument is set to 0, the
allocation to each row is N/k, with some rows getting an
increment of 1 to ensure that the sum is N. It is possible to relax the
assumption of balanced cluster sizes by setting dispersion >
0. As the dispersion increases, the variability of cluster sizes
across clusters increases.
custom
Custom distributions can be specified in defData and
defDataAdd by setting the argument dist to
“custom”. When defining a custom distribution, provide the name of the
user-defined function as a string in the formula argument. The
arguments of the custom function are listed in the variance
argument, separated by commas and formatted as “arg_1 =
val_form_1, arg_2 = val_form_2, \(\dots\), arg_K = val_form_K”. The
arg_k’s represent the names of the arguments passed to the
customized function, where \(k\) ranges
from \(1\) to \(K\). Values or formulas can be used for
each val_form_k. If formulas are used, ensure that the
variables have been previously generated. Double dot notation is
available in specifying value_formula_k. One important
requirement of the custom function is that the parameter list used to
define the function must include an argument”n = n”,
but do not include \(n\) in the
definition as part of defData or
defDataAdd.
exponential
An exponential distribution is a continuous data
distribution that takes on non-negative values. The formula
represents the mean \(\theta\) (with
the ‘identity’ link) or log of the mean (with the ‘log’ link). The
variance argument does not apply to the
exponential distribution. The variance \(\sigma^2\) is \(\theta^2\).
gamma
A gamma distribution is a continuous data distribution that
takes on non-negative values. The formula specifies the
mean \(\mu\) (with the ‘identity’ link)
or the log of the mean (with the ‘log’ link). The variance
field represents a dispersion value \(d\). The variance \(\sigma^2\) is is \(d \mu^2\).
mixture
The mixture distribution is a mixture of other predefined
variables, which can be defined based on any of the other available
distributions. The formula is a string structured with a sequence of
variables \(x_i\) and probabilities
\(p_i\): \(x_1 | p_1 + … + x_n | p_n\). All of the
\(x_i\)’s are required to have been
previously defined, and the probabilities must sum to \(1\) (i.e. \(\sum_1^n p_i = 1\)). The result of
generating from a mixture is the value \(x_i\) with probability \(p_i\). The variance and
link fields do not apply to the mixture
distribution.
negBinomial
A negative binomial distribution is a discrete data
distribution that represents the number of successes that occur in a
sequence of Bernoulli trials before a specified number of
failures occurs. It is often used to model count data more generally
when a Poisson distribution is not considered appropriate; the
variance of the negative binomial distribution is larger than the
Poisson distribution. The formula specifies the
mean \(\mu\) or the log of the mean.
The variance field represents a dispersion value \(d\). The variance \(\sigma^2\) will be \(\mu + d\mu^2\).
nonrandom
Deterministic data can be “generated” using the nonrandom
distribution. The formula explicitly represents the value
of the variable to be generated, without any uncertainty. The
variance and link fields do not apply to
nonrandom data generation.
normal
A normal or Gaussian distribution is a continuous
data distribution that takes on values between \(-\infty\) and \(\infty\). The formula
represents the mean \(\mu\) and the
variance represents \(\sigma^2\). The link field is
not applied to the normal distribution.
noZeroPoisson
The noZeroPoisson distribution is a discrete data
distribution that takes on positive integers. This is a truncated
poisson distribution that excludes \(0\). The formula specifies the
parameter \(\lambda\) (link is
‘identity’) or log() (link is log). The
variance field does not apply to this distribution. The
mean \(\mu\) of this distribution is
\(\lambda/(1-e^{-\lambda})\) and the
variance \(\sigma^2\) is \((\lambda + \lambda^2)/(1-e^{-\lambda}) -
\lambda^2/(1-e^{-\lambda})^2\). We are not typically interested
in modeling data drawn from this distribution (except in the case of a
hurdle model), but it is useful to generate positive count data
where it is not desirable to have any \(0\) values.
poisson
The poisson distribution is a discrete data distribution
that takes on non-negative integers. The formula specifies
the mean \(\lambda\) (link is
‘identity’) or log of the mean (link is log). The
variance field does not apply to this distribution. The
variance \(\sigma^2\) is \(\lambda\) itself.
trtAssign
The trtAssign distribution is an implementation of the
trtAssign functionality in the context of the
defData workflow. Sometimes, it might be convenient to
assign treatment or group membership while defining other variables. The
formula specifies the relative allocation to the different
groups. For example three-arm randomization with equal allocation to
each arm would be specified as “1;1;1”. The
variance field defines the stratification variables, and
would be specified as “s_1;s_2” if s_1 and
s_2 are the stratification variables. The link
field is used to indicate if the allocations should be perfectly
balanced; if nothing is specified (and link defaults to
identity), the allocation will be balanced; if link is
specified to be different from identity, then the allocation
will not be balanced.
