1. Designing Safe Experiments
Before we can start an experiment, to be able to do optional stopping with safe tests, we have to decide on the incoming data block size. A block is a set of data consisting of \(n_a\) outcomes in group \(a\) and \(n_b\) outcomes in group \(b\), for some pre-specified \(n_a\) and \(n_b\). One is allowed to perform a safe test and look at the results each time a new block is complete. Therefore, one should choose these blocks as small as possible, enabling the user to collect information as quickly as possible. For example, if we consider a balanced design, it is best to choose \(n_a = 1\) and \(n_b = 1\). Imagine a clinical trial researcher starting up their computer after the weekend and seeing that data of 15 patients have been processed in a control group, and 12 in a case group. The researcher could then perform the safe test with the data of the first 12 patients from each group, as 12 complete “blocks” with one patient from each group are present.
The safe tests functions expect numeric vectors giving the number of successes per block as arguments. An example of observed data in the correct format for the package with 10 blocks with \(n_a = n_b = 1\) could be:
ya <- c(1,1,1,1,1,1,0,1,1,1)
yb <- c(0,0,1,0,1,0,0,0,0,0)Case (I): Designing experiments with the minimal clinically relevant effect size known
We might know the minimal clinically relevant effect size upfront for our test of two proportions. For example, we might only be interested in further researching or developing a drug when the difference (“delta” in the code) in the proportion of cured patients in the treatment group compared to the placebo group is at least 0.3. In practice this implies, for example, that when 20% of patients get cured on average in the placebo group, we want the drug to add at least 30% to this average, so in the treated group 50% of patients should be cured. We could design a safe test that achieves a power of 80% while testing at the relatively standard significance level of 0.05 in this setting:
balancedSafeDesign <- designSafeTwoProportions(
na = 1,
nb = 1,
alpha = 0.05, #significance level for testing
beta = 1 - 0.8, #power 80% equals type II error (beta) of 0.2
delta = 0.3
)print(balancedSafeDesign)
#>
#> Safe Test of Two Proportions Design
#>
#> na±2se, nb±2se, nBlocksPlan±2se = 1±0, 1±0, 95±5.042694
#> minimal difference = 0.3
#> alternative = twoSided
#> alternative restriction = none
#> power: 1 - beta = 0.8
#> parameter: Beta hyperparameters = standard, REGRET optimal
#> alpha = 0.05
#> decision rule: e-value > 1/alpha = 20
#>
#> Timestamp: 2022-11-23 10:15:05 CET
#>
#> Note: Optimality of hyperparameters only verified for equal group sizes (na = nb = 1)For detecting this difference with a power of at least 80%, while testing at significance level 0.05, we would need to observe 95 data blocks.
A safe test could now be performed with this design object; for this, some mock data are generated below:
set.seed(19012022)
successProbabilityA <- 0.2
successProbabilityB <- 0.5
ya <- rbinom(n = balancedSafeDesign[["nPlan"]]["nBlocksPlan"], size = 1, prob = successProbabilityA)
yb <- rbinom(n = balancedSafeDesign[["nPlan"]]["nBlocksPlan"], size = 1, prob = successProbabilityB)
graphics::plot(x = seq_along(ya), y = cumsum(ya), type = "p", ylim = c(0, max(c(sum(ya), sum(yb)))),
xlab = "block number", ylab = "# successes",
main = "Total # successes per group over time")
graphics::points(x = seq_along(yb), y = cumsum(yb), type = "p", col = "grey")
graphics::legend(x = "topleft", legend = c("A", "B"), col = c("black", "grey"), pch = 1)
Performing the safe test reveals:
safe.prop.test(ya = ya, yb = yb, designObj = balancedSafeDesign)
#>
#> Safe Test of Two Proportions
#>
#> data: ya and yb. nObsA = 95, nObsB = 95
#>
#> test: Beta hyperparameters = standard, REGRET optimal
#> e-value = 48222 > 1/alpha = 20 : TRUE
#> alternative hypothesis: true difference between proportions in group a and b is not equal to 0
#>
#> design: the test was designed with alpha = 0.05
#> for experiments with na = 1, nb = 1, nBlocksPlan = 95
#> to guarantee a power = 0.8 (beta = 0.2)
#> for minimal relevant difference = 0.3 (twoSided)In this example, we would reject the null hypothesis that the true difference between the success probabilities in the groups receiving either the new treatment or a placebo is equal to 0. The interpretation of the “e-values” calculated in safe tests is the opposite of that of p-values: the higher the e-value, the more evidence for the alternative hypothesis we have. The e-value exceeds the significance threshold of \(1 / \alpha = 1 / 0.05 = 20\), hence, we reject the null hypothesis. More details and the surprisingly simple proof of the type-I error guarantee of e-values can be found in the original safe testing paper, on page 3 (Grünwald, 2019).
Case (II): Minimal clinically relevant effect size unknown, but maximum number of samples known.
We might not have enough resources to fund our study to detect the minimal difference of 0.3. For example, we might only have funds to treat 50 patients in each group, so 100 in total. If this is the case, we could inspect the minimal number of patients we need for the experiment to achieve a power of 80% at our significance level per effect size of interest in the worst case:
#make the plot voor standard parameter settings
standardPrior <- list(betaA1 = 0.18,
betaA2 = 0.18,
betaB1 = 0.18,
betaB2 = 0.18)
#set a seed for the simulations
set.seed(1082021)
sampleSizePerMinimalDifference <- simulateTwoProportions(list(standard = standardPrior),
alternativeRestriction = "none",
alpha = 0.05,
beta = 0.2,
na = 1,
nb = 1,
deltamin = 0.3,
deltamax = 0.9
)graphics::plot(sampleSizePerMinimalDifference)
The simulateTwoProportions function generates experiments based on the minimal difference corresponding to the x-axis and carries out a simulation with optional stopping, i.e., experiments were stopped when \(e > 1/\alpha=20\) was observed. The realised average number of patients in the worst case was calculated, shown as worst case, depicting the worst-case sample size we need to plan for if we want to reach 80% power. Observe that the smallest absolute difference detectable with our available resources is 0.47 if we want the worst-case power guarantee; we might have to cancel the study, or try to acquire more research funds. With our current funds, we can not guarantee a high enough power for detecting the difference between the groups we are interested in. This implies that, when a non-significant result is obtained, we would be unsure whether this was caused by our underpowered study, or because there was really no difference between the groups.
Furthermore, the plot also shows the expected sample sizes under optional stopping. To retrieve these, the simulateTwoProportions function carries out simulations with optional stopping where experiments were stopped when \(e > 1/\alpha=20\) was observed, or when the worst-case sample size for the 80% power guarantee was reached. Observe that the difference between the planned sample size and the expected sample size under optional stopping is substantial. In section 2, the behaviour of the safe test for two proportions and Fisher’s exact test under optional stopping is studied further.
Short examples of non-standard design settings
Unequal group sizes
What happens if we want to have \(n_a \neq n_b\) within the data blocks? For example, in a case-control study where we expect to collect twice as many controls as cases? We can incorporate this into our safe design:
unbalancedSafeDesign <- designSafeTwoProportions(
na = 2,
nb = 1,
alpha = 0.05,
beta = 0.2,
delta = 0.3
)We can then again perform a safe test each time a new complete data block has become available., i.e., when we have collected data of two new patients in the control group and one in the case group. The data we use for the test again comprises of the number of successes per group per data block, and could for example look something like the following:
set.seed(692021)
yaUnbalanced <- rbinom(n = unbalancedSafeDesign[["nPlan"]]["nBlocksPlan"], size = 2, prob = successProbabilityA)
ybUnbalanced <- rbinom(n = unbalancedSafeDesign[["nPlan"]]["nBlocksPlan"], size = 1, prob = successProbabilityB)
print(yaUnbalanced)
#> [1] 0 1 0 0 0 1 1 1 0 0 0 0 0 0 2 0 1 0 1 1 2 2 0 1 0 1 1 1 0 1 1 0 2 0 0 1 2 1
#> [39] 1 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 1 0 2 0 0 0 1 0 1 1 0 0 0 0 0
#> [77] 0 0 0
print(ybUnbalanced)
#> [1] 1 1 1 0 1 1 0 1 1 0 1 0 1 1 1 0 0 1 0 0 0 0 1 0 1 1 1 0 0 0 1 0 0 1 0 0 1 1
#> [39] 0 0 0 0 1 1 0 0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 0 1 1 0 1 0 0 1 1 1 1 1 0 1 1 1
#> [77] 0 0 0Notice that in the control group, we can now observe \(0\), \(1\) or \(2\) successes per data block, as we observe a total of 2 patients per block. We can perform the safe test with this design object and data exactly as before:
safeTwoProportionsTest(ya = yaUnbalanced, yb = ybUnbalanced, designObj = unbalancedSafeDesign)
#>
#> Safe Test of Two Proportions
#>
#> data: yaUnbalanced and ybUnbalanced. nObsA = 158, nObsB = 79
#>
#> test: Beta hyperparameters = standard, REGRET optimal
#> e-value = 1.8097 > 1/alpha = 20 : FALSE
#> alternative hypothesis: true difference between proportions in group a and b is not equal to 0
#>
#> design: the test was designed with alpha = 0.05
#> for experiments with na = 2, nb = 1, nBlocksPlan = 79
#> to guarantee a power = 0.8 (beta = 0.2)
#> for minimal relevant difference = 0.3 (twoSided)Non-standard Beta priors
When the safe design for the unbalanced design is printed, one might notice that a note is printed about Beta prior parameter settings:
print(unbalancedSafeDesign)
#>
#> Safe Test of Two Proportions Design
#>
#> na±2se, nb±2se, nBlocksPlan±2se = 2±0, 1±0, 79±4.043125
#> minimal difference = 0.3
#> alternative = twoSided
#> alternative restriction = none
#> power: 1 - beta = 0.8
#> parameter: Beta hyperparameters = standard, REGRET optimal
#> alpha = 0.05
#> decision rule: e-value > 1/alpha = 20
#>
#> Timestamp: 2022-11-23 10:17:06 CET
#>
#> Note: Optimality of hyperparameters only verified for equal group sizes (na = nb = 1)The test for two proportions “learns” from the data as it comes in during the optional stopping process; the speed with which this happens is influenced by these Beta prior parameter settings. These have been optimised with respect to power and a notion of performance called regret for the balanced trial setting; details can be found in Turner et al., 2021. In numerical experiments, good performance for various unbalanced designs have been observed for these standard prior settings. However, one might want to experiment with different settings, for which the following code could be used. In this example, we compare an equivalent of a uniform prior to the standard prior settings of the safe design:
#make the plot voor standard parameter settings
standardPrior <- unbalancedSafeDesign[["betaPriorParameterValues"]]
uniformLikePrior <- list(betaA1 = 2, # pretend we "have seen" 2 success and
betaA2 = 2,# 2 failure in group A before starting
betaB1 = 1, # and 1 success and failure in group B
betaB2 = 1)
#set a seed for the simulations
set.seed(1082021)
sampleSizePerMinimalDifference <- simulateTwoProportions(
hyperparameterList = list(standard = standardPrior, uniform = uniformLikePrior),
alternativeRestriction = "none",
alpha = 0.05,
beta = 0.2,
na = 2,
nb = 1,
deltamin = 0.3,
deltamax = 0.9
)graphics::plot(sampleSizePerMinimalDifference)
From this plot, it can be observed that worst-case and expected sample sizes are lower when using the uniform-like prior settings for smaller differences between groups. When carrying out the study in the unbalanced setting, it might be profitable to use this non-standard prior.
Restricting the alternative hypothesis
Above we already shortly mentioned that the safe test learns from the data stream during the optional stopping process. When one would be quite certain of a difference they expect to detect between the groups before starting the experiment, this could also be used to speed up this learning process: the safe test could be “optimized” to detect differences of only a certain size and larger. For example, we could restrict the alternative hypothesis of our safe design to all instances where the difference between the groups is 0.3 or bigger. This can be easily done by adding one argument to the designSafeTwoProportions function:
differenceRestrictedSafeDesign <- designSafeTwoProportions(
na = 1,
nb = 1,
alpha = 0.05, #significance level for testing
beta = 1 - 0.8, #power 80% equals type II error (beta) of 0.2
delta = 0.3,
alternativeRestriction = "difference" #also available: logOddsRatio
)When can use this new design on the data obtained for the unrestricted balanced safe design above:
safe.prop.test(ya = ya, yb = yb, designObj = differenceRestrictedSafeDesign)
#>
#> Safe Test of Two Proportions
#>
#> data: ya and yb. nObsA = 95, nObsB = 95
#>
#> test: Beta hyperparameters = standard, REGRET optimal
#> e-value = 337447 > 1/alpha = 20 : TRUE
#> alternative hypothesis: true difference between proportions in group a and b is greater than 0
#>
#> design: the test was designed with alpha = 0.05
#> for experiments with na = 1, nb = 1, nBlocksPlan = 54
#> to guarantee a power = 0.8 (beta = 0.2)
#> for minimal relevant difference = 0.3 (greater)We see that we now have obtained a much higher e-value, i.e., more evidence for the alternative hypothesis. We can also see in the design that if the true difference between the groups would be 0.3, we would only need to collect 54 data blocks instead of 95. However, one needs to be careful: if the true difference would be smaller than 0.3, we could miss evidence and would need to continue experiments longer than with the unrestricted safe tests: a good illustration can be found in Turner et al., 2021, figure 4.
