The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Prospective and retrospective sequential meta-analysis

Purpose

This vignette is used to introduce two types of sequential meta-analyses: prospective and retrospective. It will also describe how the different types of meta-analysis will be handled in a sequential setting using RTSA.

Introduction

A sequential meta-analysis is a meta-analysis which is updated during its lifetime e.g. when new evidence has been created. We will in this section briefly describe the difference between prospective and retrospective sequential meta-analysis. For more information and a rich description on prospective meta-analysis, the Cochrane Methods section on prospective meta-analysis is must-read and can be found here.

When is the meta-analysis prospective

For a meta-analysis to be prospective, all of the trial results, which are intended to be used in the meta-analysis, must be unknown. If this criteria is fulfilledn the research hypotheses and design of the meta-analysis must also be made public in a protocol prior to knowing the results of the trials. If any are known, the knowledge can influence the design of the meta-analysis and cause bias. For a guide on how to make a prospective meta-analysis, see Seidler et al. 2019 (https://doi.org/10.1136/bmj.l5342).

If any trial results are known or the protocol/design of the meta-analysis is based on knowledge from related trials, the meta-analysis is retrospective.

Prospective > retrospective

If meta-analysis is the gold standard of evidence, then the prospective meta-analysis must be the diamond standard of evidence. One should aim for being as close to a prospective meta-analysis as possible. This protects from several potential biases potentially present in retrospective meta-analysis. Some of the potential biases in retrospective meta-analysis includes:

Prospective and retrospective sequential meta-analysis

One can use RTSA for both prospective and retrospective sequential meta-analysis.

To make a prospective meta-analysis using RTSA, a design is required. One can design a meta-analysis using the RTSA() function by specifying that the type of analysis is design:

RTSA(type = "design",...)

Storing the RTSA design object and using it as a argument in the RTSA() function when data has been collected ensures that the design of the meta-analysis is not dependent on the data.

pro_design <- RTSA(type = "design",...)
pro_analysis <- RTSA(type = "analysis", design = pro_design, 
                     data = trials)

In the retrospective meta-analysis, there is no pre-existing design and the RTSA() function is provided with all design arguments such as alpha (type-I-error level), beta (type-II-error level), side (whether the analysis is one- or two-sided), es_alpha (the error spending function for alpha), futility (if any futility - which one), and more.

To run a retrospective meta-analysis in RTSA one sets the type to design. The difference between a retrospective and prospective analysis is that the retrospective analysis does not require a design.

retro_analysis <- RTSA(type = "analysis", alpha = alpha_level, 
                       beta = beta_level, data = trials, ...)

We will now provide two examples of respectively a prospective sequential meta-analysis and a retrospective sequential meta-analysis.

Prospective sequential meta-analysis

Let \(X\) be the treatment difference between treatment \(A\) and \(B\) such as a mean difference. Suppose that the research question of interest is: “Is X different from 0?”. Our null hypothesis is then that \(X=0\). Consider the scenario that 0.5 is a minimal clinical value of interest for the difference between treatment \(A\) and \(B\) and we expect a standard deviation of 1. We want to test at a 5% significance level at 90% power.

No trials on the topic exists, however three research centers have planned to run close to identical trials investigating the treatment difference between treatment \(A\) and \(B\). Each center can not individually recruit a sufficient number of participants to have a well-powered trial, hence they want to combine their trials in a meta-analysis. They decide that the number of participants should be split such that the first center accounts for 50% of the sample, whereas the remaining two account for 25%.

The centers want to stop and allocate the patients to the most effective treatment as soon as possible if early findings of efficacy of any of the treatments are found. They choose a Lan and DeMets’ version of O’Brien-Fleming boundaries for enabling early stopping. They are not interested in stopping for futility.

Regarding heterogeneity and the interpretation of the meta-analysis, they decide to use a fixed-effect meta-analysis based on the trials are close to homogeneous. However, presence of heterogeneity will be investigated.

We can then use RSTA to plan the prospective sequential meta-analysis. One can perform a prospective meta-analysis in RTSA by creating a design:

library(RTSA)
design_pma <- RTSA(type = "design", outcome = "MD", alpha = 0.05, beta = 0.1,
                   mc = 0.5, sd_mc = 1,side = 2, timing = c(0.5,0.75,1),
                   es_alpha = "esOF", fixed = TRUE, weights = "IV")
design_pma
## Design with Trial Sequential Analysis was computed with the following settings: 
##  
## Boundaries for a 2-sided design with a type-I-error of 0.05, and type-II-error of 0.1.
## Futility is set to: none. Alpha-spending function: esOF.
## Beta-spending function: .
## 
## The required information size is not adjusted by heterogeneity. The required information size is further increased with 2 percent due to the sequential design. The total required information size is 173.
## 
## Timing, and boundaries:
##  sma_timing upper  lower
##       0.509 2.963 -2.963
##       0.764 2.359 -2.359
##       1.018 2.014 -2.014
## sma_timing is the ratio of the required sample for a sequential meta-analysis to a non-sequential meta-analysis sample size.
## 
## Sample size calculation for standard meta-analysis:
## This is a prospective meta-analysis sample size calculation.
## The sample size calculation assumes a 2-sided test, equal group sizes,
## a type-I-error of 0.05 and a type-II-error of 0.1. 
## The minimum clinical relevant value is set to: 0.5 for outcome metric MD.
## Additional parameters for sample size are:
## Standard deviation of the mean (standard error):  1.
## 
## Fixed-effect required information size:
## 169 participants in total. 
## 
## For more information about the sample size calculation see vignette:
## 'Calculating required sample size and required number of trials'.
## 
## Sample size calculation for sequential meta-analysis:
## Fixed-effect: 173 participants.
## 
## Please note the following warnings:
## - The RTSA function is used for design. Boundaries are computed but sequential inference will not be calculated. Use the metaanalysis() function if interested in meta-analysis results.

The output provides the required number of participants and boundaries for testing in a sequential meta-analysis. Hence for the three trials, the samples can be split as follows: 87, 43.5, 43.5.

Suppose that the first trial is done. We can then update the prospective meta-analysis by using the design in the call:

set.seed(0702)
treatA <- rnorm(n = ceiling(87/2), mean = 1.0, sd = 1)
treatB <- rnorm(n = ceiling(87/2), mean = 0.5, sd = 1)
trial1 <- data.frame(mI = mean(treatB), mC = mean(treatA), sdI = sd(treatB),
                     sdC = sd(treatA), nI = ceiling(87/2), nC = ceiling(87/2))
x <- RTSA(type = "analysis", design = design_pma, data = trial1)
x
## Trial Sequential Analysis was computed with the following settings: 
##  
## Boundaries for a 2-sided design with a type-I-error of 0.05, and type-II-error of 0.1.
## Futility is set to: none. Alpha-spending function: esOF.
## Beta-spending function: .
## 
## The required information size is not adjusted by heterogeneity. The required information size is further increased with 2 percent due to the sequential design. The total required information size is 173.
## 
## Timing, boundaries, and test statistic:
##  sma_timing upper  lower fut_upper fut_lower z_fixed z_random
##       0.509 2.964 -2.964        NA        NA  -1.661   -1.661
##       1.018 1.969 -1.969        NA        NA      NA       NA
## sma_timing is the ratio of the required sample for a sequential meta-analysis to a non-sequential meta-analysis sample size.
## 
## Timing, outcomes, and confidences intervals for fixed-effect and random-effects models:
##  sma_timing MD_fixed TSA_0.95lci_fixed TSA_0.95uci_fixed
##       0.509   -0.339            -0.944             0.266
##       1.018       NA                NA                NA
##  sma_timing MD_random TSA_0.95lci_random TSA_0.95uci_random
##       0.509    -0.339             -0.944              0.266
##       1.018        NA                 NA                 NA
## lci is the lower limit of the confidence interval. uci is the upper limit of the confidence interval.
## 
## Meta-analysis results:
## Fixed pooled effect (MD): -0.34 (95% TSA-adjusted CI: -0.94;0.27), naive p-value: 0.0967
## 
## Heterogeneity results:
## tau^2: 0.00; I^2: 0.0%; D^2: 0.0%; Heterogeneity p-value: 1.0000
## 
## Please note the following warnings:
## - The order of the Trial Sequential Analysis will be based on the order of the studies in the data-set. Please add a 'order' column in the data-set to specify the order.

Type-I-error in sequential meta-analysis

The type-I-error will be controlled in a prospective sequential meta-analysis for at least fixed-effect meta-analyses, when first being designed and then updated using the same design. Changes to the original design such as having an additional trial will not impact type-I-error however it can impact the power of the meta-analysis. This is also true for unaccounted for presence of heterogeneity. If there is presence of heterogeneity then the trial will be under-powered but again it will not affect type-I-error control.

A paper reviewing some of these scenarios where we diverge from the design is under way. Both for the prospective meta-analysis and the retrospective sequential meta-analysis.

Retrospective sequential meta-analysis

The sequential retrospective meta-analysis can for many reasons be biased by the knowledge of results of trials used in the meta-analysis. Previous knowledge can affect the research hypothesis, which of the trials to be included in the meta-analysis, and more. Furthermore, the set of trials available might be overly positive due to publication bias. Most of these biases will cause a potential increase in type-I-error which cannot be handled by TSA.

Another cause of increase in type-I-error is the potential adaption of the method to the data of retrospective meta-analysis. These include leveraging information from the trials intended to use in the sequential meta-analysis. Such information could be the estimate of heterogeneity, or for binary data the probability of event in the control group. For good reasons, one might want to use the estimate of heterogeneity in the sample size calculation of the retrospective meta-analysis. However, the effect of being data-adaptive and not sticking to a pre-specified design can affect the control of type-I-error.

If one can accept not to use p-values as decision markers, a sequential retrospective meta-analysis is still of value. The method will provide the users with:

  1. Control of type-I-error in the scenario that the current state is the truth.
  2. A reminder user that the hypothesis has been tested multiple times and the type-I-error rate is affected.
  3. Quantification whether the meta-analysis is sufficiently powered or more trials are required to reach the wanted level of power.

An example of a retrospective meta-analysis is the eds data from the RTSA package. Here there is no design, so the function requires all of the relevant design settings such as setting the spending functions, type-I- and type-II-error levels, etc.

eds <- eds[order(eds$year),]
ex_retro_rtsa <- RTSA(type = "analysis", data = eds, side = 2, outcome = "MD",
                      alpha = 0.05, beta = 0.1, futility = "none", 
                      fixed = TRUE, es_alpha = "esOF", mc = -1,
                      ana_times = c(3,6,9))
## Warning in ris(outcome = outcome, mc = mc, side = side, alpha = alpha, beta =
## beta, : There is presence of tau2 (heterogeneity) and hence I2 and/or D2 is not
## 0. Consider changing fixed to FALSE.
plot(ex_retro_rtsa)

Another example is the coronary data from the package.

ex2_retro_rtsa <- RTSA(type = "analysis", data = coronary, side = 2, outcome = "OR",
                      alpha = 0.05, beta = 0.1, futility = "non-binding", 
                      fixed = FALSE, es_alpha = "esOF", es_beta = "esPoc", mc = 0.9)
plot(ex2_retro_rtsa)

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.