Example: Bootstrap Confidence Intervals for A/B Testing

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Overview

Bootstrap resampling is a powerful statistical technique for estimating confidence intervals without assuming a specific distribution. This example demonstrates parallelizing bootstrap analysis for A/B test results.

Use Case: A/B testing, hypothesis testing, statistical inference, conversion rate analysis

Computational Pattern: Embarrassingly parallel resampling with aggregation

The Problem

You’ve run an A/B test on your website with two variants: - Variant A (Control): 10,000 visitors, 850 conversions (8.5% conversion rate) - Variant B (Treatment): 10,000 visitors, 920 conversions (9.2% conversion rate)

You need to: 1. Estimate the confidence interval for the difference in conversion rates 2. Calculate the probability that B is better than A 3. Determine if the result is statistically significant

Traditional parametric tests assume normal distributions. Bootstrap resampling makes no such assumptions and provides more robust estimates.

Setup

library(starburst)

Generate Sample Data

Create synthetic A/B test data:

set.seed(42)

# Variant A (control)
n_a <- 10000
conversions_a <- 850
variant_a <- c(rep(1, conversions_a), rep(0, n_a - conversions_a))

# Variant B (treatment)
n_b <- 10000
conversions_b <- 920
variant_b <- c(rep(1, conversions_b), rep(0, n_b - conversions_b))

# Observed difference
observed_diff <- mean(variant_b) - mean(variant_a)
cat(sprintf("Observed conversion rates:\n"))
cat(sprintf("  Variant A: %.2f%%\n", mean(variant_a) * 100))
cat(sprintf("  Variant B: %.2f%%\n", mean(variant_b) * 100))
cat(sprintf("  Difference: %.2f%% (%.1f%% relative lift)\n",
            observed_diff * 100,
            (observed_diff / mean(variant_a)) * 100))

Output:

Observed conversion rates:
  Variant A: 8.50%
  Variant B: 9.20%
  Difference: 0.70% (8.2% relative lift)

Bootstrap Function

Define a function that performs one bootstrap iteration:

bootstrap_iteration <- function(iter, data_a, data_b) {
  # Resample with replacement
  n_a <- length(data_a)
  n_b <- length(data_b)

  sample_a <- sample(data_a, n_a, replace = TRUE)
  sample_b <- sample(data_b, n_b, replace = TRUE)

  # Calculate metrics
  rate_a <- mean(sample_a)
  rate_b <- mean(sample_b)
  diff <- rate_b - rate_a
  relative_lift <- diff / rate_a

  list(
    iteration = iter,
    rate_a = rate_a,
    rate_b = rate_b,
    diff = diff,
    relative_lift = relative_lift,
    b_wins = diff > 0
  )
}

Local Execution

Run a smaller bootstrap locally:

n_bootstrap_local <- 1000

cat(sprintf("Running %d bootstrap iterations locally...\n", n_bootstrap_local))
local_start <- Sys.time()

local_results <- lapply(
  1:n_bootstrap_local,
  bootstrap_iteration,
  data_a = variant_a,
  data_b = variant_b
)

local_time <- as.numeric(difftime(Sys.time(), local_start, units = "secs"))
cat(sprintf("[OK] Completed in %.2f seconds\n\n", local_time))

Illustrative output:

Running 1000 bootstrap iterations locally...
[OK] Completed in 2.1 seconds

Cloud Execution with staRburst

Run 10,000 bootstrap iterations on AWS:

n_bootstrap <- 10000

cat(sprintf("Running %d bootstrap iterations on AWS...\n", n_bootstrap))

results <- starburst_map(
  1:n_bootstrap,
  bootstrap_iteration,
  data_a = variant_a,
  data_b = variant_b,
  workers = 25,
  cpu = 1,
  memory = "2GB"
)

Illustrative output:

[Starting] Starting starburst cluster with 25 workers
[Status] Processing 10000 items with 25 workers
[Starting] Submitting 10000 tasks...
[Wait] Progress: 10000/10000 (18.0s)
[OK] Completed in 18.0 seconds
[Cost] Estimated cost: $0.01

Results Analysis

Extract and analyze the bootstrap distribution:

# Extract metrics
diffs <- sapply(results, function(x) x$diff)
relative_lifts <- sapply(results, function(x) x$relative_lift)
b_wins <- sapply(results, function(x) x$b_wins)

# Calculate confidence intervals
ci_95 <- quantile(diffs, c(0.025, 0.975))
ci_99 <- quantile(diffs, c(0.005, 0.995))

# Probability that B is better than A
prob_b_wins <- mean(b_wins) * 100

# Print results
cat("\n=== Bootstrap Results (10,000 iterations) ===\n\n")
cat(sprintf("Observed difference: %.2f%%\n", observed_diff * 100))
cat(sprintf("\n95%% Confidence Interval: [%.2f%%, %.2f%%]\n",
            ci_95[1] * 100, ci_95[2] * 100))
cat(sprintf("99%% Confidence Interval: [%.2f%%, %.2f%%]\n",
            ci_99[1] * 100, ci_99[2] * 100))
cat(sprintf("\nProbability that B > A: %.1f%%\n", prob_b_wins))

# Statistical significance
if (ci_95[1] > 0) {
  cat("\n[OK] Result is statistically significant at 95% confidence level\n")
  cat("  (95% CI does not include zero)\n")
} else {
  cat("\n[X] Result is NOT statistically significant at 95% confidence level\n")
  cat("  (95% CI includes zero)\n")
}

# Relative lift analysis
cat(sprintf("\nRelative lift: %.1f%%\n",
            median(relative_lifts) * 100))
cat(sprintf("95%% CI for relative lift: [%.1f%%, %.1f%%]\n",
            quantile(relative_lifts, 0.025) * 100,
            quantile(relative_lifts, 0.975) * 100))

Illustrative output:

=== Bootstrap Results (10,000 iterations) ===

Observed difference: 0.70%

95% Confidence Interval: [0.21%, 1.19%]
99% Confidence Interval: [0.08%, 1.32%]

Probability that B > A: 99.7%

[OK] Result is statistically significant at 95% confidence level
  (95% CI does not include zero)

Relative lift: 8.2%
95% CI for relative lift: [2.5%, 14.0%]

Visualization

Plot the bootstrap distribution:

# Create histogram
hist(diffs * 100,
     breaks = 50,
     main = "Bootstrap Distribution of Conversion Rate Difference",
     xlab = "Difference in Conversion Rate (percentage points)",
     col = "lightblue",
     border = "white")

# Add reference lines
abline(v = 0, col = "red", lwd = 2, lty = 2)
abline(v = ci_95 * 100, col = "darkblue", lwd = 2, lty = 2)
abline(v = observed_diff * 100, col = "darkgreen", lwd = 2)

# Add legend
legend("topright",
       c("Observed difference", "Zero (no effect)", "95% CI"),
       col = c("darkgreen", "red", "darkblue"),
       lwd = 2,
       lty = c(1, 2, 2))

Performance Comparison

Method	Iterations	Compute time	Cost
Local	1,000	2.1 sec	$0
Local (est.)	10,000	21 sec	$0
staRburst	10,000	18 sec (compute only)	$0.01

Reading these numbers honestly: the staRburst row is compute time only on a warm pool and excludes the one-time startup (~2 minutes) and image pull. For a job that finishes in ~20 seconds locally, that startup makes local the faster choice — the numbers here are illustrative, not a claim that bursting a 20-second job wins. Bootstrap does parallelize well; the payoff is real once iterations are expensive or you scale to 100,000+. See vignette("performance") for the sizing heuristics.

Key Insights: - Bootstrap is highly parallelizable - Minimal per-task cost even with 10,000 iterations - The win comes at scale (100,000+ iterations or expensive per-iteration work), where startup overhead is amortized away

Advanced: Multi-Metric Bootstrap

Bootstrap multiple metrics simultaneously:

bootstrap_all_metrics <- function(iter, data_a, data_b) {
  n_a <- length(data_a)
  n_b <- length(data_b)

  sample_a <- sample(data_a, n_a, replace = TRUE)
  sample_b <- sample(data_b, n_b, replace = TRUE)

  # Multiple metrics
  rate_a <- mean(sample_a)
  rate_b <- mean(sample_b)
  se_a <- sd(sample_a) / sqrt(n_a)
  se_b <- sd(sample_b) / sqrt(n_b)

  list(
    diff_rate = rate_b - rate_a,
    relative_lift = (rate_b - rate_a) / rate_a,
    z_score = (rate_b - rate_a) / sqrt(se_a^2 + se_b^2),
    effect_size = (rate_b - rate_a) / sqrt((var(sample_a) + var(sample_b)) / 2)
  )
}

# Run multi-metric bootstrap
multi_results <- starburst_map(
  1:10000,
  bootstrap_all_metrics,
  data_a = variant_a,
  data_b = variant_b,
  workers = 25
)

When to Use This Pattern

Good fit: - Need robust confidence intervals - Non-normal distributions - Small sample sizes - Complex metrics (e.g., ratios, quantiles) - Multiple hypothesis testing

Not ideal: - Very large datasets (> 1M rows per group) - Simple metrics with known distributions - Real-time analysis requirements

Running the Full Example

The complete runnable script is available at:

system.file("examples/bootstrap.R", package = "starburst")

Run it with:

source(system.file("examples/bootstrap.R", package = "starburst"))

Next Steps

Try different sample sizes
Experiment with stratified bootstrap
Compare with parametric tests (t-test, z-test)
Add multiple variants (A/B/C testing)
Bootstrap other metrics (median, quantiles, variance)

Related examples: - Monte Carlo Simulation - Similar resampling pattern - Risk Modeling - Advanced statistical analysis

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.