The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

A Complete Incrementality Analysis

Overview

Incrementality asks what happened because a treatment was applied, beyond what would have happened under control. IncrementalityTEST analyzes a collection of experiment-level treatment and control measurements.

The workflow has four steps:

  1. calculate business metrics where necessary;
  2. validate and pair treatment and control rows;
  3. calculate one effect per experiment;
  4. summarize those effects and quantify uncertainty.

Prepare group-level data

Each experiment needs exactly one row for control and one for treatment.

library(IncrementalityTEST)

results <- data.frame(
  experiment = rep(paste0("test_", 1:6), each = 2),
  group = rep(c("control", "treatment"), 6),
  transactions = c(100, 112, 80, 87, 130, 141, 92, 96, 110, 121, 70, 79),
  users = c(1000, 1005, 800, 804, 1200, 1208, 900, 902, 1050, 1053, 700, 704),
  revenue = c(2500, 2860, 1920, 2140, 3300, 3690, 2200, 2350, 2750, 3100, 1680, 1950),
  buyers = c(90, 99, 72, 78, 115, 124, 83, 86, 101, 109, 63, 70)
)

Calculate the four supported commerce metrics and attach them to the data.

metrics <- calculate_metrics(
  results$transactions,
  results$users,
  results$revenue,
  results$buyers
)
results <- cbind(results, metrics)
head(results)
#>   experiment     group transactions users revenue buyers      RPU         BR
#> 1     test_1   control          100  1000    2500     90 2.500000 0.09000000
#> 2     test_1 treatment          112  1005    2860     99 2.845771 0.09850746
#> 3     test_2   control           80   800    1920     72 2.400000 0.09000000
#> 4     test_2 treatment           87   804    2140     78 2.661692 0.09701493
#> 5     test_3   control          130  1200    3300    115 2.750000 0.09583333
#> 6     test_3 treatment          141  1208    3690    124 3.054636 0.10264901
#>        AOV      TPB
#> 1 25.00000 1.111111
#> 2 25.53571 1.131313
#> 3 24.00000 1.111111
#> 4 24.59770 1.115385
#> 5 25.38462 1.130435
#> 6 26.17021 1.137097

Inspect experiment-level effects

metric_differences() uses treatment minus control by default. A positive number therefore means that the metric was higher under treatment.

effects <- metric_differences(results, metric = "RPU")
effects
#>   experiment  control treatment difference
#> 1     test_1 2.500000  2.845771  0.3457711
#> 2     test_2 2.400000  2.661692  0.2616915
#> 3     test_3 2.750000  3.054636  0.3046358
#> 4     test_4 2.444444  2.605322  0.1608771
#> 5     test_5 2.619048  2.943970  0.3249220
#> 6     test_6 2.400000  2.769886  0.3698864

Missing groups, duplicate groups, and nonnumeric metrics produce explicit errors. Use na_action = "omit" only when dropping incomplete experiments is methodologically defensible.

Estimate the overall effect

analysis <- analyze_incrementality(
  results,
  metric = "RPU",
  conf_level = 0.95,
  bootstrap_times = 2000,
  seed = 2026
)
analysis
#> Incrementality analysis
#> Metric: RPU
#> Direction: treatment-control
#> Experiments: 6
#> 
#>     method      mean standard_error conf_level     lower     upper
#>  Student t 0.2946306     0.03069823       0.95 0.2157183 0.3735430
#>  Bootstrap 0.2946306     0.02819784       0.95 0.2358360 0.3433737

The object contains all intermediate and final results:

analysis$differences
#>   experiment  control treatment difference
#> 1     test_1 2.500000  2.845771  0.3457711
#> 2     test_2 2.400000  2.661692  0.2616915
#> 3     test_3 2.750000  3.054636  0.3046358
#> 4     test_4 2.444444  2.605322  0.1608771
#> 5     test_5 2.619048  2.943970  0.3249220
#> 6     test_6 2.400000  2.769886  0.3698864
analysis$t_interval
#>   n      mean standard_error conf_level     lower    upper
#> 1 6 0.2946306     0.03069823       0.95 0.2157183 0.373543
analysis$bootstrap_interval
#>        mean standard_error conf_level    lower     upper
#> 1 0.2946306     0.02819784       0.95 0.235836 0.3433737

The t interval assumes that experiment-level effects are independent and that their sampling distribution is reasonably approximated by a normal distribution. The percentile bootstrap makes fewer distributional assumptions, but a small or unrepresentative set of experiments still limits inference.

Report the result

A useful report states:

The package estimates an unweighted mean across experiments. If experiments have materially different precision or target populations, a hierarchical model or justified weighting strategy may be more appropriate.

Working with legacy data

For existing datasets containing iabtest_id and numeric abt_group values, the compatibility function remains available:

legacy <- transform(
  results,
  iabtest_id = experiment,
  abt_group = ifelse(group == "control", 0, 1)
)
test_metric(legacy, "RPU")
#>   iabtest_id    inc_RPU
#> 1     test_1 -0.3457711
#> 2     test_2 -0.2616915
#> 3     test_3 -0.3046358
#> 4     test_4 -0.1608771
#> 5     test_5 -0.3249220
#> 6     test_6 -0.3698864

For historical compatibility, test_metric() uses control minus treatment. New analyses should use metric_differences() so the direction is explicit.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.