Validation and comparison

This vignette provides a lightweight, fully reproducible benchmark. It is designed for package documentation, not as a final simulation study for a journal article.

The benchmark includes:

Compared methods:

Metrics:

For paper-grade evidence, increase the number of repetitions, document the simulation design, and report uncertainty around the summarized metrics.

Run benchmark suite

result <- run_validation_suite(reps = 2, grid_size_1d = 60, grid_size_2d = 16)
summary_tbl <- summarise_validation(result)
summary_tbl
#>    scenario dimension method    ise_mean elapsed_mean  ise_median
#> 1        S1         1    ASH 0.032815853       0.0010 0.032815853
#> 2        S1         1  GLBFP 0.021953737       0.0015 0.021953737
#> 3        S1         1    KDE 0.002658040       0.0000 0.002658040
#> 4        S1         1   LBFP 0.021953737       0.0015 0.021953737
#> 5        S2         1    ASH 0.020182782       0.0010 0.020182782
#> 6        S2         1  GLBFP 0.013839910       0.0020 0.013839910
#> 7        S2         1    KDE 0.005633544       0.0000 0.005633544
#> 8        S2         1   LBFP 0.013839910       0.0020 0.013839910
#> 9        S3         1    ASH 0.029677125       0.0005 0.029677125
#> 10       S3         1  GLBFP 0.021946772       0.0015 0.021946772
#> 11       S3         1    KDE 0.006869287       0.0000 0.006869287
#> 12       S3         1   LBFP 0.021946772       0.0015 0.021946772
#> 13       S4         2    ASH 0.038106086       0.0035 0.038106086
#> 14       S4         2  GLBFP 0.018294805       0.0100 0.018294805
#> 15       S4         2    KDE 0.003425851       0.0000 0.003425851
#> 16       S4         2   LBFP 0.018294805       0.0095 0.018294805
#> 17       S5         2    ASH 0.047296014       0.0035 0.047296014
#> 18       S5         2  GLBFP 0.023920100       0.0095 0.023920100
#> 19       S5         2    KDE 0.004657292       0.0005 0.004657292
#> 20       S5         2   LBFP 0.023920100       0.0090 0.023920100
#>    elapsed_median
#> 1          0.0010
#> 2          0.0015
#> 3          0.0000
#> 4          0.0015
#> 5          0.0010
#> 6          0.0020
#> 7          0.0000
#> 8          0.0020
#> 9          0.0005
#> 10         0.0015
#> 11         0.0000
#> 12         0.0015
#> 13         0.0035
#> 14         0.0100
#> 15         0.0000
#> 16         0.0095
#> 17         0.0035
#> 18         0.0095
#> 19         0.0005
#> 20         0.0090

MISE ranking by scenario

mi <- summary_tbl[order(summary_tbl$scenario, summary_tbl$ise_mean), ]
mi[, c("scenario", "dimension", "method", "ise_mean", "elapsed_mean")]
#>    scenario dimension method    ise_mean elapsed_mean
#> 3        S1         1    KDE 0.002658040       0.0000
#> 2        S1         1  GLBFP 0.021953737       0.0015
#> 4        S1         1   LBFP 0.021953737       0.0015
#> 1        S1         1    ASH 0.032815853       0.0010
#> 7        S2         1    KDE 0.005633544       0.0000
#> 6        S2         1  GLBFP 0.013839910       0.0020
#> 8        S2         1   LBFP 0.013839910       0.0020
#> 5        S2         1    ASH 0.020182782       0.0010
#> 11       S3         1    KDE 0.006869287       0.0000
#> 10       S3         1  GLBFP 0.021946772       0.0015
#> 12       S3         1   LBFP 0.021946772       0.0015
#> 9        S3         1    ASH 0.029677125       0.0005
#> 15       S4         2    KDE 0.003425851       0.0000
#> 14       S4         2  GLBFP 0.018294805       0.0100
#> 16       S4         2   LBFP 0.018294805       0.0095
#> 13       S4         2    ASH 0.038106086       0.0035
#> 19       S5         2    KDE 0.004657292       0.0005
#> 18       S5         2  GLBFP 0.023920100       0.0095
#> 20       S5         2   LBFP 0.023920100       0.0090
#> 17       S5         2    ASH 0.047296014       0.0035

Sensitivity analysis (b, m)

result$sensitivity
#>   b_scale m mean_density max_density
#> 1     0.8 1   0.01875666   0.1965201
#> 2     1.0 1   0.01847158   0.1951697
#> 3     1.2 1   0.01785758   0.1301115
#> 4     1.0 2   0.01783902   0.1514042

Notes