Forecasting pools

Rolling-origin pools

MFF aggregates point forecasts. For forecasting, the validation forecasts must be produced the way the test forecasts will be: from origins that move through time, using only the observations available at each origin. rolling_train() builds such a pool for a univariate series. For every target month \(t\) of the validation and test blocks, each model is fitted to the observations up to \(t - h\) and its \(h\)-step-ahead forecast becomes the candidate prediction for \(t\).

The models below are seasonal specifications for the log of the monthly AirPassengers series, written with the forecast package. Each model is a function of the training window y and the horizon h; it may return a numeric vector or a forecast object.

y <- log(datasets::AirPassengers)
models <- list(
  ets_aaa = function(y, h) forecast::forecast(forecast::ets(y, model = "AAA"), h = h),
  airline = function(y, h) {
    forecast::forecast(forecast::Arima(y, order = c(0, 1, 1),
                                       seasonal = c(0, 1, 1)), h = h)
  },
  stl_ets = function(y, h) forecast::stlf(y, h = h),
  seasonal_naive = function(y, h) forecast::snaive(y, h = h)
)
pool <- rolling_train(y, models, horizon = 1, n_validation = 24, n_test = 24)
pool
#> MFF prediction pool
#>   Task: forecasting | Pool: heterogeneous 
#>   Candidates: 4 | Validation: 24 | Test: 24

The last 24 months are test targets and the 24 months before them are validation targets. Origins and settings are recorded with the pool.

info <- pool_info(pool)
range(info$metadata$validation_origins)
#> [1] 1956.917 1958.833
range(info$metadata$test_origins)
#> [1] 1958.917 1960.833

Weights are learned on the validation forecasts only.

fit <- tune_mff(pool, clusters = 2:3, method = "fcm", m = c(1.5, 2),
                control = mff_control(seed = 21))
selected_configuration(fit)[c("clusters", "m")]
#> $clusters
#> [1] 3
#> 
#> $m
#> [1] 2
weights(fit)
#>        ets_aaa        airline        stl_ets seasonal_naive 
#>   3.374312e-04   5.191592e-01   4.805033e-01   1.253406e-10
evaluate(predict(fit, pool), test_observed(pool))
#>             MAE       RMSE      MAPE    SMAPE         MSE      MedAE
#> [1,] 0.02510835 0.03316024 0.4109221 0.411017 0.001099601 0.02169879
plot(fit, type = "series", newdata = pool)

Different horizons are different forecasting tasks, so each horizon has its own pool, for example rolling_train(y, models, horizon = 3, ...). For \(h > 1\) the validation block ends at the origin of the first test target, \(h\) months before it, and the \(h - 1\) months in between are left out: otherwise validation outcomes observed after a test origin would influence the selected aggregation. window = "sliding" with window_size fits every model to a fixed number of recent observations instead of an expanding window. With mff_control(seed = ) each origin receives its own seed, and parallel = TRUE distributes origins over socket workers.

Forecasts from fable

Rolling-origin forecasts produced with fable on a stretched tsibble can be converted directly. horizon selects the forecast step taken from every origin and the last n_test target times form the test block; validation targets end at the first test origin, as above. The observed tsibble must contain only the forecast series.

ap <- tsibble::as_tsibble(datasets::AirPassengers)
stretched <- tsibble::stretch_tsibble(ap, .init = 108, .step = 1)
fable_fc <- fabletools::forecast(
  fabletools::model(stretched,
                    snaive = fable::SNAIVE(log(value)),
                    ets = fable::ETS(log(value) ~ error("A") + trend("A") +
                                       season("A")),
                    drift = fable::RW(log(value) ~ drift())),
  h = 1)
fable_pool <- as_mff_pool(fable_fc, data = ap, horizon = 1, n_test = 12)
fable_pool
#> MFF prediction pool
#>   Task: forecasting | Pool: heterogeneous 
#>   Candidates: 3 | Validation: 24 | Test: 12

Externally produced forecasts

Forecasts from other software enter through mff_pool() with task = "forecasting" and time indexes. The constructor checks that the indexes are strictly increasing and that validation precedes test; it cannot verify how the external forecasts were produced, so they should come from a rolling-origin scheme like the one above.