MFF aggregates point forecasts. For forecasting, the validation
forecasts must be produced the way the test forecasts will be: from
origins that move through time, using only the observations available at
each origin. rolling_train() builds such a pool for a
univariate series. For every target month \(t\) of the validation and test blocks, each
model is fitted to the observations up to \(t
- h\) and its \(h\)-step-ahead
forecast becomes the candidate prediction for \(t\).
The models below are seasonal specifications for the log of the
monthly AirPassengers series, written with the forecast
package. Each model is a function of the training window y
and the horizon h; it may return a numeric vector or a
forecast object.
y <- log(datasets::AirPassengers)
models <- list(
ets_aaa = function(y, h) forecast::forecast(forecast::ets(y, model = "AAA"), h = h),
airline = function(y, h) {
forecast::forecast(forecast::Arima(y, order = c(0, 1, 1),
seasonal = c(0, 1, 1)), h = h)
},
stl_ets = function(y, h) forecast::stlf(y, h = h),
seasonal_naive = function(y, h) forecast::snaive(y, h = h)
)
pool <- rolling_train(y, models, horizon = 1, n_validation = 24, n_test = 24)
pool
#> MFF prediction pool
#> Task: forecasting | Pool: heterogeneous
#> Candidates: 4 | Validation: 24 | Test: 24The last 24 months are test targets and the 24 months before them are validation targets. Origins and settings are recorded with the pool.
info <- pool_info(pool)
range(info$metadata$validation_origins)
#> [1] 1956.917 1958.833
range(info$metadata$test_origins)
#> [1] 1958.917 1960.833Weights are learned on the validation forecasts only.
fit <- tune_mff(pool, clusters = 2:3, method = "fcm", m = c(1.5, 2),
control = mff_control(seed = 21))
selected_configuration(fit)[c("clusters", "m")]
#> $clusters
#> [1] 3
#>
#> $m
#> [1] 2
weights(fit)
#> ets_aaa airline stl_ets seasonal_naive
#> 3.374312e-04 5.191592e-01 4.805033e-01 1.253406e-10
evaluate(predict(fit, pool), test_observed(pool))
#> MAE RMSE MAPE SMAPE MSE MedAE
#> [1,] 0.02510835 0.03316024 0.4109221 0.411017 0.001099601 0.02169879Different horizons are different forecasting tasks, so each horizon
has its own pool, for example
rolling_train(y, models, horizon = 3, ...). For \(h > 1\) the validation block ends at the
origin of the first test target, \(h\)
months before it, and the \(h - 1\)
months in between are left out: otherwise validation outcomes observed
after a test origin would influence the selected aggregation.
window = "sliding" with window_size fits every
model to a fixed number of recent observations instead of an expanding
window. With mff_control(seed = ) each origin receives its
own seed, and parallel = TRUE distributes origins over
socket workers.
Rolling-origin forecasts produced with fable on a stretched tsibble
can be converted directly. horizon selects the forecast
step taken from every origin and the last n_test target
times form the test block; validation targets end at the first test
origin, as above. The observed tsibble must contain only the forecast
series.
ap <- tsibble::as_tsibble(datasets::AirPassengers)
stretched <- tsibble::stretch_tsibble(ap, .init = 108, .step = 1)
fable_fc <- fabletools::forecast(
fabletools::model(stretched,
snaive = fable::SNAIVE(log(value)),
ets = fable::ETS(log(value) ~ error("A") + trend("A") +
season("A")),
drift = fable::RW(log(value) ~ drift())),
h = 1)
fable_pool <- as_mff_pool(fable_fc, data = ap, horizon = 1, n_test = 12)
fable_pool
#> MFF prediction pool
#> Task: forecasting | Pool: heterogeneous
#> Candidates: 3 | Validation: 24 | Test: 12Forecasts from other software enter through mff_pool()
with task = "forecasting" and time indexes. The constructor
checks that the indexes are strictly increasing and that validation
precedes test; it cannot verify how the external forecasts were
produced, so they should come from a rolling-origin scheme like the one
above.