The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
MFF aggregates numeric predictions without requiring models to be
trained inside the package. The bundled mff_diamonds data
contain validation and test predictions of diamond prices from seven
regression learners: four linear (least squares, lasso, ridge, elastic
net) and three tree ensembles (random forest, XGBoost, LightGBM). A
random sample of 2000 diamonds from the ggplot2 diamonds
data was randomly split into 1000 training, 500 validation and 500 test
rows, and the learners were trained on the training rows only (see
?mff_diamonds). mff_example() builds these
predictions into a pool. The example demonstrates the interface; it does
not estimate general predictive superiority.
pool <- mff_example()
pool
#> MFF prediction pool
#> Task: regression | Pool: heterogeneous
#> Candidates: 7 | Validation: 500 | Test: 500
dim(validation_predictions(pool))
#> [1] 500 7
candidate_info(pool)
#> candidate family
#> 1 lm linear
#> 2 lasso linear
#> 3 ridge linear
#> 4 elastic_net linear
#> 5 random_forest tree_ensemble
#> 6 xgboost tree_ensemble
#> 7 lightgbm tree_ensembleA fixed configuration and a grid search share the same fitted-object contract.
fixed <- fit_mff(pool, clusters = 2, method = "fcm", seed = 19)
fit <- tune_mff(pool, clusters = 2:3, method = "fcm", m = c(1.5, 2),
nstart = 3, reltol = 1e-6, control = mff_control(seed = 19))
print(fit)
#> Tuned MFF fit
#> Method: fcm | Candidates: 7 | Functions: 3
#> Validation-selected function: 1 | RMSE = 1006.745
#> Convergence status: converged | Iterations: 10
#> Configurations: 4 | Successful: 4
summary(fit)
#> MFF fit summary
#> Configuration:
#> method : fcm
#> metric : RMSE
#> clusters : 3
#> m : 2
#> nstart : 3
#> iter_max : 100
#> reltol : 1e-06
#> standardize : FALSE
#> reduction : none
#> Convergence status: converged | Iterations: 10
#> Selected function: 1
#> Selected weights:
#> lm lasso ridge elastic_net random_forest
#> 0.003028213 0.001572850 0.085995441 0.001584017 0.003038848
#> xgboost lightgbm
#> 0.456845200 0.447935430
#> Validation scores:
#> MAE RMSE MAPE SMAPE MSE MedAE
#> function_1 485.0458 1006.745 12.90416 12.51245 1013536 178.3300
#> function_2 771.9162 1130.517 43.02233 41.64732 1278068 556.7292
#> function_3 643.9853 1189.650 22.41862 19.08970 1415266 314.0998
#> Search: 4 configurations; 0 failed
selected_configuration(fit)
#> method : fcm
#> metric : RMSE
#> clusters : 3
#> m : 2
#> nstart : 3
#> iter_max : 100
#> reltol : 1e-06
#> standardize : FALSE
#> reduction : none
search_results(fit)
#> m clusters seed success loss elapsed message
#> 1 1.5 2 20 TRUE 1056.685 0.003
#> 2 2.0 2 21 TRUE 1042.631 0.001
#> 3 1.5 3 22 TRUE 1019.745 0.002
#> 4 2.0 3 23 TRUE 1006.745 0.001Test outcomes are used only after fitting. Changing them cannot change weights or the selected configuration. The following compares MFF with equal averaging and the candidate chosen solely on validation RMSE.
pv <- validation_predictions(pool)
pt <- test_predictions(pool)
best <- which.min(evaluate(pv, validation_observed(pool))[, "RMSE"])
comparisons <- cbind(
MFF = predict(fit, pool),
Mean = rowMeans(pt),
Validation_best = pt[, best]
)
evaluate(comparisons, test_observed(pool))
#> MAE RMSE MAPE SMAPE MSE MedAE
#> MFF 483.6754 911.1981 13.17991 12.71691 830282.0 188.7307
#> Mean 616.3507 1032.5433 24.42174 25.44028 1066145.6 370.0113
#> Validation_best 479.6820 924.4773 12.19211 11.72586 854658.2 177.1585The heatmap exposes all candidate weights; prediction defaults to the
function selected on validation data. type = "all" means
the functions in the chosen configuration, not every configuration in
the tuning grid.
Explicit split rows make the training stage auditable. Bootstrap resampling uses training rows only. Validation/test rows are identical for each candidate.
split <- mff_split(train = 1:15, validation = 16:23, test = 24:31)
boot_pool <- boot_train(Volume ~ Girth + Height, datasets::trees, split,
times = 8, control = mff_control(seed = 44))
summary(boot_pool)
#> MFF prediction pool summary
#> Task: regression | Pool type: bootstrap
#> Candidates: 8 | Validation rows: 8 | Test rows: 8
#> Candidates:
#> candidate algorithm seed
#> bootstrap_1 lm 45
#> bootstrap_2 lm 46
#> bootstrap_3 lm 47
#> bootstrap_4 lm 48
#> bootstrap_5 lm 49
#> bootstrap_6 lm 50
#> bootstrap_7 lm 51
#> bootstrap_8 lm 52
boot_fit <- tune_mff(boot_pool, clusters = 2:3, method = "kmeans")
evaluate(predict(boot_fit, boot_pool), test_observed(boot_pool))
#> MAE RMSE MAPE SMAPE MSE MedAE
#> [1,] 12.07627 13.57597 21.17138 24.03179 184.307 11.5067Any regression method can be trained by model_train()
and boot_train() after wrapping it with
mff_learner(). The fit function receives the
model matrix built on the training rows and the training response;
predict returns one prediction per row. Named lists of
learners let one algorithm enter several times with different
controls.
ppr_learner <- mff_learner(
fit = function(x, y, nterms = 2) stats::ppr(x, y, nterms = nterms),
predict = function(object, newx) stats::predict(object, newx),
name = "ppr"
)
air <- datasets::airquality[stats::complete.cases(datasets::airquality), ]
set.seed(11)
air_rows <- sample(nrow(air))
air_split <- mff_split(air_rows[1:60], air_rows[61:85], air_rows[86:111])
air_pool <- model_train(Ozone ~ Solar.R + Wind + Temp, air, air_split,
learners = list(lm = "lm", ppr_1 = ppr_learner,
ppr_2 = ppr_learner,
ppr_3 = ppr_learner),
controls = list(ppr_1 = list(nterms = 1),
ppr_3 = list(nterms = 3)),
seed = 11)
candidate_info(air_pool)
#> candidate algorithm
#> 1 lm lm
#> 2 ppr_1 ppr
#> 3 ppr_2 ppr
#> 4 ppr_3 ppr
boot_ppr <- boot_train(Ozone ~ Solar.R + Wind + Temp, air, air_split,
times = 5, learner = ppr_learner,
control = mff_control(seed = 11))
candidate_info(boot_ppr)
#> candidate algorithm seed
#> 1 bootstrap_1 ppr 12
#> 2 bootstrap_2 ppr 13
#> 3 bootstrap_3 ppr 14
#> 4 bootstrap_4 ppr 15
#> 5 bootstrap_5 ppr 16A separate validation set is not required. With
resampling = mff_cv(), model_train() predicts
every training row from candidates fitted on the other folds, and these
out-of-fold predictions serve as validation predictions. Model-matrix
encoding and learner tuning are repeated inside each fold using that
fold’s training rows only. Test predictions come from candidates
refitted on all training rows, and test rows never enter the folds.
set.seed(7)
rows <- sample(nrow(datasets::quakes))
cv_split <- mff_split(train = rows[1:750], validation = NULL,
test = rows[751:1000])
cv_pool <- model_train(mag ~ lat + long + depth + stations, datasets::quakes,
cv_split, learners = c("lm", "ridge", "random_forest"),
resampling = mff_cv(folds = 5), seed = 7)
cv_pool
#> MFF prediction pool
#> Task: regression | Pool: heterogeneous
#> Candidates: 3 | Validation: 750 | Test: 250
table(pool_info(cv_pool)$metadata$resampling$fold_id)
#>
#> 1 2 3 4 5
#> 150 150 150 150 150
cv_fit <- tune_mff(cv_pool, clusters = 2, method = "fcm", m = c(1.5, 2))
weights(cv_fit)
#> lm ridge random_forest
#> 5.006733e-01 4.993267e-01 4.811049e-08
evaluate(predict(cv_fit, cv_pool), test_observed(cv_pool))
#> MAE RMSE MAPE SMAPE MSE MedAE
#> [1,] 0.1532053 0.1985064 3.295108 3.314259 0.03940481 0.1200164mff_cv(fold_id = ) accepts user-defined folds, for
example to keep grouped observations together. A factor level that
occurs only in a held-out fold stops with an error that names the
fold.
Models tuned in other frameworks already carry out-of-fold
predictions. as_mff_pool() converts a named list of
caret::train() models (or a caretEnsemble
caretList), tidymodels tune_grid() results and
stacks data stacks into pools. With newdata, test
predictions come from the framework’s own models fitted on all training
rows.
set.seed(3)
car_data <- datasets::mtcars
ctrl <- caret::trainControl(method = "cv", savePredictions = "final",
index = caret::createFolds(car_data$mpg[1:26], k = 4,
returnTrain = TRUE))
caret_models <- list(
linear = caret::train(mpg ~ wt + hp + disp, car_data[1:26, ], method = "lm",
trControl = ctrl),
knn = caret::train(mpg ~ wt + hp + disp, car_data[1:26, ], method = "knn",
trControl = ctrl, tuneLength = 2),
weight_only = caret::train(mpg ~ wt, car_data[1:26, ], method = "lm",
trControl = ctrl)
)
#> Zorunlu paket yükleniyor: ggplot2
#> Zorunlu paket yükleniyor: lattice
caret_pool <- as_mff_pool(caret_models, newdata = car_data[27:32, ],
test_observed = car_data$mpg[27:32])
candidate_info(caret_pool)
#> candidate method tuning
#> 1 linear lm intercept=TRUE
#> 2 knn knn k=5
#> 3 weight_only lm intercept=TRUEcar_data <- datasets::mtcars
set.seed(4)
folds <- rsample::vfold_cv(car_data[1:26, ], v = 4)
lasso <- workflows::workflow(
mpg ~ wt + hp + disp,
parsnip::set_engine(parsnip::linear_reg(penalty = tune::tune(), mixture = 1),
"glmnet"))
tuned <- tune::tune_grid(lasso, folds,
grid = data.frame(penalty = c(0.01, 0.3, 1, 3)),
control = tune::control_grid(save_pred = TRUE,
save_workflow = TRUE))
tune_pool <- as_mff_pool(tuned, newdata = car_data[27:32, ],
test_observed = car_data$mpg[27:32])
candidate_info(tune_pool)
#> candidate penalty
#> 1 pre0_mod1_post0 0.01
#> 2 pre0_mod2_post0 0.30
#> 3 pre0_mod3_post0 1.00
#> 4 pre0_mod4_post0 3.00For heterogeneous learner families, model_train() can
use an explicit selection such as
learners = c("lm", "random_forest"). The requested optional
learner packages must be installed. A single "lm" candidate
is allowed as a baseline, although fitting an MFF needs at least two
candidate columns.
Here polynomial degree is varied outside MFF; each fitted regression produces one candidate column. The candidate table retains its degree for inspection. The same approach applies to external XGBoost, SVM or forecasting settings.
dat <- datasets::trees
degrees <- 1:3
models <- lapply(degrees, function(d) {
stats::lm(Volume ~ poly(Girth, d), data = dat[1:15, ])
})
make_predictions <- function(rows) {
out <- vapply(models, function(model) {
as.numeric(predict(model, newdata = dat[rows, ]))
}, numeric(length(rows)))
colnames(out) <- paste0("degree_", degrees)
out
}
hp <- mff_pool(
make_predictions(16:23), dat$Volume[16:23],
make_predictions(24:31), dat$Volume[24:31],
candidate_info = data.frame(candidate = paste0("degree_", degrees),
algorithm = "polynomial_lm", degree = degrees),
pool_type = "hyperparameter"
)
candidate_info(hp)
#> candidate algorithm degree
#> 1 degree_1 polynomial_lm 1
#> 2 degree_2 polynomial_lm 2
#> 3 degree_3 polynomial_lm 3
hp_fit <- tune_mff(hp, clusters = 2:3, method = "fcm", m = c(1.5, 2))
evaluate(predict(hp_fit, hp), test_observed(hp))
#> MAE RMSE MAPE SMAPE MSE MedAE
#> [1,] 10.93218 12.79735 18.77613 21.2117 163.7723 10.63079The supported membership methods are fuzzy c-means
(fcm), possibilistic fuzzy c-means (pfcm),
Gustafson-Kessel (gk) and crisp k-means
(kmeans). For every new-interface method,
nstart selects the start with the lowest clustering
objective. Validation outcomes subsequently select a meta function.
The optional reltol argument controls convergence. For
FCM it is the relative objective tolerance. In PFCM it bounds the mean
absolute change of the prototype coordinates; in GK the largest change
of a membership between iterations. These are different stopping rules.
NULL retains the defaults: approximately 1.49e-8 for FCM, 1e-5 for PFCM
and 1e-3 for GK. PFCM follows Pal, Pal, Keller and Bezdek (2005) with
their protocol (a = b = 1, penalties from a terminal FCM partition, at
most 100 iterations). K-means accepts only reltol = NULL;
use iter_max there. The effective tolerance is recorded in
selected_configuration() and
fit_diagnostics(). Increasing tolerance may reduce
iterations, but can change the fitted weights and predictions.
GK estimates one fuzzy covariance matrix per cluster. With few
candidates and many validation observations these matrices are singular,
so MFF implements the modified GK algorithm of Babuska, van der Veen and
Kaymak (2002): the ratio between the largest and smallest eigenvalue of
each matrix is limited to beta (default 1e15), and
gamma (default 0) adds a scaled identity matrix.
tune_mff() can search gamma like
m. reduction = "pca" retains every nonzero
principal component in clustering space. This preserves Euclidean
distances but does not guarantee that GK or PFCM results are unchanged.
Nonfinite memberships fail explicitly rather than being replaced by
uniform weights.
fit_diagnostics(fit)
#> $selected_start
#> [1] 2
#>
#> $objective
#> [1] 21195166
#>
#> $iterations
#> [1] 10
#>
#> $tolerance
#> $tolerance$value
#> [1] 1e-06
#>
#> $tolerance$parameter
#> [1] "control$reltol"
#>
#> $tolerance$criterion
#> [1] "relative_objective_change"
#>
#>
#> $convergence
#> [1] "converged"
#>
#> $starts
#> start objective success message
#> 1 1 24862060 TRUE
#> 2 2 21195166 TRUE
#> 3 3 24862052 TRUE
#>
#> $preprocessing
#> $preprocessing$method
#> [1] "none"
#>
#> $preprocessing$original_dimensions
#> [1] 500
#>
#> $preprocessing$retained_dimensions
#> [1] 500
#>
#> $preprocessing$distance_preserving
#> [1] TRUE
#>
#>
#> $seed
#> [1] 23
weights(fit, type = "all")
#> function_1 function_2 function_3
#> lm 0.003028213 0.2704951651 0.003489439
#> lasso 0.001572850 0.2719100403 0.001818082
#> ridge 0.085995441 0.1588521724 0.177951374
#> elastic_net 0.001584017 0.2718990153 0.001831337
#> random_forest 0.003038848 0.0007193028 0.692905267
#> xgboost 0.456845200 0.0124026466 0.056735672
#> lightgbm 0.447935430 0.0137216577 0.065268830The selection metric can be MAE, RMSE, MAPE, SMAPE, MSE or MedAE.
MAPE and SMAPE use a small epsilon in the denominator; MAPE can be
unstable for responses near zero. See ?evaluate for
formulas. Tuning diagnostics record failed configurations and warnings.
The convergence element of fit_diagnostics()
is "converged" only when the selected start met its
stopping rule before iter_max without warnings;
"iteration_limit" and "backend_warning" mean
convergence is not confirmed.
For reproducible socket parallelism use
mff_control(seed = 19, parallel = TRUE, workers = 2). The
default is sequential.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.