The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Back Testing and Hyperparameter Tuning

Most models automatically included in finnts, including all multivariate models, have various hyperparameters with values that need to be chosen before a model is trained. Finn solves this by leveraging the tune package within the tidymodels ecosystem.

When prep_models() is ran, hyperparameters and back test splits are calculated and written to disk. You can get the results by calling get_prepped_models().

#> Loading required package: modeltime
#> Finn Submission Info
#> * Experiment Name: finnts_fcst
#> * Run Name: get_prepped_models-20231130T175731Z
#> 
#> i Prepping Data
#> Warning: package 'tibble' was built under R version 4.1.3
#> Warning: package 'dplyr' was built under R version 4.1.3
#> Warning: package 'timetk' was built under R version 4.1.3
#> Warning: package 'forecast' was built under R version 4.1.3
#> Warning: package 'tidyselect' was built under R version 4.1.3
#> Warning: package 'stringr' was built under R version 4.1.3
#> Warning: package 'doParallel' was built under R version 4.1.3
#> Warning: package 'iterators' was built under R version 4.1.3
#> Warning: package 'lubridate' was built under R version 4.1.3
#> Warning: package 'dials' was built under R version 4.1.3
#> Warning: package 'scales' was built under R version 4.1.3
#> Warning: package 'workflows' was built under R version 4.1.3
#> Warning: package 'Cubist' was built under R version 4.1.3
#> Warning: package 'earth' was built under R version 4.1.3
#> Warning: package 'glmnet' was built under R version 4.1.3
#> Warning: package 'purrr' was built under R version 4.1.3
#> Warning: package 'rules' was built under R version 4.1.3
#> Warning: package 'fs' was built under R version 4.1.3
#> Warning: package 'digest' was built under R version 4.1.3
#> Warning: package 'tidyr' was built under R version 4.1.3
#> Warning: package 'vroom' was built under R version 4.1.3
#> Warning: package 'cli' was built under R version 4.1.3
#> 
v Prepping Data [5.8s]
#> 
i Creating Model Workflows

v Creating Model Workflows [538ms]
#> 
i Creating Model Hyperparameters

v Creating Model Hyperparameters [823ms]
#> 
i Creating Train Test Splits

v Creating Train Test Splits [1.5s]
#> # A tibble: 31 x 4
#>    Run_Type        Train_Test_ID Train_End  Test_End  
#>    <chr>                   <dbl> <date>     <date>    
#>  1 Future_Forecast             1 2015-06-01 2015-12-01
#>  2 Back_Test                   2 2015-05-01 2015-06-01
#>  3 Back_Test                   3 2015-04-01 2015-06-01
#>  4 Back_Test                   4 2015-03-01 2015-06-01
#>  5 Back_Test                   5 2015-02-01 2015-06-01
#>  6 Back_Test                   6 2015-01-01 2015-06-01
#>  7 Back_Test                   7 2014-12-01 2015-06-01
#>  8 Back_Test                   8 2014-11-01 2015-05-01
#>  9 Back_Test                   9 2014-10-01 2015-04-01
#> 10 Validation                 10 2014-09-01 2014-10-01
#> # i 21 more rows
#> [1] "Future_Forecast" "Back_Test"       "Validation"      "Ensemble"
#> # A tibble: 3 x 3
#>   Model_Name Model_Recipe Model_Workflow
#>   <chr>      <chr>        <list>        
#> 1 arima      R1           <workflow>    
#> 2 ets        R1           <workflow>    
#> 3 xgboost    R1           <workflow>
#> # A tibble: 12 x 4
#>    Model   Recipe Hyperparameter_Combo Hyperparameters 
#>    <chr>   <chr>                 <dbl> <list>          
#>  1 arima   R1                        1 <tibble [0 x 0]>
#>  2 ets     R1                        1 <tibble [0 x 0]>
#>  3 xgboost R1                        1 <tibble [1 x 4]>
#>  4 xgboost R1                        2 <tibble [1 x 4]>
#>  5 xgboost R1                        3 <tibble [1 x 4]>
#>  6 xgboost R1                        4 <tibble [1 x 4]>
#>  7 xgboost R1                        5 <tibble [1 x 4]>
#>  8 xgboost R1                        6 <tibble [1 x 4]>
#>  9 xgboost R1                        7 <tibble [1 x 4]>
#> 10 xgboost R1                        8 <tibble [1 x 4]>
#> 11 xgboost R1                        9 <tibble [1 x 4]>
#> 12 xgboost R1                       10 <tibble [1 x 4]>

The above outputs allow a Finn user to understand what hyperparameters are chosen for tuning and how the model refitting process will work. When tuning hyperparameters, Finn uses the “Validation” train/test splits, with the final parameters chosen using RMSE. For some models like ARIMA that don’t follow a traditional hyperparameter tuning process, the model is fit from scratch across all train/test splits. After hyperparameters are chosen, the model is refit across the “Back_Test” and “Future_Forecast” splits. The “Back_Test” splits are the true testing data that will be used when selecting the final “Best-Model”. “Ensemble” splits are also created as ensemble training data if ensemble models are chosen to run. Ensemble models follow a similar tuning process.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.