To train a model on CloudML, first work the training script locally (perhaps with a smaller sample of your dataset). The script can contain arbitrary R code which trains and/or evaluates a model. Once you’ve confirmed that things work as expected, you can submit a CloudML job to perform training in the cloud.
Submitting a Job
To submit a job, call the cloudml_train() function, specifying the R script to execute for training:
All of the files within the current working directory will be bundled up and sent along with the script to CloudML.
Note that the very first time you submit a job to CloudML the various packages required to run your script will be compiled from source. This will make the execution time of the job considerably longer that you might expect. It’s only the first job that incurs this overhead though (since the package installations are cached), and subsequent jobs will run more quickly.
If you are using RStudio v1.1 or higher, then the CloudML training job is monitored (and it’s results collected) using a background terminal:
Collecting Results
When the job is complete, training results can be collected back to your local system (this is done automatically when monitoring the job using a background terminal in RStudio). A run report is displayed after the job is collected:
You can list all previous runs as a data frame using the ls_runs() function:
Data frame: 6 x 37
run_dir eval_loss eval_acc metric_loss metric_acc metric_val_loss metric_val_acc
6 runs/cloudml_2018_01_26_135812740 0.1049 0.9789 0.0852 0.9760 0.1093 0.9770
2 runs/cloudml_2018_01_26_140015601 0.1402 0.9664 0.1708 0.9517 0.1379 0.9687
5 runs/cloudml_2018_01_26_135848817 0.1159 0.9793 0.0378 0.9887 0.1130 0.9792
3 runs/cloudml_2018_01_26_135936130 0.0963 0.9780 0.0701 0.9792 0.0969 0.9790
1 runs/cloudml_2018_01_26_140045584 0.1486 0.9682 0.1860 0.9504 0.1453 0.9693
4 runs/cloudml_2018_01_26_135912819 0.1141 0.9759 0.1272 0.9655 0.1087 0.9762
# ... with 30 more columns:
# flag_dense_units1, flag_dropout1, flag_dense_units2, flag_dropout2, samples, validation_samples,
# batch_size, epochs, epochs_completed, metrics, model, loss_function, optimizer, learning_rate,
# script, start, end, completed, output, source_code, context, type, cloudml_console_url,
# cloudml_created, cloudml_end, cloudml_job, cloudml_log_url, cloudml_ml_units, cloudml_start,
# cloudml_state
You can view run reports using the view_run() function:
There are many tools available to list, filter, and compare training runs. For additional information see the documentation for the tfruns package.
