The MachineShop Package
MachineShop is a meta-package for statistical and machine learning with a common interface for model fitting, prediction, performance assessment, and presentation of results. Support is provided for predictive modeling of numerical, categorical, and censored time-to-event outcomes and for resample (bootstrap, cross-validation, and split training-test sets) estimation of model performance. This vignette introduces the package interface with a survival data analysis example, followed by applications to other types of response variables, supported methods of model specification and data preprocessing, and a list of all currently available models.
Model Fitting and Prediction
The Melanoma dataset from the MASS package (Andersen et al. 1993) contains time, in days, to (1) death from disease, (2) alive at end of study, or (3) death from other causes for 205 Denmark patients with malignant melanomas. Also provided are potential predictors of the survival outcomes. We begin by loading the MachineShop, survival, and MASS packages required for the analysis as well as the magrittr package (Bache and Wickham 2014) for its pipe (%>%) operator to simplify some of the code syntax. The dataset is split into a training set to which a survival model will be fit and a test set on which to make predictions. A global formula fo relates the predictors on the right hand side to the overall survival outcome on the left and will be used in all of the survival models in this vignette example.
Generalized boosted regression models are a tree-based ensemble method that can applied to survival outcomes. They are available in the MachineShop with the function GBMModel. A call to the function creates an instance of the model containing any user-specified model parameters and internal machinery for model fitting, prediction, and performance assessment. Created models can be supplied to the fit function to estimate a relationship (fo) between predictors and an outcome based on a set of data (train). The importance of variables in a model fit is estimated with the varimp function and plotted with plot. Variable importance is a measure of the relative importance of predictors in a model and has a default range of 0 to 100, where 0 denotes the least important variables and 100 the most.
From the model fit, predictions are obtained at 2, 5, and 10 years as survival probabilities (type = "prob") and as 0-1 death indicators (default: type = "response").
## Predict survival probabilities and outcomes at specified follow-up times
times <- 365 * c(2, 5, 10)
predict(gbmfit, newdata = test, times = times, type = "prob") %>% head
#> [,1] [,2] [,3]
#> [1,] 0.8659876 0.67458405 0.539432251
#> [2,] 0.8285838 0.59782467 0.446353240
#> [3,] 0.9837047 0.95604527 0.931946921
#> [4,] 0.8241337 0.58908117 0.436160014
#> [5,] 0.3119026 0.04127331 0.006752073
#> [6,] 0.8470514 0.63498855 0.490621484
predict(gbmfit, newdata = test, times = times) %>% head
#> [,1] [,2] [,3]
#> [1,] 0 0 0
#> [2,] 0 0 1
#> [3,] 0 0 0
#> [4,] 0 0 1
#> [5,] 1 1 1
#> [6,] 0 0 1
A call to modelmetrics with observed and predicted outcomes will produce model performance metrics. The metrics produced will depend on the type of the observed variable. In this case of a Surv variable, the metrics are area under the ROC curve (Heagerty, Lumley, and Pepe 2004) and Brier score (Graf et al. 1999) at the specified times and overall time-integrated averages.
Response Variable Types
Categorical
Categorical responses with two or more levels should be code as a factor variable for analysis. The metrics returned will depend on the number of factor levels. Metrics for factors with two levels are as follows.
- Accuracy
- Proportion of correctly classified responses.
- Kappa
- Cohen’s kappa statistic measuring relative agreement between observed and predicted classifications.
- Brier
- Brier score.
- ROCAUC
- Area under the ROC curve.
- PRAUC
- Area under the precision-recall curve.
- Sensitivity
- Proportion of correctly classified values in the second factor level.
- Specificity
- Proportion of correctly classified values in the first factor level.
- Index
- A tradeoff function of sensitivity and specificity as defined by
cutoff_index in the resampling control functions (default: Sensitivity + Specificity). The function allows for specification of tradeoffs (Perkins and Schisterman 2006) other than the default of Youden’s J statistic (Youden 1950).
Brier, ROCAUC, and PRAUC are computed directly on predicted class probabilities. The others are computed on predicted class membership. Memberships are defined to be in the second factor level if predicted probabilities are greater than a cutoff value defined in the resampling control functions (default: cutoff = 0.5).
Metrics for factors with more than two levels are as described below.
- Accuracy
- Proportion of correctly classified responses.
- Kappa
- Cohen’s kappa statistic measuring relative agreement between observed and predicted classifications.
- WeightedKappa
- Weighted Cohen’s kappa with equally spaced weights. This metric is only computed for ordered factor responses.
- Brier
- Multi-class Brier score.
- MLogLoss
- Multinomial logistic loss, or cross entropy loss.
Brier and MLogLoss are computed directly on predicted class probabilities. The others are computed on predicted class membership, defined as the factor level with the highest predicted probability.
Numerical
Numerical responses should be coded as a numeric variable. Associated performance metrics are as defined below and illustrated with Boston housing price data (Venables and Ripley 2002).
- R2
- One minus residual divided by total sums of squares, \[R^2 = 1 - \frac{\sum_{i=1}^n(y_i - \hat{y}_i)^2}{\sum_{i=1}^n(y_i - \bar{y})^2},\] where \(y_i\) and \(\hat{y}_i\) are the \(n\) observed and predicted responses.
- RMSE
- Root mean square error, \[RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^n(y_i - \hat{y}_i)^2}.\]
- MAE
- Median absolute error, \[MAE = \operatorname{median}(|y_i - \operatorname{median}(y)|).\]
Survival
Survival responses should be coded as a Surv variable. In addition to the ROC and Brier survival metrics described earlier in the vignette, the concordance index (Harrell et al. 1982) can be obtained if follow-up times are not specified for the prediction.
Model Specifications
Model specification here refers to the relationship between the response and predictor variables and the data used to estimate it. Three main types of specification are supported by the fit, resample, and tune functions: formulas, model frames, and recipes.
Model Frames
The second specification is similar to the first, except the formula and data frame pair are give in a ModelFrame. The model frame approach has a few subtle advantages. One is that cases with missing values on any of the response or predictor variables are excluded from the model frame by default. This is often desirable for models that cannot handle missing values. Note, however, that some models like GBMModel do accommodate missing values. For those, missing values can be retained in the model frame by setting its argument na.action = na.pass.
Another advantage is that case weights can be included in the model frame and will be passed on to the model fitting functions.
Preprocessing Recipes
The recipes package (Kuhn and Wickham 2018) provides a framework for defining predictor and response variables and preprocessing steps to be applied to them prior to model fitting. Using recipes helps to ensure that estimation of predictive performance accounts for all modeling step. They are also a very convenient way of consistently applying preprocessing to new data. Recipes currently support factor and numeric responses, but not generally Surv.
Available Models
Currently available model functions are summarized in the table below according to the types of response variables with which each model can be used. The package additionally supplies a generic MLModel function for users to create their own custom models.
|
|
|
Response Variable Types
|
|
|
Constructor
|
factor
|
numeric
|
ordered
|
Surv
|
|
C5.0 Classification
|
C50Model
|
x
|
|
|
|
|
Conditional Inference Trees
|
CForestModel
|
x
|
x
|
|
x
|
|
Cox Regression
|
CoxModel
|
|
|
|
x
|
|
Cox Regression (Stepwise)
|
CoxStepAICModel
|
|
|
|
x
|
|
Generalized Linear Models
|
GLMModel
|
2
|
x
|
|
|
|
Generalized Linear Models (Stepwise)
|
GLMStepAICModel
|
2
|
x
|
|
|
|
Gradient Boosted Models
|
GBMModel
|
x
|
x
|
|
x
|
|
Lasso and Elastic-Net
|
GLMNetModel
|
x
|
x
|
|
x
|
|
K-Nearest Neighbors Model
|
KNNModel
|
x
|
x
|
x
|
|
|
Feed-Forward Neural Networks
|
NNetModel
|
x
|
x
|
|
|
|
Partial Least Squares
|
PLSModel
|
x
|
x
|
|
|
|
Ordered Logistic Regression
|
POLRModel
|
|
|
x
|
|
|
Random Forests
|
RandomForestModel
|
x
|
x
|
|
|
|
Stacked Regression
|
StackedModel
|
x
|
x
|
x
|
x
|
|
Super Learner
|
SuperModel
|
x
|
x
|
x
|
x
|
|
Survival Regression
|
SurvRegModel
|
|
|
|
x
|
|
Survival Regression (Stepwise)
|
SurvRegStepAICModel
|
|
|
|
x
|
|
Support Vector Machines
|
SVMModel
|
x
|
x
|
|
|
|
Extreme Gradient Boosting
|
XGBModel
|
x
|
x
|
|
|
