setup_GLMNET() fits elastic net models with the glmnet package (Friedman et al. 2010), and setup_SPLS() fits sparse partial least squares with the spls package. This chapter fits penalized models to both datasets, returns to the expanded diabetes models of Section 5.6, and fits a multinomial model.
alpha is the elastic net mixing parameter of Equation 18.3, 1 for the lasso and 0 for ridge regression. When lambda is not set, train() calls glmnet::cv.glmnet() on the training data, which fits the full regularization path and estimates the error of each \(\lambda\) by internal 10-fold cross-validation; which_lambda_cv selects either the value with the minimum error ("lambda.min") or the largest value within one standard error of it ("lambda.1se", the default) (Section 18.5). This selection uses only the training cases: within an outer cross-validation, it is repeated in every training fold. standardize = TRUE standardizes the features internally and reports coefficients on the original scale. ifw applies inverse frequency weights to the classes in classification, and is TRUE by default; its effect is the subject of Chapter 41, and the classification examples below set it to FALSE.
11 x 1 sparse Matrix of class "dgCMatrix"
s0
(Intercept) -263.5199
age .
sex2 -20.0760
bmi 5.6486
bp 1.0494
s1 -0.1959
s2 .
s3 -0.7180
s4 1.6406
s5 47.5098
s6 0.2450
The fitted model records the selected \(\lambda\). Features with a coefficient shown as . have been set to zero. The glmnet object stores the full path, so the number of nonzero coefficients can be read at every \(\lambda\):
path<-glmnet::glmnet(model.matrix(progression~., data =db)[, -1],db[["progression"]], alpha =1)draw_line(log(path[["lambda"]]),list(`Nonzero coefficients` =path[["df"]]), points =FALSE, xlab ="log(lambda)", ylab ="Number of nonzero coefficients")
19.3 Large model classes
Section 5.6 showed that an unpenalized linear model with all two- or three-way interactions of the diabetes features overfits. The same expanded feature sets are fitted here with the lasso and assessed by 10-fold cross-validation, next to the unpenalized model:
The unpenalized model’s cross-validated \(R^2\) falls from 0.493 with main effects to -0.620 with three-way interactions. The lasso’s stays between 0.493 and 0.495 for all three feature sets. With 175 candidate terms and about 400 training cases per fold, the lasso keeps on average 12.2 nonzero coefficients and shrinks them, so that the enlarged model class does not translate into enlarged variance. The interactions add little predictive information in these data; the lasso recovers the performance of the main-effects model from a feature set in which least squares fails.
19.4 Penalized logistic regression
For the heart failure data, ridge and lasso logistic regression are compared with the two rules for choosing \(\lambda\):
With \(\lambda\) at the minimum of the internal cross-validated error, ridge regression reaches a mean test AUC of 0.721, compared with 0.709 for the unpenalized logistic regression with the same folds (Section 15.5). The one-standard-error rule, applied to training folds of about 260 patients and 49 deaths, selects larger values of \(\lambda\). For the lasso, it sets every coefficient to zero in 2 of the 10 training folds; such a model predicts the same probability for every patient, and its AUC on that fold is 0.5. Ridge regression shrinks without setting coefficients to zero; with the one-standard-error rule its mean Brier score, 0.151, is close to the 0.152 of a constant prediction equal to the prevalence.
19.5 Multinomial regression
setup_GLM() fits binary logistic regression only. For an outcome with more than two classes, setup_GLMNET() fits multinomial logistic regression. With the diabetes outcome divided into tertiles, as in Section 9.7:
<Resampled Classification Model>GLMNET (Elastic net (glmnet))
⚙ Tuned using exhaustive grid search.
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>Aggregate Confusion Matrix across resamples.
Predicted ReferenceLow Medium High Low 942 311 79
Medium 478 450 395
High 70 304 949
Showing mean (sd) across resamples.
Balanced Accuracy: 0.588 (0.011) F1: 0.578 (0.012) Accuracy: 0.588 (0.011)<Resampled Classification Test Metrics>Aggregate Confusion Matrix across resamples.
Predicted ReferenceLow Medium High Low 103 37 8
Medium 54 48 45
High 8 35 104
Showing mean (sd) across resamples.
Balanced Accuracy: 0.577 (0.095) F1: 0.566 (0.092) Accuracy: 0.577 (0.095)
The model has one coefficient vector per class, and predict() returns one probability per class:
k and eta are fixed here. Choosing them, like choosing \(\lambda\), is a problem of hyperparameter tuning, the subject of Chapter 20.
Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. 2010. “Regularization Paths for Generalized Linear Models via Coordinate Descent.”Journal of Statistical Software 33 (1). https://doi.org/10.18637/jss.v033.i01.