18  Regularization

Regularization adds to the training loss a penalty on the complexity of the model, so that the fitted model trades a small increase in bias for a large reduction in variance (Chapter 16). For linear models the penalty is on the size of the coefficients. This chapter presents ridge regression, the lasso and the elastic net, their effect on the coefficients, and sparse partial least squares, a related method that combines dimension reduction with sparsity.

18.1 Penalized least squares

A penalized linear model minimizes the training loss plus a penalty \(P\) on the coefficients, weighted by a tuning parameter \(\lambda \ge 0\):

\[ \hat\beta_\lambda = \arg\min_{\beta} \; \frac{1}{2n} \sum_{i=1}^{n} \left(y_i - \beta_0 - x_i^\top \beta\right)^2 + \lambda P(\beta). \tag{18.1}\]

The intercept \(\beta_0\) is not penalized. At \(\lambda = 0\) the solution is least squares; as \(\lambda\) increases, the coefficients shrink toward zero, and at large \(\lambda\) the model predicts the mean outcome for every case. The penalty depends on the scale of each feature, since a coefficient’s size depends on the units of its feature, so the features are standardized before fitting and the coefficients are reported on the original scale (Section 12.3). For a generalized linear model, the squared error is replaced by the negative log-likelihood, for example the log loss of logistic regression.

18.2 Ridge regression

Ridge regression penalizes the sum of squared coefficients, \(P(\beta) = \frac{1}{2}\lVert\beta\rVert_2^2 = \frac{1}{2}\sum_j \beta_j^2\) (Hoerl and Kennard 1970). Its solution has a closed form,

\[ \hat\beta_\lambda = \left(X^\top X + n\lambda I\right)^{-1} X^\top y, \tag{18.2}\]

for centered data. Adding \(n\lambda\) to the diagonal of \(X^\top X\) makes the matrix invertible for every \(\lambda > 0\), including when features are collinear or \(p > n\) (AMDS: Why regularization helps). Ridge shrinks all coefficients continuously toward zero, and shrinks the coefficients of correlated features toward each other, but sets none of them exactly to zero.

18.3 The lasso

The lasso penalizes the sum of absolute coefficients, \(P(\beta) = \lVert\beta\rVert_1 = \sum_j \lvert\beta_j\rvert\) (Tibshirani 1996). The absolute value has a corner at zero, and as \(\lambda\) increases the lasso sets coefficients exactly to zero, one after another. It therefore performs feature selection as part of the fit. Among a group of correlated features, the lasso tends to select one and exclude the others, and which one it selects can change between training sets.

The difference has a geometric description. Penalized least squares is equivalent to minimizing the residual sum of squares subject to a constraint on the size of \(\beta\) (AMDS: Penalty and barrier methods). The ridge constraint region, \(\sum_j \beta_j^2 \le t\), is a sphere; the lasso region, \(\sum_j \lvert\beta_j\rvert \le t\), is a polytope with corners on the coordinate axes. The contours of the residual sum of squares typically first touch the polytope at a corner, where some coefficients are zero.

18.4 The elastic net

The elastic net combines the two penalties with a mixing parameter \(\alpha \in [0, 1]\) (Zou and Hastie 2005):

\[ P_\alpha(\beta) = \alpha \lVert\beta\rVert_1 + \frac{1 - \alpha}{2} \lVert\beta\rVert_2^2 . \tag{18.3}\]

With \(\alpha = 1\) it is the lasso and with \(\alpha = 0\) ridge regression. Intermediate values select features, like the lasso, while keeping groups of correlated features together, like ridge. The coefficients for all values of \(\lambda\) are computed efficiently by coordinate descent along a decreasing sequence of \(\lambda\) values, the regularization path (Friedman et al. 2010).

Figure 18.1 shows the lasso and ridge paths for the ten standardized features of the diabetes data, the dataset with which Efron et al. (2004) introduced least angle regression, a method closely related to the lasso.

Figure 18.1: Coefficient paths of the lasso (top) and ridge regression (bottom) for the standardized diabetes features, against \(\log \lambda\). At the right of each panel the penalty is large and the coefficients are near zero; at the left they approach the least squares estimates.

Along the lasso path, features enter the model one at a time as \(\lambda\) decreases; along the ridge path, all coefficients are nonzero at every \(\lambda\) and grow together.

18.5 Choosing the penalty

\(\lambda\) is a hyperparameter: it is not estimated by minimizing the training loss, which always prefers \(\lambda = 0\). It is chosen by comparing the cross-validated error of the models along the path, within the training data (Chapter 20). Two rules are common: the value with the minimum cross-validated error, and the largest value whose error is within one standard error of that minimum, which gives a simpler model at little estimated cost (Hastie et al. 2009). With small samples, the second rule can select a model with all coefficients equal to zero.

18.6 Bayesian interpretation

Penalized estimates are maximum a posteriori estimates under a prior on the coefficients: ridge corresponds to independent normal priors centered at zero, and the lasso to independent Laplace priors, whose sharp peak at zero produces exact zeros in the posterior mode (AMDS: Maximum a posteriori estimation). \(\lambda\) corresponds to the precision of the prior.

18.7 Sparse partial least squares

Partial least squares (PLS) constructs a small number \(k\) of latent components, linear combinations of the features chosen to have high covariance with the outcome, and regresses the outcome on them. It is a form of supervised dimension reduction, suited to many correlated features. Sparse PLS adds an \(L_1\) penalty to the construction of the components, so that each component uses only some of the features (Chun and Keleş 2010). Its hyperparameters are the number of components \(k\) and the sparsity threshold \(\eta\).

18.8 Summary

  • Penalized models add a penalty on coefficient size to the training loss, trading bias for variance; features are standardized first.
  • Ridge regression shrinks all coefficients and stabilizes collinear features; the lasso sets some coefficients exactly to zero; the elastic net combines both.
  • The penalty \(\lambda\) is a hyperparameter, chosen by cross-validation within the training data.
  • Penalized estimates are posterior modes under normal or Laplace priors.
  • Sparse partial least squares combines supervised dimension reduction with feature selection.

18.9 Further reading