14  Linear Models

Linear models predict the outcome, or a transformation of its expected value, as a weighted sum of the features. They are the oldest and most widely used predictive models, the basis of several more flexible methods in later chapters, and the baseline against which flexible models are compared. This chapter presents linear regression, its extension to generalized linear models, and the properties that determine when a linear model predicts well.

14.1 Linear regression

Linear regression models a numeric outcome as

\[ y_i = \beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip} + \varepsilon_i = x_i^\top \beta + \varepsilon_i, \tag{14.1}\]

where \(x_i\) is extended with a leading 1 for the intercept, \(\beta\) is the vector of \(p + 1\) coefficients, and \(\varepsilon_i\) is an error with mean zero. The prediction for a case with features \(x\) is \(\hat y = x^\top \hat \beta\), an estimate of the conditional mean \(\mathbb{E}[Y \mid X = x]\), which is the optimal predictor under squared error (Equation 4.2).

The least squares estimate minimizes the residual sum of squares, \(\sum_i (y_i - x_i^\top \beta)^2 = \lVert y - X \beta \rVert^2\). Setting its gradient to zero gives the normal equations, \(X^\top X \hat\beta = X^\top y\), with solution

\[ \hat\beta = (X^\top X)^{-1} X^\top y \tag{14.2}\]

when \(X^\top X\) is invertible (AMDS: Deriving the least squares solution; The normal equations). Geometrically, the fitted values \(X\hat\beta\) are the orthogonal projection of \(y\) onto the space spanned by the columns of \(X\) (AMDS: Projection onto a subspace).

Each coefficient \(\beta_j\) is the expected difference in the outcome associated with a one-unit difference in \(x_j\), with the other features held fixed. The model is additive, since the contributions of the features add, and linear, since each contribution is proportional to the feature.

14.2 Linear in the parameters

The model is linear in the coefficients, not necessarily in the original measurements. The columns of \(X\) can include:

  • dummy-coded indicators of the levels of a categorical feature (Section 12.2);
  • transformations of a feature, such as \(\log x_j\) or \(x_j^2\);
  • products of features, \(x_j x_k\), called interactions, which allow the effect of one feature to depend on the value of another.

With such terms a linear model can represent curvature and interactions, but only those specified in advance. Chapter 22 describes methods that estimate the shape of each feature’s effect from the data.

14.3 Generalized linear models

Many outcomes are not continuous and unbounded: a binary outcome takes values 0 and 1, a count is non-negative. Generalized linear models (GLMs) extend linear regression to such outcomes with three components (Nelder and Wedderburn 1972; McCullagh and Nelder 1989):

  1. a distribution for the outcome from the exponential family, such as the normal, Bernoulli or Poisson;
  2. a linear predictor \(\eta = x^\top \beta\);
  3. a link function \(g\) that relates the mean of the outcome to the linear predictor, \(g(\mathbb{E}[Y \mid x]) = \eta\).

Linear regression is the GLM with a normal distribution and the identity link. The coefficients of a GLM are estimated by maximum likelihood, by iteratively reweighted least squares (AMDS: Maximum likelihood).

Table 14.1: Common generalized linear models
Outcome Distribution Link \(g(\mu)\) Model
Continuous Normal \(\mu\) Linear regression
Binary Bernoulli \(\log\frac{\mu}{1 - \mu}\) Logistic regression
Count Poisson \(\log \mu\) Poisson regression

14.4 Logistic regression

For a binary outcome, logistic regression models the probability of the positive class through the logit link:

\[ \log \frac{p(x)}{1 - p(x)} = x^\top \beta, \qquad p(x) = \frac{1}{1 + e^{-x^\top \beta}}. \tag{14.3}\]

The logistic function maps the linear predictor, which ranges over the real line, to a probability between 0 and 1 (Figure 14.1). The coefficient \(\beta_j\) is a difference in log odds, and \(e^{\beta_j}\) an odds ratio, per unit of \(x_j\). Maximizing the Bernoulli likelihood is equivalent to minimizing the log loss of Table 4.1.

Figure 14.1: The logistic function, which maps the linear predictor \(\eta\) to the probability \(p = 1/(1 + e^{-\eta})\).

The decision boundary of a logistic regression at threshold \(t\), the set of cases with \(p(x) = t\), is the hyperplane \(x^\top \beta = \log(t/(1-t))\): the model separates the classes with a linear boundary in feature space.

For an outcome with \(K > 2\) unordered classes, multinomial logistic regression fits one linear predictor per class and converts them to probabilities with the softmax function, \(p_k(x) = e^{x^\top \beta_k} / \sum_{l} e^{x^\top \beta_l}\), with one class as reference.

14.5 When the estimates are unstable

Two properties of the data make least squares and maximum likelihood estimates unstable.

Collinearity. When features are strongly correlated, \(X^\top X\) is close to singular, and small changes in the data produce large changes in \(\hat\beta\) (AMDS: Conditioning and numerical stability). Correlated features can receive large coefficients of opposite sign. The predictions, which depend on the combined contribution, are less affected than the individual coefficients. When \(p \ge n\), \(X^\top X\) is singular and the least squares solution is not unique.

Separation. In logistic regression, when a combination of features separates the two classes perfectly in the training data, the likelihood increases without bound as the coefficients grow, and the maximum likelihood estimate does not exist. Fitting procedures stop at very large coefficients and probabilities of 0 or 1.

Both problems are addressed by penalizing the size of the coefficients, which trades a small bias for a large reduction in variance (Chapter 18).

14.6 Constrained least squares

Constraints on the coefficients encode prior knowledge about their values. Non-negative least squares (NNLS) minimizes the residual sum of squares subject to \(\beta_j \ge 0\) for all \(j\) (Lawson and Hanson 1995). With the coefficients additionally scaled to sum to 1, the fit is a weighted average of the features. This is the form used to combine the predictions of several models in stacking (Chapter 36), where each feature is one model’s prediction and a negative weight would have no interpretation.

14.7 Properties for prediction

  • Low variance. With few features relative to cases, linear models change little between training sets, and their training and test errors are close.
  • Bias when the relationship is not linear. A linear model cannot represent curvature or interactions that are not included as terms, and the resulting error does not decrease with more data.
  • Interpretable structure. Each feature’s contribution to the linear predictor is a coefficient times its value, so the prediction decomposes exactly into feature contributions (Chapter 38).
  • A baseline. A flexible model is worth its complexity only if it predicts better, on held-out data, than a linear model of the same features. In a systematic review of clinical prediction studies, logistic regression performed comparably to machine learning methods on average (Christodoulou et al. 2019).

14.8 Summary

  • Linear regression predicts the conditional mean as a weighted sum of features, estimated by least squares.
  • The model is linear in the coefficients; transformations and interactions enter as additional columns.
  • Generalized linear models relate the mean of an exponential-family outcome to a linear predictor through a link function; logistic regression uses the logit link for binary outcomes and has a linear decision boundary.
  • Collinearity and separation make the estimates unstable; penalization stabilizes them.
  • Non-negative least squares constrains coefficients to be non-negative and is used to combine models.

14.9 Further reading