22  Additive Models and Basis Expansions

A linear model represents curvature only through terms specified in advance (Section 14.2). A basis expansion replaces each feature by a set of functions of it, chosen so that a linear combination of them can approximate a wide class of shapes, and fits the coefficients of that combination. The model remains linear in its parameters and is fitted with the tools of the previous chapters, often with a penalty that controls the roughness of the fitted function. This chapter presents splines, generalized additive models, multivariate adaptive regression splines, the highly adaptive lasso, and isotonic regression.

22.1 Basis functions

A basis expansion of a feature \(x\) represents its effect as

\[ f(x) = \sum_{m=1}^{M} \beta_m h_m(x), \tag{22.1}\]

where \(h_1, \dots, h_M\) are fixed functions, the basis functions, and the coefficients \(\beta_m\) are estimated by least squares or maximum likelihood (AMDS: Basis and dimension). Polynomials, \(h_m(x) = x^m\), are one choice; they are global, so that the fit in one region of \(x\) depends on the data in every other region, and high-degree polynomials oscillate (Figure 4.1). Bases of local functions avoid both problems:

  • Hinge functions \((x - t)_+ = \max(0, x - t)\) and \((t - x)_+\), which are zero on one side of a knot \(t\) and linear on the other, produce piecewise linear fits.
  • B-splines of degree 3 are piecewise cubic polynomials that are nonzero over a few adjacent intervals between knots; their linear combinations are the cubic splines, piecewise cubic functions with continuous first and second derivatives at the knots.
Figure 22.1: Two bases. Top: the pair of hinge functions with a knot at 0.5. Bottom: cubic B-spline basis functions with interior knots at 0.2, 0.4, 0.6 and 0.8.

22.2 Smoothing splines and penalized splines

The number and position of the knots control the flexibility of a spline. An alternative to choosing them is to place many knots and penalize roughness. A penalized regression spline minimizes

\[ \sum_{i=1}^{n} \left(y_i - f(x_i)\right)^2 + \lambda \int f''(x)^2 \, dx , \tag{22.2}\]

where the integral of the squared second derivative measures the curvature of \(f\) (Wahba 1990; Eilers and Marx 1996). At \(\lambda = 0\) the fit interpolates; as \(\lambda \to \infty\) it becomes a straight line, which has zero second derivative. The effective degrees of freedom of the fit, the trace of the matrix that maps \(y\) to the fitted values, decreases from the number of basis functions toward 2 as \(\lambda\) grows, and summarizes the flexibility the penalty allows. \(\lambda\) is chosen by generalized cross-validation or restricted maximum likelihood (Wood 2017).

22.3 Generalized additive models

A generalized additive model (GAM) replaces each linear term of a GLM by a smooth function of the feature (Hastie and Tibshirani 1986; Wood 2017):

\[ g\left(\mathbb{E}[Y \mid x]\right) = \beta_0 + f_1(x_1) + f_2(x_2) + \dots + f_p(x_p), \tag{22.3}\]

where each \(f_j\) is a penalized spline with its own smoothing parameter. The model is additive: the effect of each feature does not depend on the others. Each \(f_j\) can be plotted as a partial effect curve, which retains the interpretability of a linear model while allowing curvature. A smooth whose penalty selects 1 effective degree of freedom is linear. Interactions can be added as smooth functions of two or more features, at the cost of interpretability and data.

22.4 Multivariate adaptive regression splines

MARS builds a model from hinge functions whose knots are chosen from the data (Friedman 1991). A forward pass adds, at each step, the pair of hinge functions \((x_j - t)_+\) and \((t - x_j)_+\), for the feature \(x_j\) and knot \(t\), that most reduces the residual sum of squares, and, when interactions are allowed, products of a new hinge pair with a term already in the model. A backward pass then removes terms one at a time, keeping the subset with the lowest generalized cross-validation criterion, which penalizes the residual error by the number of terms and knots. The result is a piecewise linear model whose knots and interactions are selected adaptively, with the degree of interaction as its main hyperparameter.

22.5 The highly adaptive lasso

The highly adaptive lasso (HAL) generates a basis of indicator functions, or of their integrals, the hinge functions, at every observed value of every feature and of every product of features up to a chosen interaction order, and fits a lasso over that basis (Benkeser and van der Laan 2016; van der Laan 2017). The lasso selects and shrinks the basis functions. For functions whose sectional variation norm is bounded, HAL converges at a rate whose power of \(n\) does not depend on the number of features. The basis is large: it grows with the number of cases and combinatorially with the interaction order, which limits HAL to moderate sample sizes and few interaction levels.

22.6 Isotonic regression

Isotonic regression fits the non-decreasing function of a single feature that minimizes the squared error. The solution is a step function, computed by the pool-adjacent-violators algorithm, which merges adjacent groups of cases whose means violate the order until the sequence of means is non-decreasing (Ayer et al. 1955). It requires no choice of knots or smoothness, and imposes only monotonicity. Its main use in this book is to recalibrate predicted probabilities, which should be a non-decreasing function of the true probability (Chapter 42). Monotonic HAL provides a smooth, penalized alternative.

22.7 A comparison

Figure 22.2 fits the diabetes outcome as a function of a single feature, the serum measurement s5, with four of the methods of this chapter.

Figure 22.2: Disease progression against the serum measurement s5 in the diabetes data, with a linear fit, a penalized spline (GAM), a MARS fit and an isotonic regression.

The penalized spline uses 3.81 effective degrees of freedom for the curve, against 1 for the straight line; for body mass index alone, the same procedure selects close to 1, a straight line, since the penalty finds no curvature that the data support. MARS places knots where the slope changes. The isotonic fit is a step function that follows local averages under the constraint of monotonicity, and has no smoothness.

22.8 Summary

  • Basis expansions model nonlinear effects with linear combinations of fixed functions, and are fitted with the methods of linear models.
  • Splines use local piecewise polynomial bases; penalized splines control roughness with a penalty whose weight is chosen from the data.
  • Generalized additive models fit one smooth function per feature, and remain interpretable through partial effect curves.
  • MARS selects hinge functions, knots and interactions adaptively; HAL fits a lasso over a large basis of indicator or hinge functions.
  • Isotonic regression fits a monotone step function and is used for calibration.

22.9 Further reading