38  Interpretation

A predictive model is assessed by its error on new cases; it is also examined to understand how it uses the features, to detect errors such as leakage, and to communicate its behavior. This chapter describes global methods, which summarize the role of each feature across cases, and local methods, which decompose the prediction for one case. It closes with what such explanations describe, which is the fitted model, and what they do not, which is the process that generated the data.

38.1 Model-specific summaries

Some models are summarized by their own parameters. A linear model’s coefficients give the change in the linear predictor per unit of each feature (Section 14.1); a GAM’s smooth functions give each feature’s partial effect (Section 22.3); a small tree is a set of readable rules (Section 28.3). For other models, the fitting procedure provides an importance measure, such as the impurity reduction of a forest’s splits (Section 30.5) or the split gain of a boosted model. These measures are on different scales and measure different quantities, and are not comparable across algorithms.

38.2 Permutation importance

Permutation importance applies to any model. For feature \(j\), the values of \(x_j\) in a test set are randomly permuted, which preserves the feature’s distribution and breaks its relationship with the outcome and with the other features, and the increase in the model’s test error is recorded (Breiman 2001; Fisher et al. 2019). A feature the model does not use has importance near zero. Two properties qualify the result:

  • Correlated features share importance. When \(x_j\) is permuted, the model can still obtain the same information from a correlated feature, so each of two correlated, predictive features can appear unimportant.
  • Permutation creates unrealistic cases. Permuting one of two correlated features produces combinations of values that do not occur in the data, and the model’s predictions for them are extrapolations (Hooker et al. 2021).

38.3 Partial dependence

The partial dependence of the prediction on a feature \(x_j\) is the average prediction when \(x_j\) is set to a value \(v\) for every case, with the other features at their observed values (Friedman 2001):

\[ \mathrm{PD}_j(v) = \frac{1}{n} \sum_{i=1}^{n} \hat f\left(v, x_{i, -j}\right), \tag{38.1}\]

where \(x_{i,-j}\) are the features of case \(i\) other than \(j\). Plotted against \(v\), it shows the average shape of the model’s dependence on \(x_j\). For an additive model it equals the partial effect of \(x_j\) up to a constant. Individual conditional expectation curves plot the term inside the sum for each case separately, and reveal interactions as differing shapes (Goldstein et al. 2015). Like permutation, setting \(x_j\) to \(v\) for every case creates unrealistic combinations when features are correlated; accumulated local effects average differences in prediction over small intervals of \(x_j\) among cases with similar values, which avoids them (Apley and Zhu 2020).

38.4 Shapley values

A local explanation distributes the prediction for one case among its features. Shapley values, from cooperative game theory, do so with a unique set of properties (Shapley 1953). For a case \(x\), let \(v(S)\) be the expected prediction when only the features in the subset \(S\) are known, the others being averaged over a background distribution. The contribution of feature \(j\) is its average marginal contribution over all orderings of the features:

\[ \phi_j = \sum_{S \subseteq \{1, \dots, p\} \setminus \{j\}} \frac{\lvert S \rvert! \, (p - \lvert S \rvert - 1)!}{p!} \left[ v(S \cup \{j\}) - v(S) \right]. \tag{38.2}\]

The contributions satisfy efficiency: they add up to the difference between the prediction for the case and the average prediction over the background,

\[ \hat f(x) = \phi_0 + \sum_{j=1}^{p} \phi_j , \tag{38.3}\]

where \(\phi_0\) is the baseline (Štrumbelj and Kononenko 2014; Lundberg and Lee 2017). A feature that does not change any prediction receives zero.

Computing Equation 38.2 requires \(2^p\) subsets, and each \(v(S)\) an average over the background. For some model classes it is exact and fast: for a linear model, \(\phi_j = \beta_j (x_j - \bar x_j)\); for tree ensembles, the TreeSHAP algorithm computes exact values in polynomial time (Lundberg et al. 2020). For other models, Kernel SHAP estimates them from a sample of subsets by weighted regression (Lundberg and Lee 2017). Three choices define what is computed:

  • the background, the reference distribution, which defines the baseline \(\phi_0\);
  • the value function: interventional Shapley values average the unknown features over their marginal distribution, and attribute to the features the model uses; conditional values average over their distribution given the known features, and credit correlated proxies;
  • the scale: for classification, contributions are additive on the scale of the log odds, the margin, and not on the probability scale.

The mean absolute Shapley value of a feature across cases is a global importance measure.

38.5 What explanations describe

Every method in this chapter describes the fitted function \(\hat f\). A feature with large importance is one the model relies on; it need not cause the outcome, and a feature that causes the outcome can have small importance if a correlated feature carries the same information (Section 2.2). Two models with similar performance can rely on different features, which Fisher et al. (2019) study as the model class reliance across all well-performing models. Explanations are nevertheless useful for the predictive question itself: a feature with unexpectedly large importance can reveal leakage (Section 6.4), and an implausible partial dependence can reveal an artifact of the data.

38.6 Summary

  • Model-specific summaries, such as coefficients, partial effects and split gains, are on different scales and not comparable across algorithms.
  • Permutation importance measures the increase in test error when a feature’s values are permuted; correlated features share importance.
  • Partial dependence shows the average dependence of the prediction on one feature; ICE curves show it per case; ALE avoids unrealistic combinations.
  • Shapley values decompose one prediction into feature contributions that add up to the prediction minus a baseline; their meaning depends on the background, value function and scale.
  • Explanations describe the fitted model, not the data-generating process.

38.7 Further reading