46  Dimensionality Reduction

The chapters so far have addressed predictive questions, in which an outcome defines the target and the error on new cases defines success. Unsupervised learning has no outcome. Dimensionality reduction, or decomposition, represents each case by a small number of new variables, components, derived from the original features, so that the data can be visualized, summarized, compressed, or passed to a supervised model with fewer inputs. This chapter presents linear methods, distance-preserving methods, nonlinear embeddings and autoencoders, and the criteria by which they are judged.

46.1 Principal component analysis

Principal component analysis (PCA) finds orthogonal directions in feature space along which the data vary most (Pearson 1901; Hotelling 1933). For centered data \(X\) (\(n \times p\)), the first principal component direction \(v_1\) is the unit vector that maximizes the variance of the projections \(X v_1\); each subsequent direction maximizes the variance among directions orthogonal to the previous ones. The directions are the eigenvectors of the covariance matrix \(X^\top X / (n - 1)\), and the variance of component \(j\) is its eigenvalue \(\lambda_j\) (AMDS: Principal component analysis from the SVD). Equivalently, with the singular value decomposition \(X = U D V^\top\), the scores of the cases on the components are \(U D\), and the loadings, the weights of the features in each component, are the columns of \(V\).

The proportion of variance explained by the first \(k\) components is \(\sum_{j \le k} \lambda_j / \sum_j \lambda_j\), and the first \(k\) components give the best rank-\(k\) approximation of \(X\) in squared error (AMDS: Low-rank approximation). PCA depends on the scale of the features, and features measured in different units are standardized first, which makes it a decomposition of the correlation matrix (Jolliffe and Cadima 2016).

Figure 46.1: Two correlated features and their principal component directions, drawn as segments of length four standard deviations of each component.

In Figure 46.1, the first component, along the shared variation of the two features, explains 92.5% of the variance.

46.2 Other linear methods

  • Independent component analysis (ICA) seeks components that are statistically independent, rather than merely uncorrelated, by maximizing their non-Gaussianity (Hyvärinen and Oja 2000). It separates mixed signals, such as sources recorded together by several sensors.
  • Non-negative matrix factorization (NMF) approximates a non-negative matrix by the product of two non-negative matrices, \(X \approx W H\) (Lee and Seung 1999). Since components can only add, they tend to represent parts of the data, such as groups of co-occurring features.

46.3 Distance-preserving methods

Multidimensional scaling (MDS) places cases in a low-dimensional space so that their distances approximate given dissimilarities. Classical MDS, also called principal coordinates analysis (PCoA), computes the coordinates from the eigendecomposition of the doubly centered matrix of squared distances (Torgerson 1952; Gower 1966); with Euclidean distances it is equivalent to PCA. Other dissimilarities, such as Gower’s for mixed feature types, make it applicable where PCA is not. Isomap replaces Euclidean distances by geodesic distances along a neighborhood graph, so that a curved low-dimensional surface embedded in many dimensions can be unrolled (Tenenbaum et al. 2000).

46.4 Nonlinear embeddings for visualization

t-SNE converts distances between cases into probabilities that two cases are neighbors, in the original space and in a two-dimensional embedding, and moves the embedded points to minimize the Kullback–Leibler divergence between the two distributions (van der Maaten and Hinton 2008) (AMDS: Kullback–Leibler divergence). Its perplexity sets the effective number of neighbors. UMAP builds a weighted neighborhood graph and optimizes an embedding whose graph is similar, with a different objective that also preserves more of the global arrangement (McInnes et al. 2018). Both reveal local neighborhoods and groups. The sizes of groups and the distances between them in the embedding are not interpretable as in the original space, and different settings or random seeds produce different pictures from the same data.

46.5 Autoencoders

An autoencoder is a neural network trained to reproduce its input through a narrow hidden layer, the code (Hinton and Salakhutdinov 2006). The encoder maps the input to the code and the decoder maps the code back; with linear activations and squared error, the optimal code spans the same subspace as PCA. With nonlinear activations, it learns a nonlinear decomposition. A variational autoencoder learns a probability distribution over codes, with a penalty that keeps it close to a standard normal distribution (Kingma and Welling 2014).

46.6 Application to new cases

Some methods define a function that maps any new case to the components: PCA, ICA and NMF by their loadings, PCoA by Gower’s out-of-sample formula, UMAP by its graph and autoencoders by their encoder. Others, such as t-SNE, Isomap and classical MDS as usually implemented, produce coordinates only for the cases they were fitted to. Only methods of the first kind can serve as a preprocessing step for a predictive model, since the same transformation must be applied to the test cases and to future cases. As an estimated transformation, it is fitted within each training set (Section 12.1).

46.7 Evaluation

Without an outcome, a decomposition is judged by what it preserves: the proportion of variance explained or the reconstruction error for methods that reconstruct the input, the preservation of neighborhoods for embeddings, and, when it serves a predictive model, the cross-validated performance of that model. A decomposition that explains most of the variance of the features need not retain the information that predicts an outcome, since the outcome may depend on a direction of small variance.

46.8 Summary

  • PCA finds orthogonal directions of maximal variance from the eigendecomposition of the covariance or correlation matrix.
  • ICA seeks independent components; NMF non-negative parts.
  • MDS and PCoA preserve distances; Isomap preserves geodesic distances.
  • t-SNE and UMAP produce nonlinear embeddings for visualization, in which global distances are not interpretable.
  • Autoencoders learn nonlinear codes with neural networks.
  • A decomposition used before a predictive model must apply to new cases and is fitted within each training set.

46.9 Further reading