8  Performance Metrics

A performance metric summarizes how close a model’s predictions are to the observed outcomes. Different metrics measure different properties of the predictions, and the choice among them follows from how the predictions will be used (Section 1.3). This chapter defines the standard metrics for regression and classification, distinguishes metrics of class predictions from metrics of probabilities, and describes the baselines against which every metric is read.

8.1 Regression metrics

For outcomes \(y_i\) and predictions \(\hat y_i\) on \(n\) test cases, the common regression metrics are:

\[ \begin{aligned} \mathrm{MSE} &= \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat y_i)^2, & \mathrm{RMSE} &= \sqrt{\mathrm{MSE}}, \\ \mathrm{MAE} &= \frac{1}{n} \sum_{i=1}^{n} \lvert y_i - \hat y_i \rvert, & R^2 &= 1 - \frac{\sum_i (y_i - \hat y_i)^2}{\sum_i (y_i - \bar y)^2}. \end{aligned} \tag{8.1}\]

The mean squared error (MSE) is the average of the squared-error loss; it weights large errors heavily. The root mean squared error (RMSE) is in the units of the outcome. The mean absolute error (MAE) weights errors in proportion to their size and is less influenced by a few large errors. These three are minimized, respectively, by predicting the conditional mean (MSE, RMSE) and the conditional median (MAE).

\(R^2\) compares the model’s squared error to that of a constant prediction equal to the mean outcome \(\bar y\) of the cases evaluated. It equals 1 for perfect predictions and 0 for predictions no better than the constant. On test data it can be negative, when the model’s predictions are worse than the constant. Because it is relative to the variance of the outcome in the evaluated cases, \(R^2\) depends on that variance: the same model has a lower \(R^2\) in a sample whose outcomes vary less.

8.2 Class predictions and the confusion matrix

For a binary outcome with a positive class (for example, death) and a negative class (survival), each class prediction falls into one of four cells:

Table 8.1: Confusion matrix

	Predicted positive	Predicted negative
Observed positive	True positive (TP)	False negative (FN)
Observed negative	False positive (FP)	True negative (TN)

The metrics of class predictions are ratios of these counts:

Table 8.2: Metrics of class predictions

Metric	Definition	Interpretation
Sensitivity (recall, true positive rate)	\(\mathrm{TP} / (\mathrm{TP} + \mathrm{FN})\)	Proportion of positive cases predicted positive
Specificity (true negative rate)	\(\mathrm{TN} / (\mathrm{TN} + \mathrm{FP})\)	Proportion of negative cases predicted negative
Positive predictive value (precision, PPV)	\(\mathrm{TP} / (\mathrm{TP} + \mathrm{FP})\)	Proportion of positive predictions that are correct
Negative predictive value (NPV)	\(\mathrm{TN} / (\mathrm{TN} + \mathrm{FN})\)	Proportion of negative predictions that are correct
Accuracy	\((\mathrm{TP} + \mathrm{TN}) / n\)	Proportion of all predictions that are correct
Balanced accuracy	\((\text{sensitivity} + \text{specificity}) / 2\)	Mean of the per-class recalls
F1	\(2 \cdot \mathrm{PPV} \cdot \text{sensitivity} / (\mathrm{PPV} + \text{sensitivity})\)	Harmonic mean of precision and recall

Sensitivity and specificity are conditional on the observed class, and do not depend on the proportion of positive cases, the prevalence. PPV, NPV and accuracy do depend on it. By Bayes’ theorem (AMDS: Bayes’ theorem), with prevalence \(\pi\),

\[ \mathrm{PPV} = \frac{\text{sens} \cdot \pi}{\text{sens} \cdot \pi + (1 - \text{spec})(1 - \pi)}, \tag{8.2}\]

so a model with fixed sensitivity and specificity has a lower PPV in a population with lower prevalence.

Accuracy is dominated by the more frequent class. When 18.6% of patients die, as in the heart failure data, a constant prediction of survival for every patient has an accuracy of 81.4%, sensitivity 0 and specificity 1. Balanced accuracy gives the two classes equal weight, and equals 0.5 for any constant prediction.

8.3 Thresholds and the ROC curve

A probabilistic classifier produces a score \(\hat p(x)\), and a class prediction requires a threshold \(t\): predict positive if \(\hat p(x) \ge t\). Each threshold yields a different confusion matrix. Lowering the threshold predicts positive more often, which increases sensitivity and decreases specificity.

Figure 8.1 shows simulated scores for 800 negative and 200 positive cases, and Figure 8.2 the sensitivity and specificity obtained at each threshold.

Figure 8.1: Distribution of predicted probabilities among negative and positive cases (simulated).

Figure 8.2: Sensitivity and specificity as functions of the classification threshold, for the scores of Figure 8.1.

The receiver operating characteristic (ROC) curve plots sensitivity against \(1 - \text{specificity}\) over all thresholds (Fawcett 2006) (Figure 8.3). A model whose scores carry no information about the outcome follows the diagonal; a model whose scores separate the classes perfectly passes through the upper left corner.

Figure 8.3: ROC curve for the scores of Figure 8.1.

The area under the ROC curve (AUC), also called the concordance statistic or C-statistic, equals the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case (Hanley and McNeil 1982):

\[ \mathrm{AUC} = P\big(\hat p(X_+) > \hat p(X_-)\big), \tag{8.3}\]

with ties counted as one half. For the simulated scores, the AUC is 0.892. AUC measures discrimination, the ability of the scores to rank positive cases above negative ones. It is unaffected by any strictly increasing transformation of the scores, and so it does not measure whether the scores are accurate probabilities. It equals 0.5 for uninformative scores.

When positive cases are rare and the use of the model concerns them, the precision–recall curve, which plots PPV against sensitivity, shows the trade-off on the scale of positive predictions; unlike the ROC curve, it depends on prevalence (Saito and Rehmsmeier 2015).

8.4 Metrics of probabilities

A model that outputs probabilities can be evaluated on the probabilities themselves, without choosing a threshold. With \(y_i \in \{0, 1\}\) and predicted probabilities \(\hat p_i\) of the positive class:

\[ \text{Brier score} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat p_i)^2, \qquad \text{Log loss} = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log \hat p_i + (1 - y_i) \log (1 - \hat p_i) \right]. \tag{8.4}\]

Both are strictly proper scoring rules: their expected value is minimized only by reporting the true conditional probability (Gneiting and Raftery 2007; Brier 1950). They therefore reward probabilities that are both discriminating and calibrated, meaning that among cases assigned a probability near \(p\), a proportion near \(p\) are positive (Chapter 42). The log loss penalizes confident errors without bound: a predicted probability of 0 for a case that is positive incurs infinite loss.

The Brier score of a constant prediction equal to the prevalence \(\pi\) is \(\pi(1 - \pi)\), which is the reference value for a model with no discrimination. For the heart failure data, with \(\pi =\) 0.186, this is 0.152.

8.5 Multiclass metrics

With \(K > 2\) classes, the confusion matrix is \(K \times K\). Per-class sensitivity, specificity and PPV are computed by treating each class in turn as positive and the others as negative (one-vs-rest). Balanced accuracy is the mean of the per-class sensitivities. Summaries across classes are either macro-averaged, giving each class equal weight, or weighted by class frequency. The multiclass Brier score sums the squared differences over the \(K\) class probabilities (Table 4.1).

8.6 Baselines

A metric value is informative only relative to a baseline that uses no information from the features:

Table 8.3: Baselines for common metrics

Metric	Baseline	Baseline value
MSE, RMSE, MAE	Constant prediction: training mean (or median for MAE)	Depends on outcome scale
\(R^2\)	Constant prediction: mean of the evaluated outcomes	0
Accuracy	Most frequent class	Proportion of the most frequent class
Balanced accuracy	Any constant prediction	0.5
AUC	Uninformative scores	0.5
Brier score	Constant probability equal to prevalence	\(\pi(1 - \pi)\)

8.7 Choosing a metric

The metric follows from the use of the model:

• When predictions inform decisions at a fixed threshold, the metrics of class predictions at that threshold, together with the costs of false positives and false negatives, describe the consequences. Decision curve analysis summarizes the net benefit of a model across a range of thresholds (Vickers and Elkin 2006).
• When the model ranks cases, for example to prioritize patients for review, AUC measures the quality of the ranking.
• When the probabilities themselves are communicated or used, for example as a risk estimate given to a clinician, a proper scoring rule such as the Brier score or log loss, together with an assessment of calibration, is required.

Several metrics are usually reported together, since each describes one aspect of performance (Steyerberg 2019). The metric used to select among models, for example in hyperparameter tuning (Chapter 20), is chosen before the analysis.

8.8 Summary

• MSE, RMSE and MAE measure regression error on the outcome scale; \(R^2\) compares the error to that of the mean.
• Class predictions are summarized by the confusion matrix; sensitivity and specificity do not depend on prevalence, while PPV, NPV and accuracy do.
• The ROC curve shows the trade-off between sensitivity and specificity over all thresholds; the AUC measures discrimination.
• The Brier score and log loss are proper scoring rules for probabilities and reward both discrimination and calibration.
• Every metric is read against a baseline that uses no feature information.

8.9 Further reading

• AMDS: Probability (conditional probability and Bayes’ theorem), Information Theory (cross-entropy).
• Fawcett (2006) on ROC analysis; Hanley and McNeil (1982) on the interpretation of the AUC; Saito and Rehmsmeier (2015) on precision–recall curves.
• Gneiting and Raftery (2007) on proper scoring rules; Vickers and Elkin (2006) on decision curve analysis; Steyerberg (2019) on the evaluation of clinical prediction models.
• Wikipedia: Confusion matrix, Sensitivity and specificity, Receiver operating characteristic, Brier score, Scoring rule.

Brier, Glenn W. 1950. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78 (1): 1–3. https://doi.org/10.1175/1520-0493(1950)0782.0.co;2.

Fawcett, Tom. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8): 861–74. https://doi.org/10.1016/j.patrec.2005.10.010.

Gneiting, Tilmann, and Adrian E Raftery. 2007. “Strictly Proper Scoring Rules, Prediction, and Estimation.” Journal of the American Statistical Association 102 (477): 359–78. https://doi.org/10.1198/016214506000001437.

Hanley, J A, and B J McNeil. 1982. “The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve.” Radiology 143 (1): 29–36. https://doi.org/10.1148/radiology.143.1.7063747.

Saito, Takaya, and Marc Rehmsmeier. 2015. “The Precision-Recall Plot Is More Informative Than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.” PLOS ONE 10 (3): e0118432. https://doi.org/10.1371/journal.pone.0118432.

Steyerberg, Ewout W. 2019. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. In Statistics for Biology and Health. Springer International Publishing. https://doi.org/10.1007/978-3-030-16399-0.

Vickers, Andrew J., and Elena B. Elkin. 2006. “Decision Curve Analysis: A Novel Method for Evaluating Prediction Models.” Medical Decision Making 26 (6): 565–74. https://doi.org/10.1177/0272989x06295361.