42  Calibration

A predicted probability is useful as a risk estimate only if it means what it says: among patients assigned a risk of 20%, about 20% should experience the event. This property, calibration, is distinct from discrimination, the ranking of patients, which the AUC measures (Section 8.3). A model can rank perfectly and be badly calibrated. This chapter defines calibration, describes how it is assessed and why models are miscalibrated, and presents methods that recalibrate a model’s probabilities.

42.1 Definition

A model with predicted probability \(\hat p(x)\) for a binary outcome is calibrated if

\[ P\left(Y = 1 \mid \hat p(X) = p\right) = p \quad \text{for all } p . \tag{42.1}\]

Van Calster et al. (2019) distinguish levels of calibration of increasing strength:

  • Mean calibration, or calibration-in-the-large: the mean predicted probability equals the observed proportion of events.
  • Weak calibration: the calibration intercept is 0 and the calibration slope is 1, in a logistic regression of the outcome on the logit of the predicted probability (COX 1958). A slope below 1 means that the predictions are too extreme, too close to 0 and 1, which is the signature of overfitting; a slope above 1 means they are too moderate.
  • Moderate calibration: Equation 42.1 holds, which is assessed with a calibration curve.

42.2 Assessing calibration

A calibration curve, or reliability diagram, plots the observed proportion of events against the predicted probability. Cases are grouped into bins of predicted probability, or the relationship is smoothed, and a calibrated model follows the diagonal. Calibration is assessed on held-out data: a model’s calibration on its training data reflects its fit, not its behavior on new cases.

The Brier score combines calibration and discrimination. It decomposes into a reliability term, which is zero for a calibrated model, a resolution term, which rewards predictions that differ from the prevalence, and an uncertainty term, \(\pi(1 - \pi)\), which depends only on the data (Murphy 1973).

42.3 Sources of miscalibration

  • Overfitting. A model that fits noise produces predictions that are too extreme on new data: calibration slope below 1.
  • Reweighting and resampling. Case weights and balanced resampling shift the probabilities toward the minority class (Section 40.3).
  • The algorithm. Some methods produce scores that are not probabilities, such as the decision values of a support vector machine. Others produce systematically distorted probabilities: averaging over the trees of a random forest pulls probabilities toward the middle, and boosting can push them toward 0 and 1 (Niculescu-Mizil and Caruana 2005).
  • A different population. A model applied where the prevalence or the case mix differs from its training data is miscalibrated there, even if it was calibrated where it was developed.

Figure 42.1 illustrates the first source. A logistic regression with 40 features, of which 5 are associated with the outcome, is fitted to 200 cases and evaluated on 5000 new cases.

Figure 42.1: Calibration on 5000 test cases of two logistic regressions fitted to 200 simulated cases: one with all 40 features, of which 35 are unrelated to the outcome, and one with the 5 related features.

The model with 40 features has a calibration slope of 0.32 on the test cases: its predictions are too extreme, and cases it assigns probabilities near 0 or 1 experience the event at rates much closer to the average. The model with the 5 related features has a slope of 0.72, closer to 1; its 6 coefficients, estimated from 200 cases, also fit some noise.

42.4 Recalibration

A miscalibrated model can be corrected by fitting a calibration map, a non-decreasing function from its predicted probabilities to calibrated ones, on cases not used to fit the model:

  • Platt scaling fits a logistic regression of the outcome on the model’s score or the logit of its probability, a two-parameter map that corrects the intercept and slope (Platt 1999).
  • Isotonic regression fits a non-decreasing step function (Section 22.6), which can correct any monotone distortion, and requires more calibration cases (Zadrozny and Elkan 2002).
  • Monotonic penalized fits, such as a monotonic highly adaptive lasso, give smooth non-decreasing maps.

Because the map is non-decreasing, it preserves the ranking of cases: a strictly increasing map leaves the AUC unchanged, and a step function changes it only by creating ties. The calibration cases must be separate from the cases used to fit the model, whose predictions are optimistic, and from the cases used to assess the calibrated model. In practice, the out-of-fold predictions of a cross-validation on the development data serve as calibration cases, and the calibrated model is assessed on a separate test set or by an outer resampling.

42.5 Summary

  • A calibrated model’s predicted probabilities equal the observed event rates; calibration is distinct from discrimination.
  • Calibration is assessed on held-out data by the mean, the calibration intercept and slope, and the calibration curve.
  • Overfitting, reweighting, the algorithm and a change of population cause miscalibration.
  • Platt scaling and isotonic regression fit a non-decreasing map from predicted to calibrated probabilities on cases not used to fit the model.

42.6 Further reading