hf30 <- read_heart_failure(horizon = 30L)
table(hf30[["death"]])
No Yes
259 35 cv <- setup_KFold(n_resamples = 10L, seed = 2026L)
ex <- setup_SerialExecution(seed = 2026L)41 Class Imbalance in rtemis
Every rtemis classification learner has an ifw hyperparameter that applies inverse frequency weighting: each case is weighted by the inverse of its class’s frequency in the training set, scaled so that the majority class has weight 1. This chapter uses the heart failure data with a 30-day horizon, in which the imbalance is stronger than at 60 days, to follow the order of considerations of Section 40.4.
35 of the 294 patients, 11.9%, died within 30 days. Each training fold of a 10-fold cross-validation contains about 32 deaths. class_imbalance() summarizes the imbalance as \(K \sum_k (n_k/N - 1/K)^2\), which is 0 for balanced classes and approaches 1 as one class dominates:
class_imbalance(hf30[["death"]])[1] 0.580541.1 Training with and without weights
The helper below trains a model with 10-fold cross-validation and summarizes its out-of-fold predictions: AUC and Brier score, the mean predicted probability, its standard deviation across patients, and the sensitivity at a threshold of 0.5.
summarize_cv <- function(hp, label) {
mod <- train(
hf30,
hyperparameters = hp,
outer_resampling_config = cv,
execution_config = ex,
verbosity = 0L
)
oof <- oof_predictions(mod)
list(
model = mod,
summary = data.table(
model = label,
auc = mod@metrics_test@mean_metrics[["auc"]],
brier = mod@metrics_test@mean_metrics[["brier_score"]],
mean_prob = mean(oof[["prob"]]),
sd_prob = sd(oof[["prob"]]),
sensitivity = mod@metrics_test@mean_metrics[["sensitivity"]]
)
)
}
runs <- list(
glm = summarize_cv(setup_GLM(), "GLM"),
glm_ifw = summarize_cv(setup_GLM(ifw = TRUE), "GLM, ifw"),
glmnet = summarize_cv(setup_GLMNET(ifw = FALSE), "GLMNET"),
glmnet_ifw = summarize_cv(setup_GLMNET(ifw = TRUE), "GLMNET, ifw"),
lgb = summarize_cv(setup_LightGBM(), "LightGBM"),
lgb_ifw = summarize_cv(setup_LightGBM(ifw = TRUE), "LightGBM, ifw")
)Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!
Warning in eval(family$initialize): non-integer #successes in a binomial glm!rbindlist(lapply(runs, `[[`, "summary")) model auc brier mean_prob sd_prob sensitivity
<char> <num> <num> <num> <num> <num>
1: GLM 0.6988 0.09838 0.1180 0.135694 0.1333
2: GLM, ifw 0.6808 0.19984 0.3904 0.241385 0.5083
3: GLMNET 0.5777 0.10455 0.1188 0.003273 0.0000
4: GLMNET, ifw 0.6973 0.23725 0.4880 0.035894 0.6667
5: LightGBM 0.7041 0.09960 0.1178 0.061187 0.0000
6: LightGBM, ifw 0.6585 0.15527 0.2891 0.224927 0.4000Three patterns appear.
The unweighted models are calibrated in the mean. The mean predicted probability of the unweighted logistic regression is 0.118, close to the prevalence of 0.119, and its Brier score, 0.098, is below the baseline of 0.105. Its sensitivity at 0.5 is 0.133: few patients reach a probability of 0.5, although the AUC shows that the probabilities rank patients who died above those who did not.
Weighting shifts the probabilities. With inverse frequency weights, the mean predicted probability of the logistic regression rises to 0.390, 3.3 times the prevalence. The AUC changes little, sensitivity at 0.5 rises, and the Brier score, 0.200, is worse than the baseline: the probabilities do not describe the risk of death in this population. The same holds for gradient boosting.
Some models fail without weights. The elastic net with the default one-standard-error rule (Section 18.5) has a standard deviation of predicted probabilities across patients of 0.0033 and a mean AUC of 0.578: with about 31 deaths per training fold, the selected penalty shrinks the coefficients to zero or near zero, and the model predicts nearly the prevalence for every patient. With weights, the deaths contribute as much to the loss as the survivors; the AUC rises to 0.697, and the mean predicted probability to 0.488.
41.2 Calibration curves
draw_calibration() plots, for groups of patients with similar predicted probability, the observed proportion of deaths against the mean predicted probability. A calibrated model follows the diagonal:
Above predicted probabilities of about 0.1, the weighted model’s curve lies below the diagonal: it overestimates the risk of death. The unweighted model’s groups lie closer to the diagonal, with the scatter expected from groups that contain few deaths.
41.3 Prior correction
For logistic regression, the shift is removed by Equation 40.2. ifw weights each death by \(n_0 / n_1\) relative to each survivor, the ratio of survivors to deaths in the training set, so the correction subtracts \(\log(n_0 / n_1)\) from the log odds. The ratio is computed in each outer training fold:
mod_ifw <- runs[["glm_ifw"]][["model"]]
log_ratio <- vapply(mod_ifw@outer_resampler@resamples, function(idx) {
y <- hf30[idx, death]
log(sum(y == "No") / sum(y == "Yes"))
}, numeric(1))
names(log_ratio) <- names(mod_ifw@y_test)
oof_glm_ifw[, prob_corrected := plogis(qlogis(prob) - log_ratio[resample])]
y01 <- as.integer(oof_glm_ifw[["observed"]] == "Yes")
c(
mean_prob_weighted = mean(oof_glm_ifw[["prob"]]),
mean_prob_corrected = mean(oof_glm_ifw[["prob_corrected"]]),
brier_weighted = mean((y01 - oof_glm_ifw[["prob"]])^2),
brier_corrected = mean((y01 - oof_glm_ifw[["prob_corrected"]])^2),
brier_unweighted = mean((y01 - oof_glm[["prob"]])^2)
) mean_prob_weighted mean_prob_corrected brier_weighted brier_corrected
0.39035 0.11823 0.19993 0.09876
brier_unweighted
0.09856 The corrected probabilities have a mean close to the prevalence, and their Brier score is close to that of the unweighted model. For algorithms without a closed-form correction, such as the weighted elastic net and gradient boosting, Chapter 43 fits a calibration model on held-out cases.
41.4 Choosing the threshold
With calibrated probabilities, the threshold follows from the costs of the errors (Equation 40.1). If missing a death is taken to cost nine times as much as a false alarm, \(t^* = 1 / (1 + 9) = 0.1\). Applied to the out-of-fold probabilities of the unweighted logistic regression, and compared with the weighted model at 0.5:
at_threshold <- function(observed, prob, t) {
predicted <- factor(ifelse(prob >= t, "Yes", "No"), levels = c("No", "Yes"))
m <- classification_metrics(observed, predicted, prob)@metrics[["overall"]]
m[c("sensitivity", "specificity", "ppv")]
}
rbind(
`GLM, threshold 0.1` = at_threshold(oof_glm[["observed"]], oof_glm[["prob"]], 0.1),
`GLM, ifw, threshold 0.5` = at_threshold(oof_glm_ifw[["observed"]], oof_glm_ifw[["prob"]], 0.5)
) sensitivity specificity ppv
GLM, threshold 0.1 0.6571 0.6332 0.1949
GLM, ifw, threshold 0.5 0.5143 0.7066 0.1915Both produce a trade-off between sensitivity and specificity far from that of the unweighted model at 0.5. The unweighted model reaches it by an explicit decision on the threshold, and keeps probabilities that can be reported as risks; the weighted model reaches a similar trade-off through distorted probabilities.