db <- read_diabetes()
hf <- read_heart_failure()
cv <- setup_KFold(n_resamples = 10L, seed = 2026L)
inner <- setup_KFold(n_resamples = 5L, seed = 2027L)37 Stacking in rtemis
rtemis fits stacked ensembles with setup_SuperLearner(), setup_ConditionalSuperLearner() and setup_ModalityStacking(). Each takes a named list of base learner configurations, a meta learner, and the inner resampling that produces the cross-validated predictions. This chapter combines the learners of the previous chapters on both datasets.
37.1 A Super Learner
The library combines a linear model, an additive model, a nearest-neighbor model and a random forest. The default meta learner is non-negative least squares with coefficients normalized to sum to 1:
library_db <- list(
GLM = setup_GLM(),
GAM = setup_GAM(),
KNN = setup_KNN(k = 30L),
Ranger = setup_Ranger(seed = 2026L)
)
sl_db <- setup_SuperLearner(
base_learners = library_db,
inner_resampling_config = inner
)Fitted to all cases, the ensemble reports each learner’s weight and the cross-validated risk of its predictions:
mod_sl <- train(
db,
hyperparameters = sl_db,
execution_config = setup_SerialExecution(seed = 2026L),
question = diabetes_question,
verbosity = 0L
)
varimp_table(get_varimp(mod_sl)) variable weight cv_risk
1 GLM 0.21887 2988
2 GAM 0.69015 2930
3 KNN 0.09097 3308
4 Ranger 0.00000 3282GAM receives the largest weight. A learner with zero weight (Ranger) contributes nothing to the combination given the others; its own cross-validated risk can still be close to the others’. The ensemble’s performance is assessed by an outer cross-validation, within each fold of which the weights are estimated anew:
mod_sl_cv <- train(
db,
hyperparameters = sl_db,
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
question = diabetes_question
)<Resampled Regression Model>
SuperLearner (SuperLearner: cross-validated stacked ensemble)
⟳ Tested using 10 independent folds.
<Resampled Regression Training Metrics>
Showing mean (sd) across resamples.
MAE: 40.188 (1.394)
MSE: 2506.763 (177.221)
RMSE: 50.037 (1.832)
R²: 0.577 (0.030)
<Resampled Regression Test Metrics>
Showing mean (sd) across resamples.
MAE: 43.284 (4.253)
MSE: 2907.220 (479.560)
RMSE: 53.750 (4.498)
R²: 0.507 (0.076)
For the heart failure data, with a library of a logistic regression, a ridge logistic regression, a random forest and a nearest-neighbor model:
library_hf <- list(
GLM = setup_GLM(),
Ridge = setup_GLMNET(alpha = 0, which_lambda_cv = "lambda.min", ifw = FALSE),
Ranger = setup_Ranger(min_node_size = 20L, seed = 2026L),
KNN = setup_KNN(k = 60L)
)
mod_sl_hf <- train(
hf,
hyperparameters = setup_SuperLearner(
base_learners = library_hf,
inner_resampling_config = inner
),
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question
)<Resampled Classification Model>
SuperLearner (SuperLearner: cross-validated stacked ensemble)
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 78 408
No 0 2124
Showing mean (sd) across resamples.
Sensitivity: 0.160 (0.094)
Specificity: 1.000 (0.000)
Balanced Accuracy: 0.580 (0.047)
PPV: 1.000 (0.000)
NPV: 0.839 (0.015)
F1: 0.266 (0.137)
Accuracy: 0.844 (0.017)
AUC: 0.955 (0.021)
Brier Score: 0.094 (0.009)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 3 51
No 0 236
Showing mean (sd) across resamples.
Sensitivity: 0.057 (0.092)
Specificity: 1.000 (0.000)
Balanced Accuracy: 0.528 (0.046)
PPV: NA (NA)
NPV: 0.823 (0.020)
F1: NA (NA)
Accuracy: 0.824 (0.022)
AUC: 0.736 (0.121)
Brier Score: 0.136 (0.015)
37.2 A Conditional Super Learner
A Conditional Super Learner with two experts, a linear model and a random forest, and a depth-2 tree as oracle:
mod_csl <- train(
db,
hyperparameters = setup_ConditionalSuperLearner(
base_learners = list(GLM = setup_GLM(), Ranger = setup_Ranger(seed = 2026L)),
meta_learner = setup_CART(maxdepth = 2L),
inner_resampling_config = inner
),
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)During fitting, the region assigned to the random forest falls below the minimum region size (min_region_size, 10 cases), and the forest is refitted on the whole training set instead. The oracle routes few or no cases to the forest, and the mean test \(R^2\), 0.493, is close to that of the linear model: the cross-validated losses identify no region of the feature space in which the forest predicts better.
37.3 Stacking feature groups
The diabetes features form two groups: clinical measurements (age, sex, BMI, blood pressure) and the six serum measurements. setup_ModalityStacking() fits one model per group, here a linear model for each, and combines their predictions. With setup_NNLS(normalize = FALSE) as meta learner, the weights need not sum to 1, so that the two groups’ predictions can add:
mod_ms <- train(
db,
hyperparameters = setup_ModalityStacking(
feature_groups = list(
clinical = c("age", "sex", "bmi", "bp"),
serum = c("s1", "s2", "s3", "s4", "s5", "s6")
),
base_learners = list(clinical = setup_GLM(), serum = setup_GLM()),
meta_learner = setup_NNLS(normalize = FALSE),
inner_resampling_config = inner
),
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)37.4 Comparison
diabetes rsq
<char> <num>
1: GLM 0.4935
2: GAM 0.5112
3: Super Learner 0.5071
4: Conditional Super Learner 0.4934
5: Modality stacking 0.4640 heart_failure auc brier_score
<char> <num> <num>
1: GLM 0.7085 0.1395
2: Ranger 0.7451 0.1363
3: Super Learner 0.7356 0.1365The Super Learner’s mean test \(R^2\), 0.507, is close to that of its best member, the GAM. Modality stacking, at 0.464, is below the single linear model on all features, 0.493. Each group’s model is fitted without the other group’s features, so the coefficients of each reflect the omitted group’s contribution through their correlation, and the two predictions are combined with only two weights. Combining learners cannot add information that none of them extracts; it protects against the choice of a weak learner, at the cost of fitting all of them.