db <- read_diabetes()
hf <- read_heart_failure()
cv <- setup_KFold(n_resamples = 10L, seed = 2026L)29 Decision Trees in rtemis
rtemis fits CART trees with setup_CART(), using rpart; single trees in LightGBM with setup_LightCART(); model-based trees with setup_GLMTree(), using partykit; and linear additive trees with setup_LINAD(). This chapter fits and reads a CART tree, examines the effect of depth and the instability of the splits, and compares the tree variants that fit linear models in their leaves.
29.1 A classification tree
A tree of depth at most 3, fitted to all heart failure patients:
mod_cart <- train(
hf,
hyperparameters = setup_CART(maxdepth = 3L),
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question,
verbosity = 0L
)
mod_cart@modeln= 290
node), split, n, loss, yval, (yprob)
* denotes terminal node
1) root 290 54 No (0.8138 0.1862)
2) serum_creatinine< 1.815 244 33 No (0.8648 0.1352)
4) age< 81.5 233 27 No (0.8841 0.1159) *
5) age>=81.5 11 5 Yes (0.4545 0.5455)
10) age>=83.5 8 3 No (0.6250 0.3750) *
11) age< 83.5 3 0 Yes (0.0000 1.0000) *
3) serum_creatinine>=1.815 46 21 No (0.5435 0.4565)
6) age< 74 31 11 No (0.6452 0.3548)
12) ejection_fraction< 32.5 16 3 No (0.8125 0.1875) *
13) ejection_fraction>=32.5 15 7 Yes (0.4667 0.5333) *
7) age>=74 15 5 Yes (0.3333 0.6667)
14) serum_sodium>=133.5 8 3 No (0.6250 0.3750) *
15) serum_sodium< 133.5 7 0 Yes (0.0000 1.0000) *Each line is a node: its split, the number of cases, the number misclassified, the predicted class and the class proportions; * marks a leaf. The first split is on serum creatinine, and the second-level splits on age. Several leaves contain fewer than 10 patients, and two are pure, with predicted probabilities of 0 or 1.
29.2 Depth and flexibility
depths <- c(1L, 2L, 3L, 4L, 6L, 10L)
cart_depth <- rbindlist(lapply(depths, function(d) {
mod <- train(
db,
hyperparameters = setup_CART(maxdepth = d, cp = 0.001),
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
data.table(
depth = d,
rsq_training = mod@metrics_training@mean_metrics[["rsq"]],
rsq_test = mod@metrics_test@mean_metrics[["rsq"]]
)
}))
draw_line(
depths,
list(Training = cart_depth[["rsq_training"]], Test = cart_depth[["rsq_test"]]),
xlab = "Maximum depth",
ylab = "R-squared"
)Training \(R^2\) increases with depth, and test \(R^2\) peaks at depth 4, at 0.323, well below the linear model’s 0.493. The result is consistent with the near-linear dependence of progression on several features (Section 23.1): a step function needs many leaves to approximate a linear effect, and with about 400 training cases the leaves of a deep tree contain few cases.
29.3 Instability
Refitting the depth-3 tree to 10 bootstrap samples of the heart failure data and recording the first split of each:
boot <- resample(hf, setup_StratBoot(n_resamples = 10L, seed = 2026L))
root_splits <- vapply(boot@resamples, function(idx) {
mod <- train(
hf[idx],
hyperparameters = setup_CART(maxdepth = 3L),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
paste0(mod@model[["frame"]][["var"]][1], " < ", round(mod@model[["splits"]][1, "index"], 2))
}, character(1))
table(root_splits)root_splits
age < 86.5 serum_creatinine < 1.81 serum_creatinine < 1.86
4 5 1 The first split alternates between serum creatinine and age across bootstrap samples, and every split below it changes accordingly. The rules a tree displays describe one sample, and a similar sample can yield different rules.
29.4 Linear models in the leaves
setup_GLMTree() fits model-based recursive partitioning. By default, all features are both regressors of the leaf models and candidate partitioning variables:
mod_glmtree <- train(
db,
hyperparameters = setup_GLMTree(),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
mod_glmtree@modelLinear model tree
Model formula:
progression ~ age + sex + bmi + bp + s1 + s2 + s3 + s4 + s5 +
s6 | age + sex + bmi + bp + s1 + s2 + s3 + s4 + s5 + s6
Fitted party:
[1] root: n = 442
(Intercept) age sex2 bmi bp s1
-357.42679 -0.03636 -22.85965 5.60296 1.11681 -1.09000
s2 s3 s4 s5 s6
0.74645 0.37200 6.53383 68.48312 0.28012
Number of inner nodes: 0
Number of terminal nodes: 1
Number of parameters per node: 11
Objective function (residual sum of squares): 1263986On the full diabetes data, the parameter instability tests find no significant change in the linear model across any partitioning variable, at the default significance level alpha = 0.05 with Bonferroni correction, and the tree has a single node: it is the linear model of Section 15.1.
setup_LINAD() fits a linear additive tree. The number of leaves is a hyperparameter, tuned here by nested resampling with force_max_leaves = TRUE, which grows each candidate tree to exactly max_leaves leaves:
mod_linad <- train(
hf,
hyperparameters = setup_LINAD(
max_leaves = tune_over(1L, 2L, 4L, 8L, 16L),
force_max_leaves = TRUE
),
tuner_config = setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L),
metric = "brier_score",
maximize = FALSE
),
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question
)<Resampled Classification Model>
LINAD (Linear Additive Tree)
⚙ Tuned using exhaustive grid search.
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 25 461
No 9 2115
Showing mean (sd) across resamples.
Sensitivity: 0.051 (0.014)
Specificity: 0.996 (1.5e-03)
Balanced Accuracy: 0.524 (0.007)
PPV: 0.750 (0.096)
NPV: 0.821 (2.1e-03)
F1: 0.096 (0.026)
Accuracy: 0.820 (2.3e-03)
AUC: 0.802 (0.027)
Brier Score: 0.127 (3.2e-03)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 2 52
No 2 234
Showing mean (sd) across resamples.
Sensitivity: 0.037 (0.078)
Specificity: 0.992 (0.018)
Balanced Accuracy: 0.514 (0.037)
PPV: NA (NA)
NPV: 0.819 (0.018)
F1: NA (NA)
Accuracy: 0.814 (0.025)
AUC: 0.716 (0.118)
Brier Score: 0.139 (0.014)
Fold_1 Fold_2 Fold_3 Fold_4 Fold_5 Fold_6 Fold_7 Fold_8 Fold_9 Fold_10
8 16 16 8 4 4 8 8 8 16 plot_varimp() shows the two importance measures of a LINAD model, the standardized coefficients of the leaf models and the split gain, for the model of the first outer fold:
plot_varimp(mod_linad@models[[1]])29.5 Comparison
tree_models <- list(
`CART, tuned` = list(
hp = setup_CART(
cp = tune_over(0.001, 0.005, 0.01, 0.02, 0.05),
maxdepth = tune_over(2L, 3L, 4L, 6L)
),
metric = "brier_score"
),
GLM = list(hp = setup_GLM(), metric = NULL)
)
compare_hf <- rbindlist(lapply(names(tree_models), function(nm) {
tm <- tree_models[[nm]]
mod <- train(
hf,
hyperparameters = tm[["hp"]],
tuner_config = if (!is.null(tm[["metric"]])) {
setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L),
metric = tm[["metric"]],
maximize = FALSE
)
},
outer_resampling_config = cv,
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
data.table(model = nm, mod@metrics_test@mean_metrics[c("auc", "brier_score")])
}))
compare_hf <- rbind(
compare_hf,
data.table(model = "LINAD, tuned", mod_linad@metrics_test@mean_metrics[c("auc", "brier_score")])
)
compare_hf model auc brier_score
<char> <num> <num>
1: CART, tuned 0.5544 0.1749
2: GLM 0.7085 0.1395
3: LINAD, tuned 0.7161 0.1389The tuned CART tree has the lowest AUC and highest Brier score of the three: its predictions take a few distinct values, one per leaf, and its leaves contain few deaths. LINAD, whose leaves carry shrunken linear models, performs comparably to the logistic regression.