db <- read_diabetes()
hf <- read_heart_failure()
cv <- setup_KFold(n_resamples = 10L, seed = 2026L)
ex <- setup_SerialExecution(seed = 2026L, device = "cpu")35 Neural Networks in rtemis
setup_MLP() fits multilayer perceptrons with the torch package, and setup_TabNet() fits TabNet models with the tabnet package. Both run on the device named by the execution configuration; the examples here use the CPU. This chapter fits MLPs to the diabetes and heart failure data, shows early stopping on a validation set, and tunes the architecture and weight decay.
35.1 Configuration
The architecture is given by hidden_units, one width per hidden layer, or generated from a shape. By default, three layers taper from a width derived from the number of inputs (“funnel”). Training uses the AdamW optimizer, a learning rate of 0.001, batches of 256 cases, and at most 100 epochs. Numeric features are standardized, and factors are represented by learned embeddings:
setup_MLP(hidden_units = c(32L, 16L), dropout = 0.1, weight_decay = 0.01, seed = 2026L)<MLPHyperparameters>
hyperparameters:
hidden_units: <int> 32, 16
activation: <chr> relu
norm_first: <lgc> FALSE
bias: <lgc> TRUE
residual: <lgc> FALSE
dropout: <nmr> 0.10
input_dropout: <nmr> 0.00
weight_decay: <nmr> 0.01
l1_penalty: <nmr> 0.00
embeddings: <lgc> TRUE
embedding_dropout: <nmr> 0.00
optimizer: <chr> adamw
lr: <nmr> 1e-03
batch_size: <int> 256
max_epochs: <int> 100
patience: <int> 10
seed: <int> 2026
num_workers: <int> 0
drop_last: <lgc> FALSE
ifw: <lgc> FALSE
unset: <chr> shape, shape_layers, shape_max_units, norm, embedding_dim, beta1, (+6 more)
tunable_hyperparameters: <int> 19
fixed_hyperparameters: <int> 13
tuned: <int> -1
resampled: <int> 0
n_workers: <int> 1
No search values defined for tunable hyperparameters.
35.2 Early stopping
With a validation set, training stops when the validation loss has not improved for patience epochs, and the weights of the best epoch are kept. A network trained on 80% of the diabetes cases, with the remaining 20% for validation:
split <- resample(db, setup_StratSub(n_resamples = 1L, train_p = 0.8, seed = 2026L))
mod_es <- train(
db[split[[1]]],
dat_validation = db[-split[[1]]],
hyperparameters = setup_MLP(
hidden_units = c(64L, 32L),
lr = 0.01,
max_epochs = 500L,
patience = 30L,
seed = 2026L
),
execution_config = ex,
verbosity = 0L
)
plot_learning(mod_es)The training and validation losses fall together over the first epochs. The validation loss reaches its minimum at epoch 48 and training stops 30 epochs later, while the training loss continues to fall.
35.3 Cross-validated performance
mlp_configs <- list(
`MLP, default` = setup_MLP(seed = 2026L),
`MLP, 32-16, lr 0.001` = setup_MLP(
hidden_units = c(32L, 16L), dropout = 0.1, weight_decay = 0.01,
lr = 0.001, seed = 2026L
),
`MLP, 32-16, lr 0.01` = setup_MLP(
hidden_units = c(32L, 16L), dropout = 0.1, weight_decay = 0.01,
lr = 0.01, seed = 2026L
)
)
mlp_db <- rbindlist(lapply(mlp_configs, function(hp) {
mod <- train(
db,
hyperparameters = hp,
outer_resampling_config = cv,
execution_config = ex,
verbosity = 0L
)
data.table(
rsq_training = mod@metrics_training@mean_metrics[["rsq"]],
rsq_test = mod@metrics_test@mean_metrics[["rsq"]]
)
}), idcol = "model")
mlp_db model rsq_training rsq_test
<char> <num> <num>
1: MLP, default 0.4944 0.4552
2: MLP, 32-16, lr 0.001 -1.3291 -1.3546
3: MLP, 32-16, lr 0.01 0.5519 0.5072The network with two layers of 32 and 16 units, dropout and weight decay reaches a training \(R^2\) of only -1.329 with the default learning rate of 0.001; a training \(R^2\) below 0 shows that the optimization has not converged in 100 epochs. With a learning rate of 0.01, the same network reaches a training \(R^2\) of 0.552 and a test \(R^2\) of 0.507. The learning rate and the number of epochs determine whether training converges, and are tuned with the architecture.
mod_mlp_hf <- train(
hf,
hyperparameters = setup_MLP(seed = 2026L),
outer_resampling_config = cv,
execution_config = ex,
question = hf_question
)<Resampled Classification Model>
MLP (Multilayer perceptron (torch))
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 210 276
No 64 2060
Showing mean (sd) across resamples.
Sensitivity: 0.431 (0.144)
Specificity: 0.970 (0.020)
Balanced Accuracy: 0.701 (0.064)
PPV: 0.784 (0.070)
NPV: 0.883 (0.024)
F1: 0.539 (0.109)
Accuracy: 0.870 (0.016)
AUC: 0.873 (0.021)
Brier Score: 0.098 (0.005)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 10 44
No 17 219
Showing mean (sd) across resamples.
Sensitivity: 0.193 (0.189)
Specificity: 0.928 (0.041)
Balanced Accuracy: 0.560 (0.099)
PPV: 0.365 (0.329)
NPV: 0.834 (0.041)
F1: NA (NA)
Accuracy: 0.790 (0.054)
AUC: 0.705 (0.102)
Brier Score: 0.148 (0.032)
35.4 Tuning
Architectures are tuned by listing candidate vectors in tune_over(), each a complete architecture (Chapter 21):
mod_mlp_tuned <- train(
db,
hyperparameters = setup_MLP(
hidden_units = tune_over(c(16L), c(32L, 16L), c(64L, 32L, 16L)),
lr = tune_over(0.001, 0.01),
dropout = 0.1,
weight_decay = 0.01,
seed = 2026L
),
tuner_config = setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L)
),
outer_resampling_config = cv,
execution_config = ex,
question = diabetes_question,
verbosity = 0L
)
rbindlist(lapply(mod_mlp_tuned@models, function(m) {
hp <- m@hyperparameters@hyperparameters
list(
hidden_units = paste(hp[["hidden_units"]], collapse = "-"),
lr = hp[["lr"]]
)
}), idcol = "fold") fold hidden_units lr
<char> <char> <num>
1: Fold_1 64-32-16 0.01
2: Fold_2 64-32-16 0.01
3: Fold_3 64-32-16 0.01
4: Fold_4 32-16 0.01
5: Fold_5 64-32-16 0.01
6: Fold_6 64-32-16 0.01
7: Fold_7 64-32-16 0.01
8: Fold_8 32-16 0.01
9: Fold_9 64-32-16 0.01
10: Fold_10 64-32-16 0.0135.5 TabNet
mod_tabnet <- train(
db,
hyperparameters = setup_TabNet(epochs = 200L),
outer_resampling_config = cv,
execution_config = ex,
verbosity = 0L
)35.6 Comparison
model rsq_test
<char> <num>
1: GLM 0.4935
2: MLP, default 0.4552
3: MLP, 32-16, lr 0.001 -1.3546
4: MLP, 32-16, lr 0.01 0.5072
5: MLP, tuned 0.5018
6: TabNet 0.1301On the diabetes data, the networks with a converged optimization perform comparably to the linear model, within the variation across folds; TabNet is far below it with 400 training cases. On the heart failure data, the default MLP has a mean test AUC of 0.705 and Brier score of 0.148. With a few hundred cases and ten features, the additional flexibility of a network does not translate into better predictions than a linear or additive model, and it brings optimization hyperparameters that must be tuned.