27  Support Vector Machines in rtemis

setup_LinearSVM() and setup_RadialSVM() fit support vector machines with a linear and a radial basis function kernel, using e1071::svm(). The backend standardizes the features, and for regression the outcome, before fitting; rtemis one-hot encodes factors. This chapter fits support vector regression to the diabetes data, examines the effects of \(C\) and \(\gamma\), and tunes them jointly.

db <- read_diabetes()
cv <- setup_KFold(n_resamples = 10L, seed = 2026L)
setup_RadialSVM()

<RadialSVMHyperparameters>
        hyperparameters: 
                         kernel: <chr> radial
                           cost: <nmr> 1.00
                          gamma: <nmr> 0.01
                            ifw: <lgc> FALSE
tunable_hyperparameters: <int> 3
  fixed_hyperparameters: <int> 0
                  tuned: <int> -1
              resampled: <int> 0
              n_workers: <int> 1

  No search values defined for tunable hyperparameters.

cost is \(C\) of Equation 26.2 and gamma is \(\gamma\) of the RBF kernel (Table 26.1).

27.1 Linear kernel

svm_linear <- rbindlist(lapply(c(0.01, 0.1, 1, 10), function(cost) {
  mod <- train(
    db,
    hyperparameters = setup_LinearSVM(cost = cost),
    outer_resampling_config = cv,
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
  data.table(cost = cost, rsq_test = mod@metrics_test@mean_metrics[["rsq"]])
}))
svm_linear

    cost rsq_test
   <num>    <num>
1:  0.01   0.4935
2:  0.10   0.4897
3:  1.00   0.4889
4: 10.00   0.4882

With a linear kernel, support vector regression fits a linear function with the \(\varepsilon\)-insensitive loss and a ridge penalty, and its performance is close to that of the least squares fit for every value of \(C\) examined.

27.2 Radial basis function kernel

The RBF kernel adds the width \(\gamma\). A grid of values shows how the two hyperparameters interact:

grid_svm <- CJ(gamma = c(0.01, 0.1), cost = c(0.1, 1, 10, 100))
svm_radial <- rbindlist(lapply(seq_len(nrow(grid_svm)), function(i) {
  mod <- train(
    db,
    hyperparameters = setup_RadialSVM(cost = grid_svm[i, cost], gamma = grid_svm[i, gamma]),
    outer_resampling_config = cv,
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
  data.table(
    grid_svm[i],
    rsq_training = mod@metrics_training@mean_metrics[["rsq"]],
    rsq_test = mod@metrics_test@mean_metrics[["rsq"]]
  )
}))
svm_radial

   gamma  cost rsq_training rsq_test
   <num> <num>        <num>    <num>
1:  0.01   0.1       0.4455   0.4332
2:  0.01   1.0       0.5394   0.5048
3:  0.01  10.0       0.5720   0.4927
4:  0.01 100.0       0.6088   0.4477
5:  0.10   0.1       0.5227   0.4695
6:  0.10   1.0       0.6598   0.4669
7:  0.10  10.0       0.8224   0.2786
8:  0.10 100.0       0.9601  -0.3496

Large \(\gamma\) and large \(C\) together produce the most flexible fit: at \(\gamma\) = 0.1 and \(C\) = 100, the training \(R^2\) is 0.960 and the test \(R^2\) -0.350. The best combination in this grid, \(\gamma\) = 0.01 and \(C\) = 1, has a test \(R^2\) of 0.505. The choice of the best cell from this table uses the outer test folds, so its value is optimistic (Section 20.3); nested tuning gives an estimate free of this selection.

27.3 Nested tuning

mod_svm <- train(
  db,
  hyperparameters = setup_RadialSVM(
    cost = tune_over(0.1, 1, 10, 100),
    gamma = tune_over(0.001, 0.01, 0.1)
  ),
  tuner_config = setup_GridSearch(
    resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L)
  ),
  outer_resampling_config = cv,
  execution_config = setup_SerialExecution(seed = 2026L),
  question = diabetes_question
)

<Resampled Regression Model>
RadialSVM (SVM with radial kernel (e1071))
⚙ Tuned using exhaustive grid search.
⟳ Tested using 10 independent folds.

  <Resampled Regression Training Metrics>
  Showing mean (sd) across resamples.
   MAE: 41.565 (0.611)
   MSE: 2730.074 (55.819)
  RMSE: 52.248 (0.534)
    R²: 0.540 (0.009)

  <Resampled Regression Test Metrics>
  Showing mean (sd) across resamples.
   MAE: 43.677 (4.785)
   MSE: 2926.214 (549.015)
  RMSE: 53.879 (5.090)
    R²: 0.504 (0.088)

rbindlist(lapply(mod_svm@models, function(m) {
  as.list(m@hyperparameters@hyperparameters[c("cost", "gamma")])
}), idcol = "fold")

       fold  cost gamma
     <char> <num> <num>
 1:  Fold_1     1 0.010
 2:  Fold_2     1 0.010
 3:  Fold_3   100 0.001
 4:  Fold_4     1 0.010
 5:  Fold_5     1 0.010
 6:  Fold_6     1 0.010
 7:  Fold_7     1 0.010
 8:  Fold_8     1 0.010
 9:  Fold_9     1 0.010
10: Fold_10     1 0.010

The nested estimate of the tuned RBF support vector regression is a mean test \(R^2\) of 0.504, comparable to the linear and additive models of the previous chapters on these data.