13  Preprocessing in rtemis

rtemis applies preprocessing in two settings. preprocess() transforms a dataset once, with a configuration from setup_Preprocessor(); it serves data preparation and fixed transformations. train() accepts a configuration from setup_SupervisedPreprocessor() through its preprocessor_config argument, and estimates the transformation within every training set, as Section 12.1 requires. This chapter uses both, and then measures the bias that results from estimating a feature selection on the full data.

13.1 Standalone preprocessing

setup_Preprocessor() lists every available operation, with defaults that leave the data unchanged. The configuration below standardizes the numeric features and one-hot encodes the factors:

hf <- read_heart_failure()
x <- features(hf)
pp_config <- setup_Preprocessor(scale = TRUE, one_hot = TRUE)

preprocess() returns a Preprocessor object holding the transformed data and the fitted configuration. It modifies a data.table in place, so a copy is passed to keep x unchanged:

pp <- preprocess(copy(x), config = pp_config)
head(preprocessed(pp), 3)

       age anaemia_No anaemia_Yes creatinine_phosphokinase diabetes_No
     <num>      <num>       <num>                    <num>       <num>
1:  1.1771          1           0                -0.004465           1
2: -0.4902          1           0                 7.463250           1
3:  0.3434          1           0                -0.451768           1
   diabetes_Yes ejection_fraction high_blood_pressure_No
          <num>             <num>                  <num>
1:            0         -1.525588                      0
2:            0         -0.009295                      1
3:            0         -1.525588                      1
   high_blood_pressure_Yes platelets serum_creatinine serum_sodium sex_Female
                     <num>     <num>            <num>        <num>      <num>
1:                       1   0.02433          0.48301      -1.4782          0
2:                       0   0.00766         -0.28330      -0.1365          0
3:                       0  -1.02148         -0.09173      -1.7019          0
   sex_Male smoking_No smoking_Yes
      <num>      <num>       <num>
1:        1          1           0
2:        1          1           0
3:        1          0           1

Each factor with two levels becomes two indicator columns. The fitted configuration stores the estimated parameters, here the training means and standard deviations:

fitted <- fitted_config(pp)
rbind(center = fitted@scale_centers, scale = fitted@scale_coefficients)

         age creatinine_phosphokinase ejection_fraction platelets
center 60.88                    586.4             38.11    262.60
scale  12.00                    974.7             11.87     98.49
       serum_creatinine serum_sodium
center            1.396      136.610
scale             1.044        4.472

apply_preprocessor() applies the fitted transformation to new data, using the stored parameters rather than re-estimating them:

apply_preprocessor(pp, x[1:3])

       age anaemia_No anaemia_Yes creatinine_phosphokinase diabetes_No
     <num>      <num>       <num>                    <num>       <num>
1:  1.1771          1           0                -0.004465           1
2: -0.4902          1           0                 7.463250           1
3:  0.3434          1           0                -0.451768           1
   diabetes_Yes ejection_fraction high_blood_pressure_No
          <num>             <num>                  <num>
1:            0         -1.525588                      0
2:            0         -0.009295                      1
3:            0         -1.525588                      1
   high_blood_pressure_Yes platelets serum_creatinine serum_sodium sex_Female
                     <num>     <num>            <num>        <num>      <num>
1:                       1   0.02433          0.48301      -1.4782          0
2:                       0   0.00766         -0.28330      -0.1365          0
3:                       0  -1.02148         -0.09173      -1.7019          0
   sex_Male smoking_No smoking_Yes
      <num>      <num>       <num>
1:        1          1           0
2:        1          1           0
3:        1          0           1

The operations run in a fixed order, documented in ?setup_Preprocessor: filters of features and cases first, then type conversions, missingness indicators, imputation, scaling, and one-hot encoding last.

13.2 Preprocessing within training

setup_SupervisedPreprocessor() offers the operations that a fitted model can carry and apply to new cases. It excludes operations that remove cases, such as keeping only complete cases, since a model must return one prediction for every case it is given.

The configuration below standardizes the numeric features within each training set of a 10-fold cross-validation:

mod_scaled <- train(
  hf,
  preprocessor_config = setup_SupervisedPreprocessor(scale = TRUE),
  hyperparameters = setup_GLM(),
  outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
  execution_config = setup_SerialExecution(seed = 2026L),
  question = hf_question
)

<Resampled Classification Model>
GLM (GLM (generalized linear model))
▣ Preprocessed using centering, scaling.
⟳ Tested using 10 independent folds.

  <Resampled Classification Training Metrics>
  Aggregate Confusion Matrix across resamples.
                     Predicted
          Reference  Yes  No    
                Yes  111   375
                 No   62  2062

  Showing mean (sd) across resamples.
        Sensitivity: 0.228 (0.025)
        Specificity: 0.971 (0.006)
  Balanced Accuracy: 0.600 (0.011)
                PPV: 0.644 (0.042)
                NPV: 0.846 (3.8e-03)
                 F1: 0.336 (0.027)
           Accuracy: 0.833 (4.7e-03)
                AUC: 0.780 (0.010)
        Brier Score: 0.123 (1.6e-03)

  <Resampled Classification Test Metrics>
  Aggregate Confusion Matrix across resamples.
                     Predicted
          Reference  Yes  No   
                Yes    9   45
                 No    9  227

  Showing mean (sd) across resamples.
        Sensitivity: 0.170 (0.147)
        Specificity: 0.962 (0.047)
  Balanced Accuracy: 0.566 (0.065)
                PPV: NA (NA)
                NPV: 0.836 (0.029)
                 F1: NA (NA)
           Accuracy: 0.814 (0.040)
                AUC: 0.709 (0.117)
        Brier Score: 0.140 (0.015)

Each fold’s model carries its own fitted preprocessor, estimated on that fold’s training set and applied to its test set. The centers used for serum creatinine differ slightly between folds:

vapply(
  mod_scaled@models,
  function(m) m@preprocessor@values[["scale_centers"]][["serum_creatinine"]],
  numeric(1)
)

 Fold_1  Fold_2  Fold_3  Fold_4  Fold_5  Fold_6  Fold_7  Fold_8  Fold_9 Fold_10 
  1.372   1.421   1.405   1.413   1.380   1.412   1.370   1.407   1.388   1.389 

The coefficients of the fitted logistic regressions are per standard deviation of each numeric feature:

coef(mod_scaled@models[[1]]@model)

             (Intercept)                      age               anaemiaYes 
                 -2.6077                   0.5750                   0.5247 
creatinine_phosphokinase              diabetesYes        ejection_fraction 
                  0.1716                   0.3344                  -0.2256 
  high_blood_pressureYes                platelets         serum_creatinine 
                  1.2434                  -0.1436                   0.5270 
            serum_sodium                  sexMale               smokingYes 
                 -0.2453                  -0.1225                   0.2611 

Standardization does not change the predictions of an unpenalized linear model (Section 12.3): the mean test AUC is 0.709 with standardization and 0.709 without. It matters for the penalized, distance-based and gradient-trained models of later chapters, several of which standardize internally.

13.3 Feature selection outside and inside resampling

The following experiment replaces the heart failure features with 1000 features of pure noise, drawn independently of the outcome. No model can predict death from them, and the true AUC of any model is 0.5:

set.seed(2026L)
noise <- as.data.table(matrix(rnorm(nrow(hf) * 1000L), nrow = nrow(hf)))
setnames(noise, sprintf("noise_%04d", 1:1000))
dat_noise <- cbind(noise, death = hf[["death"]])

A common procedure selects the features most associated with the outcome and then fits a model to them. The function below selects the 10 features with the smallest t-test p-values comparing patients who died with those who did not:

select_top <- function(dat, k = 10L) {
  died <- dat[["death"]] == "Yes"
  features <- setdiff(names(dat), "death")
  p <- vapply(features, function(v) {
    t.test(dat[[v]][died], dat[[v]][!died])[["p.value"]]
  }, numeric(1))
  names(sort(p))[seq_len(k)]
}

Selection outside resampling. The 10 features are selected once, using all 290 patients, and a logistic regression on them is assessed by 10-fold cross-validation:

cv <- setup_KFold(n_resamples = 10L, seed = 2026L)
top_all <- select_top(dat_noise)
mod_outside <- train(
  dat_noise[, c(top_all, "death"), with = FALSE],
  hyperparameters = setup_GLM(),
  outer_resampling_config = cv,
  execution_config = setup_SerialExecution(seed = 2026L),
  verbosity = 0L
)

Selection inside resampling. The selection is repeated within each training fold, using only that fold’s cases, and the model is tested on the held-out fold:

folds <- resample(dat_noise, cv)
auc_inside <- vapply(folds@resamples, function(idx) {
  dat_training <- dat_noise[idx]
  dat_test <- dat_noise[-idx]
  top <- select_top(dat_training)
  mod <- train(
    dat_training[, c(top, "death"), with = FALSE],
    dat_test = dat_test[, c(top, "death"), with = FALSE],
    hyperparameters = setup_GLM(),
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
  mod@metrics_test@metrics[["overall"]][["auc"]]
}, numeric(1))

c(
  selection_outside = auc_outside,
  selection_inside = mean(auc_inside)
)

selection_outside  selection_inside 
           0.8153            0.5332 

With selection on all cases, the cross-validated AUC is 0.815, an apparent ability to predict death from noise. The selection used the test folds: among 1000 noise features, some are associated with the outcome in this sample by chance, and the selection found exactly those associations, including the parts contributed by the cases later used for testing. With selection inside each training fold, the AUC is 0.533, close to the true value of 0.5 (Ambroise and McLachlan 2002).

The same principle applies to every estimated step. When preprocessing is specified through preprocessor_config, or feature reduction through a decomposition configuration or a penalized algorithm, train() fits it within each training set. Steps performed on the data before it is passed to train() are outside that guarantee, and review() reports this as a limitation of every assessment (Section 9.8).

Ambroise, Christophe, and Geoffrey J. McLachlan. 2002. “Selection Bias in Gene Extraction on the Basis of Microarray Gene-Expression Data.” Proceedings of the National Academy of Sciences 99 (10): 6562–66. https://doi.org/10.1073/pnas.102102699.