Each resampling design of Chapter 10 has a setup_*() function in rtemis. resample() applies a configuration to a dataset and returns the training indices of every resample; train() applies it through its outer_resampling_config argument. This chapter creates resamples of the heart failure data, measures how the variability of a performance estimate depends on the design, and shows the effect of grouped resampling on the Parkinson’s data.
11.1 Resampling configurations
Table 11.1: Resampling configurations
Design
Function
Main arguments
k-fold cross-validation
setup_KFold()
n_resamples (the number of folds), stratify_var, id_strat, seed
k-fold cross-validation, subsampling and the stratified bootstrap stratify on the outcome unless stratify_var names another column; a numeric outcome is stratified within strat_n_bins quantile bins. id_strat names a column of group identifiers, such as patient IDs, and keeps all cases of a group in the same side of every split.
The Resampler object holds the training indices of each resample in its resamples property. The test cases of a resample are the cases not in its training set:
A resampling configuration passed to train() as outer_resampling_config determines how the model is assessed. Two designs are compared here for the logistic regression of the heart failure data. The first is a single 75/25 holdout split; with setup_StratSub(n_resamples = 50L), one call fits 50 models on 50 independent random splits, and the AUC of each split is the estimate that a single holdout split would have produced:
The second is 10-fold cross-validation, repeated with 50 different random fold assignments; each repetition yields one cross-validation estimate, the mean of its 10 test-fold AUCs:
The 50 holdout estimates range from 0.567 to 0.812, with a standard deviation of 0.058. The 50 cross-validation estimates range from 0.668 to 0.739, with a standard deviation of 0.015. The AUC reported by a single holdout split of these data depends strongly on which patients fall in the test set; a 10-fold cross-validation estimate depends much less on the random division. Both distributions describe variation due to the division of one dataset, and neither includes the variation that a different sample of 290 patients would produce.
11.4 Resampling in train()
The training and test metrics of every resample are stored in the fitted object (Section 9.1). The outer resampler, with its indices, is stored in outer_resampler, so the assessment can be reproduced and inspected:
The seed of the resampling configuration determines the division into folds; the seed of the execution configuration determines any randomness in model fitting. Both are recorded with the model.
11.5 Leave-one-out cross-validation
Leave-one-out cross-validation fits one model per case. For the diabetes data, it fits 442 linear models, each tested on one patient:
Per-resample \(R^2\) is undefined for a single test case, so the metrics are computed on the pooled out-of-fold predictions.
11.6 Grouped resampling
The Parkinson’s data contain between 101 and 168 recordings per patient (Section 7.4). Two linear models of total UPDRS are assessed by 5-fold cross-validation. The first assigns recordings to folds at random, ignoring the patient; the column subject is removed so that it is not used as a feature:
The second assigns patients to folds with id_strat = "subject". train() uses the named column to group the resamples and excludes it from the features:
With random assignment, every patient in a test fold also contributes about four fifths of their recordings to the training set, and the model is assessed on its ability to predict other recordings of patients it has seen. With assignment by patient, it is assessed on patients it has not seen, which is the situation of the question. The negative \(R^2\) shows that the linear model, applied to a new patient, predicts total UPDRS less accurately than the mean of that test fold. The difference between the two estimates is a measure of how much the random design overstates performance on new patients.