hf_question <- paste(
"Among patients aged 40 years or older admitted with heart failure due to",
"left ventricular systolic dysfunction (NYHA class III or IV), will the",
"patient die within 60 days of baseline, given age, sex, comorbidities and",
"laboratory measurements recorded at baseline?"
)3 Questions and Generalization in rtemis
This chapter fits one logistic regression model to the heart failure data and examines it twice: as a description of the association between baseline measurements and death in the population sampled, and as a predictive model whose performance on new patients is estimated by resampling (Chapter 2).
3.1 The question
The predictive question is written down first and stored in a variable, so that it can be passed to every model that addresses it:
The same definition is stored in the shared setup file, from which later chapters use it. read_heart_failure() reads the data and derives the 60-day outcome, as described in Chapter 7:
The data contain 290 patients, of whom 54 (18.6%) died within 60 days. The outcome death is a factor whose second level, "Yes", is the positive class: rtemis treats the second level of a binary outcome as the event of interest.
3.2 A descriptive analysis
train() fits a logistic regression when given setup_GLM() and a binary factor outcome. Without a resampling configuration, it fits one model to all cases. train() prints a summary of the fitted model; verbosity = 0L suppresses it here, since the summary concerns prediction and is examined in the next section:
fit <- train(
hf,
hyperparameters = setup_GLM(),
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question,
verbosity = 0L
)The fitted glm object is stored in the model property. Exponentiated coefficients are odds ratios, and exponentiated Wald intervals (Equation 2.2) are their 95% confidence intervals:
| term | OR | lower | upper | p |
|---|---|---|---|---|
| age | 1.055 | 1.025 | 1.086 | 0.000 |
| anaemiaYes | 1.699 | 0.857 | 3.370 | 0.129 |
| creatinine_phosphokinase | 1.000 | 1.000 | 1.001 | 0.061 |
| diabetesYes | 1.682 | 0.847 | 3.340 | 0.137 |
| ejection_fraction | 0.969 | 0.940 | 0.999 | 0.042 |
| high_blood_pressureYes | 2.976 | 1.509 | 5.868 | 0.002 |
| platelets | 0.999 | 0.996 | 1.003 | 0.697 |
| serum_creatinine | 1.528 | 1.140 | 2.048 | 0.005 |
| serum_sodium | 0.946 | 0.882 | 1.015 | 0.123 |
| sexMale | 0.997 | 0.450 | 2.210 | 0.995 |
| smokingYes | 1.247 | 0.569 | 2.730 | 0.582 |
Each odds ratio describes the association of one feature with death among patients with equal values of the other features. For example, the odds of death within 60 days are 1.71 times higher for a patient 10 years older (95% CI 1.28 to 2.28), and 1.53 times higher for each additional 1 mg/dL of serum creatinine (95% CI 1.14 to 2.05).
These intervals generalize from the sample to the population of patients like those enrolled in the study. They rest on the assumptions of the logistic regression model, including linearity of each feature’s effect on the log odds, and on the sample being representative. They do not license a causal reading of any coefficient (Section 2.2).
3.3 A predictive analysis
The same fitted model is also a function that returns a probability of death for any patient’s measurements. Two summaries of how well these probabilities separate and match outcomes are used here and defined fully in Chapter 8: the area under the ROC curve (AUC), the probability that a randomly chosen patient who died receives a higher predicted probability than a randomly chosen patient who survived, where 0.5 corresponds to chance; and the Brier score, the mean squared difference between the predicted probability and the outcome coded as 0 or 1, where lower is better.
Evaluated on the cases used to fit it, the model’s performance is:
fit@metrics_training<Training Classification Metrics>
Predicted
Reference Yes No
Yes 12 42
No 7 229
Overall
Sensitivity 0.222
Specificity 0.970
Balanced Accuracy 0.596
PPV 0.632
NPV 0.845
F1 0.329
Accuracy 0.831
AUC 0.779
Brier Score 0.124
Positive Class Yes
This apparent AUC of 0.779 is optimistic, since the coefficients were chosen to fit these same cases (Equation 2.4). An estimate of performance on new patients requires cases that were not used for fitting. outer_resampling_config specifies how the data are divided into training and test sets. setup_KFold() divides the cases into 10 folds; the model is fitted 10 times, each time on 9 folds, and assessed on the remaining fold. The printed summary reports the mean and standard deviation of each metric across the training folds and across the test folds:
mod <- train(
hf,
hyperparameters = setup_GLM(),
outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question
)<Resampled Classification Model>
GLM (GLM (generalized linear model))
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 111 375
No 62 2062
Showing mean (sd) across resamples.
Sensitivity: 0.228 (0.025)
Specificity: 0.971 (0.006)
Balanced Accuracy: 0.600 (0.011)
PPV: 0.644 (0.042)
NPV: 0.846 (3.8e-03)
F1: 0.336 (0.027)
Accuracy: 0.833 (4.7e-03)
AUC: 0.780 (0.010)
Brier Score: 0.123 (1.6e-03)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 9 45
No 9 227
Showing mean (sd) across resamples.
Sensitivity: 0.170 (0.147)
Specificity: 0.962 (0.047)
Balanced Accuracy: 0.566 (0.065)
PPV: NA (NA)
NPV: 0.836 (0.029)
F1: NA (NA)
Accuracy: 0.814 (0.040)
AUC: 0.709 (0.117)
Brier Score: 0.140 (0.015)
Across the 10 test folds, the mean AUC is 0.709 (SD 0.117) and the mean Brier score is 0.140. On the training folds of the same resamples, the mean AUC is 0.780. The difference between training and test performance, 0.072 in AUC, estimates the optimism of the training-set evaluation.
plot_roc() draws the ROC curves of the training and test predictions, pooled across folds:
plot_roc(mod)The fold-level estimates vary considerably. Each test fold contains 28 to 30 patients, of whom 5 to 6 died, so a single fold gives an imprecise estimate. plot_metric() shows the AUC of every fold, for the training and test sets:
plot_metric(mod, metric = "auc")Test-fold AUC ranges from 0.533 to 0.875. This variation, and the choice of resampling design that controls it, are the subject of Chapter 11.
3.4 What each analysis estimated
The two analyses used the same model specification and the same data, and answered different questions.
- The descriptive analysis estimated population parameters, the odds ratios, and quantified their uncertainty analytically, from the sampling distribution of maximum likelihood estimates. Its validity rests on the model assumptions.
- The predictive analysis estimated the performance on new patients of the procedure “fit a logistic regression with these 11 features to about 260 patients”. It quantified generalization empirically, by fitting on some cases and testing on others. It made no use of the coefficients’ standard errors, and its validity does not depend on the model being correctly specified: the test cases measure the performance of whatever function was fitted.
The remaining chapters address the predictive question. Every model is assessed on cases not used to fit it, and the question is passed to train() so that it is stored with the model:
mod@question[1] "Among patients aged 40 years or older admitted with heart failure due to left ventricular systolic dysfunction (NYHA class III or IV), will the patient die within 60 days of baseline, given age, sex, comorbidities and laboratory measurements recorded at baseline?"