ml-001 · Train/validation/test splitting — Concepts

The big idea

A model’s accuracy on the rows it was fitted on tells you almost nothing about how it will behave on rows it has never seen — flexible models can memorize noise. Held-out data exists to simulate the future: we hide part of the sample, pretend it does not exist while fitting, and only then ask how well the model does on it. A three-way split refines this further. The training set fits parameters, the validation set compares candidate models and tunes hyperparameters, and the test set answers one final question — “how good is the model we already committed to?” — exactly once. Every decision you make while looking at a data split “spends” that split: after enough peeks, its score is no longer an honest estimate. The same logic extends to preprocessing: any statistic computed from held-out rows (a mean for scaling, a per-category target mean) quietly moves information across the wall, and some of those leaks are catastrophic while others are negligible. This practice makes both kinds concrete.

Definitions

Training set — rows used to fit model parameters (coefficients, tree splits). Here: 60% of the data.
Validation set — rows used to compare models and choose hyperparameters. The model never trains on them, but you repeatedly look at them, so their score drifts optimistic as you tune. Here: 20%.
Test set — rows used once, at the very end, to report performance of the single model you already selected. Here: 20%.
Stratified split — a split that preserves the class proportions of the full sample inside every fold. With a 15% positive class, a plain random split can hand a small validation set 10% or 20% positives by luck; stratification removes that variance.
Data leakage — any flow of information from held-out rows into choices made during training, including preprocessing statistics.
Target encoding — replacing a categorical value with the mean of the target among rows sharing that value. Powerful, and the classic leakage trap: if the means are computed on all rows, each held-out row’s own label is baked into its feature.
Generalization gap — the difference between training accuracy and held-out accuracy; large gaps signal overfitting.

Why it matters

Every applied ML paper, Kaggle submission, and production model review hinges on whether the reported number is honest. Leakage through preprocessing is the most common way honest people fool themselves: the pipeline “works”, the validation score is great, and the deployed model falls flat because the deployment data cannot leak its own labels. Knowing which preprocessing steps are dangerous (anything involving the target, row identity, or time) versus benign-in-practice (means and variances of features, on reasonably sized samples) is what lets you review a pipeline quickly. The train/val/test discipline is also the substrate for everything later: cross-validation, early stopping, and model selection all assume you understand why the test set is touched once.

Pitfalls and misconceptions

“I standardized before splitting, my results are invalid.” Fitting a scaler on all rows is technically leakage, but with n in the hundreds the pooled mean and standard deviation are nearly identical to the training-only ones — the effect on accuracy is typically zero to the third decimal. Fix it for hygiene, but do not expect results to change.
“Target encoding is safe because the model never sees the label column.” The encoded feature is a function of labels. If held-out rows contributed to the category means, their labels are inside their own features, and validation scores become fiction.
“Validation accuracy is my final result.” You chose the model because its validation score was highest, so that score is biased upward. The test set exists to remove that selection bias — which only works if you did not use it while selecting.
“The split proportions are what matters.” The independence of the splits matters more: shuffle before splitting, stratify on rare outcomes, and split before any target-aware preprocessing.
Unstratified splits on imbalanced data silently change the base rate each fold sees, which moves accuracy even if the model is unchanged.

Mental model

Think of the three splits as three rooms with one-way doors:

Room	Who may enter	How often
Training	the fitting algorithm	as much as it wants
Validation	you, comparing models	many times, score decays with use
Test	you, reporting the final number	exactly once

Preprocessing is part of the model, so it lives in the training room: any statistic it needs must be computed there and carried out to the other rooms, never recomputed inside them. A leak is any statistic that walked through a door the wrong way — and leaks involving the target are orders of magnitude worse than leaks involving feature moments.

Check your understanding

Q. Your tuned model gets 91% on validation and 88% on test. Which number do you report, and why are they different?

Report 88%. The validation score is biased upward because you selected the model that maximized it — over many candidate models, the winner partly wins by luck on those specific 300 rows. The test set was never used for any decision, so its score is an unbiased estimate of performance on new data. The 3-point drop is the price of model selection, not a bug.

Q. Why does target-encoding a category using all rows leak, when standardizing a feature using all rows is nearly harmless?

Standardization uses only feature moments (mean, sd), which are almost the same whether computed on 900 or 1500 rows — and they contain no label information, so nothing about the answer crosses the split boundary. Target encoding computed on all rows embeds each held-out row’s own label into its feature value: in a small category, one row’s label visibly moves the category mean. The model then “predicts” validation labels partly by reading them back out of the feature — signal that vanishes on genuinely new data.