ml-001 · Train/validation/test splitting — Python

Problem statement

Open this practice in GitHub Codespaces runs in your browser; open this folder under practices/ after it starts

Generate a synthetic imbalanced binary-classification dataset (n = 1500, about 15% positive, 4 informative + 6 noise features, plus one high-cardinality categorical that is pure noise), all in-script with a fixed seed. Then:

1. Write a stratified 60/20/20 train/validation/test split by hand and verify each split preserves the 15% positive share.
2. Show that standardizing before vs after splitting (scaler fit on all rows vs on training rows only) leaves a logistic regression’s validation accuracy essentially unchanged.
3. Show that target-encoding the noise categorical with all rows (peeking) makes it look predictive on validation, while a train-only encoding correctly shows no signal. Use AUC alongside accuracy.
4. Fit an unconstrained decision tree vs a regularized one, report the train/val/test accuracy of each, then tune max_depth on validation and evaluate the winner on the test set exactly once.

You know it is correct when the splits partition the rows with preserved class balance, the leaky encoding’s validation AUC is far above 0.5 while the proper one sits near 0.5, and the overfit tree shows a much larger train-val gap than the regularized one.

Learning objectives

After completing this practice you should be able to:

• explain why held-out data exists and what each of the three splits is for
• implement a stratified multi-way split from scratch with numpy
• distinguish harmless preprocessing-before-split (feature scaling) from catastrophic leakage (target encoding on pooled labels), and demonstrate the difference empirically
• read a train/val/test accuracy table and diagnose overfitting from the gap
• run an honest tuning loop: select on validation, touch test once

Concepts and definitions

See concepts for the shared background. Python-specific notes: numpy.random.default_rng(seed) gives an isolated generator so the data and the split can be reproduced independently; sklearn.linear_model.LogisticRegression applies an L2 penalty by default (C=1.0), so feature scaling can move its predictions slightly — unpenalized models are affine-invariant and would not move at all.

Input / output description

Item	Description
Input	Nothing external — make_synthetic_data(seed=42) builds X (1500 x 10 float), y (1500 ints, 15% ones), group (1500 ints in 0..399, independent of y) in memory
Output	Printed report: split sizes and positive shares; validation accuracy with scaler fit on all vs train rows; validation accuracy + AUC for proper vs leaky target encoding; train/val/test accuracy for the overfit and regularized trees; tuned depth and the single test-set evaluation. run_experiments() also returns all numbers as a dict

Before-coding checklist

• I can restate the problem without looking at it
• I know what the output should look like for a tiny example
• I have identified the tricky part
• I know how I will check my answer

My attempt

Work here. Run cells interactively; nothing below overwrites this section.

# Milestone 1: generate the data (X, y, group) with a fixed seed and
# confirm the class imbalance is ~15%.

# Milestone 2: stratified 60/20/20 split by hand; check the three index
# sets are disjoint, cover everything, and preserve the positive share.

# Milestone 3: standardization experiment — scaler fit on ALL rows vs on
# train only; compare logistic-regression validation accuracy.

# Milestone 4: target-encoding experiment — encode `group` with all labels
# vs train labels only; compare validation accuracy AND AUC.

# Milestone 5: overfit vs regularized tree (train/val/test table), then
# tune max_depth on validation and touch the test set exactly once.

Hint 1

TipHint 1 — a nudge

For the stratified split, do not try to split the whole index vector at once. Loop over the two classes: collect that class’s indices with np.flatnonzero(y == cls), shuffle them, and slice off 60% / 20% / the rest. Concatenating the per-class slices automatically preserves the class balance in every split.

Hint 2

TipHint 2 — almost a spoiler

Write target_encode(group, y, fit_idx) so it only ever reads y[fit_idx]: compute each group’s mean of y over fit_idx rows and map it onto all rows (unseen groups get the global mean over fit_idx). Then the leak is a one-argument change — fit_idx=train_idx is proper, fit_idx=np.arange(n) is peeking. For the demonstration, evaluate with AUC as well as accuracy: with a 15% positive class, accuracy hides the leak because predicted probabilities rarely cross 0.5.

Complete solution

NoteOfficial solution — open only after a real attempt

The official, tested script lives at solutions/solution.py. Run it with:

python practices/ml-001-train-validation-split/solutions/solution.py

Core code (the script adds a full report and returns every number):

def stratified_three_way_split(y, train_frac=0.6, val_frac=0.2, seed=42):
    rng = np.random.default_rng(seed)
    train_parts, val_parts, test_parts = [], [], []
    for cls in np.unique(y):
        idx = np.flatnonzero(y == cls)
        rng.shuffle(idx)
        n_train = int(round(train_frac * idx.size))
        n_val = int(round(val_frac * idx.size))
        train_parts.append(idx[:n_train])
        val_parts.append(idx[n_train:n_train + n_val])
        test_parts.append(idx[n_train + n_val:])
    return (np.sort(np.concatenate(train_parts)),
            np.sort(np.concatenate(val_parts)),
            np.sort(np.concatenate(test_parts)))


def target_encode(group, y, fit_idx):
    fit_group, fit_y = group[fit_idx], y[fit_idx]
    global_mean = float(fit_y.mean())
    means = {int(g): float(fit_y[fit_group == g].mean())
             for g in np.unique(fit_group)}
    return np.array([means.get(int(g), global_mean) for g in group])


# The two encodings differ by ONE argument:
enc_proper = target_encode(group, y, tr)              # leak-free
enc_leaky = target_encode(group, y, np.arange(y.size))  # peeking

Key output (seed 42): splits 900/300/300 all at 15.0% positive; standardize-before vs -after validation accuracy 0.937 vs 0.937 (difference 0.000); target encoding on the pure-noise categorical gives validation AUC 0.457 proper vs 0.857 leaky; overfit tree train/val/test = 1.000/0.883/0.887 (gap 0.117) vs regularized 0.909/0.893/0.890 (gap 0.016); tuned max_depth=7 scores 0.910 on validation and 0.907 on the once-touched test set.

Line-by-line explanation

NoteWhat each part does

• make_synthetic_data fixes the imbalance exactly (225 ones, shuffled), then builds informative features by adding np.outer(y, shifts) — each positive row’s informative features are shifted up by 0.8-1.6 standard deviations. group = rng.integers(0, 400, size=n) is drawn without looking at y, which is what makes any apparent predictive power of its encoding provably fake.
• stratified_three_way_split shuffles within class and slices 60/20/20. int(round(...)) on per-class counts gives 135/45/45 positives and 765/255/255 negatives — exactly 15% everywhere.
• standardize(fit_x, *apply_x) computes mu/sd from fit_x only. Calling it as standardize(x, x) reproduces the “fit on everything” mistake; standardize(x[tr], x[tr], x[va]) is the correct pattern.
• target_encode maps each group to the mean of y over fit_idx rows. The means.get(..., global_mean) fallback handles groups that appear in validation but not in the fitting rows.
• auc_score is the Mann-Whitney statistic via scipy.stats.rankdata (average ranks, so ties are handled): the probability a random positive outscores a random negative.
• In run_experiments, experiment 2 fits a logistic regression on the encoded feature alone; a leaky feature with no real signal reaching validation AUC 0.857 is the whole lesson in one number.
• The tuning loop fits one tree per depth 1..12, keeps the best validation accuracy, and only the final refit model ever sees te — the single test-set touch.

Design decisions

• Hand-rolled split instead of train_test_split twice. The point of the practice is the mechanics: per-class shuffling makes stratification transparent, and rounding per class shows why the counts come out exact.
• A pure-noise categorical for the leak demo. Because group is independent of y by construction, any validation lift is proof of leakage — no need to argue about how much signal is real.
• AUC alongside accuracy. With 15% positives, a model can leak heavily yet keep accuracy near the 0.85 majority baseline because probabilities rarely cross 0.5. The rank-based AUC exposes the leak (0.457 vs 0.857).
• One target_encode function with a fit_idx argument rather than two functions: it makes “the leak is just a wrong argument” literal, and it is exactly how leaks happen in real pipelines.
• min_samples_leaf=20 kept fixed while tuning depth so validation compares a one-dimensional family; tuning two knobs on 300 rows invites selection noise.

Common mistakes

• Splitting after target-encoding “because the encoder is just feature engineering” — the encoder is part of the model and must be fit on train.
• Stratifying only train/test and letting the validation set drift off the base rate; with 300 rows and 15% positives, the share can easily land at 11% or 19% unstratified.
• Reporting the best validation accuracy (0.910) as the result instead of the once-touched test accuracy (0.907).
• Using accuracy alone to check for leakage on imbalanced data — the leaky encoding here moves accuracy by 2 points but AUC by 40 points.
• Re-tuning “just once more” after seeing the test score. That converts the test set into a second validation set, permanently.
• Forgetting a fallback for categories unseen in the training rows, which crashes (or silently NaNs) the proper encoding on validation.

Alternative solution

NoteAnother way to do it

The scikit-learn idiom: two chained train_test_split calls with stratify=, and a Pipeline so preprocessing is structurally unable to leak:

from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

x_tmp, x_te, y_tmp, y_te = train_test_split(
    x, y, test_size=0.2, stratify=y, random_state=42)
x_tr, x_va, y_tr, y_va = train_test_split(
    x_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=42)  # 0.25*0.8=0.2

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
pipe.fit(x_tr, y_tr)          # scaler statistics come from x_tr only
pipe.score(x_va, y_va)

Prefer this in real projects: Pipeline.fit guarantees every step’s statistics come from the training rows, and cross-validation composes with it. The hand-rolled version remains better for learning because nothing is hidden. For leak-resistant target encoding, TargetEncoder in sklearn.preprocessing uses internal cross-fitting.

Extensions and harder variants

• Repeat the split with 50 different seeds and plot the distribution of validation accuracy — how much of the train-val gap is split luck?
• Replace the 60/20/20 split with 5-fold cross-validation for tuning and keep the same untouched test set; compare the chosen depth.
• Make the leak subtler: target-encode with additive smoothing ((sum + m*prior) / (count + m)) and find the smoothing m at which the leaky validation AUC stops being distinguishable from 0.5.
• Add a time column and show why a random split leaks when the deployment regime is “train on the past, predict the future”.
• Shrink the positive class to 3% and observe stratification going from nice-to-have to mandatory (unstratified validation folds can lose the positive class almost entirely).

Review questions

Answer out loud before expanding.

CautionQ1. Why do we need a validation set at all — why not tune hyperparameters directly on the test set?

Every tuning decision made while looking at a dataset optimizes for that dataset’s quirks. If you tune on the test set, the final score inherits that optimism and you have no clean data left to measure it. The validation set is the designated place to burn optimism; the test set stays clean precisely because it never influenced any choice.

CautionQ2. The scaler fit on all 1500 rows changed validation accuracy by 0.000. Is standardizing before splitting therefore fine?

It is technically leakage but practically negligible here: pooled feature means/sds are nearly identical to training-only ones at this sample size, and they carry no label information. It becomes dangerous when the statistic involves the target (target encoding), row identity, or when n is tiny or distributions shift across the split. Fix it for hygiene — a Pipeline makes it free — but do not expect conclusions to change.

CautionQ3. The group feature is independent of y, yet the leaky encoding reaches validation AUC 0.857. Where exactly does the signal come from?

From the validation labels themselves. With 400 groups over 1500 rows, each group has ~4 members, so one row’s label moves its group mean by ~0.25. Encoding with all rows bakes each validation row’s own label into its feature value; the model then partially reads the answer out of the feature. On new data (whose labels cannot be in the encoding) that signal does not exist — which the train-only encoding correctly shows (AUC 0.457, i.e., chance).

CautionQ4. The overfit tree scores 1.000 on train and 0.883 on validation; the regularized tree 0.909 and 0.893. What do the gaps tell you, and which model do you prefer?

The 0.117 gap says the unconstrained tree memorized training noise — perfect recall of the fit data, no better generalization. The regularized tree’s 0.016 gap says its training score is an honest preview of held-out performance. Prefer the regularized tree: equal-or-better validation accuracy with far less variance, and its simpler structure is more stable across resamples.

CautionQ5. Depth 7 won on validation (0.910) and scored 0.907 on test. If test had come back at 0.85, what would be the correct response?

Report 0.85 — that is the honest estimate. Do not go back and pick a different depth, because choosing a model using the test score makes the test set a validation set and the new number optimistic. A large val-test drop is itself information: the validation set was overused or too small, so next time hold out more data or use cross-validation for tuning.

Personal reflection

2026-08-10 — first pass. The AUC-vs-accuracy contrast in the leak demo is the part worth re-deriving from scratch next review: I initially expected the leak to show up in accuracy and it barely did.

References and related practices

• concepts — shared background for this practice
• ml-002 (as-of joins) — a time-ordered cousin of the same leakage idea
• ml-003 (random forest vs xgboost) — uses this split discipline for model comparison
• Hastie, Tibshirani, Friedman, The Elements of Statistical Learning, ch. 7 (model assessment and selection)
• scikit-learn user guide: “Common pitfalls and recommended practices” (data leakage section)