lesson
50 min

Tuning, Cross-Validation, and Leaks

Pick model settings with cross-validation, and stop data leaks that fake good results.

Objective

By the end of this lesson, you can tune a model with cross-validation, explain why the test set stays locked until the end, and spot data leakage.

Hyperparameters

A hyperparameter is a setting you choose before training, like a tree's max_depth or the number of clusters k. The model does not learn it. You have to pick it.

If you try many settings and pick the one that scores best on the test set, the test set stops being a fair test. You have tuned to it.

Cross-validation

Cross-validation solves this. Split the training data into, say, 5 parts (called folds). Train on 4 and score on the 5th. Repeat 5 times so each part gets a turn. Average the scores. Pick the setting with the best average. Only then use the test set, once.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

search = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    param_grid={"max_depth": [2, 3, 4, 6, 8, None]},
    cv=5,
    scoring="recall",      # choose the score that matches the harm you care about
)
search.fit(X_train, y_train)
print("Best setting:", search.best_params_)
print("Cross-validated recall:", round(search.best_score_, 3))
print("Test recall (used once):", round(search.score(X_test, y_test), 3))

Notice scoring="recall". What you optimize is a choice. Pick the score that matches the mistake you most need to avoid (Module 11).

Data leakage

Leakage happens when information the model would not have at prediction time sneaks into training. It makes results look great and then fail in real use. Common leaks:

  • Scaling or filling missing values using the whole dataset before splitting. Fix: fit scalers on training data only (a scikit-learn Pipeline does this for you).
  • A column that is recorded after the outcome. Predicting whether a patient was admitted using a column filled in only after admission.
  • Shuffling time series so the model trains on the future.
  • The same person or place appearing in both train and test sets.

If results look too good, suspect a leak first.

A national example

Leakage check

0 of 6 done

Ask. Answer. Build together.