lesson
45 min

Overfitting: When a Model Memorizes

Why a perfect score on training data is a warning sign, and how to see overfitting in your numbers.

Objective

By the end of this lesson, you can explain overfitting and underfitting, and diagnose them by comparing training and test error.

Memorizing versus learning

A student who memorizes last year's answers can ace last year's test and fail this year's. Models do the same thing.

  • Overfitting: the model learns the training data too closely, including its noise. Training error is low. Test error is much higher.
  • Underfitting: the model is too simple to capture the real pattern. Both errors are high.
  • A good fit: training and test error are both reasonably low and close together.

See it in numbers

The "lookup" model memorizes every training point. The "line" model is a simple straight line.

Run it. Compare training and test error for each model. The lookup model is perfect on training data. Is it better on new data?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The lookup model has zero training error. The simple line does better on the new points. A perfect training score is a warning, not a win.

What causes overfitting

  • A model with too much freedom (a very deep decision tree, too many inputs).
  • Too little data for the model's size.
  • Training too long (for neural networks, Module 15).

How to fight it

  • Use simpler models or limit their size (max_depth for trees).
  • Get more data, if it can be collected fairly.
  • Regularization: a penalty that keeps model weights small. In scikit-learn, Ridge and Lasso are regularized linear regression.
  • Always judge on a test set the model never saw.

A national example

Reflection

Private. Saved only in this browser. No one else can see it.

Ask. Answer. Build together.