lesson
50 min

Gradient Descent: How Networks Train

Measure how wrong a model is with a loss, then walk downhill one small step at a time.

Objective

By the end of this lesson, you can explain loss, gradient descent, and learning rate, and run gradient descent on a one-weight model.

Loss: one number for "how wrong"

A loss is a single number that measures how wrong a model's predictions are. Training means making the loss smaller. A common loss for numbers is mean squared error: square each miss and average them. Squaring makes big misses count much more than small ones.

Walking downhill

Picture the loss as a hilly landscape. Each spot is one choice of weights. You want the lowest valley.

Gradient descent finds it step by step:

  1. Measure the slope of the loss at your current weights. This slope is the gradient.
  2. Take a small step downhill.
  3. Repeat.

The size of the step is the learning rate. Too small and training takes forever. Too large and you jump past the valley and the loss bounces or grows.

Run it. The true rule is y = 3x. Watch w move toward 3 and the loss shrink. Then set rate to 0.1 and run it. What happens when the steps are too big?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

A real network does exactly this, for millions of weights at once. A method called backpropagation computes all the gradients efficiently, layer by layer, from the output back to the input.

Reading training curves

When you train in Keras, track loss on both training data and validation data (a slice held out from training):

  • Both going down: still learning.
  • Training loss down, validation loss up: overfitting (Module 12). Stop earlier.
  • Both flat and high: the model is too simple, or the learning rate is wrong.
history = model.fit(X_train, y_train, epochs=30, validation_split=0.2, verbose=0)

import matplotlib.pyplot as plt
plt.plot(history.history["loss"], label="training loss")
plt.plot(history.history["val_loss"], label="validation loss")
plt.xlabel("epoch")
plt.legend()
plt.show()

Keras's EarlyStopping callback stops training automatically when validation loss stops improving.

A national example

Reflection

Private. Saved only in this browser. No one else can see it.

Ask. Answer. Build together.