Gradient Descent: How Networks Train
Measure how wrong a model is with a loss, then walk downhill one small step at a time.
Objective
By the end of this lesson, you can explain loss, gradient descent, and learning rate, and run gradient descent on a one-weight model.
Loss: one number for "how wrong"
A loss is a single number that measures how wrong a model's predictions are. Training means making the loss smaller. A common loss for numbers is mean squared error: square each miss and average them. Squaring makes big misses count much more than small ones.
Walking downhill
Picture the loss as a hilly landscape. Each spot is one choice of weights. You want the lowest valley.
Gradient descent finds it step by step:
- Measure the slope of the loss at your current weights. This slope is the gradient.
- Take a small step downhill.
- Repeat.
The size of the step is the learning rate. Too small and training takes forever. Too large and you jump past the valley and the loss bounces or grows.
A real network does exactly this, for millions of weights at once. A method called backpropagation computes all the gradients efficiently, layer by layer, from the output back to the input.
Reading training curves
When you train in Keras, track loss on both training data and validation data (a slice held out from training):
- Both going down: still learning.
- Training loss down, validation loss up: overfitting (Module 12). Stop earlier.
- Both flat and high: the model is too simple, or the learning rate is wrong.
history = model.fit(X_train, y_train, epochs=30, validation_split=0.2, verbose=0)
import matplotlib.pyplot as plt
plt.plot(history.history["loss"], label="training loss")
plt.plot(history.history["val_loss"], label="validation loss")
plt.xlabel("epoch")
plt.legend()
plt.show()
Keras's EarlyStopping callback stops training automatically when validation loss stops improving.
A national example
Reflection
Discussion
Ask. Answer. Build together.
Loading…