lesson
50 min

Evaluating Regression Honestly

Split data, measure error with MAE and R squared, and read residuals to find where a model fails.

Objective

By the end of this lesson, you can split data into train and test sets, compute MAE and R squared, and use residuals to find where a model is weak.

Train and test

If you grade a model on the same data it learned from, it will look better than it is. So split the data:

  • Training set: the model learns from this (often about 80 percent).
  • Test set: held back. Used once, at the end, to measure real performance.

Two scores

  • MAE (mean absolute error), from Module 8: the average miss, in the data's units.
  • R squared: how much of the variation the model explains, compared with always guessing the average. 1.0 is perfect. 0 means no better than the average. It can go below 0 on test data when a model is worse than the average.

A residual is one miss: actual minus predicted. Residuals tell you where the model fails.

Run it. Five made-up test days compare actual and predicted demand. Look at the errors. Which day did the model miss most, and in which direction?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

R squared is high. But the last day was over-predicted by 9. If that pattern holds on the hottest days, the overall score hides it.

In scikit-learn

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print("MAE:", mean_absolute_error(y_test, pred))
print("R2:", r2_score(y_test, pred))

For time series, split by date instead (Module 8).

Check errors by group

A model can be accurate on average and wrong for one group. Always break down error by group: region, size of place, season. If rural counties have twice the error of urban ones, report it.

A national example

Judgment check

Never report only one score. Report the overall error, the error by group, and the worst misses. If you would not be comfortable explaining a model's worst miss to the person it affected, the model is not ready.

Regression evaluation checklist

0 of 5 done

Ask. Answer. Build together.