Evaluating Regression Honestly
Split data, measure error with MAE and R squared, and read residuals to find where a model fails.
Objective
By the end of this lesson, you can split data into train and test sets, compute MAE and R squared, and use residuals to find where a model is weak.
Train and test
If you grade a model on the same data it learned from, it will look better than it is. So split the data:
- Training set: the model learns from this (often about 80 percent).
- Test set: held back. Used once, at the end, to measure real performance.
Two scores
- MAE (mean absolute error), from Module 8: the average miss, in the data's units.
- R squared: how much of the variation the model explains, compared with always guessing the average. 1.0 is perfect. 0 means no better than the average. It can go below 0 on test data when a model is worse than the average.
A residual is one miss: actual minus predicted. Residuals tell you where the model fails.
R squared is high. But the last day was over-predicted by 9. If that pattern holds on the hottest days, the overall score hides it.
In scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print("MAE:", mean_absolute_error(y_test, pred))
print("R2:", r2_score(y_test, pred))
For time series, split by date instead (Module 8).
Check errors by group
A model can be accurate on average and wrong for one group. Always break down error by group: region, size of place, season. If rural counties have twice the error of urban ones, report it.
A national example
Judgment check
Never report only one score. Report the overall error, the error by group, and the worst misses. If you would not be comfortable explaining a model's worst miss to the person it affected, the model is not ready.
Regression evaluation checklist
0 of 5 doneDiscussion
Ask. Answer. Build together.
Loading…