The Confusion Matrix: Which Mistakes Matter?
Count each kind of right and wrong answer, compute precision and recall, and pick a threshold on purpose.
Objective
By the end of this lesson, you can build a confusion matrix, compute precision and recall, and explain which kind of mistake is worse for a given use.
Accuracy can lie
Suppose 1 in 100 messages is a real emergency. A model that says "not an emergency" every time is 99 percent accurate, and useless. When one category is rare, accuracy hides failure.
Four outcomes
For a yes/no classifier, every prediction lands in one box:
| Predicted yes | Predicted no | |
|---|---|---|
| Actually yes | True positive (TP) | False negative (FN): missed |
| Actually no | False positive (FP): false alarm | True negative (TN) |
This table is a confusion matrix.
- Precision = TP divided by (TP + FP). Of the ones we flagged, how many were right?
- Recall = TP divided by (TP + FN). Of the real ones, how many did we catch?
Which mistake is worse?
It depends on what happens next.
| Use | Worse mistake | So favor |
|---|---|---|
| Spam filter | Blocking a real message (FP) | Precision |
| Flagging possible fraud for a human to review | Missing real fraud (FN) | Recall |
| Screening that leads to a free follow-up test | Missing a real case (FN) | Recall |
| Any model that denies people a service | Wrongly denying (FP on "deny") | Precision, plus human appeal |
Moving the threshold trades one for the other. Lower threshold: more catches, more false alarms. Higher threshold: fewer false alarms, more misses.
from sklearn.metrics import confusion_matrix, classification_report
proba = logreg.predict_proba(X_test)[:, 1]
for threshold in [0.3, 0.5, 0.7]:
pred = (proba >= threshold).astype(int)
print(threshold, confusion_matrix(y_test, pred).ravel())
print(classification_report(y_test, logreg.predict(X_test)))
A national example
Judgment check
Before you pick a threshold, write one sentence for each mistake: "When the model says yes and is wrong, this happens to this person." Then decide. If the harm of a false positive lands on people who cannot appeal, do not automate the decision.
Prompt Lab
Use an AI assistant to brainstorm who is affected by each kind of mistake. Then check its ideas against people who know the setting.
Your prompt
Act as a careful reviewer who thinks about who is harmed by automated decisions. Format: Two short lists: harms of false positives and harms of false negatives, then one question I should ask the people affected
Discussion
Ask. Answer. Build together.
Loading…