lesson
50 min

The Confusion Matrix: Which Mistakes Matter?

Count each kind of right and wrong answer, compute precision and recall, and pick a threshold on purpose.

Objective

By the end of this lesson, you can build a confusion matrix, compute precision and recall, and explain which kind of mistake is worse for a given use.

Accuracy can lie

Suppose 1 in 100 messages is a real emergency. A model that says "not an emergency" every time is 99 percent accurate, and useless. When one category is rare, accuracy hides failure.

Four outcomes

For a yes/no classifier, every prediction lands in one box:

Predicted yesPredicted no
Actually yesTrue positive (TP)False negative (FN): missed
Actually noFalse positive (FP): false alarmTrue negative (TN)

This table is a confusion matrix.

  • Precision = TP divided by (TP + FP). Of the ones we flagged, how many were right?
  • Recall = TP divided by (TP + FN). Of the real ones, how many did we catch?
Run it. 1 means yes and 0 means no, for ten made-up cases. Then change one prediction to create another false alarm, and watch precision drop.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

Which mistake is worse?

It depends on what happens next.

UseWorse mistakeSo favor
Spam filterBlocking a real message (FP)Precision
Flagging possible fraud for a human to reviewMissing real fraud (FN)Recall
Screening that leads to a free follow-up testMissing a real case (FN)Recall
Any model that denies people a serviceWrongly denying (FP on "deny")Precision, plus human appeal

Moving the threshold trades one for the other. Lower threshold: more catches, more false alarms. Higher threshold: fewer false alarms, more misses.

from sklearn.metrics import confusion_matrix, classification_report

proba = logreg.predict_proba(X_test)[:, 1]
for threshold in [0.3, 0.5, 0.7]:
    pred = (proba >= threshold).astype(int)
    print(threshold, confusion_matrix(y_test, pred).ravel())
print(classification_report(y_test, logreg.predict(X_test)))

A national example

Judgment check

Before you pick a threshold, write one sentence for each mistake: "When the model says yes and is wrong, this happens to this person." Then decide. If the harm of a false positive lands on people who cannot appeal, do not automate the decision.

Prompt Lab

Use an AI assistant to brainstorm who is affected by each kind of mistake. Then check its ideas against people who know the setting.

Who should the AI act as?

What exactly should it do?

What does it need to know?

What shape should the answer take?

Your prompt

Act as a careful reviewer who thinks about who is harmed by automated decisions.

Format: Two short lists: harms of false positives and harms of false negatives, then one question I should ask the people affected

Do not paste private details (full name, address, ID numbers) into any AI tool.

Ask. Answer. Build together.