lesson
50 min

Text Classification

Build a spam classifier from word counts, then train a real one on a public dataset and check who it fails.

Objective

By the end of this lesson, you can build a simple word-count classifier, train a TF-IDF classifier on a public dataset, and test it for uneven errors.

Learning word evidence

A text classifier learns which words are evidence for each label. The simplest version counts how often each word appeared under each label in the training data, then scores new messages by adding up those counts.

Run it. The classifier learns from four made-up messages. Then classify "free noon meeting". Is one word enough to flip the label?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

"Ham" is a common name for messages that are not spam. A real method called Naive Bayes uses the same idea with probabilities, and it is still used for text because it is fast and works with little data.

A real classifier

The SMS Spam Collection from the UC Irvine Machine Learning Repository (CC BY 4.0) has a few thousand real text messages labeled spam or ham. Download it, upload the file to Google Colab, and run:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.metrics import classification_report

df = pd.read_csv("SMSSpamCollection", sep="\t", names=["label", "text"])
X_train, X_test, y_train, y_test = train_test_split(
    df["text"], df["label"], test_size=0.2, random_state=0, stratify=df["label"])

model = make_pipeline(TfidfVectorizer(), MultinomialNB()).fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))

Then print 10 messages it got wrong. What do they have in common?

Who does the model fail?

Language models learn from the text they are trained on. Text in some dialects, slang, or languages is often rare in training data. Models can then make more mistakes on it. For a spam filter, that means one community's real messages get blocked more often.

A national example

Judgment check

Test your text model on the kinds of writing your real users produce, including other languages and dialects. If you cannot test on those, say so clearly, and keep a human review path for anything the model blocks.

Quiz: Text classification check

  1. 01What does a simple word-count classifier learn?

    Choose one.

  2. 02A spam filter blocks real messages in one dialect more often. What should you do?

    Choose one.

Ask. Answer. Build together.