Text Classification
Build a spam classifier from word counts, then train a real one on a public dataset and check who it fails.
Objective
By the end of this lesson, you can build a simple word-count classifier, train a TF-IDF classifier on a public dataset, and test it for uneven errors.
Learning word evidence
A text classifier learns which words are evidence for each label. The simplest version counts how often each word appeared under each label in the training data, then scores new messages by adding up those counts.
"Ham" is a common name for messages that are not spam. A real method called Naive Bayes uses the same idea with probabilities, and it is still used for text because it is fast and works with little data.
A real classifier
The SMS Spam Collection from the UC Irvine Machine Learning Repository (CC BY 4.0) has a few thousand real text messages labeled spam or ham. Download it, upload the file to Google Colab, and run:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.metrics import classification_report
df = pd.read_csv("SMSSpamCollection", sep="\t", names=["label", "text"])
X_train, X_test, y_train, y_test = train_test_split(
df["text"], df["label"], test_size=0.2, random_state=0, stratify=df["label"])
model = make_pipeline(TfidfVectorizer(), MultinomialNB()).fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))
Then print 10 messages it got wrong. What do they have in common?
Who does the model fail?
Language models learn from the text they are trained on. Text in some dialects, slang, or languages is often rare in training data. Models can then make more mistakes on it. For a spam filter, that means one community's real messages get blocked more often.
A national example
Judgment check
Test your text model on the kinds of writing your real users produce, including other languages and dialects. If you cannot test on those, say so clearly, and keep a human review path for anything the model blocks.
Quiz: Text classification check
Discussion
Ask. Answer. Build together.
Loading…