Turning Text Into Numbers
Models need numbers. Learn tokens, bag of words, and TF-IDF, the first steps of every language tool.
Objective
By the end of this lesson, you can tokenize text, build a bag-of-words table, and explain what TF-IDF adds.
What NLP is
Natural language processing (NLP) is the part of AI that works with human language: spam filters, search, translation, voice assistants, and chatbots. Every NLP system starts the same way. It turns words into numbers.
Tokens
A token is one piece of text, often a word. Splitting text into tokens is tokenizing. Before that, text is usually normalized: lowercased, and sometimes stripped of punctuation.
Even this step involves choices. Is "don't" one token or two? Should "Bus" and "bus" match? Tools for large language models split words into smaller pieces, so rare words still get tokens.
Bag of words
A bag of words counts how often each word appears in each document, and ignores word order. Each document becomes a row of numbers. Each word in the vocabulary becomes a column.
Bag of words loses order. "Not good, very bad" and "not bad, very good" get the same counts. Later methods, like transformers (Module 17), keep order and context.
TF-IDF
Common words like "the" appear everywhere and say little. TF-IDF (term frequency times inverse document frequency) gives a word a high score when it is common in one document but rare across all documents. That highlights the words that make a document distinct.
from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["bus late again", "bus on time", "late late late"]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)
print(vec.get_feature_names_out())
print(X.toarray().round(2))
A national example
Reflection
Discussion
Ask. Answer. Build together.
Loading…