lesson
45 min

Turning Text Into Numbers

Models need numbers. Learn tokens, bag of words, and TF-IDF, the first steps of every language tool.

Objective

By the end of this lesson, you can tokenize text, build a bag-of-words table, and explain what TF-IDF adds.

What NLP is

Natural language processing (NLP) is the part of AI that works with human language: spam filters, search, translation, voice assistants, and chatbots. Every NLP system starts the same way. It turns words into numbers.

Tokens

A token is one piece of text, often a word. Splitting text into tokens is tokenizing. Before that, text is usually normalized: lowercased, and sometimes stripped of punctuation.

Even this step involves choices. Is "don't" one token or two? Should "Bus" and "bus" match? Tools for large language models split words into smaller pieces, so rare words still get tokens.

Bag of words

A bag of words counts how often each word appears in each document, and ignores word order. Each document becomes a row of numbers. Each word in the vocabulary becomes a column.

Run it. Three short made-up rider comments become rows of word counts. Add a fourth comment, "bus late", and run it again.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

Bag of words loses order. "Not good, very bad" and "not bad, very good" get the same counts. Later methods, like transformers (Module 17), keep order and context.

TF-IDF

Common words like "the" appear everywhere and say little. TF-IDF (term frequency times inverse document frequency) gives a word a high score when it is common in one document but rare across all documents. That highlights the words that make a document distinct.

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["bus late again", "bus on time", "late late late"]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)
print(vec.get_feature_names_out())
print(X.toarray().round(2))

A national example

Reflection

Private. Saved only in this browser. No one else can see it.

Ask. Answer. Build together.