Attention and Transformers
How modern language models use context: attention weights, transformers, and next-token prediction.
Objective
By the end of this lesson, you can explain attention in plain words, describe what a transformer does, and explain why large language models can sound right and be wrong.
The problem with word order
Bag of words (Module 16) ignores order and context. But "bank" means something different in "river bank" and "bank account." A model needs to look at the other words to know which one you mean.
Attention
Attention lets a model decide, for each word, how much to look at every other word. It gives each other word a score. Then it turns the scores into weights that add up to 1, using a function called softmax. High weight means "pay attention to this word."
"Bank" attends mostly to "river" and "flooded," so the model reads it as the side of a river. In a real model, the scores are learned, not typed in.
Transformers
A transformer is a neural network built from layers of attention. It was introduced in a 2017 research paper titled "Attention Is All You Need." Transformers look at all the words at once, which makes them fast to train on very large amounts of text.
Large language models
A large language model (LLM) is a very large transformer trained to predict the next token in text. It learns from a huge collection of text. Then it is usually tuned further with human feedback to follow instructions.
This explains both what LLMs do well and how they fail:
- They produce fluent, likely-sounding text, because that is what they were trained to do.
- They can produce false facts, fake citations, and wrong numbers that sound confident. This is often called hallucination. Likely-sounding is not the same as true.
- They reflect patterns, including biases, in their training text.
- They do not know about events after their training data ends, unless a tool gives them that information.
A national example
Reflection
Discussion
Ask. Answer. Build together.
Loading…