Chunking and Embeddings
Split documents into retrievable pieces and turn them into vectors, with the trade-offs that decide whether RAG works.
Objective
By the end of this lesson, you can choose a chunking strategy for a document set, explain how embeddings make meaning searchable, and name the choices that most affect retrieval quality.
The RAG pipeline, end to end
Retrieval-augmented generation (RAG) answers questions from your own documents. You met the idea in AI Engineering. As an architect, you design each stage:
- Ingest: collect documents with known owners, dates, and permissions.
- Chunk: split them into passages.
- Embed: turn each passage into a vector.
- Index: store vectors and metadata for fast search.
- Retrieve: find the best passages for a question.
- Generate: answer only from those passages, with citations.
- Evaluate: measure retrieval and answers separately.
Most RAG failures happen in stages 1 to 5, before the model writes a word.
Chunking: the most underrated decision
A chunk is a passage small enough to search and fit into a prompt. Chunks that are too big bury the answer in noise. Chunks that are too small lose the context that makes them make sense.
Common strategies:
- Fixed size with overlap: every N words, with some words repeated between neighbors so facts at the edges stay whole.
- Structure-aware: split on headings, sections, list items, or table rows. Best for policies, handbooks, and forms.
- Semantic: split where the topic changes. More work, sometimes worth it.
Always keep metadata with each chunk: source file, section heading, date, language, and who may see it. You will need it for citations, filtering, and access control.
The first run downloads Python into your browser (several MB).
Embeddings: meaning as numbers
An embedding is a list of numbers (a vector) that an embedding model produces for a piece of text. Texts with similar meaning get vectors that point in similar directions. "Do I need an appointment?" lands near "Walk-ins are welcome on Saturdays," even though they share few words.
Similarity is usually measured with cosine similarity: 1.0 means the same direction, 0 means unrelated. You will compute it by hand in the next lesson.
Design choices that matter:
- Language coverage. Test the embedding model in every language your users speak. Quality can drop sharply outside English.
- Same model for documents and questions. Vectors from different models do not compare.
- Re-embed on change. Switching embedding models means re-processing every chunk.
- Domain words. Program names, form numbers, and local slang may embed poorly. Hybrid search helps (next lesson).
A national example
Quiz: Chunking and embeddings check
Go deeper: Machine Learning, Minus the Hype: A Practical Playbook to Ship Useful Models
Discussion
Ask. Answer. Build together.
Loading…