lesson
60 min

Chunking and Embeddings

Split documents into retrievable pieces and turn them into vectors, with the trade-offs that decide whether RAG works.

Objective

By the end of this lesson, you can choose a chunking strategy for a document set, explain how embeddings make meaning searchable, and name the choices that most affect retrieval quality.

The RAG pipeline, end to end

Retrieval-augmented generation (RAG) answers questions from your own documents. You met the idea in AI Engineering. As an architect, you design each stage:

  1. Ingest: collect documents with known owners, dates, and permissions.
  2. Chunk: split them into passages.
  3. Embed: turn each passage into a vector.
  4. Index: store vectors and metadata for fast search.
  5. Retrieve: find the best passages for a question.
  6. Generate: answer only from those passages, with citations.
  7. Evaluate: measure retrieval and answers separately.

Most RAG failures happen in stages 1 to 5, before the model writes a word.

Chunking: the most underrated decision

A chunk is a passage small enough to search and fit into a prompt. Chunks that are too big bury the answer in noise. Chunks that are too small lose the context that makes them make sense.

Common strategies:

  • Fixed size with overlap: every N words, with some words repeated between neighbors so facts at the edges stay whole.
  • Structure-aware: split on headings, sections, list items, or table rows. Best for policies, handbooks, and forms.
  • Semantic: split where the topic changes. More work, sometimes worth it.

Always keep metadata with each chunk: source file, section heading, date, language, and who may see it. You will need it for citations, filtering, and access control.

Run it. Compare chunk 2 with and without overlap. Without overlap, the ID rule is cut mid-sentence. Then try size 8 with overlap 2. How many chunks? Is each one still understandable on its own?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The first run downloads Python into your browser (several MB).

Embeddings: meaning as numbers

An embedding is a list of numbers (a vector) that an embedding model produces for a piece of text. Texts with similar meaning get vectors that point in similar directions. "Do I need an appointment?" lands near "Walk-ins are welcome on Saturdays," even though they share few words.

Similarity is usually measured with cosine similarity: 1.0 means the same direction, 0 means unrelated. You will compute it by hand in the next lesson.

Design choices that matter:

  • Language coverage. Test the embedding model in every language your users speak. Quality can drop sharply outside English.
  • Same model for documents and questions. Vectors from different models do not compare.
  • Re-embed on change. Switching embedding models means re-processing every chunk.
  • Domain words. Program names, form numbers, and local slang may embed poorly. Hybrid search helps (next lesson).

A national example

Quiz: Chunking and embeddings check

  1. 01Why store the section heading and date with each chunk?

    Choose one.

  2. 02You switch to a new embedding model. What must happen?

    Choose one.

Go deeper: Machine Learning, Minus the Hype: A Practical Playbook to Ship Useful Models

Ask. Answer. Build together.