lesson
55 min

Evaluating Retrieval

Measure whether retrieval finds the right passages with recall@k and MRR, and separate retrieval failures from generation failures.

Objective

By the end of this lesson, you can build a labeled retrieval test set, compute recall@k and mean reciprocal rank, and use the results to decide what to fix.

Two questions, measured separately

A RAG answer can fail in two places:

  1. Retrieval failed: the right passage never reached the model.
  2. Generation failed: the passage arrived, but the model ignored it, misread it, or added things.

If you only score final answers, you cannot tell these apart. You might spend weeks tuning prompts when the real problem is chunking. Measure retrieval on its own.

Building a retrieval test set

For each test question, a person marks which chunks contain the answer. That is a labeled set.

  • Start with 50 to 100 real questions, de-identified. Pull from call logs, emails, or front-desk notes with permission.
  • Include questions in every language you serve.
  • Include unanswerable questions, where no chunk is relevant. The right result is "nothing above the floor."
  • Re-label when documents change.

The metrics

  • recall@k: of the relevant chunks, what share appear in the top k results? If you send the model 5 chunks, recall@5 is the one that matters.
  • MRR (mean reciprocal rank): how high the first relevant chunk ranks, averaged across questions. First place scores 1, second scores 0.5, third scores 0.33, not found scores 0.
  • Never found: list the questions where nothing relevant appeared at all. Read them one by one.
Run it. Then look at q3: the relevant document d0 is never retrieved. Change retrieved for q3 so d0 is in second place. How do recall@3 and MRR change? Which single fix helped most?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The first run downloads Python into your browser (several MB).

From numbers to fixes

What you seeLikely causeTry
Low recall, relevant chunk never appearsChunking split the answer, or vocabulary mismatchStructure-aware chunking, hybrid search
Relevant chunk appears but low in the listRankingReranker, better metadata filters
Good recall, wrong answersGenerationPrompt, passage order, model choice
Worse scores in one languageEmbedding model coverageTest another embedding model, add translated docs
Answers on unanswerable questionsFloor too lowRaise the similarity floor, tune on test set

Always report metrics by language and by question type. An average of 0.8 can hide 0.5 for Spanish questions.

RAG readiness check

0 of 7 done

Go deeper: Machine Learning, Minus the Hype: A Practical Playbook to Ship Useful Models

Ask. Answer. Build together.