Evaluating Retrieval
Measure whether retrieval finds the right passages with recall@k and MRR, and separate retrieval failures from generation failures.
Objective
By the end of this lesson, you can build a labeled retrieval test set, compute recall@k and mean reciprocal rank, and use the results to decide what to fix.
Two questions, measured separately
A RAG answer can fail in two places:
- Retrieval failed: the right passage never reached the model.
- Generation failed: the passage arrived, but the model ignored it, misread it, or added things.
If you only score final answers, you cannot tell these apart. You might spend weeks tuning prompts when the real problem is chunking. Measure retrieval on its own.
Building a retrieval test set
For each test question, a person marks which chunks contain the answer. That is a labeled set.
- Start with 50 to 100 real questions, de-identified. Pull from call logs, emails, or front-desk notes with permission.
- Include questions in every language you serve.
- Include unanswerable questions, where no chunk is relevant. The right result is "nothing above the floor."
- Re-label when documents change.
The metrics
- recall@k: of the relevant chunks, what share appear in the top k results? If you send the model 5 chunks, recall@5 is the one that matters.
- MRR (mean reciprocal rank): how high the first relevant chunk ranks, averaged across questions. First place scores 1, second scores 0.5, third scores 0.33, not found scores 0.
- Never found: list the questions where nothing relevant appeared at all. Read them one by one.
The first run downloads Python into your browser (several MB).
From numbers to fixes
| What you see | Likely cause | Try |
|---|---|---|
| Low recall, relevant chunk never appears | Chunking split the answer, or vocabulary mismatch | Structure-aware chunking, hybrid search |
| Relevant chunk appears but low in the list | Ranking | Reranker, better metadata filters |
| Good recall, wrong answers | Generation | Prompt, passage order, model choice |
| Worse scores in one language | Embedding model coverage | Test another embedding model, add translated docs |
| Answers on unanswerable questions | Floor too low | Raise the similarity floor, tune on test set |
Always report metrics by language and by question type. An average of 0.8 can hide 0.5 for Spanish questions.
RAG readiness check
0 of 7 doneGo deeper: Machine Learning, Minus the Hype: A Practical Playbook to Ship Useful Models
Discussion
Ask. Answer. Build together.
Loading…