A Practical Guide to RAG Evaluation
Retrieval-augmented generation is easy to demo and hard to measure. Here is how to separate retrieval quality from generation quality.
Two systems wearing one coat
A RAG pipeline is really two systems: a retriever that finds context and a generator that writes an answer from it. When the output is wrong, the first question is always *which half failed*. Evaluate them separately or you will tune blind.
Measuring retrieval
Retrieval is an information-retrieval problem, so use IR metrics. **Recall@k** asks whether the relevant chunk made it into the top k. **MRR** rewards putting it near the top.
def recall_at_k(retrieved_ids, relevant_ids, k):
top = retrieved_ids[:k]
hits = sum(1 for r in relevant_ids if r in top)
return hits / max(len(relevant_ids), 1)If recall@k is low, no amount of prompt engineering saves you — the answer was never in the context window.
Measuring generation
Given good context, does the model use it faithfully? Two properties matter: **faithfulness** (every claim is grounded in the retrieved text) and **answer relevance** (it actually addresses the question). An LLM judge with a strict rubric works well here, but pin the judge model and temperature so your numbers are reproducible across runs.
Build a real eval set
Hand-label 100–200 question/answer/source triples from your own domain. Golden sets beat generic benchmarks every time, because they measure the distribution you actually serve. Track recall, faithfulness, and end-to-end accuracy on every change and you will stop shipping regressions you cannot see.
Written by the AI Blog editorial team
Deep dives written by ML engineers and researchers who ship models in production. Replace this bio with your own — a line about your background and the systems you build goes a long way with technical readers.