How to evaluate a RAG system before your users do

Most RAG projects are tested the same way: someone asks it ten questions, the answers look fine, it ships. Then real users ask the eleventh question. If you want to evaluate a RAG system properly, you need something repeatable — a test you can run after every change to chunking, prompts, models or data, and a number that tells you whether you just made things better or worse.

It does not need a platform or a research team. It needs a spreadsheet’s worth of questions and a script.

Split the problem in two

A RAG system can fail in two different places, and you need to know which one:

• Retrieval failed — the passage with the answer was never found, so the model had nothing to work with.
• Generation failed — the right passage was found, but the model ignored it, misread it or added things that were not there.

The fixes are completely different. Retrieval problems are solved with chunking, metadata filters and search tuning. Generation problems are solved with prompts, models and answer format. Measure them separately or you will fix the wrong thing.

Step 1: build a test set from real questions

Collect 50–100 questions. Pull them from support tickets, sales emails, search logs, whatever real people have actually asked. For each one, record:

• The question, in the user’s own words
• The document or passage that contains the answer
• A short reference answer, written by someone who knows the subject

Include the uncomfortable ones: questions with no answer in your documents, questions with outdated answers, and questions that use different words than your documentation does. Those are where systems break.

Step 2: measure retrieval

For each question, run the search and check whether the expected passage appears in the top results. Two numbers are enough to start:

• Hit rate @ k — in what share of questions does the right passage appear in the top k (say, 5)?
• Rank — when it appears, how high? Mean reciprocal rank captures this in one number.

This part needs no LLM at all, runs in seconds, and is where most improvement happens. If the hit rate is low, stop — no prompt will save you.

Step 3: measure answers

For the generated answers, I score three things:

1. Correctness — does it agree with the reference answer?
2. Faithfulness — is every claim supported by the retrieved passages, or did the model add something?
3. Refusal when it should — for questions with no answer in the documents, does it say so instead of inventing one?

Scoring can be done with a second model acting as a grader, given the question, reference answer, retrieved context and the system’s answer, and asked for a pass/fail with a reason. It is not perfect, so spot-check its verdicts by hand every run — especially the failures.

Step 4: run it on every change

The value is not in one score, it is in the comparison. Store every run with its settings: chunk size, embedding model, top k, prompt version, LLM. Then when someone asks “should we switch models?” you run the suite and look at the numbers instead of arguing.

I run a fast retrieval-only check on every deploy and the full answer evaluation when prompts, models or the index change.

Step 5: keep learning from production

Your test set goes stale as your users and documents change. In production, log every question, the retrieved chunks and the answer. Add a thumbs-up/down or a “this didn’t help” link. Every week, move the worst real failures into the test set. That loop is what makes a RAG system get better over time instead of slowly drifting.

What good looks like

• You can say what your retrieval hit rate is, today, in one number.
• You know whether last week’s change helped.
• Unanswerable questions get “I don’t know” instead of confident nonsense.
• Every answer shows its sources, so users can check.

If you already run a RAG system and nobody can answer those four points, that is the first thing I would build — before any new feature. My AI agents and RAG systems work always starts there. Get in touch if you want a second pair of eyes on yours.