How to evaluate a RAG system before production
Evaluate RAG with questions people already ask of your documents. For each question, record the correct source, an acceptable answer and when the system should refuse. Measure retrieval and the final answer separately. That tells you whether to fix data, search or the model.
7 min read
Which questions should be in the test set?
Start with questions employees or customers have actually asked. Include routine and difficult cases: a current document, an outdated version, a table, a restricted section and a question with no answer in the corpus. Assign a person who can confirm the correct outcome for each question.
- The question and the person or role asking it.
- The document and passage containing the answer, or a clear no-answer label.
- The document version and access rights.
- The conditions an acceptable answer must meet.
How do you test retrieval separately?
Inspect the passages retrieval passes to the model. Is the correct passage present, and may this user see it? If the source is absent, changing the prompt will not restore it. Check OCR, chunking, metadata, access filters and index freshness. Record how often the correct source appears among the first results.
How do you test the answer?
When retrieval finds the right source, compare each material claim in the answer with that source. Record omissions, incorrect citations and confident answers where the documents contain no answer. Keep an error category as well as a pass or fail result; the next run can then show which failure a model or prompt change fixed.
Which failures should block launch?
Set acceptance rules with the process owner before testing. In a sensitive workflow, an answer from a restricted document or a fabricated citation may block launch regardless of the average score. Define acceptable error rates separately for lower-risk questions. A single quality percentage hides the difference between a typo and a data leak.
What happens after the first passing run?
Keep the questions, expected sources and results as a regression set. Run it after changes to documents, index, model or access rules. Add new production failures to the set. A RAG system reads a changing corpus, so a passing pre-launch test needs continued checks after release.