Evaluating RAG Quality

Evaluation

Part of: Build a RAG System

"It seems to work" is not a metric. You built a RAG pipeline; now a stakeholder asks, "How good is it?" If your answer is a shrug and a couple of cherry-picked demos, you can't tune it, can't compare two versions, and can't catch the day it silently regresses. RAG quality has to be measured , and it splits into two halves that fail independently. What it is A RAG system has two stages, so it has two things to measure: - Retrieval quality : did the search return the right chunks? Measured with precision and recall . - Answer quality : given the retrieved chunks, is the final answer actually supported by them? Measured with faithfulness . You need both, because they break separately. Perfect retrieval with an unfaithful model still hallucinates. A faithful model fed the wrong chunks faithfully summarizes the wrong thing. How it works For one test question you have a set of gold chunks (the ones that truly answer it) and the set your system retrieved . A hit is a retrieved chunk that's also gold. - Precision = hits / retrieved. Of what you pulled, how much was relevant? Low precision = you're dragging in junk. - Recall = hits / gold. Of what you needed, how much did you find? Low reca

Challenge: The RAG Scorecard