How to Evaluate a RAG System: Recall, Grounding, and Abstention

RAG evaluation is the practice of scoring a retrieval-augmented system against a golden set — a curated list of questions with known correct sources and expected answers — so that changes to chunking, search, reranking, or prompts can be measured instead of vibed. It is the least glamorous part of the stack and the only thing standing between you and a system that got quietly worse.

The failure mode it prevents is universal: someone tweaks a threshold, spot-checks three answers, ships it, and a regression on the questions nobody spot-checked surfaces weeks later as user complaints. Spot-checking cannot catch this, for a structural reason — a RAG pipeline is a chain of interacting stages, and a change that improves one stage routinely degrades another. A more aggressive relevance filter sharpens the answers that survive it and silently kills the borderline questions that used to work. Only a fixed question set, scored the same way every time, makes that trade visible.

The evaluation stack has three layers, and they must be measured separately: retrieval (did the right passages reach the model?), answers (did the model use them faithfully?), and abstention (did the system decline the questions it should have declined?). Skipping any layer leaves a class of regressions you cannot see.

A golden set is questions, expected source documents, and expected key facts. Its quality determines the value of everything downstream, and three sourcing rules matter more than size.

First, harvest real questions, not invented ones. Support tickets, Slack threads, and your assistant's own conversation logs contain questions phrased the way users actually phrase them — including the vagueness, the wrong vocabulary, and the exact identifiers. Questions written by the person who wrote the docs share the docs' vocabulary and inflate retrieval scores.

Second, label sources, not just answers. For each question, record which document (ideally which section) should ground the answer. This is what lets you score retrieval independently of generation — a wrong answer with the right passages retrieved is a different bug than a wrong answer with the wrong passages, and they are fixed in different places.

Third, include questions your corpus does not cover, on purpose. More on this below, because it is the part everyone skips. A workable starting set is a few dozen questions per knowledge base, grown from every incident: when a user reports a bad answer, the fix ships with that question added to the set, the same way a bug fix ships with a test.

Evaluate retrieval first, because it sets the ceiling on everything downstream — a passage retrieval misses can never appear in an answer. The standard trio:

Because they fail differently and are fixed differently. Low recall@k with good answer scores means the model is doing well with what it gets, and the work is upstream: chunking, hybrid-search weighting, or the corpus itself. Good recall with a low grounded rate means retrieval is delivering and generation is drifting — a prompt, model, or context-assembly problem. A single end-to-end "answer quality" score collapses these into one number and points at nothing. The layered report is what turns "the bot is worse this week" into "MRR dropped after Tuesday's chunking change, revert it."

With retrieval measured, score what the model produced. These are judged by an LLM acting as grader — the only way to score free-text answers at scale — with periodic human spot-audits to keep the judge honest. The metrics that matter:

A golden set built only from answerable questions contains a trap: it scores a system that answers everything confidently as an improvement. You must include questions your corpus does not cover — plausible ones, phrased like real user questions, adjacent to topics the corpus does cover — and measure two things: abstention precision (when the system declined, was declining correct?) and abstention recall (of the questions it should have declined, how many did it actually decline?).

These two numbers pull against each other, which is exactly why both must be tracked. Loosen the confidence threshold and abstention recall falls — the system starts improvising answers to uncovered questions, the exact behavior that ends up in a screenshot. Tighten it and abstention precision falls the other way: the system declines questions it could have answered, and users learn the assistant is useless. The threshold is a product decision, but it can only be a decision if both curves are measured. This is also why enforced abstention — structurally withholding passages when confidence is low, rather than politely asking the model to say "I don't know" — is the mechanism worth testing: a request in the prompt is the one part of the pipeline your eval cannot pin down.

Mechanically: a harness replays every golden-set question through the live pipeline, records what was retrieved and what was generated, scores each layer, and emits one report — recall@k, MRR, nDCG, fact coverage, grounded rate, and abstention precision/recall. BeforeQuery ships this as its ragEval tool, driven by a YAML golden set, but the shape is the same whatever you build: fixed inputs, the real pipeline, layered scores, one comparable artifact per run.

The comparison is the point. A single report tells you almost nothing — is 0.86 grounded rate good? — but a diff against the last report tells you everything: the reranker swap bought four points of MRR and cost one point of abstention recall, and now that trade is a decision instead of a surprise.

Evaluation pays off when it runs on every meaningful change: re-derive scores after switching rerankers, embedding models, or chunking strategies, and fail the change if it regresses the stored baseline. Treat retrieval quality like test coverage — a number that is someone's job to keep from falling, wired into the same discipline as CI: baseline in the repo, diff on every pipeline change, red means stop.

The objection is always cost — building a golden set takes days, and grading runs cost tokens. The comparison to make is the cost of the alternative: every tuning decision made blind, every provider swap a leap of faith, and the strongest regressions discovered by your angriest users. Teams with an eval gate tune fearlessly, because any mistake is caught in minutes. Teams without one stop touching their pipeline out of fear — which is its own slow regression, compounding quietly while the corpus and the questions drift.