How to evaluate a RAG system (2026)

The short answer

A RAG evaluation has two jobs: tell you whether the system is good enough, and tell you where it failed. Most teams try to do both with one end-to-end score, which is why the result is unactionable.

Score retrieval and generation separately. Build a golden set from real inputs. Write the rubric before you look at model outputs. Calibrate any LLM judge against human labels. Then wire the whole thing into a gate that runs before a release.

That is the difference between "the chatbot feels better" and "retrieval recall improved by 14 points, answer correctness moved by 6, and the refusal behaviour regressed on two cases."

Why do end-to-end scores hide the problem?

Consider a query that fails. The assistant gives a confident, wrong answer. There are at least four possible causes:

1. The right document is not in the index.
2. The right document is in the index but the retriever returned a different chunk.
3. The right chunk was returned but the prompt put it too far down the context, or buried it among distractors.
4. The chunk was there and the model ignored it, misread it or chose to answer from its prior.

An end-to-end correctness score treats all four as the same failure. A retrieval score separates the first two. A groundedness score separates the last two. Without that separation, the team spends a sprint rewriting prompts when the real problem is chunking, or re-indexing documents when the real problem is that the model refuses to cite anything.

Research on long-context models has shown that position matters: models use information at the beginning and end of a context more reliably than information in the middle. That is a prompt and ranking problem, not a knowledge problem. An end-to-end score cannot see it.

How do you build a golden set that survives?

A golden set is a small, curated set of queries with expected answers and expected source documents. It is not a dump of production logs, and it is not a benchmark borrowed from a paper.

Start with 30 to 80 cases, stratified by:

• Common queries. The requests that make up most of the traffic.
• Known failures. The tickets that already exist, with the real query text.
• Edge cases. Empty results, ambiguous questions, multi-hop questions, questions about documents that were recently updated.
• Refusals. Questions the system should decline or escalate, not answer.
• Adversarial inputs. Prompt-injection attempts and questions that tempt the model to answer beyond its sources.

For each case, record:

Remove or mask personal data before the set leaves the customer environment. A golden set is still production data.

Refresh the set when the product changes. Add a case every time a real failure escapes. Remove cases that no longer represent the product. The set is a living asset, not a one-time deliverable.

Score retrieval and generation separately

The exact metrics matter less than scoring the two layers independently. A practical set looks like this.

For agents, the final answer is not enough. An agent can produce a correct answer through a dangerous tool call, or a wrong answer through a sensible one. Score the trajectory as well as the output.

How do you write a rubric that holds up?

This is the discipline that separates an evaluation from a rationalisation. If the rubric is written after seeing the model's answers, it will drift toward what the model already does.

A usable rubric has three levels and a short rationale for each:

• Pass. The answer is correct, complete enough for the user's purpose, and supported by the retrieved context.
• Partial. The answer is directionally correct but misses a required element, or cites a source that only partly supports it.
• Fail. The answer is wrong, unsupported, unsafe, or answers a question that should have been refused.

Grade a sample twice: once by two humans independently, then compare. If agreement is below roughly 80%, the rubric is ambiguous, not the graders. Rewrite the rubric and repeat. This calibration step is what makes an LLM judge defensible later.

Use an LLM judge without fooling yourself

An LLM judge is useful because human grading is slow. It is dangerous because a judge can be confidently wrong, and its errors are correlated with the same model family that produced the answer.

Four rules keep it honest:

1. Calibrate against human labels. Measure agreement on a sample before using the judge at scale. Report the agreement number alongside the score.
2. Pin the judge version. A judge model upgrade can change your historical numbers. Record the model, prompt and decoding settings with every run.
3. Hide the model identity. Do not tell the judge which system produced the answer. Position bias and self-preference are real.
4. Spot-check the failures. Read the cases the judge marked as failures, and a sample of the passes. The judge is a filter, not an oracle.

Use deterministic checks wherever possible. JSON schema validity, citation presence, required section order and refusal patterns do not need a judge. Save the model for the parts that genuinely require judgement.

How do you put a gate in front of releases?

An evaluation that runs once is a report. An evaluation that runs before every release is a control.

The gate needs three things:

• A fixed test set. The same cases, with a version number.
• A threshold and a regression budget. For example: answer correctness must stay above 82%, and no category may drop more than 3 points from the last release.
• A report that a human can read. The score, the delta, the failing cases and the category breakdown.

Run the gate on every change to the prompt, model, index, chunking, re-ranker or retrieval configuration. Add a scheduled run for drift, because the documents change even when the code does not.

The gate does not need to block every merge. It needs to make a regression visible before a user finds it. A weekly report that nobody reads is not a gate.

A ten-day audit plan

This is the shape I use for a focused RAG or agent audit.

The output is not a dashboard. It is a harness, a rubric and a prioritised list of fixes, with the evidence behind each one.

What good looks like

A good RAG evaluation has five properties:

• It runs on real inputs, not synthetic questions.
• It scores retrieval and generation separately.
• It has a written rubric that two people can apply consistently.
• It produces a failure analysis, not just a number.
• It can be rerun by the team without the consultant in the room.

If you are preparing for a model upgrade, a prompt rewrite or an enterprise review, an internal assistant build is the smallest useful engagement. If the problem is already in production, the workflow automation service covers the retrieval, guardrail and observability work that follows.