Evaluate retrieval and answers separately in RAG

A weak RAG answer may come from missing evidence, poor ranking, noisy context, or generation that does not follow the supplied material. The final paragraph alone cannot distinguish these causes.

Evaluate stages separately so you can see what a change improved and what it broke.

Check whether evidence was retrieved

Label supporting source identifiers for a collection of queries. Recall@k measures how many labeled relevant sources appear in the first k results. It helps expose missed evidence.

Allow multiple valid sources when a question has several supporting documents. Labeling only one can incorrectly penalize another correct source. Also specify whether identifiers refer to whole documents or individual chunks.

MRR measures the rank of the first relevant result. It helps detect evidence placed too far down the list and answers a different question from Recall@k.

Compute retrieval metrics locally

This example uses constructed document identifiers. A query with no relevant sources receives no retrieval score and should be evaluated separately for appropriate abstention.

def retrieval_metrics(ranked, relevant, k):
    if not relevant:
        return {"recall": None, "mrr": None}
    top = ranked[:k]
    recall = len(set(top) & relevant) / len(relevant)
    first = next((i for i, doc in enumerate(top, 1) if doc in relevant), None)
    return {"recall": recall, "mrr": 1 / first if first else 0}

score = retrieval_metrics(["doc-c", "doc-b"], {"doc-a", "doc-b"}, k=2)
assert score == {"recall": 0.5, "mrr": 0.5}
assert retrieval_metrics(["doc-c"], set(), 1)["recall"] is None
print(score)

The MRR calculation here considers only the first k results. State that definition in a real report. For averages, identify which unanswerable or unlabeled cases were excluded.

Check whether the answer uses its evidence

Inspect whether claims are supported by the supplied passages, citations lead to the right locations, and the response answers the actual question. A citation is not proof that the evidence was used correctly.

Model-assisted scoring can help, but the scoring model can also fail. Keep manually judged examples, sample across question categories, and check agreement with human judgments.

Work such as RAGAS separates context quality, faithfulness, and answer quality. Use these dimensions to design an interpretable evaluation rather than treating a single score as self-explanatory.

Include missing-answer and permission cases

Test questions with no answer in the collection, outdated sources, and sources the user cannot access. The correct behavior may be explaining missing evidence, asking for clarification, or stopping access rather than generating a long answer.

After a permission change, check cached contexts and saved retrieval results as well as the index. Authorization cannot be a one-time import check.

Try it next

Fix a small query set and record retrieval, ranking, evidence support, and final-answer results separately. Change one major variable at a time, such as chunk size or the ranking method.

Diagnose the failure category before choosing a component to adjust. Add discovered failures to regression cases instead of expanding only the successful examples.