Two numbers came out of the same evaluation run, printed one line apart. Retrieval hit rate at 5, 0.858. Answer correctness, 0.617. I had spent the previous fortnight assuming those two things moved together, and here was a quarter of my queries where the assistant had the right paragraph sitting in its context window and still produced a wrong answer.

That gap is the whole subject of this part. A retrieval augmented system has two independent places it can fail, and almost every team I have worked with reports a single blended accuracy figure that hides which one broke. You then spend your budget on whichever component is more fun to tune, which is never the one at fault.

Assistant so far, and what we still cannot see

Last part the assistant gained a second retriever and a cross encoder: BM25 alongside dense search, fused with reciprocal rank fusion, reranked down to 6 chunks that go into the prompt. This part stops improving it and starts measuring it properly, because every decision from here to Part 30 depends on being able to tell whether a change helped.

Here is my worst two weeks on this project, and I tell it because I have watched three other teams repeat it. Answer accuracy sat at 0.62. Convinced that retrieval was starving the model, I swapped embedding models twice, tuned chunk size, added the reranker, and pushed recall@10 from 0.71 to 0.86. Answer accuracy after all of that: 0.64. Fifteen points of retrieval quality bought two points of what the user actually experiences. I had no instrument capable of telling me that most of my failures were downstream of retrieval, so I optimised the component I could measure by feel.

Two questions hiding inside one score

Split the system at the prompt boundary and two separate questions fall out. Did the retriever put at least one chunk containing the answer into the context? That is a question about ranked lists of document ids, and information retrieval has had rigorous answers to it since the 1960s. Given those chunks, did the model produce a correct, grounded answer? That is a question about text, and it needs a human or a judge model.

Cost differs sharply too, and that shapes how often you can run each one. Scoring retrieval over 120 queries is a numpy operation that finishes in about 40 milliseconds and costs nothing, so it belongs in continuous integration on every commit. Scoring answers means 120 judge calls, roughly 0.09 USD and ninety seconds at our prompt sizes on a small judge model, which is cheap enough nightly and irritating enough per commit. Treating them as one pipeline forces the cheap check to run at the expensive check’s cadence.

Building an eval set in an afternoon

Labels come first and there is no way around it. I pulled 120 questions from six months of support tickets, weighted toward what people actually ask rather than what the docs cover well, and for each one recorded the chunk ids that genuinely answer it. An afternoon of work. Keep it in JSONL in the same repository as the code, because an eval set that lives in a spreadsheet on somebody’s laptop stops being maintained within a month.

# tested with ranx 0.3.21, numpy 2.2.6, numba 0.66.0, Python 3.10.12
# eval/queries.jsonl, one object per line:
#   {"qid": "q000", "text": "how do I rotate an API key",
#    "gold": {"c18432": 2, "c18433": 1}, "slice": "conceptual"}
import json
from ranx import Qrels

rows = [json.loads(l) for l in open('eval/queries.jsonl')]
qrels = Qrels({r['qid']: r['gold'] for r in rows})

print('queries   ', len(rows))
print('judgements', sum(len(r['gold']) for r in rows))
print('slices    ', {s: sum(r['slice'] == s for r in rows)
                     for s in {r['slice'] for r in rows}})

queries    120
judgements 171
slices     {'conceptual': 89, 'identifier': 31}

One decision here matters more than the labelling effort: label chunks, not documents. My first version recorded the source document for each question, which felt more durable. It was useless, because a 40 page API reference is one document and the assistant retrieves 300 token chunks. Every retriever scored a perfect hit and the metric had no discriminating power at all. Rebuilding against chunk ids means the labels break whenever you change the chunker, which is annoying and correct: a different chunker genuinely is a different retrieval system.

Metrics that answer the retrieval question

Four metrics cover everything I have needed. Hit rate asks whether at least one relevant chunk appeared, which is the closest proxy for whether the model had any chance. Recall asks what fraction of relevant chunks were found. Mean reciprocal rank rewards putting the first relevant chunk near the top. Normalised discounted cumulative gain grades the whole ordering and is the one to watch when you are tuning a reranker.

Implementing these correctly is fiddly and unnecessary. ranx wraps the standard definitions, runs them through numba, and gives you paired significance testing for free. A run is just a dictionary of query id to a dictionary of document id and score, which is exactly what your retriever already produces.

from ranx import Run, evaluate, compare

METRICS = ['hit_rate@1', 'recall@5', 'recall@10', 'mrr@10', 'ndcg@10']

def to_run(retriever, name, depth=10):
    scored = {}
    for r in rows:
        hits = retriever(r['text'], limit=depth)      # [(chunk_id, score), ...]
        scored[r['qid']] = {cid: float(s) for cid, s in hits}
    run = Run(scored)
    run.name = name
    return run

runs = [to_run(dense_search, 'dense'),
        to_run(bm25_search, 'bm25'),
        to_run(hybrid_search, 'hybrid_rrf'),
        to_run(hybrid_reranked, 'hybrid_rerank')]

for run in runs:
    scores = evaluate(qrels, run, METRICS)
    print(f"{run.name:<15}", {k: round(float(v), 3) for k, v in scores.items()})

dense           {'hit_rate@1': 0.417, 'recall@5': 0.7, 'recall@10': 0.708, 'mrr@10': 0.54, 'ndcg@10': 0.582}
bm25            {'hit_rate@1': 0.425, 'recall@5': 0.633, 'recall@10': 0.642, 'mrr@10': 0.518, 'ndcg@10': 0.549}
hybrid_rrf      {'hit_rate@1': 0.458, 'recall@5': 0.858, 'recall@10': 0.858, 'mrr@10': 0.615, 'ndcg@10': 0.677}
hybrid_rerank   {'hit_rate@1': 0.633, 'recall@5': 0.858, 'recall@10': 0.858, 'mrr@10': 0.735, 'ndcg@10': 0.767}

My first attempt at this crashed, and the failure is worth showing because it catches everyone who evaluates a retriever that returns nothing for some queries. A BM25 retriever with no lexical overlap returns an empty list, my code skipped writing those query ids into the run, and ranx refused to score a run that does not cover the answer key:

Traceback (most recent call last):
  File 'eval/run_retrieval.py', line 31, in <module>
    scores = evaluate(qrels, run, METRICS)
  File '.../ranx/meta/evaluate.py', line 91, in evaluate
    assert qrels.keys() == run.keys(), (
AssertionError: Qrels and Run query ids do not match. Pass `make_comparable=True` to add empty results for queries missing from the run and remove those not appearing in qrels.

Passing make_comparable=True fixes it by scoring the missing queries as zero, which is the honest treatment. Better still, write the empty dictionary yourself so the shape of your run always matches the shape of your answer key and the library never has to guess.

Once two configurations are close, eyeballing the third decimal place stops being defensible. Paired significance testing over 120 queries takes one call:

report = compare(qrels, runs,
                 metrics=['recall@10', 'ndcg@10', 'mrr@10'],
                 stat_test='student', max_p=0.05)
print(report)

#    Model          Recall@10    NDCG@10    MRR@10
---  -------------  -----------  ---------  --------
a    dense          0.708        0.582      0.540
b    bm25           0.642        0.549      0.518
c    hybrid_rrf     0.858ᵃᵇ      0.677ᵇ     0.615
d    hybrid_rerank  0.858ᵃᵇ      0.767ᵃᵇᶜ   0.735ᵃᵇᶜ

Diagnosing failures by splitting the score

Here is where the separation pays for itself. Join the retrieval outcome and the answer outcome per query and cross tabulate them. Twenty lines of code, and it is the single most useful artifact this series has produced so far.

import collections

# answers.jsonl holds one graded verdict per query, from a human or a judge:
#   {"qid": "q000", "correct": true}
verdicts = {json.loads(l)['qid']: json.loads(l)['correct']
            for l in open('eval/answers.jsonl')}

prod = runs[3].to_dict()                      # the configuration we ship
retrieved = {r['qid']: bool(set(r['gold']) & set(list(prod[r['qid']])[:6]))
             for r in rows}

table = collections.Counter((retrieved[q], verdicts[q]) for q in verdicts)
n = len(verdicts)
hit = sum(retrieved.values())
ok  = sum(verdicts.values())
print('retrieval hit_rate@6  :', round(hit / n, 3))
print('answer accuracy       :', round(ok / n, 3))
print('answer given retrieved:', round(table[(True, True)] / hit, 3))
print('answer given missed   :', round(table[(False, True)] / (n - hit), 3))

retrieval hit_rate@6  : 0.858
answer accuracy       : 0.617
answer given retrieved: 0.689
answer given missed   : 0.176

Slicing goes one level deeper for free, because you recorded the slice at labelling time. Identifier queries and conceptual queries behave differently at both gates, and averaging them hides the only actionable signal in the set.

Look at the rightmost pair. Twelve pricing and quota questions retrieved at 0.92 and answered at 0.33, because the docs give a per seat rate and a formula and the model was doing the arithmetic in prose. That slice needs a calculator tool, which is Part 14, and no amount of retrieval work would have surfaced it. Identifier queries show the opposite shape: retrieval is still the weak gate at 0.74 even after adding BM25, so that slice does justify more retrieval effort.

Two disciplines keep this honest over time. Freeze a portion of the set and never look at it while tuning, exactly as you would with a classifier; my 120 splits 90 for iteration and 30 held out, and the held out gap has caught two cases of me overfitting prompt wording to specific questions. Then version the qrels file alongside the chunker configuration, because a change to chunking invalidates the labels and a silently stale answer key is worse than no answer key. Drift applies to eval sets as much as to models, a point monitoring models in production makes at length, and your support inbox keeps generating new question shapes whether or not you add them.

Evaluation setup I would ship for a docs assistant

Keep 120 labelled questions in a JSONL file in the repository, split 90 for iteration and 30 held out. Score retrieval with ranx on every commit using hit_rate and nDCG at your real prompt depth, because it costs nothing and finishes in under a second. Score answers nightly with a judge and store the per query verdicts, not just the mean. Regenerate the failure attribution table on every run and route the work by which cell grew.

Avoid the tempting shortcut of using a framework’s reference free RAG metrics as your primary signal. Ragas offers LLMContextPrecisionWithoutReference, which judges retrieved chunks against the generated response rather than a labelled answer, and it is genuinely useful for monitoring live traffic you cannot label. As a development metric it is circular, because it grades retrieval by an answer that retrieval produced, and it will happily report 0.9 while your labelled recall sits at 0.6. Reference free metrics are for production monitoring. Labelled metrics are for decisions.

One thing to do on Monday: take twenty of your own failing queries, mark for each whether the correct passage was actually in the prompt, and count the two columns. If your split looks anything like my 32 against 14, close the browser tab with the embedding model benchmarks and go and read your prompt instead. If retrieval concepts underneath this are still hazy, what RAG actually is covers the shape of the system, and getting data into Python covers the JSONL and file handling this harness leans on. Part 14 starts the next phase and gives the assistant tools, beginning with tool calling from first principles, which is what those 12 pricing questions have been waiting for.

References

• Metrics, ranx documentation, for the metric aliases, the at k syntax and the exact definitions of hit rate, recall, MRR and nDCG used above.
• Compare, ranx documentation, for the paired statistical tests and the superscript notation in the comparison report.
• Cumulated gain based evaluation of IR techniques, Järvelin and Kekäläinen, ACM TOIS 2002, the paper that defines discounted cumulative gain and its normalised form.
• Context Precision, Ragas documentation, for the reference free and reference based context precision variants discussed in the closing section.
• trec_eval, NIST, the reference implementation these metric definitions descend from and the source of the qrels file format.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts