Our support lead asked me a question I could not answer for two days. Why, she wanted to know, could the assistant handle "how do I rotate an API key" perfectly but return complete rubbish for "E4021", when the error code table was right there in the docs she had written herself.

I assumed the ingestion pipeline had dropped the table. It had not. Chunk 18432 held the row for E4021, indexed and healthy. Cosine similarity between the query embedding and that chunk came back at 0.71. Similarity between the same query and an unrelated troubleshooting paragraph came back at 0.70. An embedding model has no mechanism for telling E4021 from E4012 from E4201, because those strings tokenise into near identical fragments and carry almost no semantic content. Two days of my life, and the fix was a retrieval method invented in 1994.

Assistant so far, and the queries it fails

Last part we put 50,000 chunks of product docs, support tickets and changelog entries into three vector stores and picked Qdrant, with pgvector as the fallback for teams already running Postgres. This part adds a second retriever alongside the dense one and decides how to combine them.

To measure anything I needed labels, so I hand tagged 120 real questions pulled from our support inbox with the chunk that actually answers each one. That took an afternoon and it is the single highest value afternoon in this whole series. Part 13 turns this into a proper retrieval evaluation harness; for now, treat it as a crude ruler that is still infinitely better than eyeballing results.

Splitting those 120 queries by shape explained everything. Eighty nine were conceptual, phrased in the reader’s own words: how do I invite a teammate, what happens when a trial expires. Dense retrieval answered those at recall@10 of 0.86. Thirty one contained a literal token that appears verbatim in the corpus: an error code, a config key like retry_backoff_ms, a ticket reference, a version string. Dense retrieval answered those at 0.29. Averaged together you get 0.71 and a completely misleading picture of a system that works fine most of the time.

Sparse retrieval and BM25 in plain terms

BM25 scores a document by how many query terms it contains, weighted so that rare terms count for more than common ones, and damped so that a document repeating a term forty times does not beat one that uses it twice in a relevant sentence. Length matters too: a short chunk containing E4021 outranks a long one mentioning it in passing. Nothing about meaning enters the calculation, which is exactly why it works where embeddings do not. E4021 is a rare term, and rare terms are BM25’s entire reason for existing.

Sparse is the standard name because the representation is a vector with one dimension per vocabulary term, almost all of them zero. You do not need a new database for it. Postgres ships a full text engine, Qdrant stores sparse vectors natively alongside dense ones, and OpenSearch and Elasticsearch have done this for a decade. I used Postgres for the first version because the corpus already sat there.

Running lexical and dense retrieval side by side

Two independent queries, two ranked lists of chunk ids. Postgres gives you both from one connection when pgvector and the built in text search sit in the same table. Credentials come from an environment variable, never from the file.

# tested with PostgreSQL 17, pgvector 0.8.2, psycopg 3.2.9, numpy 2.3.4, Python 3.10.12
import os
import numpy as np, psycopg
from pgvector.psycopg import register_vector

conn = psycopg.connect(os.environ['PG_DSN'], autocommit=True)
register_vector(conn)

# one time: a generated tsvector column plus a GIN index over it
conn.execute('''ALTER TABLE chunks
  ADD COLUMN IF NOT EXISTS body_tsv tsvector
  GENERATED ALWAYS AS (to_tsvector('english', body)) STORED''')
conn.execute('CREATE INDEX IF NOT EXISTS chunks_tsv_idx ON chunks USING GIN (body_tsv)')

def dense_search(qvec, limit=40):
    rows = conn.execute(
        'SELECT id FROM chunks ORDER BY embedding <=> %s LIMIT %s',
        (qvec, limit)).fetchall()
    return [r[0] for r in rows]

def bm25_search(text, limit=40):
    rows = conn.execute('''SELECT id,
               ts_rank_cd(body_tsv, websearch_to_tsquery('english', %s)) AS s
          FROM chunks
         WHERE body_tsv @@ websearch_to_tsquery('english', %s)
      ORDER BY s DESC LIMIT %s''', (text, text, limit)).fetchall()
    return [r[0] for r in rows]

q = 'E4021'
qvec = np.load('query_vectors.npy')[7]   # embedded with the Part 10 model
print('dense ', dense_search(q_vec := qvec)[:5])
print('sparse', bm25_search(q)[:5])

dense  [9137, 22015, 3388, 41902, 17760]
sparse [18432, 18433, 5091, 27714, 902]

Before that worked I spent twenty minutes on a mistake worth showing, because everyone makes it once. My first version ran the match against the text column directly:

Traceback (most recent call last):
  File 'retrieve.py', line 31, in bm25_search
    rows = conn.execute('''SELECT id ... WHERE body @@ websearch_to_tsquery(...)''')
psycopg.errors.UndefinedFunction: operator does not exist: text @@ tsquery
LINE 4:          WHERE body @@ websearch_to_tsquery('english', $1)
                      ^
HINT:  No operator matches the given name and argument types.

Note which function I reached for. Use websearch_to_tsquery, not to_tsquery. Raw user text fed to to_tsquery raises a syntax error the first time somebody types an apostrophe or an ampersand, and you will find out in production. On ranking, ts_rank_cd factors in how close the matched terms sit to each other, which suits chunk sized passages better than plain ts_rank. Neither is true BM25, and for a 50,000 chunk corpus that difference cost me about 0.01 nDCG, which I was happy to trade for zero new infrastructure.

Fusing two ranked lists

Two lists, two incompatible score scales. A cosine distance of 0.31 and a ts_rank_cd of 0.084 have no common unit, and normalising them is guesswork that breaks whenever the score distribution shifts. Reciprocal rank fusion sidesteps the problem by throwing the scores away and keeping only the positions. A document’s fused score is the sum, over every list it appears in, of one divided by k plus its rank. Cormack and colleagues published it in 2009 and it has quietly outlived most of what replaced it.

Twelve lines, no dependencies, and it is the piece of code from this part I have copied into the most projects. Keep it standalone so you can unit test it against hand written lists.

def rrf(ranked_lists, k=2, weights=None):
    '''Fuse ranked id lists. Rank is zero based, best first.'''
    weights = weights or [1.0] * len(ranked_lists)
    scores = {}
    for lst, w in zip(ranked_lists, weights):
        for rank, doc_id in enumerate(lst):
            scores[doc_id] = scores.get(doc_id, 0.0) + w / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

dense  = dense_search(qvec, 40)
sparse = bm25_search('E4021', 40)
fused  = rrf([dense, sparse])
print(fused[:5])
print('18432 rank:', fused.index(18432))

[18432, 9137, 18433, 22015, 5091]
18432 rank: 0

If your store does fusion natively, use it, because one round trip beats two plus application side merging. Qdrant runs both retrievers as prefetches and fuses server side:

# tested with qdrant-client 1.18.0 against Qdrant server 1.16
from qdrant_client import QdrantClient, models

client = QdrantClient(url=os.environ['QDRANT_URL'], api_key=os.environ['QDRANT_API_KEY'])

hits = client.query_points(
    collection_name='docs_chunks',
    prefetch=[
        models.Prefetch(
            query=models.SparseVector(indices=sparse_idx, values=sparse_val),
            using='sparse', limit=40),
        models.Prefetch(query=qvec.tolist(), using='dense', limit=40),
    ],
    query=models.RrfQuery(rrf=models.Rrf(k=2)),
    limit=40,
    with_payload=True,
).points
print(len(hits), hits[0].id, round(hits[0].score, 4))

My first attempt at that call failed because the collection had been created in Part 11 with one unnamed dense vector and no sparse configuration:

qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 400 (Bad Request)
Raw response content:
b'{"status":{"error":"Wrong input: Not existing vector name error: sparse"},"time":0.0}'

Qdrant also offers distribution based score fusion through models.FusionQuery(fusion=models.Fusion.DBSF), which normalises each retriever’s score distribution instead of discarding scores. It beat RRF by 0.01 nDCG on my set and lost by 0.02 on a noisier internal corpus. Not worth the reasoning cost. Start with RRF.

Reranking with a cross encoder

Fusion got the right chunk into the candidate list. Getting it into the top three is a different job. An embedding model encodes the query and the chunk separately and never sees them together, which is what makes it fast enough to index 50,000 documents in advance. A cross encoder does the opposite: it takes the query and one chunk as a single input, runs the whole transformer over the pair, and emits one relevance score. No precomputation is possible, which is why you only ever run it over a handful of survivors.

# tested with sentence-transformers 5.6.0, torch 2.9.1, on 8 vCPU, no GPU
from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L6-v2', max_length=512)

candidates = fetch_bodies(fused[:40])          # list of chunk texts, same order
ranked = reranker.rank(query, candidates, top_k=6, return_documents=False,
                       batch_size=32)
for r in ranked[:3]:
    print(fused[r['corpus_id']], round(r['score'], 3))

18432 8.914
18433 5.207
27714 1.883

Latency and quality of each stage

Every measurement below comes from the same 120 queries against the same 50,000 chunks on one 8 vCPU box with no GPU, p95 over five runs, July 2026. Retrieval latency excludes the generation call, which dominates end to end time anyway and is the subject of Part 26.

Read those two recall columns again. Fusion moved recall by 15 points. Reranking moved it by zero, because a reranker can only reorder what retrieval already found. If your evaluation says the right chunk is missing from the candidate list, a bigger reranker is a waste of money and you should widen retrieval instead.

Candidate depth deserves the same scepticism. Advice to retrieve 100 and rerank down to 10 is everywhere. Going from 40 to 100 candidates on my set added 340 ms of reranker time and gained 0.006 nDCG, which is noise. Forty was the knee for this corpus. Measure yours, because the answer moves with chunk size and corpus breadth.

On hardware: the 568M parameter BGE model is genuinely the better ranker, and at 1.87 seconds on CPU it is unusable for anything interactive. On a single L4 at fp16 the same 40 pairs took 118 ms, which changes the decision completely. Hosted reranking endpoints from Cohere, Jina and Voyage remove the hardware question and add a per query fee and a second provider dependency. For an internal tool answering a few thousand questions a day, MiniLM on the CPU you already pay for is the honest answer.

Symptom lookup for hybrid retrieval

Bookmark this one. Every row is something that cost me at least an hour.

Row two deserves a sentence of its own. English text search configurations stem aggressively, and a token like retry_backoff_ms can end up split or reduced in ways that stop it matching the query form. I found this only because a single query kept failing after everything else worked. Indexing identifier heavy fields with the simple configuration alongside the english one fixed it and cost nothing.

Retrieval stack I would ship for a docs assistant

Run dense and BM25 in parallel, take 40 from each, fuse with RRF at k of 2, rerank the 40 with cross-encoder/ms-marco-MiniLM-L6-v2, and pass the top 6 into the prompt. That configuration cost me 196 ms and 0.31 USD per thousand queries, and it took nDCG@10 from 0.58 to 0.77 with no change to the model, the prompt or the chunker. Reach for bge-reranker-v2-m3 only when a GPU is already in the budget, and skip hosted reranking APIs until you have proven a local model is the bottleneck.

Sequence matters more than any single choice here. Add BM25 first and measure, because it is free and it fixed the largest failure class in our corpus. Add fusion second. Add a reranker last, and only after your numbers show that the right chunk is already in the candidate list and merely sitting too low. Skipping to the reranker is the expensive mistake, and it is the one I see most often, because reranking is the interesting part and BM25 is not.

One thing to do on Monday: split your evaluation queries into those containing a literal identifier and those that do not, then score each group separately. If the gap looks anything like my 0.86 against 0.29, you need a lexical retriever more than you need a better embedding model. On the concepts underneath all of this, what RAG actually is and vector databases explained stay useful, and for building an honest evaluation split without leaking, model evaluation and data leakage applies here exactly as it does to a classifier. Part 13 takes the 120 query set seriously and separates retrieval evaluation from answer evaluation, which is where most teams conflate two very different problems.

References

• Hybrid Queries, Qdrant documentation, for the prefetch model, the RRF formula, the default k of 2 and distribution based score fusion.
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, Cormack, Clarke and Buettcher, SIGIR 2009, the original RRF paper and the source of k of 60.
• CrossEncoder API reference, Sentence Transformers 5.6.0, for rank(), predict(), max_length and batch_size.
• Controlling Text Search, PostgreSQL documentation, for websearch_to_tsquery, ts_rank_cd and text search configurations.
• cross-encoder/ms-marco-MiniLM-L6-v2, Hugging Face model card, for the reranker used throughout this part.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts