The answer was fluent, confident, and quoted a refund window we retired eighteen months ago. The model did nothing wrong. It answered from the paragraph it was handed, and the retrieval step handed it a stale chunk that keyword search had ranked first. That is the failure mode this part is about. A grounded assistant is only ever as good as what its search index returns on the query, and on Azure that index is Azure AI Search.
What RAG is, and why the index does the work
Retrieval-augmented generation, RAG, is a pattern where you search a knowledge store for text relevant to the user question, then send that text to the model along with the question and ask it to answer from what you gave it. The model is not retrained and it learns nothing permanent. Each request is a fresh prompt that happens to carry a few paragraphs of your data. That is the entire trick, and it is why a company can ground a general model on private documents without ever touching the model weights.
Because the model only sees what retrieval selected, the search step is where accuracy is won or lost. Azure AI Search is the managed search service that holds your indexed content and answers those queries, over plain keywords, over vectors, or both at once. It is a separate resource from Azure OpenAI, billed on its own tier, and it does the unglamorous half of RAG: store the chunks, find the right ones, hand back a ranked list. Get that list right and a small model looks brilliant. Get it wrong and the best model on the market quotes your retired refund policy with total confidence.
From documents to a searchable index
Before you can search anything, raw files have to become index records. Two operations make that happen. Chunking splits a long document into passages small enough for a model to embed and for the prompt to hold, because a fifty page PDF is useless as a single retrieval unit. Embedding turns each chunk into a vector, a list of numbers that places similar meanings near each other in space, so a query about refunds lands next to a passage about returns even when the words differ. You can write both steps yourself, or you can let Azure AI Search do them inside its indexing pipeline.
That built-in path is called integrated vectorization, and it is the one I reach for first. You point an indexer at a data source such as Blob Storage, attach a Text Split skill to chunk and an AzureOpenAIEmbedding skill to vectorize, and the indexer runs the whole thing on a schedule. When a document changes, the indexer notices, re-chunks only what moved, calls your embedding deployment, and writes both the text and its vector into the index. It is available on every tier and region, and it removes the standing pipeline that teams otherwise babysit. The embedding skill calls a model you deployed, so the same deployment choices from Part 3 apply here.
Keyword, vector, or both?
Two retrieval styles live in the same index, and they fail in opposite ways. Keyword search, scored by an algorithm called BM25, matches the literal terms in the query. It is exact and unbeatable on part numbers, error codes, and proper nouns, and it goes blind the moment the user asks for a refund while your document says return. Vector search matches on meaning, so it connects refund and return with no shared word, and it degrades on rare exact strings because a product code has no semantic neighborhood to sit in. Each covers the other one gap.
Hybrid search runs both at once against the same index and combines the two ranked lists into one. You almost always want it. The keyword leg keeps you honest on identifiers and quotes, the vector leg rescues the paraphrased and the synonym-heavy question, and the merge gives you a single ordering that is better than either input. The cost is small: two searches instead of one over the same data, and a merge step measured in milliseconds. The table below is the mental model I use when someone asks why we are not just doing vectors.
| Mode | Matches on | Strong at | Weak at |
|---|---|---|---|
| Keyword, BM25 | Literal terms | Codes, names, exact quotes | Synonyms, paraphrase |
| Vector | Meaning | Rephrased questions | Rare exact strings |
| Hybrid | Both, then merged | Most real questions | Slightly more compute |
The weak column is why I have never shipped vector-only retrieval to production. Someone always pastes an order number.
How hybrid search merges two rankings
The two legs produce scores that live on different scales, so you cannot just add them. BM25 scores are unbounded and vary with term frequency; vector similarity sits in a fixed range. Azure AI Search sidesteps the mismatch with Reciprocal Rank Fusion, RRF, which throws away the raw scores and keeps only the rank position. Each document gets a contribution of one divided by a constant plus its rank in each list, and the contributions are summed. The constant, k, is set to 60 in the service. A document ranked first in a list contributes one over sixty one; ranked eighth, one over sixty eight.
The effect is that agreement wins. A chunk that both legs like, even at middling ranks, floats above a chunk that one leg loved and the other never surfaced. That is exactly the behavior you want for grounding, because a passage two different methods both consider relevant is a safer thing to feed the model than a single method fluke. RRF is not tunable in the usual sense; you influence the merge by improving each leg, not by turning a fusion dial.
Worked example
Take four candidate chunks for one query. Chunk A ranks first on vector and eighth on keyword, so its RRF score is 1/61 plus 1/68, about 0.0311. Chunk B ranks second on keyword and fifteenth on vector, 1/62 plus 1/75, about 0.0294. Chunk C shows up only on keyword at rank third, 1/63, about 0.0159, and Chunk D only on vector at rank second, 1/62, about 0.0161. Final order: A, then B, then D, then C. The chunk both legs ranked well, A, wins even though neither leg put it strictly on top. The chart below plots those four scores.
Reranking the shortlist with the semantic ranker
RRF gives you a good ordering cheaply, but it never reads the passages against the question; it only fuses ranks. The semantic ranker does read them. It takes the top results from the BM25 or RRF pass, up to 50 of them, and rescores each one with a language model trained to judge how well a passage answers a query, producing a reranker score on a scale from 0 to about 4. Results that fooled the first pass on shallow term overlap fall; results that truly answer the question rise. This second stage is where a lot of the felt quality of a RAG system comes from.
It also offers query rewriting, where the ranker expands the original query into as many as ten variants that fix typos or add synonyms before searching, which lifts recall on messy human phrasing. The trade is cost and latency. The semantic ranker is a billed add-on: the first 1,000 queries each month are free, and past that you pay per thousand, and it needs a Basic tier search service or higher, not the free one. It adds a little time per query too. So you turn it on for the user-facing path and leave it off for bulk or internal jobs that do not need the extra precision.
A retrieval query in Python
Here is one hybrid query with the semantic ranker turned on, using the azure-search-documents library and no key, the same identity-based auth from Part 10. Because the index was built with integrated vectorization, the query text is embedded for you server side through a VectorizableTextQuery, so you send words, not vectors.
from azure.identity import DefaultAzureCredential
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery
client = SearchClient(
endpoint="https://my-search.search.windows.net",
index_name="kb-docs",
credential=DefaultAzureCredential(),
)
vector_q = VectorizableTextQuery(
text="what is the refund window", # service embeds this text
k_nearest_neighbors=50,
fields="content_vector",
)
results = client.search(
search_text="refund window", # BM25 keyword leg
vector_queries=[vector_q], # vector leg
query_type="semantic", # reranker on
semantic_configuration_name="default",
top=5,
)
for r in results:
print(round(r["@search.reranker_score"], 2), r["title"])Expected output: up to five lines, each a reranker score on the 0 to 4 scale next to a document title, ordered highest score first. The top line is the passage the model should ground on.
Failure mode: a KeyError or None on reranker_score means semantic ranking did not run, so check the tier and the semantic plan. An error on VectorizableTextQuery means the index has no vectorizer defined, so either add one or pass a precomputed vector instead. A 403 means the identity lacks the Search Index Data Reader role on the service.
Agentic retrieval for the hard questions
Single-query retrieval, even hybrid with a reranker, breaks on a question that secretly contains three. Ask which plan changed the most between two releases and also whether it affects enterprise customers, and one search string cannot serve all of it. Agentic retrieval is Azure AI Search answering that class of question by decomposing it. A newer Knowledge Base object, exposed through a preview API version, takes the full question, splits it into focused sub-queries, runs each as its own hybrid search with a semantic rerank over the top candidates, and merges the results into one grounded set for the model.
It is genuinely better on multi-part and comparison questions, and it costs more, because you are now paying for several searches and rerank passes per user turn instead of one. I treat it as the third thing to try, not the first. Get hybrid plus semantic solid and measured, and only reach for agentic retrieval when your evaluation shows single queries failing on real multi-part questions. Because it rides a preview surface, confirm the current API version and object names before you build on it, since preview contracts move [VERIFY preview API version and Knowledge Base naming].
Sizing the index and its cost
The vector field carries most of the storage weight, and its size is set by the embedding model you chose. Each vector holds one float per dimension, so a model with more dimensions makes a bigger index and a heavier search. The two common Azure OpenAI text embedding models sit far apart here, and the larger one is not automatically the right call. More dimensions can mean better recall on subtle distinctions, but it also multiplies the bytes you store and move on every query.
| Embedding model | Dimensions | Relative index size | Reach for it when |
|---|---|---|---|
| text-embedding-3-small | 1536 | Baseline | Default, cost-sensitive corpora |
| text-embedding-3-large | 3072 | About twice the vector bytes | Recall matters more than storage |
One hard rule: the dimensions on the index field must match the model exactly. Configure a field for 3072 and feed it vectors from the 1536 model and the index rejects the write. Pick the model first, set the field to its dimension count, and do not mix models in one field. If you later switch models, you reindex; there is no in-place resize of a vector field.
Start hybrid, add the semantic ranker, measure before agentic
If you build one thing from this part, build an index with integrated vectorization on text-embedding-3-small, query it hybrid, and turn the semantic ranker on for the user-facing path. That combination handles the large majority of grounding work, and every piece of it is a setting, not a pipeline you maintain. Keep the ranker off for batch and internal retrieval so the meter only runs where precision pays. Leave agentic retrieval for later, gated behind an evaluation that shows single queries actually failing on multi-part questions, and confirm its preview contract before you commit code to it.
The retrieval decisions here mirror the ones on other clouds; if you also run AWS, the same pattern is managed for you in Amazon Bedrock Knowledge Bases, and the vendor-neutral view lives in the core series. Next in this series we hand this retrieval to something that can act on it: the Azure AI Agent Service, where the model calls search as a tool inside a loop rather than in a single hop. Before then, index a folder of your own documents today, run one hybrid query with the ranker on, and read the top reranker score. If it is not pointing at the passage you would have picked, fix chunking before you touch anything else.
References
- RAG and generative AI in Azure AI Search
- Integrated vectorization in Azure AI Search
- Hybrid search scoring with RRF
- Semantic ranking in Azure AI Search
About The Author
DrJha