TL;DR
- Grounding data is the private content you feed a model at query time so it answers from your facts instead of its training. Preparing it well is most of the work in a RAG system, and it happens before any prompt runs.
- Four stages turn a folder of documents into a searchable index: pull the source, crack it into plain text, split that text into chunks, and turn each chunk into a vector. On Azure, an indexer plus a skillset runs all four for you.
- Chunk size and overlap decide retrieval quality far more than the embedding model does. A 512 token chunk with about 128 tokens of overlap is a sane starting point, then you measure.
- Embedding cost barely moves with chunk size. Index size and retrieval precision move a lot. So tune chunking for answer quality, not to shave a few dollars off the embedding bill.
Most bad RAG answers are not the model failing. They are baked in at ingestion, the moment a paragraph gets sliced down the middle and the half that held the answer lands in a different chunk that retrieval never returns alongside the question. The model then writes a confident, wrong reply from the fragment it did get. You can swap in a smarter model, raise the temperature, rewrite the system prompt, and none of it helps, because the answer was never in the context window to begin with. The fix is upstream, in how you prepared the data.
What grounding data actually is
Grounding data is the content you put in front of a model at answer time so its reply comes from your material and not from whatever it absorbed during training. A retrieval augmented generation system, RAG for short, works in two beats: retrieve the handful of passages most relevant to a question, then hand those passages to the model as context and ask it to answer using them. The model half of that gets all the attention. The retrieval half is where projects actually succeed or fail, and retrieval can only return what you indexed, in the shape you indexed it.
So the real job is turning a pile of source material, PDFs, wiki pages, support tickets, contracts, into an index that returns the right passage for a question. That job has a fixed shape. You start from a data source, the place your documents live, such as Azure Blob Storage or a database. You crack each document, which means pulling the readable text out of a format that was built for humans, not machines. You chunk that text into passages small enough to retrieve precisely. You embed each chunk, turning it into a vector, a list of numbers that captures its meaning so similar passages sit near each other in vector space. Then you write the chunks and their vectors into a search index. Every term in that sentence is a knob you will tune, and the two that matter most are the ones people skip: how you chunk, and what you embed with.
How chunking decides your answers
A chunk is the unit retrieval actually returns. Ask for the top five results and you get five chunks, and the model sees only those. If a chunk is too big, it carries several ideas and the vector that represents it is a muddy average of all of them, so it matches weakly and, when it does match, it floods the context window with text that is off topic. If a chunk is too small, it is precise but it loses the surrounding sentences that made it mean something, and the answer that needed two adjacent paragraphs never arrives together. Chunking is the act of choosing that unit, and it is the single decision that most shapes whether retrieval works.
Azure AI Search gives you three built in ways to chunk, and they climb in how much structure they respect. The Text Split skill cuts by a fixed length, either pages or sentences, which is fast, free, and blind to layout, so it will happily cut through the middle of a table. The Document Layout skill reads the structure of a document first, its headings, paragraphs, and tables, and splits along those boundaries so a chunk lines up with a real section rather than an arbitrary character count. The Azure Content Understanding skill goes further into multimodal material, handling images and complex documents. For most text corpora the Text Split skill is where you begin, and you move up only when layout aware splitting measurably helps.
| Chunking skill | How it splits | Reach for it when | Cost |
|---|---|---|---|
| Text Split | Fixed length, pages or sentences | Plain text, a sane default | Free |
| Document Layout | Along headings, paragraphs, tables | Structured docs, contracts, manuals | Billed per document |
| Content Understanding | Semantic, layout and image aware | Multimodal, images and mixed media | Billed per document |
My take
Do not reach for the fanciest chunker first. I start almost every corpus on the Text Split skill with a 512 token size, get the whole pipeline working end to end, and only then look at whether bad answers trace back to bad splits. Nine times out of ten the fixed splitter is fine and the Document Layout skill is a per document bill I did not need to pay. The exception is tables and forms, where fixed splitting genuinely mangles the content, and there the layout aware skill earns its cost. Prove you have the problem before you pay to solve it.
Which embedding model to choose
An embedding is the vector a model produces from a chunk, and the embedding model is what decides how well two passages that mean the same thing end up near each other. On Azure OpenAI you have three in common use. The oldest, text-embedding-ada-002, produces 1536 dimensions and is fine but superseded. The current pair is text-embedding-3-small at 1536 dimensions and text-embedding-3-large at 3072 dimensions, and the large model retrieves noticeably better on hard queries at a higher price and a bigger index. There is a lever people miss: the 3 series supports asking for fewer dimensions than the maximum through a dimensions parameter, so you can take text-embedding-3-large down from 3072 to, say, 1024 and keep most of the quality while shrinking every stored vector. That trade, smaller vectors for a small quality give up, is often the right call at scale.
Price is the part that surprises people, because it is small. At the time of writing, text-embedding-3-large costs about 0.13 dollars per million input tokens on Azure OpenAI, and you only pay for input, there is no output token charge on an embedding call. So embedding a large corpus once is cheap, and re embedding it when you change chunking is cheap too. The expensive resource is not the embedding call, it is the index that holds the vectors and the retrieval quality you get from it, which is why I pick the model for recall on my own test queries first and treat the token price as a rounding error. If you have not built a habit of choosing models against your own eval set, Part 3 on choosing a model is the companion to this section. [VERIFY: current text-embedding-3-large Azure price per million tokens and regional availability]
Building the index with integrated vectorization
Integrated vectorization is the Azure AI Search feature that runs chunking and embedding inside the service, so you do not stand up a separate job to call the embedding model and stitch vectors back in. You define a skillset, a chunking skill followed by an embedding skill, attach it to an indexer, and the indexer chunks and vectorizes every document as it ingests. Here is the skillset that does the two stages that matter, split then embed, defined as a REST payload and pushed with the management API. It uses the Text Split skill at 512 tokens with 128 tokens of overlap, then the Azure OpenAI embedding skill against a text-embedding-3-large deployment.
import os, requests
endpoint = os.environ['SEARCH_ENDPOINT'] # https://YOUR-SEARCH.search.windows.net
key = os.environ['SEARCH_ADMIN_KEY']
api = '2024-07-01' # stable REST api-version [VERIFY latest]
skillset = {
'name': 'rag-skillset',
'skills': [
{
'@odata.type': '#Microsoft.Skills.Text.SplitSkill',
'textSplitMode': 'pages',
'maximumPageLength': 512,
'pageOverlapLength': 128,
'unit': 'azureOpenAITokens', # [VERIFY exact unit value]
'inputs': [{'name': 'text', 'source': '/document/content'}],
'outputs': [{'name': 'textItems', 'targetName': 'chunks'}],
},
{
'@odata.type': '#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill',
'resourceUri': 'https://YOUR-AOAI.openai.azure.com',
'deploymentId': 'text-embedding-3-large',
'modelName': 'text-embedding-3-large',
'dimensions': 3072,
'context': '/document/chunks/*',
'inputs': [{'name': 'text', 'source': '/document/chunks/*'}],
'outputs': [{'name': 'embedding', 'targetName': 'vector'}],
},
],
}
r = requests.put(
f'{endpoint}/skillsets/rag-skillset?api-version={api}',
headers={'api-key': key, 'Content-Type': 'application/json'},
json=skillset,
)
print(r.status_code)Expected output: 201 on first create, 204 when you update an existing skillset. Failure mode one: a 400 that names deploymentId, which means the embedding deployment does not exist under that Azure OpenAI resource, so create the text-embedding-3-large deployment first. Failure mode two: the indexer runs green but retrieval returns nothing, because the vector field in your index was defined with a different dimension count than 3072, and the mismatch is silent until you query. Set the index field dimensions to match the model, or set both to the reduced number you chose. [VERIFY: current stable Search REST api-version and the SplitSkill unit field values]
Chunk size and overlap in numbers
The two chunking dials are size and overlap. Size is how long each chunk is; overlap is how many tokens the end of one chunk repeats at the start of the next, so a sentence that straddles a boundary survives in at least one chunk whole. A common starting point is 512 tokens with about 128 tokens, one quarter, of overlap. Smaller chunks retrieve more precisely but fracture context; larger chunks preserve context but blur the vector and pull in noise. What most people get wrong is assuming the choice is a cost decision. Run the numbers and it is not.
Worked example
Take a corpus of 10,000 documents averaging 4,000 tokens each, so 40 million source tokens. Chunk at 256 tokens with 64 overlap and you get about 21 chunks per document, roughly 210,000 chunks, and you embed about 53.8 million tokens, which at 0.13 dollars per million is about 6.99 dollars. Chunk at 512 with 128 overlap and you get about 11 chunks per document, about 110,000 chunks, embedding about 56.3 million tokens for about 7.32 dollars. Chunk at 1024 with 128 overlap and you get about 5 chunks per document, about 50,000 chunks, embedding about 51.2 million tokens for about 6.66 dollars. The embedding bill lands near 7 dollars in every case. The chunk count, and so the size of your vector index and how granular retrieval is, swings by more than four times. Chunk size is a retrieval quality decision wearing a cost costume. [VERIFY: current embedding price; efficiency and per document chunk counts vary with your real content]
| Chunk size | Overlap | Chunks | Embedded tokens | Embedding cost |
|---|---|---|---|---|
| 256 tokens | 64 | about 210,000 | about 53.8M | about 6.99 dollars |
| 512 tokens | 128 | about 110,000 | about 56.3M | about 7.32 dollars |
| 1024 tokens | 128 | about 50,000 | about 51.2M | about 6.66 dollars |
Where grounding pipelines go wrong
The failures cluster. The first is stale data. An index is a snapshot, and if the indexer does not run on a schedule against a change tracked source, your model answers from last month while the source moved on, and nobody notices until an answer is confidently out of date. Set the indexer schedule and use change detection so only new and modified documents get reprocessed. The second is a dimension mismatch, the silent one from the code section: the vector field and the embedding model disagree on how many numbers a vector holds, so writes succeed and searches quietly return nothing useful.
The third is missing metadata. If you index only the chunk text and its vector, you cannot filter a query to one customer, one product, or one document type, and every search runs against the whole corpus whether that makes sense or not. Carry the fields you will filter on, the source path, a title, a date, a tenant id, into the index alongside the vector, because retofitting them later means a full reindex. The fourth is the quietest and the most common: garbage in the source. A chunk drawn from a scanned PDF that cracked into broken text, or from a page that was mostly navigation boilerplate, embeds into a vector that matches nothing sensible and pollutes results. No chunk size rescues bad extracted text. Look at your cracked output before you tune anything downstream, because grounding data quality is upstream of every other dial. AWS builds the same ingestion machinery into its managed knowledge bases, and the trade offs there are the same; I walk through them in the Bedrock data prep part.
Fix the corpus before you touch the model
If a RAG system gives wrong answers, my order of investigation is fixed and it starts nowhere near the model. First I look at the cracked text, because broken extraction beats every downstream fix. Then I look at the chunks, reading a dozen of them to see whether a real answer survives inside one chunk or gets split across two. Then I check retrieval, whether the right chunk is even in the top results for a failing question. Only after all of that would I consider the generation model, and by then the problem is almost always already found. The model is the last suspect, not the first.
So the recommendation is plain. Start on the Text Split skill at 512 tokens with 128 overlap and text-embedding-3-large, get the indexer running on a schedule with change detection, carry your filter fields into the index, and build a small set of real questions with known answers before you tune anything. That eval set is what turns chunk size from a guess into a measurement. This is the plumbing every agent and every chat feature later in the series sits on top of, so it is worth getting boring and right. Pick ten questions your users actually ask, write down the answer each should return, and run them against your index this week. The gap between what comes back and what should is your whole tuning list.
References
- Integrated vectorization overview, Azure AI Search
- Develop a RAG solution, the chunking phase, Azure Architecture Center
- Grounding data design, Azure Well-Architected Framework
About The Author
DrJha