Here is chunk five of a nine chunk split of the Acme rate limits page, printed in full, nothing removed: | Enterprise | custom | custom | negotiated |. Forty five characters. No header row, so nothing says the first column is a plan name or that the second is requests per minute. That chunk went into an index, scored well against a query about enterprise limits, and gave a model four words to reason from. Chunking is the step where a corpus that survived parsing gets quietly destroyed, and unlike a parsing bug it leaves no traceback behind.

Where the assistant stands, and what a chunk boundary decides

Last part the assistant gained a corpus. Sentry style MDX and an internal handbook PDF now arrive as clean Markdown with headings intact, tables re-rendered as pipe tables, and six metadata fields attached to each document. Nothing is searchable yet. This part cuts those documents into retrievable units, and Part 10 turns each unit into a vector.

A chunk is simply the unit you embed and the unit you hand back to the model, and those two jobs pull in opposite directions. Retrieval wants small chunks, because one embedding vector averages everything inside it and a chunk covering four topics sits in the middle of all four, close to nothing in particular. Generation wants large chunks, because a model answering from three sentences with no surrounding context will hedge or invent. Every chunking decision you make is a position taken on that tension, and there is no setting that dissolves it.

Two branches in that diagram are the entire argument of this part. Structural boundaries come first and size comes second, rather than the other way round, which is how nearly every quickstart does it. And a block that would be ruined by cutting is exempt from the size cap rather than subject to it.

Recursive character splitting on a real docs page

Start with the default everyone starts with. Recursive character splitting walks an ordered list of separators, paragraph break first, then line break, then space, then bare character, and uses the coarsest one that produces chunks under the cap. Reasonable behaviour, and it is the right first thing to try. Below it runs on a 1,845 character rate limits page from the Acme docs, the sort of page that carries a heading tree, prose, and one table that matters more than all the prose around it.

# tested with langchain-text-splitters 1.1.2, Python 3.10.12
from langchain_text_splitters import RecursiveCharacterTextSplitter

text = open('rate_limits.md').read()
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=0)
chunks = splitter.split_text(text)
print('chunks:', len(chunks))
for i in (4, 5):
    print('--- chunk', i, 'len', len(chunks[i]))
    print(chunks[i])

# ---- output ----
# chunks: 16
# --- chunk 4 len 169
# | Plan | Requests per minute | Burst | Overage |
# | --- | --- | --- | --- |
# | Free | 60 | 90 | blocked |
# | Team | 600 | 900 | queued |
# | Business | 3000 | 4500 | queued |
# --- chunk 5 len 45
# | Enterprise | custom | custom | negotiated |

Nothing errored. No warning appeared. Sixteen chunks came back and fifteen of them are fine, which is exactly why this bug survives to production: your smoke test passes, your character counts reconcile, and one chunk in sixteen is a fact stripped of its meaning. Worse, that orphan is short, and short chunks are dense in embedding space. A query about enterprise pricing will match it strongly, because almost nothing in it is diluting the signal.

Chunk size, overlap and what each one actually costs

Two knobs exist and both are routinely set by superstition. Sweep them instead. Below, chunk_size runs from 200 to 1,200 characters, each with no overlap and with overlap at one eighth of chunk size, and the last column reports how many total characters get embedded relative to the original document. That ratio is your embedding bill and your storage footprint, in one number.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = open('rate_limits.md').read()
print('doc chars', len(text))
for size in (200, 300, 500, 800, 1200):
    for ovl in (0, size // 8):
        cs = RecursiveCharacterTextSplitter(
            chunk_size=size, chunk_overlap=ovl).split_text(text)
        L = [len(c) for c in cs]
        print(f'{size:5}{ovl:5}{len(cs):6}{sum(L)//len(L):6}{max(L):6}'
              f'{sum(L)/len(text):8.2f}x')

# ---- output ----
# doc chars 1845
#  size  ovl chunks  mean   max inflation
#   200    0     16   112   197    0.97x
#   200   25     16   124   197    1.07x
#   300    0      9   203   282    0.99x
#   300   37      9   216   297    1.06x
#   500    0      5   367   474    1.00x
#   500   62      5   377   474    1.02x
#   800    0      3   613   725    1.00x
#   800  100      3   624   725    1.01x
#  1200    0      2   921  1131    1.00x
#  1200  150      2   921  1131    1.00x

Read the last two rows carefully, because they show something the parameter documentation will not tell you: overlap silently does nothing when a document splits on paragraph boundaries that already fall well under the cap. Requesting 150 characters of overlap and receiving 1.00x inflation is not a bug, it is the splitter deciding it never needed to slice inside a paragraph. Teams see a configured overlap value in a YAML file and assume protection they are not getting.

Since a sweep is only useful if you know how to read it against your own content, here is the sizing table I actually work from. Character counts assume English prose; multiply by roughly 0.25 for a rough token figure, and measure properly with a tokeniser before you commit to a budget, which Part 5 covers.

Header aware splitting and atomic tables

Fixing the orphan takes two changes, and both come from treating Markdown as structure rather than as a long string. Split on headings first so every chunk inherits a section path, then protect table blocks from the size cap entirely. Around 25 lines of code, and it is the version I ship.

import re
from langchain_text_splitters import (
    RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter)

TABLE = re.compile(r'(?:^|.*|s*$n?)+', re.M)

def split_keeping_tables(text, splitter):
    out, last = [], 0
    for m in TABLE.finditer(text):
        prose = text[last:m.start()]
        if prose.strip():
            out.extend(splitter.split_text(prose))
        out.append(m.group().strip())          # table stays whole
        last = m.end()
    if text[last:].strip():
        out.extend(splitter.split_text(text[last:]))
    return out

text = open('rate_limits.md').read()
headers = MarkdownHeaderTextSplitter(
    [('#', 'h1'), ('##', 'h2')], strip_headers=False)
prose = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=25)

chunks = []
for section in headers.split_text(text):
    path = ' > '.join(section.metadata.values())
    for c in split_keeping_tables(section.page_content, prose):
        chunks.append((path + 'n' + c) if path else c)

print('chunks:', len(chunks), 'max len:', max(len(c) for c in chunks))
print([c for c in chunks if '| Team |' in c][0])

# ---- output ----
# chunks: 17 max len: 246
# Rate Limits > Ceilings by plan
# | Plan | Requests per minute | Burst | Overage |
# | --- | --- | --- | --- |
# | Free | 60 | 90 | blocked |
# | Team | 600 | 900 | queued |
# | Business | 3000 | 4500 | queued |
# | Enterprise | custom | custom | negotiated |

Note strip_headers=False, which keeps the heading line inside the section body as well as in metadata. Default behaviour removes it, on the reasonable theory that you already have it in metadata, and the result is a chunk whose embedded text never mentions the topic it is about. Metadata is not embedded unless you put it there yourself. Prepending the path costs about 30 characters per chunk and is the highest return per byte of anything in this part.

I learned the orphan lesson expensively. On a hardware documentation assistant I shipped fixed 256 token chunks with 32 tokens of overlap, because that is what the tutorial I copied used, and the corpus was mostly prose so it looked fine in review. Six weeks later a field engineer specified the wrong power supply for an install, having asked the assistant for the rack unit draw and received a chunk of orphaned numbers. When I finally built a 60 question eval set restricted to specification tables, the assistant scored 22 out of 60. Making tables atomic and prepending heading paths took two days and moved that to 51 out of 60. Nothing else in the pipeline changed, no reranker, no better embedding model, no prompt work.

Semantic and contextual chunking, and when they pay for themselves

Two fancier approaches deserve an honest hearing. Semantic chunking embeds each sentence, measures similarity between neighbours, and cuts where the topic shifts. Elegant, and on documentation with clean heading structure it mostly reproduces boundaries the headings already gave you for free, at the cost of one embedding call per sentence. Reach for it on transcripts, meeting notes and long unstructured reports, which have topic drift and no headings. Skip it on a docs corpus.

Contextual chunking is the more interesting one, and the numbers behind it are public. Anthropic reported that generating a short context sentence for each chunk with a cheap model, then embedding chunk plus context together, cut top 20 retrieval failure rate by 35 percent, from 5.7 to 3.7 percent, and by 49 percent when combined with contextual BM25. Prepending a heading path, which the code above does for nothing, is a crude version of the same idea. Full contextual chunking costs one small model call per chunk at ingestion, which for a 50,000 chunk corpus is a real but bounded one time bill, and prompt caching makes it cheaper still since the surrounding document is repeated across every call from that document.

My sequencing is firm on this. Get heading paths and atomic blocks working first, since they are free and they fix the failures that actually hurt. Add contextual chunking when your eval set shows retrieval failing on chunks that are individually well formed but ambiguous out of context, such as a paragraph that says this limit applies per key without ever naming the limit. Do not start there.

Failure lookup for chunking bugs

One chunking error is loud, and you will hit it the first time you try to make overlap generous on small chunks.

splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=400)

# ---- output ----
# Traceback (most recent call last):
#   File <stdin>, line 1, in <module>
#   File langchain_text_splitters/character.py, line 103, in __init__
#     super().__init__(keep_separator=keep_separator, **kwargs)
#   File langchain_text_splitters/base.py, line 84, in __init__
#     raise ValueError(msg)
# ValueError: Got a larger chunk overlap (400) than chunk size (200),
# should be smaller.

# ---- fix: express overlap as a fraction, and validate at config load ----
def make_splitter(size, overlap_frac=0.1):
    assert 0 <= overlap_frac < 0.5, 'overlap fraction must be under 0.5'
    return RecursiveCharacterTextSplitter(
        chunk_size=size, chunk_overlap=int(size * overlap_frac))

print(make_splitter(200)._chunk_overlap, make_splitter(800)._chunk_overlap)

# ---- output ----
# 20 80

Every other chunking bug is silent, which is what makes the table below worth keeping open beside your splitter module. Each row is a symptom you will observe in retrieval results rather than in logs.

Structure first, then size, for a documentation corpus

My recommendation for the assistant, and for most documentation corpora, is narrow enough to argue about: split on h2 and h3 first, prepend the heading path to every chunk, exempt tables and fenced code from the size cap, and set chunk size at 300 to 500 characters with zero overlap. Overlap is the parameter I would drop entirely on structured content, since heading paths do the continuity job better and cost less. Semantic chunking is the option I would avoid on a docs corpus, because you pay an embedding call per sentence to rediscover boundaries the author already wrote down as headings.

On Monday, do one thing. Dump 50 random chunks from your current index to a text file and read them cold, without the source document open. Any chunk you cannot understand on its own is a chunk your retriever cannot use, and my experience is that between one in ten and one in five will fail that test on a first pass. Fix boundaries before you touch embedding models or rerankers. Part 10 takes these chunks and turns them into vectors, where the choice of embedding model sets both your recall ceiling and a cost you will live with for a long time.

References

• LangChain, splitting recursively, for the separator order, chunk_size and chunk_overlap semantics used throughout this part. Code here was run against langchain-text-splitters 1.1.2 on Python 3.10.12.
• Chroma, Evaluating Chunking Strategies for Retrieval, for the recall, precision and IoU comparison across strategies, and the finding that small recursive chunks perform consistently well.
• Anthropic, Contextual Retrieval, for the 35 percent and 49 percent reductions in top 20 retrieval failure rate cited above.
• getsentry/sentry-docs, the public documentation corpus this series follows along with.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts