Ingestion is where most retrieval projects quietly fail. Teams argue for six weeks about which vector store to buy and then spend one afternoon on the code that decides what text ever reaches it, which is the wrong ratio by about an order of magnitude. A perfect embedding model cannot recover a sentence your parser dropped, and no amount of reranking will find a rate limit table that arrived in your index as a vertical column of orphaned numbers.

Where ingestion sits in the assistant

Last part ended with a decision and no code: retrieval rather than fine tuning for the Acme documentation assistant, on the grounds that a docs update should cost minutes and not days. Right now that assistant is still one model call with a hand pasted prompt, reliable under load thanks to Parts 5 and 6 but ignorant of anything Acme has ever written. This part gives it a corpus. It does not yet chunk, embed or retrieve anything; those arrive in Parts 9 through 11.

So you can follow along on a real corpus rather than a toy one, I use the public Sentry documentation repository at github.com/getsentry/sentry-docs. It is genuinely representative: MDX source files with frontmatter and custom components, a develop-docs tree, and enough platform specific structure to be awkward. Alongside it I use a two page internal handbook PDF I generated with reportlab, containing exactly what real internal PDFs contain, which is a running header, a page footer and a table nobody will ever export to CSV. Ticket archives get their own treatment in Part 12.

Two design choices in that diagram carry the whole part. First, every format converges on Markdown rather than raw text, because Markdown keeps heading hierarchy and table structure in a form that both a chunker and a language model already understand. Second, a document that yields almost no text is quarantined, not indexed. Both choices exist because I got them wrong first.

Formats in a real documentation corpus

Before choosing parsers, inventory what you actually have. In four separate engagements the shape was similar enough to be worth writing down: roughly three quarters of files were plain markup from a docs repository, a small tail of PDFs carried disproportionate authority because legal and finance wrote them, and a handful of spreadsheets held facts that appeared nowhere else. Sizes in the table below are from the Acme corpus, and your tail will differ, but the failure column travels well.

Row five deserves a warning. A docx with tracked changes will hand you both the deleted text and its replacement unless you flatten revisions first, so an assistant reading a policy draft can confidently quote a clause that legal removed. I found that one in a compliance corpus and it was the only bug in the project that made me feel ill.

PDF text extraction with pypdf

Start with the cheapest thing that works, which for a machine generated PDF is pypdf. It reads the text layer directly and does no layout analysis, so it is fast and it is honest about being fast. Here it is on page one of the Acme handbook, which contains a heading, a paragraph, a four column rate limit table, a running header and a page footer.

# tested with pypdf 6.13.1, pdfplumber 0.11.9, Python 3.10
from pypdf import PdfReader

reader = PdfReader('handbook.pdf')
text = reader.pages[0].extract_text()
for line in text.splitlines()[:14]:
    print(line)

# ---- output ----
# Acme SaaS Internal Handbook  |  Confidential
# Page 1 of 2   Revised 2026-03-11
# Rate Limits by Plan
# Every API key is subject to a per minute request ceiling. Exceeding it returns HTTP 429 with a
# Retry-After header.
# Plan
# Requests per minute
# Burst
# Overage
# Free
# 60
# 90
# blocked
# Team
# 600

Read that output as a retrieval engineer rather than as a programmer. Every character from the table survived, so a naive test that counts characters or greps for the number 600 passes. And yet the document no longer says that the Team plan allows 600 requests per minute. It says the words Team, 600, 900 and queued happen to appear near each other in a list. When a user later asks what the Team ceiling is, a retrieved chunk containing that fragment gives the model four plausible numbers and no way to bind one to a plan.

Two other things in that output are noise rather than content. A confidentiality banner and a page number appear on every page of the file, which means once you chunk the document, every single chunk carries them. Across a 40 page PDF that is 80 lines of repeated text competing for space in your context window, and worse, it makes every chunk slightly similar to every other chunk in embedding space. Strip running headers and footers before you do anything else. Position is the reliable signal, not content: text consistently within about 50 points of the page top or bottom edge across three or more pages is furniture.

Tables, headers and footers that poison a chunk

pdfplumber solves the table problem properly because it works from the drawn rectangles and rules on the page rather than from the text stream. Same page, same file, and this time the structure survives. I have also timed both parsers over 20 runs on the same page so the trade off is a number and not an opinion.

import time, pdfplumber
from pypdf import PdfReader

t0 = time.perf_counter()
for _ in range(20):
    PdfReader('handbook.pdf').pages[0].extract_text()
print('pypdf   ms/page', round((time.perf_counter() - t0) / 20 * 1000, 1))

t0 = time.perf_counter()
for _ in range(20):
    with pdfplumber.open('handbook.pdf') as pdf:
        page = pdf.pages[0]
        page.extract_text()
        page.extract_tables()
print('plumber ms/page', round((time.perf_counter() - t0) / 20 * 1000, 1))

with pdfplumber.open('handbook.pdf') as pdf:
    page = pdf.pages[0]
    for row in page.extract_tables()[0]:
        print(row)
    print('table bbox', page.find_tables()[0].bbox)

# ---- output ----
# pypdf   ms/page 3.6
# plumber ms/page 20.6
# ['Plan', 'Requests per minute', 'Burst', 'Overage']
# ['Free', '60', '90', 'blocked']
# ['Team', '600', '900', 'queued']
# ['Business', '3000', '4500', 'queued']
# ['Enterprise', 'custom', 'custom', 'negotiated']
# table bbox (165.26, 138.0, 430.02, 228.0)

That bounding box is the whole trick. Run pypdf for prose, ask pdfplumber for table boxes, drop any text that falls inside a box from the prose stream, and re-render each table as a Markdown table appended after the paragraph that introduced it. You end up with one document where the sentence and the table both survive, which is what you wanted from the start. Serialising the table as Markdown rather than as CSV matters too, since models read pipe delimited tables considerably more reliably in my testing, and the header row repeats visually next to each value.

Cost of that decision is easy to overstate. pdfplumber is 5.7 times slower per page, which sounds like a reason to avoid it until you notice that only pages containing ruled tables need it. In the Acme corpus that was 12 percent of pages, so the blended cost lands at 5.6 ms per page rather than 20.6, and a 5,000 page corpus finishes its parse pass in about 28 seconds instead of 103.

I learned the middle bar the slow way. On my first documentation assistant I shipped a pypdf only pipeline because the eval numbers looked acceptable and the deadline did not, and the corpus was 90 percent Markdown so PDFs felt like a rounding error. Support started forwarding wrong answers about entitlement limits. When I finally built a 40 question eval set aimed only at facts that lived in tables, the assistant got 18 of 40 wrong, and every single miss traced to a collapsed table. Adding the table pass took a day and a half and moved that to 34 of 40. Cost of the delay was three weeks of an assistant confidently telling account managers the wrong plan limits, which is worse than an assistant that says it does not know.

HTML and MDX, easier and still lossy

Markup feels like the easy case, and it is, right up to the point where you measure how much of each file is not prose. Sentry style MDX carries YAML frontmatter, import statements and JSX components, none of which a reader ever sees and all of which a naive loader will happily embed. Here is a representative file put through a cleaner of about eight lines.

import re

raw = open('sample.mdx').read()
print('raw chars:', len(raw))

body = re.sub(r'^---n.*?n---n', '', raw, flags=re.S)   # frontmatter
body = re.sub(r'^import .*$', '', body, flags=re.M)              # imports
body = re.sub(r'</?[A-Z][A-Za-z]*[^>]*>', '', body)             # JSX components
body = re.sub(r'n{3,}', 'nn', body).strip()
print('clean chars:', len(body))
print(body)

# ---- output ----
# raw chars: 380
# clean chars: 151
# Ceilings are enforced per key, not per user.
#
# Every API key is subject to a per minute request ceiling.
#
# Use the `backoff` helper shipped with the SDK.

Notice what the cleaner keeps. Text inside a warning component is content and often the most important content on the page, so strip the tag and keep the sentence. A cleaner that removes whole component blocks will delete precisely the caveats your support team needs. Frontmatter is different again: do not discard it, lift it into metadata, because a title and a sidebar position are the cheapest source of the section path you will want in Part 9.

Rendered HTML deserves one blunt rule. If the source repository is public or internally accessible, parse the source and never the rendered site. Scraping your own docs site drags in navigation trees, version switchers, cookie banners and footers, and I have watched a retriever return a sidebar three times in one answer because sidebars are short, repetitive and embed close to almost any query. When you have no choice, select the main content element and nothing else.

Metadata to capture at ingestion time

Whatever you fail to record now is unrecoverable later, because by the time a chunk reaches a retriever it is a float vector and a string. Six fields have earned their place in every pipeline I have built, and one of them is the field teams most often skip.

import hashlib, pathlib
from dataclasses import dataclass, asdict

MIN_CHARS = 200   # tuned on the corpus; below this a doc is almost always broken

@dataclass
class IngestedDoc:
    doc_id: str
    source_uri: str
    parser: str
    content_sha256: str
    modified_at: float
    access_scope: str
    text: str

def ingest(path, text, parser, scope='internal'):
    digest = hashlib.sha256(text.encode('utf-8')).hexdigest()
    return IngestedDoc(
        doc_id=digest[:16],
        source_uri=str(path),
        parser=parser,
        content_sha256=digest,
        modified_at=pathlib.Path(path).stat().st_mtime,
        access_scope=scope,
        text=text,
    )

from pypdf import PdfReader
for name in ('handbook.pdf', 'scan.pdf'):
    body = PdfReader(name).pages[0].extract_text()
    if len(body) < MIN_CHARS:
        print('QUARANTINE', name, 'chars:', len(body))
        continue
    doc = ingest(name, body, 'pypdf-6.13.1')
    print('OK', name, doc.doc_id, 'chars:', len(doc.text))

# ---- output ----
# OK handbook.pdf 8f2c1d4a9b03e771 chars: 508
# QUARANTINE scan.pdf chars: 0

access_scope is the field teams skip, and skipping it is how an internal assistant reads salary bands to a contractor. Resolve permissions at ingestion, store the scope on the record, and filter on it at query time in the vector store; retrofitting access control after a corpus is embedded means re-ingesting everything. Vector databases covers how metadata filters work, and Part 11 compares which stores filter efficiently.

content_sha256 earns its place a month later. Re-embedding a 5,000 document corpus nightly is a waste of money when 30 documents changed, and a content hash makes incremental ingestion a three line comparison. Recording the parser and its version alongside is what lets you answer the question that will eventually arrive in a review: which documents were indexed by the old broken parser and need reprocessing. Structuring this as an importable package rather than a notebook cell pays off quickly, and From Notebook to Python Package covers that shape.

Failure lookup for ingestion errors

Loud failures are the kind ones. Here is the traceback you will meet within your first hundred files, because someone in finance password protected a PDF in 2019 and nobody remembers doing it.

reader = PdfReader('locked.pdf')
print('pages:', len(reader.pages))

# ---- output ----
# Traceback (most recent call last):
#   File <stdin>, line 2, in <module>
#   File pypdf/_reader.py, line 544, in get_object
#     raise FileNotDecryptedError('File has not been decrypted')
# pypdf.errors.FileNotDecryptedError: File has not been decrypted

# ---- fix: check first, decrypt from the environment, never from source ----
import os
reader = PdfReader('locked.pdf')
if reader.is_encrypted:
    if reader.decrypt(os.environ['PDF_PASSWORD']) == 0:
        raise SystemExit('wrong password for locked.pdf')
print('pages:', len(reader.pages))

# ---- output ----
# pages: 2

Keep the table below next to your ingestion module. It is the artifact from this part worth bookmarking, because every row cost me at least half a day to diagnose the first time.

Parse to Markdown before you chunk

My recommendation for this stage is narrow and I would defend it in any review: normalise every format to Markdown, with headings preserved and tables re-rendered as pipe tables, before a chunker ever sees the text. Markdown is the only intermediate representation that a splitter can reason about structurally and a model can read natively, which means one format choice buys you better chunk boundaries in Part 9 and better answers in Part 12. Plain text throws away the hierarchy you need. Rich element objects keep it but tie your pipeline to one library.

On Monday, do one thing. Take ten documents from your corpus, run them through whatever parser you currently use, and read the extracted text yourself, end to end, without skimming. Not the retrieval results, not the eval score, just the text. Ten minutes of that will tell you more about your assistant than a week of prompt tuning, and if any table in those ten documents survived intact I would be surprised. Part 9 takes this normalised text and cuts it into chunks, which is the next place a corpus goes quietly wrong.

References

• pypdf documentation, Extract Text from a PDF, for extraction modes and the explicit statement that pypdf is not OCR software and cannot detect OCR failures.
• pdfplumber on GitHub, for extract_tables, find_tables and the table_settings options that control cell detection.
• unstructured, Partitioning, for the generic partition function and the full list of supported file types.
• getsentry/sentry-docs, the public MDX documentation corpus used as the follow along data set for this part.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts