Most RAG pipelines quietly throw away half of every document. A PDF goes in, a text extractor strips it to a wall of characters, and the table that held the actual answer becomes a run-on line with no rows, no columns, and no idea which number belonged to which quarter. The model then hallucinates a value because you handed it mush. Fix the ingestion step and a surprising number of retrieval bugs disappear. On watsonx, the tool that fixes it is Docling, and this part is about using it well.
What Docling actually does
Docling is an open-source toolkit, MIT licensed, that reads a document and produces a single structured representation of it. It parses PDF, DOCX, PPTX, XLSX, HTML, EPUB, images such as PNG and TIFF, and even audio formats, then normalizes all of them into one Python object called a DoclingDocument. That object is a Pydantic data type, which just means a typed, validated structure, and it holds the text, the tables as real rows and columns, the pictures, the document hierarchy of headings and sections, and the location of each element on the page.
Two specialized models do the heavy lifting inside it. Layout analysis, deciding what is a heading versus a paragraph versus a table, runs on a model trained on DocLayNet, IBM’s document layout dataset. Table structure recognition, working out the grid of a table even when the borders are invisible, runs on TableFormer. In December 2025 IBM added a faster layout model called Heron that speeds up PDF parsing without dropping accuracy. You do not call these models directly. You call Docling and it orchestrates them.
The output of a DoclingDocument exports to Markdown, JSON, HTML, or a format called DocTags. Markdown is what most people grab first because it looks clean. That is also the mistake, and it is worth its own section.
DocTags and why Markdown loses information
DocTags is a markup format from IBM Research that describes every element on a page, a chart, a table, a form field, a code block, an equation, a footnote, a caption, together with how they relate to each other and where they sit in the layout. Think of it as a structural transcript of the page rather than a flattened copy of the words. Markdown, by contrast, throws away the coordinates and most of the relationships. A merged table cell becomes ambiguous. A figure caption floats free of its figure. For a human reader that is fine. For a retrieval system that needs to answer which region had the highest churn, it is a slow leak of exactly the context that made the answer findable.
My take
Export to DocTags or JSON when the target is a RAG index, and keep Markdown only for a human preview. I have watched teams debug a retrieval problem for a week when the fix was that their chunker had been fed flattened Markdown and could no longer tell one table row from the next. Chunk on the structure, not on the prose.
How a page becomes a DoclingDocument
The pipeline branches on one early decision: is this document born digital, meaning the text is already embedded and selectable, or is it a scan, meaning the page is really an image of text. Born-digital files skip optical character recognition entirely and parse fast. Scans have to go through OCR, which reads pixels back into characters, and that step is where almost all of your processing time goes. Everything after the branch is shared: layout analysis, table recognition, then assembly into the DoclingDocument, then export.
flowchart LR
A[PDF or DOCX] --> B{Born digital?}
B -->|Yes| D[Layout DocLayNet]
B -->|No| C[OCR pass]
C --> D
D --> E[Tables TableFormer]
E --> F[Assemble DoclingDocument]
F --> G[Export MD JSON DocTags]
G --> H[Chunk to watsonx.data Milvus]
Granite-Docling versus the bigger vision models
There are two distinct jobs here and the industry keeps confusing them. The first is conversion: turn a page into faithful structure. The second is understanding: answer a question about what the page means. Docling with its default models handles conversion. IBM also ships Granite-Docling-258M, a vision-language model, meaning it reads image and text together, that does end-to-end conversion in a single 258M-parameter model under an Apache 2.0 license. At that size it runs on a CPU or a small GPU, and IBM reports its accuracy rivals systems several times larger. It added experimental support for Arabic, Chinese, and Japanese, with a roadmap toward larger variants around 900M parameters.
When the task shifts to understanding, you want a bigger vision model. Granite Vision 3.2 2B was IBM’s first official Granite vision-language model, built specifically for document understanding, and on the standard enterprise benchmarks DocVQA, ChartQA, AI2D, and OCRBench it matches models roughly five times its size such as Llama 3.2 11B and Pixtral 12B. IBM trained it by running Docling over 85 million PDFs and generating 26 million synthetic question-answer pairs, so the toolkit and the model are two ends of the same effort. Granite 4.0 3B Vision, released in 2026, goes further on structured extraction, chart, table, and semantic key-value pair pulling, and ships as a LoRA adapter on top of Granite 4.0 Micro so one deployment serves both text-only and multimodal traffic.
Which extractor for which document?
| Model | Size | Best at | Runs where | When not |
|---|---|---|---|---|
| Docling default | DocLayNet + TableFormer | Fast conversion of born-digital files | Local, CPU | Heavy handwriting or noisy scans |
| Granite-Docling-258M | 258M | End-to-end structure to DocTags, equations, code | Local CPU or small GPU | You need answers, not structure |
| Granite Vision 3.2 2B | 2B | Visual Q&A over charts and infographics | watsonx.ai hosted | Plain conversion (overkill and slower) |
| Granite 4.0 3B Vision | 3.5B + 0.5B LoRA | Chart, table, and key-value extraction | Hugging Face; watsonx hosting | Simple digital PDFs with clean tables |
Read that table as a ladder. Start at the top. Move down only when the row above cannot do the job. Most enterprise corpora are 80 percent born-digital PDFs where the default Docling path is enough, and paying for a 2B or 4B vision model on every page is money set on fire. Save the vision models for the pages that actually need reading: a scanned form, a dense chart, a photographed receipt.
Converting a real PDF and keeping its tables
Here is the smallest useful Docling program. It converts a file, exports both a structural DocTags view and a human Markdown view, and pulls each table out as a dataframe so you can validate the grid before it ever reaches your index. Install with pip install docling.
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert('quarterly-report.pdf')
doc = result.document
# Structure for the index, prose for humans
doctags = doc.export_to_doctags()
markdown = doc.export_to_markdown()
print('tables found:', len(doc.tables))
for i, t in enumerate(doc.tables):
df = t.export_to_dataframe()
print('table', i, 'shape', df.shape)Expected output on a clean four-table report:
tables found: 4
table 0 shape (12, 5)
table 1 shape (8, 3)
table 2 shape (20, 6)
table 3 shape (5, 4)Gotcha
If that same script prints tables found: 0 on a document you know has tables, the file is almost certainly a scan and no OCR engine is active. Docling ships with OCR backends such as EasyOCR, Tesseract, and RapidOCR, but you have to enable OCR in the pipeline options for image-only PDFs. A born-digital file needs no OCR and runs far faster, so check the document type before you blame the code.
Chunking on structure, not character counts
Getting a clean DoclingDocument is half the win. The other half is how you cut it into chunks for the vector index, because a chunk is the smallest thing your retriever can return, and a bad cut destroys good structure just as thoroughly as a bad parse did. The naive approach splits on a fixed character count, say every 1,000 characters, which happily slices a table in two or severs a heading from the paragraph it introduces. You end up retrieving half a table and the model fills in the rest from imagination.
Docling ships chunkers that respect the structure it worked so hard to recover. The HierarchicalChunker walks the document tree and keeps each element, a table, a section, a list, intact as its own chunk, carrying the heading path along as metadata so a retrieved row still knows which section and which document it came from. The HybridChunker layers a tokenizer on top of that: it keeps the structural boundaries but also merges tiny neighbouring chunks and splits any element that overflows your embedding model’s token window, so nothing silently gets truncated at index time. For most watsonx.data indexes the HybridChunker tuned to your embedding model’s context length is the right default, and it is a two-line change from the naive splitter.
The payoff shows up at query time in a way that is easy to measure. Keep the heading path and page provenance on every chunk and your retrieval can filter by section before it ever scores similarity, which cuts false hits on documents with repetitive boilerplate such as contracts or compliance filings. It also means a citation back to the user can name the page and section, not just paste a paragraph, and that traceability is exactly what the evaluation harness from Part 17 needs to score groundedness honestly. Chunk carelessly and even a perfect parse degrades into the same soup you started with.
Sizing an ingestion pipeline
The single number that decides your compute bill is the share of scanned documents, because OCR is the slow path. IBM reports Docling running up to roughly 30 times faster on born-digital parsing than traditional OCR-based methods. Treat that as a ratio, not a promise, and the sizing falls out cleanly. Suppose a corpus of 50,000 documents at an average of 12 pages each, and 30 percent of them are scans.
| Segment | Documents | Pages | Relative cost per page | Time units |
|---|---|---|---|---|
| Born-digital | 35,000 | 420,000 | 1 | 420,000 |
| Scanned + OCR | 15,000 | 180,000 | 30 | 5,400,000 |
| Total | 50,000 | 600,000 | mixed | 5,820,000 |
Worked example
The scanned 30 percent of documents consume 5,400,000 of 5,820,000 time units, about 93 percent of the whole job, while producing under a third of the pages. The lesson: benchmark your OCR throughput on real scans, provision GPUs for that branch, and let the born-digital majority ride on cheap CPU workers. If you can push even a slice of the scans through a cleaner source, the bill drops far more than any model swap would give you.
Where Docling stops and a vision model starts
Docling gives you faithful structure. It does not reason about content. If the question is what does this contract clause mean or which line in this chart is trending down, structure alone will not answer it, and that is the handoff point to a vision model like Granite Vision 3.2 2B on watsonx.ai. A practical split works well: run Docling on the whole corpus to produce clean chunks, index them as you did in Part 9, and route only the pages your retrieval flags as image-heavy or chart-heavy to the vision model at query time. You pay the vision premium on a fraction of pages instead of all of them. This is the same instinct behind Amazon Bedrock Data Automation on the AWS side, and comparing the two managed approaches is worth an afternoon if you run both clouds.
One governance note before the next part. Everything Docling extracts becomes training-adjacent data the moment it lands in an index, so the provenance fields in a DoclingDocument, which page and which coordinates a chunk came from, are not a nicety. They are your audit trail. The next part, on watsonx.governance, is where those trails get formalized into risk records and factsheets, and clean provenance from ingestion is what makes that step painless instead of a reconstruction project. Granite models were themselves partly trained on Docling-processed corpora, and IBM has said DocTags terms are being folded into the Granite tokenizer, so the format you export today is the format the models increasingly expect. For the model lineage behind all this, see Part 3 on the Granite family.
Docling for structure, a vision model only for the hard pages
If you build one thing from this part, make it a Docling front-end on your ingestion, exporting DocTags or JSON rather than Markdown, feeding the same watsonx.data index you already run. That single change recovers the tables and reading order most pipelines lose, and it costs almost nothing on born-digital files. Add a vision model only where a page genuinely needs reading, and size your hardware around the scanned minority because that is where the time and money actually go. Convert everything, understand selectively.
Next, take a document you already tried to index and run the small script above on it. Count the tables it recovers that your old extractor flattened. That number is your case for switching.
References
- IBM, Granite-Docling end-to-end document understanding
- Docling project on GitHub
- IBM Granite 3.2, open-source reasoning and vision
- Granite 4.0 3B Vision model card, Hugging Face
About The Author
DrJha