A support bot at one of my customers answered a billing question by quoting a policy document. The policy had been rewritten two quarters earlier. The bot was right about the retrieval and wrong about the world, because the vector store still held the old text and nobody had told it to forget. That is the whole problem this part is about. A large language model on its own has no memory of your data, so retrieval augmented generation, RAG, bolts an external search step in front of the model and feeds it the passages it needs. On watsonx, that search step lives in watsonx.data, and the piece doing the searching is a vector database called Milvus.
What watsonx.data Milvus actually is
Start with the words. An embedding is a list of numbers, a vector, that a small model produces from a chunk of text so that passages with similar meaning land close together in that number space. A vector database stores those vectors and answers one question fast: given this query vector, which stored vectors are nearest? Milvus is the open-source vector database that IBM embeds inside watsonx.data, the lakehouse layer of the watsonx stack. RAG is the pattern that stitches these together. You embed your documents once, store the vectors, and at question time you embed the user query, find the nearest chunks, and paste them into the prompt so the model answers from real text instead of guessing.
You could run a standalone Milvus and wire it to watsonx.ai, and plenty of people do. The reason to use the one inside watsonx.data is not raw speed. It is that the vectors sit under the same access controls, network isolation, and audit trail as the rest of your governed data. Retrieval becomes a governed operation rather than a side channel that quietly reads whatever it was pointed at. On a regulated account that difference is the reason the project ships at all. IBM positions the integrated vector database exactly this way, as a place to unify and prepare data for AI without moving it out of the lakehouse.
Milvus here is not a stripped-down copy. It does vector similarity search across millions of vectors in seconds, and it does hybrid search, which combines the vector match with ordinary filtering on scalar fields like a tenant id, a document date, or a category. Hold on to hybrid search. It is the feature that later saves you from the stale-document problem in the opening story.
How a query becomes a grounded answer
There are two flows, and mixing them up is the most common design mistake I see. Ingest runs offline, on a schedule or a trigger, and it is where the cost and the correctness live. Query runs online, per request, and it needs to be cheap and fast. Draw them separately or you will end up re-embedding your whole corpus on every user question, which is a real thing I have watched a team do for three weeks before anyone checked the bill.
flowchart TD
subgraph Ingest [Ingest offline]
A[Source docs] --> B[Chunk text]
B --> C[Embed with watsonx.ai]
C --> D[Upsert vectors to Milvus]
end
subgraph Query [Query online]
E[User question] --> F[Embed query]
F --> G[Search Milvus top k]
G --> H[Build prompt with chunks]
H --> I[Granite model answers]
end
D -.stored vectors.-> G
Chunking is the step people skip and regret. If you embed a whole 40-page PDF as one vector, every query matches it weakly and nothing precisely. Split documents into passages of a few hundred tokens with a little overlap so a sentence that straddles a boundary is not lost. Store the source id, a version stamp, and the page as scalar fields next to each vector. Those fields are free to add and they are what make hybrid filtering possible later.
One more decision hides in the query flow: how many chunks to retrieve, the top k. Too few and the model misses the passage that held the answer. Too many and you flood the prompt with weakly related text, pay for the extra tokens, and give the model more room to pick the wrong sentence. I start at k of 5 for a focused knowledge base and tune from there against real questions. When precision matters more than the raw vector order, add a reranker, a second small model that reorders the retrieved chunks by relevance before they reach the prompt. watsonx.ai hosts reranker models for exactly this, and the pattern is retrieve wide with the vector search, then rerank narrow. It costs a little latency and usually buys back more answer quality than a bigger embedding model would.
Which embedding model to pick
watsonx.ai hosts the embedding models you call during ingest and query. The default is slate-30m, a small English retrieval model that returns a 384-dimension vector and is fast enough to embed large batches cheaply. slate-125m is the bigger sibling: two to three times slower, slightly better recall. The Granite embedding models, granite-embedding-107m-multilingual and granite-embedding-278m-multilingual, are the choice when your documents are not all English. Here are the ones I reach for first.
| Model id | Dimensions | Reach for it when |
|---|---|---|
| ibm/slate-30m-english-rtrvr-v2 | 384 | English corpus, latency and cost matter, this is the default |
| ibm/slate-125m-english-rtrvr-v2 | 768 | English, recall is the bottleneck and you can pay 2 to 3x latency |
| ibm/granite-embedding-107m-multilingual | 384 | Mixed languages, want the smaller and faster multilingual option |
| ibm/granite-embedding-278m-multilingual | 768 | Mixed languages, recall matters more than throughput |
Sizing the Milvus service
When you add Milvus to watsonx.data you choose a size, and the size is expressed as a recommended vector count. Get this roughly right and forget about it; get it wrong and you either overpay for idle capacity or hit a wall mid-ingest. The tiers below are the current recommendations, all assuming a 384-dimension vector and default segment settings.
| Size | Recommended vectors | Fits |
|---|---|---|
| Lite | 10,000 | A demo or a unit test |
| Starter | 1 million | A single team knowledge base |
| Small | 10 million | A department, most first production loads |
| Medium | 50 million | An enterprise document set |
| Large | 100 million | Multi-tenant or web-scale corpus |
| Custom | up to 3 billion | You know exactly why you are here |
Worked example
You have 800,000 documents that average five chunks each, so 4 million vectors. That is above the Starter recommendation of 1 million and comfortably inside Small at 10 million, with headroom to double the corpus before you touch Medium. Do not jump to Medium out of caution; you would pay for 46 million vectors of empty space. Start at Small, watch the fill rate, and resize when you cross roughly 70 percent. The numbers here map straight onto Figure 2, Small versus Medium.
Index types and the recall latency trade-off
An index is how Milvus avoids comparing your query against every stored vector. Three choices cover almost every case. FLAT does no approximation; it scans everything, returns exact nearest neighbors, and gets slow as the collection grows. IVF_FLAT clusters vectors into buckets and only searches the nearest buckets, trading a little recall for a lot of speed on large sets. HNSW builds a navigable graph and gives high recall at low latency, at the cost of more memory to hold the graph. The picture below is the mental model, not a benchmark.
My default is HNSW with COSINE distance, because slate and Granite embeddings are trained to be compared by cosine similarity and HNSW gives the latency users expect. I move to IVF_FLAT only when the collection is large enough that the HNSW graph strains the node memory, and I keep FLAT for tiny collections and for the offline job that checks how much recall the approximate index is costing me. That last use is worth building; it is the only honest way to know your index is not silently dropping answers.
Hybrid search and why metadata earns its keep
Pure vector search answers one question, which chunks mean the same thing as the query. It cannot answer the questions a real application asks alongside that, like which chunks belong to this customer, which are still current, and which come from a document the user is allowed to see. Those are scalar filters, and hybrid search is Milvus applying them in the same call that does the vector match. You store the fields once at ingest, and every query narrows the candidate set before or during the nearest-neighbor step.
Two filters carry most of the weight. A tenant id keeps one customer from ever seeing another customer’s passages, which on a multi-tenant deployment is a hard security boundary, not a nicety. A validity window, an effective date and a retirement date, lets you drop stale text at query time even before the ingest job has caught up. The billing bot from the opening would have skipped the rewritten policy the moment its retirement date passed, because the filter would have removed it from the candidate set regardless of how well it still matched on meaning.
The cost is discipline at ingest. Every field you want to filter on has to be written next to the vector when the chunk goes in, because you cannot filter on data you never stored. That is the argument for spending time on the ingest schema up front. Decide the scalar fields before the first load, not after you have three million vectors with no tenant id and a migration to run. In the governed watsonx.data setting this pays a second dividend, since the same fields feed the access policies and the audit trail rather than living in a separate lookup nobody reviews.
A minimal ingest and query in Python
Here is the smallest end-to-end path that is still real: embed two documents with slate-30m, create a collection, insert, then embed a query and search. It uses the ibm-watsonx-ai SDK for embeddings and the pymilvus client for the vector store.
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import Embeddings
from pymilvus import MilvusClient
creds = Credentials(url='https://us-south.ml.cloud.ibm.com', api_key='YOUR_KEY')
emb = Embeddings(model_id='ibm/slate-30m-english-rtrvr-v2',
credentials=creds, project_id='YOUR_PROJECT')
docs = ['watsonx.data ships Milvus version 2.5.12.',
'slate-30m returns 384 dimension vectors.']
vectors = emb.embed_documents(texts=docs) # list of 384 dim vectors
# token format is ibmlhapikey plus your api key
client = MilvusClient(uri='https://YOUR-MILVUS-HOST:8443',
token='ibmlhapikey:YOUR_KEY')
client.create_collection(collection_name='kb', dimension=384,
metric_type='COSINE')
client.insert(collection_name='kb',
data=[{'id': i, 'vector': v, 'text': docs[i]}
for i, v in enumerate(vectors)])
q = emb.embed_query(text='which Milvus version ships with watsonx.data?')
hits = client.search(collection_name='kb', data=[q], limit=1,
output_fields=['text'])
print(hits[0][0]['entity']['text'])Expected output, the first document, retrieved because its vector is nearest the query about the Milvus version:
watsonx.data ships Milvus version 2.5.12.The failure mode to expect: dimension mismatch. Create the collection with dimension 384, then later query with a 768-dimension model like slate-125m, and Milvus rejects the search because the query width does not match the collection. The fix is not a cast. You drop the collection and re-embed everything with the new model, which is why the model choice earlier is load-bearing.
Gotcha
This is the stale-document trap from the opening. Nothing keeps Milvus in sync with the source. When a document is edited or deleted, its old vector sits there answering queries until your ingest job overwrites or removes it. Two habits fix it. Store a content hash and a version stamp as scalar fields, and on re-ingest delete by source id before inserting the new chunks. Then use hybrid search to filter out anything past its retirement date at query time. The vector store is a cache, and caches lie unless you invalidate them.
Start on Starter, slate-30m, and HNSW
If you are standing up RAG on watsonx this week, here is where I would put the pins. Use the Milvus that ships inside watsonx.data rather than a standalone one, because on a governed account the colocated access control is the entire point. Embed with slate-30m and only move up to slate-125m or a Granite multilingual model when you have measured recall and found it wanting, not on a hunch. Size at Starter or Small to start and resize on fill rate. Index with HNSW and COSINE, keep a small FLAT collection around to audit recall, and wire a version stamp into every vector on day one so the stale-document problem never gets a chance to embarrass you in front of a customer.
Next in the series, retrieval is only half of trustworthy answers. Part 10 covers Granite Guardian and hallucination detection, the layer that checks whether the model actually used the chunks you retrieved or wandered off. Pull one real document set into a Starter Milvus this week and run the snippet above against it before you read on. If you want the platform context around this, the series guide maps where RAG sits in the wider watsonx stack, and if you are comparing approaches across clouds, see how Amazon Bedrock Knowledge Bases handles the same job with a managed pipeline.
References
- IBM, watsonx.data integrated vector database announcement
- IBM Documentation, Milvus overview in watsonx.data 2.2.x
- IBM Documentation, supported encoder foundation models in watsonx.ai
- IBM watsonx.ai Python SDK, RAG and MilvusVectorStore
About The Author
DrJha