Our internal support assistant answered general product questions well and got exact error-code lookups wrong about a third of the time. Same Granite model, same prompt, same guardrails from the last part. That gap was retrieval, and moving it onto the platform default made it worse before it made it better.

Where the support assistant stands now

Last part the assistant got guardrails: an input detector that refuses questions it should never answer and an output detector that redacts what it should never return, both wired around one generation call through the Guardrails Orchestrator (Part 26). It is safe now, but it still answers from what Granite learned in pretraining, which means it invents specifics about our products because our docs were never in its training set. This part grounds it: staff questions get answered from the product docs, the support ticket archive and the changelog, all held in a vector store inside the cluster.

RAG, in one clause, retrieves the passages most relevant to a question and hands them to the model as context so the answer reflects your data rather than the model’s memory (the concept is covered in what RAG is). What changes on OpenShift AI is not the idea, it is that every moving piece, the embedder, the index and the model, runs on hardware you control. Nothing leaves the namespace.

Worth saying plainly, because it comes up in every review: we already tuned Granite on company data earlier in this series, so why retrieve at all? Tuning taught the model our tone and our common tasks; it did not teach it today’s changelog or a ticket filed this morning, and retraining every time a document changes is neither fast nor cheap. RAG covers the freshness problem that tuning cannot, and it does it without touching model weights. The two are complements: a tuned model that answers in the right register, grounded on retrieved passages that are current. This part adds the second half.

What RAG on OpenShift AI assembles

You do not glue a RAG stack together by hand here. Instead, the Llama Stack Operator manages a custom resource called a LlamaStackDistribution, and creating one deploys a unified runtime that exposes OpenAI-compatible endpoints for inference, embeddings, vector stores and a Responses API with a file_search tool. Llama Stack is upstream, from the Llama ecosystem; what Red Hat adds is the operator, the hardened images and the integration with OpenShift AI projects. Being blunt about maturity matters here: as of OpenShift AI 2.25 the entire Llama Stack path ships as Technology Preview, so treat it as production-shaped, not production-blessed.

That OpenAI-compatible surface is the quietly valuable part. Because the endpoints speak the same schema as a hosted API, the client code that already talks to a commercial provider mostly works unchanged, which is the same portability argument made in the AI Engineering series. You can drive retrieval two ways: a low-level rag_tool for one-off lookups, or the higher-level Responses API with a file_search tool when you want the model to decide when to retrieve. Both hit the same Milvus store, so the choice is about how much control you want, not about wiring. A LlamaStackDistribution also lives inside a single OpenShift AI project, so its RBAC, quota and network policy are the ones you set up in the platform architecture (Part 13); a second team’s assistant sits in its own namespace and cannot read your vectors.

flowchart LR
  subgraph Ingest
    D[Company docs] --> DL[Docling convert]
    DL --> EM1[Granite embedding]
    EM1 --> V[(Milvus vector store)]
  end
  subgraph Query
    Q[Staff question] --> LS[Llama Stack]
    LS --> EM2[Granite embedding]
    EM2 --> V
    V -->|top passages| LS
    LS --> G[Granite on vLLM, KServe]
    G --> A[Grounded answer]
  end

Two pieces you already have get reused. Granite served through KServe from Part 17 becomes the generation endpoint, pointed at by a VLLM_URL environment variable. And the embedding model, the small network that turns text into a 768-dimension vector, runs inline inside the Llama Stack pod as ibm-granite/granite-embedding-125m-english, so even the embedding step never calls out. Getting the source documents into that pipeline is ordinary data engineering, the same parsing and cleaning discipline as getting data into Python, just aimed at Docling instead of pandas.

Choosing a self hosted vector store

Milvus is the vector database Llama Stack supports on OpenShift AI, and it comes in two shapes. Inline Milvus Lite runs embedded in the Llama Stack pod and stores vectors in a local SQLite file, which is fine for a laptop-scale proof but forgets everything when the pod restarts. Remote Milvus runs as its own standalone deployment backed by etcd and a persistent volume, which is what you want the moment the assistant is real. If you have read vector stores in practice, the trade space is familiar; the constraint here is that the platform picks Milvus for you.

One thing to budget for: remote Milvus is not free to run. Standalone Milvus, its etcd instance and the persistent volume are three more workloads sitting in your project around the clock, and they hold vectors in memory for fast search, so a large corpus wants real RAM. For the support assistant, a single-node standalone Milvus with a few gigabytes of RAM handled roughly a hundred thousand chunks comfortably; past that you are into clustered Milvus and a genuine storage conversation. None of this touches a GPU, which is worth remembering when the next part gets into where the real cost lives.

Standing up remote Milvus and Llama Stack

Deploy Milvus with its etcd sidecar first, then point a LlamaStackDistribution at both the vector store and your Granite endpoint. The Milvus service exposes a gRPC port for client traffic and a separate HTTP port purely for health checks; confusing the two is the single most common way this goes wrong. Ingestion into Milvus usually runs through Docling, a document converter that turns PDFs, HTML and office files into clean Markdown before embedding, wired as a data science pipeline so re-ingestion stays a scheduled job rather than a manual notebook run.

# Tested against OpenShift AI 2.25 self-managed (EUS); Llama Stack Distribution
# 0.2.17 (Technology Preview); Milvus v2.6.0; etcd v3.5.5; embedding model
# ibm-granite/granite-embedding-125m-english (768 dim); Granite 3.3 8B Instruct
# served on KServe from Part 17; OpenShift 4.17+. Run in the model namespace.

# Milvus standalone, connected to etcd and a PVC (abridged to the parts that bite)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: milvus-standalone
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: milvus
          image: milvusdb/milvus:v2.6.0
          args: ["milvus", "run", "standalone"]
          ports:
            - containerPort: 19530   # gRPC, client traffic
            - containerPort: 9091    # HTTP, health only
          livenessProbe:
            exec:
              command: ["curl", "-f", "http://localhost:9091/healthz"]

Apply it, then confirm the container is actually listening for clients:

$ oc apply -f milvus-standalone.yaml
deployment.apps/milvus-standalone created
service/milvus-service created

$ oc logs deploy/milvus-standalone | grep -i listening
[INFO] Proxy successfully started
Listening on 0.0.0.0:19530 (gRPC)

Now the distribution. It reads the Granite endpoint from a secret so no token is ever written into the custom resource, and it names the Milvus gRPC endpoint. Note the port on MILVUS_ENDPOINT.

# Secrets first. VLLM_URL points at the Granite predictor from Part 17.
# Never hardcode the token; it is read from the mounted secret at runtime.
$ oc create secret generic llama-stack-inference-model-secret 
    --from-literal=INFERENCE_MODEL="granite-33-8b-instruct" 
    --from-literal=VLLM_URL="https://granite-33-8b-instruct-predictor:8443/v1" 
    --from-literal=VLLM_API_TOKEN="$GRANITE_SA_TOKEN"

$ oc create secret generic milvus-secret 
    --from-literal=MILVUS_ENDPOINT="tcp://milvus-service:19530" 
    --from-literal=MILVUS_TOKEN="$MILVUS_ROOT_TOKEN"

# LlamaStackDistribution, remote Milvus
apiVersion: llamastack.io/v1alpha1
kind: LlamaStackDistribution
metadata:
  name: lsd-support-assistant
spec:
  replicas: 1
  server:
    distribution:
      name: rh-dev
    containerSpec:
      port: 8321
      env:
        - name: VLLM_URL
          valueFrom:
            secretKeyRef: { name: llama-stack-inference-model-secret, key: VLLM_URL }
        - name: MILVUS_ENDPOINT
          valueFrom:
            secretKeyRef: { name: milvus-secret, key: MILVUS_ENDPOINT }

Here is the failure I hit, and you will too if you copy the health port out of habit. Set MILVUS_ENDPOINT to port 9091 and the pod starts clean, then the first vector operation dies:

$ oc logs deploy/lsd-support-assistant | tail -3
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses; last error:
              connection refused: milvus-service:9091"

# Fix: 9091 is the health endpoint. Client traffic is gRPC on 19530.
# MILVUS_ENDPOINT=tcp://milvus-service:19530

Ingesting company docs and running a query

With the distribution up, the rest is a notebook talking to one endpoint on port 8321. List the models to confirm both the Granite generator and the embedder are registered, create a vector store, ingest documents and query. For a corpus you can actually follow along with, the official examples ingest a public web page; swap that URL for your own product docs once it works.

from llama_stack_client import RAGDocument, LlamaStackClient

client = LlamaStackClient(base_url="http://lsd-support-assistant-service:8321/")

models = client.models.list()
embed = next(m for m in models if m.model_type == "embedding")
print(embed.identifier, embed.metadata["embedding_dimension"])
# ibm-granite/granite-embedding-125m-english 768.0

store = client.vector_stores.create(
    name="support_kb",
    embedding_model=embed.identifier,
    embedding_dimension=int(embed.metadata["embedding_dimension"]),
    provider_id="milvus-remote",   # "milvus" for inline Lite
)
print("vector store:", store.id)
# vector store: vs_9f2a1c7b

Ingest a document, then ask a question. That rag_tool.insert call chunks the text for you at the size you pass; rag_tool.query does the retrieval and hands passages to Granite. Chunk size is a real knob, not a formality: 100 tokens keeps a chunk tight enough that a retrieved passage is mostly signal, while very large chunks pull in surrounding noise that dilutes the match and burns context window on the generation call. Start near 100 to 200 tokens for support docs and only grow it if answers are getting cut off mid-procedure.

doc = RAGDocument(
    document_id="changelog_4_2",
    content="https://raw.githubusercontent.com/opendatahub-io/rag/main/README.md",
    mime_type="text/html",
    metadata={"source": "changelog"},
)
client.tool_runtime.rag_tool.insert(
    documents=[doc], vector_db_id=store.id, chunk_size_in_tokens=100,
)
# 34 chunks inserted

result = client.tool_runtime.rag_tool.query(
    vector_db_ids=[store.id],
    content="How do I roll back after a failed upgrade?",
)
print(result.content[0].text[:120])
# Retrieved 4 passages from support_kb reflecting the rollback procedure...

For the actual assistant you want the conversational form rather than a one-shot lookup, so build an Agent with a builtin::rag/knowledge_search tool bound to the store, create a session and stream turns; the retrieval happens automatically inside each turn and the model keeps context across the conversation. That is the surface the support team talks to, and it sits on the same served Granite model, so everything you learned about serving models for real-time requests still applies to the generation side. Retrieval is now just another dependency in front of it.

A dimension mismatch is the honest failure in this step. Create the store at 768, then later point embedding_model at a smaller model that emits 384-dimension vectors, and every insert is rejected because the collection schema is fixed at creation:

pymilvus.exceptions.MilvusException: (code=1100, message=the dim (384) of
    field data(vector) is not equal to schema dim (768): invalid parameter)

# Fix: the store's dimension is set when you create it and cannot be changed.
# Either keep the 768-dim Granite embedder, or drop and recreate the store
# with embedding_dimension matching the new model.

Where Milvus retrieval quietly loses recall

Here is the part the docs mention once and everyone skims past. Milvus under Llama Stack currently provides vector search only. Keyword and hybrid search are not supported. That flatly contradicts the retrieval advice that holds everywhere else, including hybrid search and reranking, where blending dense vectors with keyword scoring is the default because pure vector search misses exact tokens.

I learned this the expensive way. Our assistant’s retrieval originally ran on Qdrant with hybrid search, and it was good. Moving it onto Llama Stack plus Milvus so the whole stack sat inside OpenShift AI looked like a clean win, until a support engineer asked what error E1442 meant and got a confident, generic non-answer. Vector search encodes E1442 into a dense vector that sits near other error strings, so the exact token no longer gets an exact-match boost. Recall on error-code questions fell from about 0.89 to 0.61 over one afternoon of testing, while recall on prose questions barely moved. A full day went to blaming the embedding model before I reread the one line that said hybrid search is unsupported. What saved the next migration was reading recall per query type rather than as one average number: a single blended score of 0.8 hid a healthy 0.9 on prose and a broken 0.61 on codes, and only slicing the eval set the way you would slice any evaluation exposed it. If you keep one habit from this part, make it that slice.

Grounding recommendation for a self hosted assistant

Use inline Milvus Lite for the first hour and remote Milvus for everything after, keep the Granite 768-dimension embedder so you never fight a dimension mismatch, and point MILVUS_ENDPOINT at gRPC 19530, never the health port. One thing to do before you trust this in front of users: run your existing eval set and read recall separately for prose questions and for exact-token questions. If the exact-token number sags, the platform is behaving as designed and you need the keyword prefilter, not a new embedding model. This grounds the assistant on private data end to end; next part turns to what all this GPU time actually costs and how to bring the unit cost down. Retrieval quality is also what the cost math trades against, so carry these numbers forward.

References

• Red Hat OpenShift AI 2.25, Working with Llama Stack and RAG
• opendatahub-io/rag, the RAG Stack demos and benchmarks
• Red Hat Developer, Demystify RAG with OpenShift AI

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts