Most watsonx projects that stall do not stall on the model. They stall on the wiring around it. A team picks Granite, gets a good answer in Prompt Lab, and then spends three months arguing about where the vectors live, who signs off on an agent that can move money, and what the auditor will ask for. That argument has known answers, because the same three shapes keep recurring, and this part draws them.
Twenty two parts in, you have seen every piece in isolation: RAG on watsonx.data, agents on Orchestrate, factsheets in governance. A reference architecture is how those pieces sit together for a real workload, with the seams named. I use three: enterprise RAG, agentic, and a regulated overlay that sits on top of either. Here is each one, what it costs, and where it breaks.
TL;DR
Enterprise RAG grounds a Granite model on watsonx.data with an embedded Milvus vector store, a reranker, and a Granite Guardian check on the way out. Agentic adds watsonx Orchestrate as the control plane that routes work across tools and agents, with evaluation nodes in the loop. The regulated overlay is not a fourth pattern; it is watsonx.governance wrapped around either one, capturing an AI factsheet and an EU AI Act risk classification for every asset. Pick RAG first. Add agents only when a single grounded call cannot finish the task. Wire governance from the first commit, because retrofitting an audit trail after launch is the most expensive thing on this list.
Three patterns, one platform
A reference architecture is a reusable blueprint: the components, how requests flow through them, and the decisions already made so you do not remake them per project. On watsonx the same platform services recombine into three shapes, and knowing which shape you are building saves the three month argument. Enterprise RAG answers questions from your own documents. Agentic gets work done across systems, not just answered. The regulated overlay is a governance wrapper you drop over either when a rule says every decision must be explainable and logged.
The mistake I see most is reaching for agents first because they demo well. An agent that plans, calls tools, and loops is harder to test, harder to cost, and harder to defend to a risk officer than a single grounded call. If a well built RAG pipeline answers the question, that is the architecture. Reserve the agentic pattern for work that genuinely needs multiple steps against multiple systems. The table sets the three side by side.
| Dimension | Enterprise RAG | Agentic | Regulated overlay |
|---|---|---|---|
| Job | Answer from your data | Complete multi step work | Prove it was safe |
| Core services | watsonx.data, watsonx.ai | watsonx Orchestrate, watsonx.ai | watsonx.governance |
| Primary risk control | Granite Guardian on output | Policy enforcement and eval nodes | Factsheet and risk tier |
| Hardest part | Retrieval quality | Bounding what agents can do | Evidence completeness |
| Build first when | A grounded answer is enough | One call cannot finish it | A regulator is in scope |
Pattern one, enterprise RAG on watsonx.data
RAG, retrieval augmented generation, means fetching relevant text from your own corpus and handing it to the model as context so the answer is grounded in your data rather than the model memory. On watsonx the store is watsonx.data, the lakehouse whose embedded vector database is built on open source Milvus, so vectorized embeddings live next to the governed tables they came from. IBM added that integrated Milvus store specifically to keep RAG data trusted and in place. The request path has five stops, and Figure 1 traces them.
flowchart LR U[User question] --> E[Embed query on watsonx.ai] E --> V[Milvus vector search in watsonx.data] V --> R[Rerank top passages] R --> G[Granite generates grounded answer] G --> Gd[Granite Guardian output check] Gd --> U
Two design decisions carry this pattern. First, the reranker. Vector search is fast but blunt; a reranker scores the top passages a second time and reorders them, and it is the single cheapest lift for answer quality I know on this stack. Second, the guardrail placement. Granite Guardian, IBM open safety model family, checks the generated answer for hallucination and harm before it reaches the user, so the last thing in the path is a safety gate, not the model. Skip either and quality or safety leaks in ways a demo never shows.
The part that decides RAG quality is upstream of the diagram: how you chunk and embed the corpus before any query arrives. Chunk too large and a passage carries three topics, so retrieval returns something loosely on point and the model pads its answer with the irrelevant two thirds. Chunk too small and a single fact gets split across two vectors, so neither ranks high enough to retrieve. I start around 500 to 800 tokens per chunk with a small overlap and adjust against the eval set, not against a hunch. Part 9 covers the ingestion mechanics; the design point here is that the offline pipeline, not the live path, is where most RAG failures are actually born.
How much latency does each stage cost?
A grounded answer that takes four seconds fails a chat use case no matter how correct it is, so you budget latency per stage before you build. The small script below sums a per stage budget and tells you whether it clears a service level target. It makes no network calls, so it runs as is, and you feed it your measured numbers to sanity check a design. The numbers here are representative for a small Granite model with a reranker.
# rag latency budget, pure python, no api calls
# sums per stage milliseconds and checks a service level target
STAGES = {
'embed_query': 40,
'vector_search': 60,
'rerank': 90,
'generation': 850,
'guardrail': 120,
}
SLA_MS = 1500
def budget(stages=STAGES, sla=SLA_MS):
total = sum(stages.values())
headroom = sla - total
return {
'total_ms': total,
'fits_sla': headroom >= 0,
'headroom_ms': headroom,
'slowest': max(stages, key=stages.get),
}
if __name__ == '__main__':
print(budget())
Expected output:
{'total_ms': 1160, 'fits_sla': True, 'headroom_ms': 340, 'slowest': 'generation'}The failure mode is the one the sum hides. This budget assumes retrieval returns on the first try; add a re retrieval loop, a larger model, or a second guardrail pass and generation plus retries can blow past the 1500 ms target while every individual stage still looks fine. When it fails, the fix is almost always the slowest stage the script names, generation, so cache what you can and keep the model as small as the task allows. Figure 2 plots the same budget so the shape is obvious.
| Stage | Latency | Share | Note |
|---|---|---|---|
| Embed query | 40 ms | 3% | Cheap, leave it alone |
| Vector search | 60 ms | 5% | Milvus in watsonx.data |
| Rerank | 90 ms | 8% | Best quality lift per ms |
| Generation | 850 ms | 73% | Optimise here or nowhere |
| Guardrail | 120 ms | 10% | Granite Guardian, non negotiable |
| Total | 1160 ms | 100% | vs 1500 ms target |
Pattern two, agentic systems through watsonx Orchestrate
An agent is a model given tools and a goal, allowed to plan and take steps rather than answer once. When one grounded call cannot finish the job, because it needs to read a ticket, check inventory, and file an order, you move to the agentic pattern, and on watsonx the coordinator is watsonx Orchestrate. Orchestrate is the agentic control plane: a centralized layer that observes, governs, and routes work across agents no matter where they were built or run. It is the difference between a pile of agents and a system you can operate.
The control plane earns its place through what it enforces around the agents. Pre deployment evaluation measures accuracy, tool call reliability, and completion rate, and only validated agents publish. Policy enforcement bounds which tools and models each agent may touch. Audit and traceability log every decision. Those are not add ons; IBM builds them into the plane, which is exactly what a regulated buyer needs. Figure 2 covered the RAG budget; Figure 3 shows the agentic loop with those controls in place.
One structural choice sets the tone of the whole build: a single agent with many tools, or a supervisor that delegates to narrow sub agents. A single agent is simpler to reason about and cheaper to run, because every step is one model deciding what to do next. A supervisor pattern, where one coordinating agent hands subtasks to specialists, scales to messier work but multiplies model calls and therefore cost and latency, since each hop is another round trip. My default is a single agent until a real task proves it needs specialists, because a supervisor you did not need is just a slower, pricier version of the thing you already had. Add the second layer when the tools stop fitting in one agent head, not before.
flowchart TD
Req[User request] --> Sup[Supervisor agent on Orchestrate]
Sup --> Pol{Policy check}
Pol -->|Allowed| Tools[Call tools and sub agents]
Pol -->|Blocked| Deny[Refuse and log]
Tools --> RAG[Grounded lookup pattern one]
Tools --> Act[Action tool, ticket or order]
RAG --> Ev[Evaluation node]
Act --> Ev
Ev --> Log[Audit trail and factsheet]
Log --> Req
Gotcha
The dangerous edge is the action tool, not the model. A grounded lookup that returns a wrong passage is a quality bug; an action tool that files the wrong order is an incident. So bound tools tightly: give each agent the narrowest set of actions it needs, require a policy check before any state changing call, and keep a human approval step on anything that moves money or touches a customer record. The evaluation node catches bad answers; only the policy gate catches bad actions.
Pattern three, the regulated-industry overlay
In banking, healthcare, insurance, and other regulated sectors, a working answer is not enough; you have to prove the system was safe and show your evidence on demand. That is why I treat governance as an overlay rather than a pattern. watsonx.governance wraps a RAG or agentic build and produces the artifacts a regulator asks for, chiefly the AI factsheet, an automatically generated lifecycle record capturing intended purpose, risk classification, data provenance, validation results, and the approval chain for every model or agent. Risk officers read it for model risk reviews; auditors treat it as primary evidence.
The regulatory clock is real. Under the EU AI Act, obligations for high risk AI systems, including conformity assessment and post market monitoring, apply from 2 August 2026, and watsonx.governance can classify each model against those risk categories and flag the high risk ones automatically. In 2025 IBM extended governance to agents too, with new AI agent object types and evaluation that scores answer relevance and policy alignment on live decisions. The overlay is drawn in Figure 4 as coverage, because the question a regulator asks is how much of your evidence is captured automatically versus assembled by hand the night before an audit.
Agentic systems raise the governance bar rather than clearing it, and that is the honest catch in the overlay. A RAG answer is a single decision you can log, cite, and score in one shot. An agent that took six steps made six decisions, each with its own inputs, tool call, and outcome, and a regulator can ask about any of them. That is why the agentic bars in Figure 4 start lower: the surface area of what must be recorded is larger, so more of the evidence tends to fall to manual reconstruction unless the control plane logs each step as it happens. The practical rule is to lean harder on the evaluation and audit nodes in the agentic pattern than you would for plain RAG, because the thing you will be asked to prove is not the final answer but the path that produced it.
Worked example
A bank builds a RAG assistant for policy questions. Without governance, at audit the team spends roughly two weeks reconstructing which model version answered which query and where each passage came from. With watsonx.governance wired from the start, the factsheet already holds the model version, the source tables behind each answer through the watsonx.data lineage, the validation scores, and the EU AI Act risk tier. The audit becomes a review of an existing record, not a reconstruction project. The cost of the overlay is paid during the build; the saving lands every time someone asks for evidence.
What the three patterns share underneath
The reason these are variations and not three separate stacks is that they sit on one substrate. Every pattern runs inference through watsonx.ai, leans on a Granite model as the default engine, and ships through a deployment space, the production boundary covered in Part 22. Retrieval, when a pattern needs it, comes from the same watsonx.data lakehouse whether the caller is a chat endpoint or an agent tool. Identity and encryption come from the shared security layer, not per pattern. So the agentic pattern is the RAG pattern with a control plane on top, and the regulated overlay is either one with governance turned on. You are composing, not choosing between three worlds.
That shared base is the practical argument for standardising early. If you build your first RAG pipeline with embeddings colocated in watsonx.data, a Granite Guardian gate, and a factsheet, the jump to agents later reuses all three. Teams that instead bolt each pattern onto different infrastructure pay a migration tax the first time a RAG assistant needs to become an agent tool. Pick the substrate once, and let the pattern be the only thing that changes.
Which pattern fits your problem?
Work down three questions in order. Can a single grounded call answer it? Build enterprise RAG and stop. Does the task need several steps across several systems, with actions and not just answers? Move to the agentic pattern, and bound the action tools hard. Is a regulator in scope, or personal or financial data in the request? Add the governance overlay from the first commit regardless of which of the other two you chose. Most workloads I meet are RAG with the overlay. Genuinely agentic, genuinely regulated systems are the minority, and they are the ones worth slowing down for.
Start with one pattern, wire governance from day one
My recommendation is narrow on purpose. Build the enterprise RAG pattern first, colocate embeddings with your governed tables in watsonx.data, put a reranker in and Granite Guardian on the output, and turn on watsonx.governance before the first user sees it. Reach for watsonx Orchestrate and the agentic pattern only when a real workload cannot be finished by one grounded call, and when you do, treat the action tools as the risk and the policy gate as the control. Three patterns exist, but you should be building one at a time.
This week, take one workload and label it: RAG, agentic, or regulated. If you cannot label it in a sentence, it is not ready to build. The next and final part closes the series with a cost recap and a straight verdict on where watsonx wins and where it does not. Compare these shapes with the other clouds as you go, starting with the Bedrock reference architectures in the AWS series.
References
- watsonx.data integrated vector database, IBM announcement
- Agentic control plane, watsonx Orchestrate
- Preparing for the EU AI Act, IBM Think
- IBM watsonx.governance product overview
Related: Reference Architectures on Google Cloud in the Google Cloud series.
About The Author
DrJha