A bank asked me to put a generative AI feature into production in a regulated environment. The model was the easy part. The hard part was proving, to an auditor, where the training data came from, who could call the endpoint, and what the thing would refuse to say. That is the gap IBM watsonx is built to close, and it is why the platform is three products, not one.

What watsonx actually is

watsonx is IBM’s enterprise AI and data platform. Read that as a bundle, not a single app. The name covers three products that share one account, one set of identities, and one governance spine. IBM sells them together because the pain they solve shows up together: you cannot ship a trusted AI feature if the data is ungoverned, and you cannot pass an audit if the model is a black box no one is watching.

Compare that to the plain approach of calling a hosted model API. The API gives you tokens in and tokens out. It does not give you a governed place for your documents, a record of which model version answered a given request, or a dashboard an auditor can read. For a hobby project that is fine. For a bank, an insurer, or a hospital, the missing pieces are the whole job. watsonx exists to supply them without you stitching five vendors together.

One term to fix early. A foundation model is a large model pretrained on broad data that you then adapt by prompting or tuning. IBM’s own foundation models are branded Granite. watsonx also hosts selected third party and open models, so Granite is the default, not the only option.

Where each watsonx module fits

Think of three planes stacked on your data. At the bottom, watsonx.data holds and prepares the data. In the middle, watsonx.ai is where people build, prompt, tune, and serve models. Across the top, watsonx.governance watches everything and keeps the paper trail. A request moves up through the build plane; oversight comes down from the governance plane.

Here is the same split as a table, with who tends to live in each module and the one artifact that matters most there.

Governance deserves a word now, because it is the piece people underestimate. watsonx.governance keeps an AI factsheet for every model, a living record of its data lineage, version, owner, and approved use. It monitors production traffic for drift and for outputs that breach a policy, and it maps your models against regulatory frameworks including the EU AI Act. IBM was named a Leader in the 2026 Gartner Magic Quadrant for AI Governance Platforms, which matters less than the practical point behind it: when a regulator asks how a decision was made, the factsheet is the document you hand over. Build without it and that document does not exist, so you reconstruct it under pressure later.

Granite, and the models you actually run

Granite is IBM’s own family of foundation models, and it is the default choice inside watsonx.ai. The reason IBM pushes its own models is trust and cost, not raw benchmark bragging. Granite models are released under an Apache 2.0 license, trained on curated and IP cleared data, and IBM offers an uncapped indemnity for third party IP claims on content Granite generates when you run it on watsonx.ai. For a legal team, that indemnity is often the deciding line.

Granite 4.0 landed on 2 October 2025 with a change worth knowing even at Part 1 level. Instead of a pure transformer, the hybrid models mix a small number of transformer attention layers with a majority of Mamba layers in a 9 to 1 ratio. Mamba is a different sequence design whose memory stays roughly flat as input grows, so the hybrid needs far less RAM on long inputs. IBM reports over 70 percent less memory on long context and many concurrent sessions versus a comparable transformer. In plain terms, the same workload runs on cheaper GPUs. Granite 4.0 was also the first open model family to earn ISO 42001 certification, and every checkpoint is cryptographically signed so you can verify it is the real thing.

The lineup is deliberately small. You pick a size for your latency and cost budget, not a single giant model for everything.

The distinction between total and active parameters matters for your bill. In a mixture of experts model, only a slice of the network fires per token. H Small carries 32B parameters but activates about 9B per token, so it costs closer to a 9B model to run while reasoning like something larger. Granite 4.1 arrived in 2026 with dense 3B, 8B, and 30B variants plus refreshed speech, vision, embedding, and Guardian models, so the exact ids move over time. Check the model list in your region before you hardcode one. Later parts cover choosing and tuning these; Part 3 goes deep on the Granite family and licensing.

How a prompt flows from data to deployment

The typical path is short. Your documents land in watsonx.data and get chunked and vectorised into a Milvus index, which is an open source vector store IBM embeds in the lakehouse. In watsonx.ai you open Prompt Lab, a visual workspace for writing and testing prompts, and wire the retrieval to a Granite model. Once the prompt behaves, you deploy it as an endpoint. Every model you promote gets an AI factsheet in watsonx.governance, and live traffic is monitored for drift and policy violations. That loop is the product.

One acronym to define, since it runs through the whole series. RAG, short for retrieval augmented generation, means the model pulls relevant chunks from your own data at query time and reads them before answering, rather than relying only on what it learned in training. watsonx.data is where those chunks live as vectors, and the Milvus index is what makes the retrieval fast. Most enterprise watsonx projects are some flavour of RAG, because it is how you ground a general model in your own facts without retraining it. Part 9 builds a full RAG pipeline on watsonx.data.

At the code level, calling a served Granite model is a few lines with the ibm-watsonx-ai Python SDK. This is the smallest useful example: authenticate, pick a model, send a prompt.

from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference

creds = Credentials(
    url='https://us-south.ml.cloud.ibm.com',
    api_key='YOUR_IBM_CLOUD_API_KEY',
)

model = ModelInference(
    model_id='ibm/granite-4-0-h-small',   # confirm exact id in your model list
    credentials=creds,
    project_id='YOUR_PROJECT_ID',
    params={'max_new_tokens': 120},
)

print(model.generate_text(prompt='Explain watsonx in two sentences.'))

Expected output: a two sentence plain text answer printed to your console. Failure modes: a 404 model_not_supported means the model_id or region is wrong; a 401 means the api_key or project_id do not match; an empty string usually means max_new_tokens is too low. The model_id above is a placeholder pattern, so verify the live id before you run it.

Orchestrate, Assistant, and InstructLab

Three more names round out the map, and you will hit all of them in later parts. watsonx Orchestrate is the agent layer. It builds and runs AI agents that take actions across your business apps, so a request can become a chain of real steps rather than a single answer. watsonx Assistant is the conversational layer, used to build the chat front ends that sit in front of those agents and models. Neither replaces watsonx.ai; they consume it. You still build and serve the underlying model in watsonx.ai, and Orchestrate or Assistant call it.

InstructLab is the odd one out and worth knowing early. It is an open source project, co created by IBM and Red Hat, for teaching a Granite model new skills and knowledge using generated synthetic data, so you can improve a model with far fewer hand labelled examples than full fine tuning needs. If your first instinct is to fine tune Granite on a giant dataset you do not have, InstructLab is often the cheaper route to the same result. Part 12 covers it properly.

Where watsonx runs

You have three deployment shapes, and the choice usually comes down to where your data is allowed to live. The first is SaaS on IBM Cloud, where IBM runs everything and you consume it through a browser and an API. It is the fastest way to start and the right call for most teams. The second is Cloud Pak for Data, IBM’s software layer that runs the same watsonx capabilities on a cluster you control. The third is self managed on Red Hat OpenShift, which is how banks and government tenants keep the entire stack inside their own boundary, on premises or in an air gapped site.

My rule of thumb: start on SaaS to prove the use case, then move to OpenShift only when a data residency or air gap requirement forces it. The self managed path gives you control and hands you the operational burden of GPUs, upgrades, and capacity planning. Do not take that on before you have a working feature that justifies it.

What it costs, in plain terms

watsonx.ai meters two things. Model inference is billed in Resource Units, where one Resource Unit equals 1,000 tokens counted across both your input and output. Everything else in the runtime, such as tuning jobs and notebook compute, is billed in Capacity Unit Hours. The mental model you need is simple: inference cost scales with tokens, and token count is the number you can actually forecast.

Start in watsonx.ai, prove value, then wire in governance

If you are deciding where to put your hands first, start in watsonx.ai on the SaaS plan and get one Granite prompt working against your own data through watsonx.data. That single loop teaches you more than any diagram. Bring in watsonx.governance the moment the feature is headed for production or touches regulated data, and not a day later, because the factsheet and monitoring are far cheaper to set up before launch than to reconstruct after an auditor asks. The whole reason to pick watsonx over a bare model API is that governance plane, so do not skip the part you are paying for.

Next in this series I open the first box: watsonx.ai studio and Prompt Lab, where you will build that first prompt hands on. Go set up a free watsonx.ai trial before you read it, so you can follow along in your own account.

References

IBM watsonx platform overview
IBM Granite 4.0 announcement
IBM Research: Granite 4.1 family
watsonx.data integrated vector database
watsonx.ai pricing
Companion vendor neutral GenAI series

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts