A retrain job ran green at 2am. The pipeline pulled fresh data, tuned Granite, pushed the new version to the serving space, and posted a success in Slack. By 9am the support queue was worse, because the model that shipped scored three points lower on answer relevance than the one it replaced, and nothing in the pipeline had checked. Green did not mean better.
That gap is the whole job of LLMOps. LLMOps, the operational discipline for running large language model features in production, is MLOps with two extra problems: prompts are code you have to version, and quality is a moving target you have to measure on every release. On the IBM stack the work sits across two planes, watsonx.ai for the model, the prompts, and the governance record, and Red Hat OpenShift AI for the pipelines and the GPUs. This part wires them into one loop, from a notebook experiment to a promotion gate that refuses to ship a regression.
Key takeaways
watsonx.ai is the control plane for models, prompts, factsheets, and managed endpoints. OpenShift AI is where training and pipelines actually run, on Kubeflow Pipelines 2.x over Argo Workflows. Promotion moves an asset from a project to a deployment space, and that boundary is the right place to hang a quality gate. Version prompts and eval sets like source code. Gate on evaluation deltas, not on job success. Serve with vLLM through KServe when you want control and cost headroom, or a managed watsonx endpoint when you want speed to production. Let watsonx.governance record lineage on every promotion so a green pipeline still leaves an audit trail.
What LLMOps means on this stack
MLOps, machine learning operations, is the practice of shipping and running models like software: version them, test them, deploy them through a pipeline, watch them in production. LLMOps is that same practice bent around generative models, and it adds two artifacts that classical MLOps never had to track. The first is the prompt. A prompt template is application logic now, and a one-line edit to it can swing quality as hard as a retrain. The second is the evaluation set, the fixed batch of questions and expected answers you score every candidate against. If that set is not versioned, you cannot tell whether last night was better than the night before.
So three things need version control on watsonx, not one: model weights, prompt templates, and eval sets. Miss any of them and your pipeline can run flawlessly and still degrade the product, which is exactly the 2am story above. The rest of this part is about building the loop that catches it.
Where the work runs, watsonx.ai or OpenShift AI
The IBM answer to LLMOps is two planes that hand off to each other. watsonx.ai is the managed control plane: Prompt Lab for authoring, the tuning studio, deployment spaces, managed inference endpoints, and the governance layer that records what shipped. Red Hat OpenShift AI is the execution plane that runs on your own Kubernetes: Jupyter notebooks, Data Science Pipelines, GPU scheduling, and model serving through KServe. Data Science Pipelines is OpenShift AI branding for Kubeflow Pipelines, the open source workflow engine for ML, and the current release runs Kubeflow Pipelines 2.x with Argo Workflows underneath. The plane you choose per stage decides how much you operate yourself.
| LLMOps stage | On watsonx.ai | On OpenShift AI |
|---|---|---|
| Experiment | Prompt Lab | Jupyter notebook |
| Version | Prompt templates as assets | Pipeline code in git |
| Train or tune | Tuning studio | Kubeflow pipeline on GPU |
| Evaluate | Governance evaluations | Pipeline eval step |
| Promote | Deployment space | Pipeline promote task |
| Serve | Managed endpoint | KServe with vLLM |
| Monitor | Factsheets and drift | Prometheus and serving metrics |
Read left to right and the loop appears. You experiment, you version, you train, you evaluate, and then one edge matters more than any other: promotion, the step that moves an asset out of a development project and into a production deployment space. The diagram below is the pipeline I build, with the gate sitting on that promotion edge.
flowchart LR
Nb[Notebook or Prompt Lab] --> Ver[Version prompt and code]
Ver --> Pipe[OpenShift AI pipeline]
Pipe --> Tune[Tune Granite on GPU]
Tune --> Eval[Evaluate against baseline]
Eval --> Gate{Passes gate}
Gate -->|Yes| Promote[Promote to deployment space]
Gate -->|No| Stop[Fail run and keep old model]
Promote --> Serve[Serve endpoint]
Serve --> Mon[Monitor and factsheet]
Mon --> Nb
How a promotion actually flows
A project on watsonx.ai is your workbench, where notebooks, prompt templates, and tuned models live while you iterate. A deployment space is the production room, where an asset becomes a running deployment with its own environment and access controls. Promotion is the one-way door between them. IBM documents the Studio user interface as the recommended way to promote, because it carries an asset and its dependencies across in one complete definition, and there is a programmatic path for pipelines through the watsonx.ai SDK. Either way, watsonx.governance can capture the lineage at that moment, so the factsheet records which base model, which prompt, and which evaluation produced the thing now serving traffic.
Gotcha
Promotion does not preserve version history. Promote version B of a model to a space and the space shows only asset A at its current state, with the earlier version gone from that space. So a deployment space is not your rollback store. If you need to fall back, keep the previous deployment live behind the endpoint until the new one has proven itself on real traffic, or archive versioned artifacts outside the space in object storage. Treat the space as the current truth, not the history.
Build a promotion gate in code
The gate is the smallest piece of the whole system and the one that earns the most. It takes the candidate model evaluation scores, compares them to the live baseline, and returns a promote or reject decision that a pipeline step can act on. It has no external calls, so you can run it as is, and you can drop the same function into a Kubeflow pipeline step so a regression fails the run instead of shipping. Here is the core.
# watsonx promotion gate, pure python, no api calls
# rejects a candidate that regresses against the live baseline
BASELINE = {'faithfulness': 0.86, 'answer_relevance': 0.83, 'toxicity': 0.02}
# quality metrics may drop at most 0.02; toxicity may rise at most 0.01
TOLERANCE = {'faithfulness': -0.02, 'answer_relevance': -0.02, 'toxicity': 0.01}
def gate(candidate, baseline=BASELINE, tol=TOLERANCE):
failures = []
for metric, base in baseline.items():
delta = round(candidate[metric] - base, 3)
limit = tol[metric]
regressed = delta > limit if metric == 'toxicity' else delta < limit
if regressed:
failures.append((metric, delta))
return {'promote': not failures, 'failures': failures}
if __name__ == '__main__':
candidate = {'faithfulness': 0.88, 'answer_relevance': 0.80, 'toxicity': 0.02}
print(gate(candidate))
Expected output:
{'promote': False, 'failures': [('answer_relevance', -0.03)]}The candidate gained two points of faithfulness and held toxicity flat, but answer relevance fell three points, past the two point tolerance, so the gate rejects it. That is the 2am regression caught before it serves. In the pipeline, a rejection raises a non zero exit so the promote task never runs and the old model keeps serving. On a pass, the pipeline calls the watsonx.ai SDK to promote, roughly client.spaces.promote with the asset id, source project, and target space.
The failure mode is the one the code cannot fix, and it matters more than the code. The gate is only as honest as the eval set behind it. If that set is stale, too small, or does not resemble live traffic, a candidate can clear every threshold and still be worse in production, because you measured the wrong thing. The gate gives you a decision; the eval set gives you the truth, and the eval set is the part you have to keep earning. Refresh it from real conversations on a schedule, and version it alongside the model.
What a nightly retrain costs in time and CUH
A pipeline is not free to run, and the cost is mostly wall clock time on the GPU. Capacity Unit Hours, CUH, is the watsonx compute meter I covered in Part 20, and a tuning run in a GPU environment is where it accrues fastest. Below is the shape of a representative nightly loop: pull data, tune Granite, evaluate, gate and promote. The tuning stage dominates everything else combined.
Worked example
A nightly pipeline spends about 12 minutes pulling and chunking data from watsonx.data, 95 minutes tuning a small Granite model on a GPU environment, 18 minutes running the evaluation against the baseline, and 3 minutes gating and promoting. That is roughly 128 minutes end to end, and near 10 CUH, of which the tune is about 8. The read is blunt: cut cost by cutting tuning frequency or GPU size, not by shaving the cheap stages. Moving from nightly to twice weekly on this profile removes more CUH than any other single change.
| Stage | Wall clock | CUH (illustrative) | Where it runs |
|---|---|---|---|
| Data pull and chunk | 12 min | ~0.5 | CPU, watsonx.data |
| Tune Granite | 95 min | ~8 | GPU environment |
| Evaluate vs baseline | 18 min | ~1.5 | Eval harness |
| Gate and promote | 3 min | ~0 | Control plane call |
Serving models, vLLM on OpenShift or a managed watsonx endpoint
Once a model clears the gate, it has to serve traffic, and here the two planes genuinely diverge. A managed watsonx.ai endpoint is the fast path: you deploy the asset in the space, IBM runs the inference, and you pay per token or per CUH. No cluster to operate, quick to production, and the governance record comes attached. The trade is less control over the runtime and cost that tracks the meter rather than your hardware.
The OpenShift AI path serves through KServe, the Kubeflow serving component, with vLLM as the inference engine. vLLM is an open source server built for high throughput LLM inference, and KServe offers it in two modes: a Serverless mode, shown as Advanced in the dashboard, that scales to zero between requests, and a RawDeployment mode, shown as Standard, that holds fixed replicas. You run the cluster and the GPUs, which is real operational weight, but you get autoscaling, your choice of accelerator, and cost that is your own capacity rather than a per token rate. At high, steady volume that usually wins on money.
My verdict is sequencing, not either or. Ship the first version on a managed watsonx endpoint, because time to production beats a cost optimisation you have not earned yet. Move to vLLM on OpenShift when volume is steady and large enough that owning the serving pays for the operations it adds, or when a data residency rule forces the model onto your own cluster. Do not stand up a KServe stack for a pilot that serves a hundred requests a day; you will spend more on the platform than on the tokens.
Where pipelines leak, and how to plug them
Four leaks account for most of the trouble I see. The first is gating on job success instead of quality, the 2am story, and the gate above is the fix. The second is a frozen eval set: it passes everything because it stopped resembling production months ago, so refresh it from live traffic on a cadence. The third is the prompt edited straight into production outside the pipeline, which no gate ever scores; close it by making git the only road into the serving space. The fourth is a GPU environment left running after a pipeline finishes, quietly metering CUH, which is the same idle leak Part 20 warned about, so scale tuning environments down when the run ends.
Under all four is one habit. A pipeline you set up once and never watch will drift, because the world it was tuned against keeps moving. The point of LLMOps is not that the loop runs by itself. It is that the loop tells you the truth when it runs.
My default pipeline for watsonx LLMOps
If you build one thing from this part, build the gate before you build the automation. A manual promote with a real quality check beats a fully automated pipeline that ships whatever tuning produced. My default is a single OpenShift AI pipeline with five steps, prep, tune, evaluate, gate, promote, driven by a git repository that holds the pipeline code, the prompt templates, and the eval set together. Serving starts on a managed watsonx endpoint and moves to vLLM on OpenShift only when volume or residency justifies the operations. Governance records lineage on every promotion, so even the green nights leave a trail an auditor can follow.
This week, add one gate step to whatever promote flow you already run, and let it fail a build on a genuine regression before you automate anything else. Part 23 takes this operational base into full reference architectures for enterprise RAG, agentic, and regulated workloads. For the same discipline on another cloud, compare this with Vertex AI Pipelines for LLMOps on Google Cloud.
References
- What is new with data science pipelines in Red Hat OpenShift AI
- Promoting assets to a deployment space, IBM Documentation
- Extend your AI capabilities with watsonx, IBM MLOps guide
- Autoscaling vLLM with OpenShift AI, Red Hat Developer
Related: LLMOps and CI/CD for Amazon Bedrock and SageMaker in the AWS series.
About The Author
DrJha