The tune job was submitted, the cluster had four A100s, and the pod would not start. oc get pod showed it stuck in Pending, and the event log read 0/6 nodes are available: insufficient nvidia.com/gpu. The GPUs were in the rack. OpenShift just did not know they were usable yet. That gap, between hardware existing and the scheduler agreeing to place a pod on it, is what this part is about.

Parts 11 through 13 treated the trainer as a given. You wrote a taxonomy, you cleaned a dataset, you clicked tune. Underneath that button sit a scheduler, a set of GPU operators, an autoscaler, and a queue. When one of them is misconfigured, your job either waits forever or quietly costs you a node you forgot to turn off. This part is the infrastructure layer, from a bare GPU worker to a job that scales cleanly and stops when it is done.

Where a watsonx tune actually runs

When you deploy watsonx.ai as software rather than SaaS, it installs on IBM Software Hub, the platform that used to be called Cloud Pak for Data, and that platform runs on Red Hat OpenShift. OpenShift is a Kubernetes distribution, so the mental model is simple: a tune is a container that needs a slice of a GPU worker node, and OpenShift is the thing that finds a node with a free GPU and starts the container there.

Tuning Studio, the watsonx.ai feature you clicked in Part 11, does not talk to a GPU directly. It creates a training job, that job becomes one or more pods, and the pods carry a resource request that reads nvidia.com/gpu: 2 or similar. The OpenShift scheduler reads that request and looks for a node advertising at least that many free GPUs. No node advertises GPUs, no placement, and the pod sits in Pending exactly as it did in the opener.

So the whole chapter comes down to three questions. What makes a node advertise its GPUs? How do you get a node when there is none free? And how do you stop three teams from grabbing the same GPUs at once? The diagram below is the path a single tune takes from click to running pod.

flowchart TD
  A[Submit tune in Tuning Studio] --> B{Kueue quota free?}
  B -- no --> Q[Job waits in the queue]
  Q --> B
  B -- yes --> C[Training job creates pods]
  C --> D{GPU node with free GPUs?}
  D -- no --> S[Cluster autoscaler adds a node]
  S --> E[NFD labels node, GPU Operator installs driver]
  E --> D
  D -- yes --> F[Pod placed, tune runs]
  F --> G[Checkpoints written to a shared PVC]

Two operators stand between you and a usable GPU

A fresh GPU worker is invisible to the scheduler until two operators finish their work. The first is the Node Feature Discovery operator, or NFD, which inspects each node and writes labels describing its hardware. On a GPU box it adds a label marking the presence of an NVIDIA PCI device, and that label is what the second operator watches for.

The second is the NVIDIA GPU Operator. It sees the NFD label, then rolls out the driver, the container toolkit, and the device plugin onto that node. The device plugin is the piece that matters for scheduling: it is what registers nvidia.com/gpu as an allocatable resource so the scheduler can count it. It also installs DCGM, the tool that exports GPU metrics you will want later for utilization tracking. Until the device plugin reports in, the node shows zero allocatable GPUs, and this is the honest reason for the Pending pod at the top of this part. The hardware was fine. The driver stack was still coming up.

Two required operators, in a fixed order, is the rule to remember. If you want the step by step install, I covered it in Part 6, and the internals of how a driver talks to the card sit in the NVIDIA AI Series piece on the GPU data path. Here the point is narrower: no NFD label, no GPU Operator action, no schedulable GPU.

watsonx also gives you a choice about which engine runs the training. Tuning Studio inside Software Hub is one path. Red Hat OpenShift AI, a separate product that installs on the same cluster, is another; it ships the distributed training machinery covered later in this part. And plain SaaS skips the cluster entirely. Where each one fits:

Why tuning wants every GPU on one node

IBM is explicit that when a foundation model tune needs more than one GPU, all of those GPUs must sit on a single OpenShift worker node. This is not a soft preference. A two GPU tune will not place across a GPU here and a GPU there on a different node. It waits until one node has both free.

The reason is bandwidth. Inside one node the GPUs talk over NVLink, which is far faster than the network hop between nodes, and the tuning runtime assumes that intra node speed for its gradient exchange. Striping a single tune across nodes would slow it to a crawl, so watsonx does not try. That means your node shape matters more than your total GPU count. Eight A100s spread as one GPU across eight nodes cannot run a two GPU tune. Two nodes with four A100s each can run several.

Large scale pretraining is the exception, and it is a different workload. When you genuinely need to span nodes, that is the job of the OpenShift AI Training Operator or a Ray cluster, both of which are built for multi node distributed training. Tuning Studio itself stays single node. So size your GPU workers for the widest single tune you expect, buy fewer, denser nodes rather than many thin ones, and you avoid the quiet Pending that comes from GPUs that exist but cannot be gathered onto one host.

Scale GPU nodes to zero between jobs

GPU nodes are the expensive line on the bill, and tuning is bursty. You run a job for a few hours, then nothing until next week. Paying for an idle A100 node the rest of the time is the most common waste I see in on prem watsonx builds. The fix is a MachineSet plus the cluster autoscaler.

A MachineSet is the OpenShift object that provisions worker nodes of a given type, a GPU node in this case. A MachineAutoscaler attached to that MachineSet gives it a floor and a ceiling, say zero to four. The cluster wide ClusterAutoscaler then watches for pods that cannot be scheduled. When your tune pod goes Pending for want of a GPU, the autoscaler grows the GPU MachineSet from zero to one, a node boots, the operators bring the GPU up, and the pod lands. When the job finishes and the node sits idle past a timeout, it scales back to zero.

There is one trap. Because a new GPU node reports zero allocatable GPUs for the minutes the driver takes to install, the autoscaler can think the node did not help and add a second one. That is overprovisioning, and on GPU nodes it is costly. The cure is patience in the config: a generous delay after adding a node, so the autoscaler waits for the GPU Operator to finish before it decides whether more capacity is needed.

# 1. Find your GPU machineset (replicas may already be 0)
oc get machinesets -n openshift-machine-api

# 2. Apply cluster autoscaler limits and a per machineset autoscaler
oc apply -f gpu-autoscale.yaml

# gpu-autoscale.yaml
apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  resourceLimits:
    gpus:
    - type: nvidia.com/gpu
      min: 0
      max: 8
  scaleDown:
    enabled: true
    delayAfterAdd: 20m
    unneededTime: 10m
---
apiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  name: gpu-a100
  namespace: openshift-machine-api
spec:
  minReplicas: 0
  maxReplicas: 4
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: cluster-gpu-a100-us-east-1a

Expected: oc apply reports the ClusterAutoscaler and MachineAutoscaler created. Submit a tune and, within a minute, the Pending GPU pod triggers the MachineSet to go from 0 to 1. The node registers a few minutes later once the GPU Operator finishes.

Failure mode: the node shows Ready but the pod still reads insufficient nvidia.com/gpu. The device plugin has not registered yet. Do not add a node by hand, that is the overprovisioning trap. Watch the node until its nvidia.com/gpu allocatable count turns nonzero, then the scheduler places the pod on its own.

Should you slice GPUs with MIG?

MIG, or Multi-Instance GPU, partitions one A100 or H100 into as many as seven isolated instances, each with its own slice of memory and compute. For an inference tier it is genuinely useful. You can pack several small Granite endpoints onto one card, give each tenant hard isolation, and stop a single model from starving the rest. For interactive notebooks it is the same story, many light users sharing one physical GPU.

Tuning is where it bites. IBM documents that MIG partitioned GPUs are incompatible with foundation model tuning, so a node you plan to tune on must expose whole, unpartitioned GPUs. If you MIG your only GPU pool to squeeze more inference onto it, you have just made those cards unusable for tuning without noticing, until a tune sits Pending forever because no whole GPU exists. The answer is two pools. Keep a MIG pool for inference and dev, and a separate whole GPU pool, its own MachineSet, for tuning.

Share GPUs with a queue, not a scramble

Once more than one team tunes on the same cluster, first come scheduling turns into a scramble. Someone launches a wide job at 9am, takes every GPU, and the rest of the org waits with no idea when they will get a turn. Kueue fixes this. Kueue is a Kubernetes native job queue, and it ships with Red Hat OpenShift AI as part of the distributed workloads set alongside the Training Operator and the Ray based CodeFlare stack.

The model is two objects. A ClusterQueue holds the total GPU budget and the sharing rules for the cluster. A LocalQueue lives in a team namespace and points at the ClusterQueue. A job submitted to a LocalQueue does not run until its share of the quota is free; if the budget is spent, it waits in the queue rather than failing or grabbing GPUs it should not have. You get fair sharing and priorities instead of a race, and jobs that would have errored out now simply hold.

Own the cluster only when SaaS tuning runs out

Here is my recommendation, and it is a real one. For most teams, tune on watsonx SaaS and skip this entire chapter. You rent GPUs by the job, IBM runs the operators and the scaling, and you never wake up to an idle A100. Owning an OpenShift GPU cluster earns its keep in three cases: data that legally cannot leave your premises, an air gapped environment, or utilization high and steady enough that renting costs more than owning. If none of those is true, the SaaS path is the honest answer.

If you do own the cluster, three moves pay off before anything else. Scale your GPU MachineSets to zero between jobs, because idle GPU time is the bill. Keep a whole GPU tuning pool separate from any MIG inference pool, so a capacity push never makes your tuning nodes unusable. And put Kueue quotas in front on day one, before two teams collide, not after. Do those three and the rest of the tuning stack, the part you actually came for, has a floor it can stand on. Next, in Part 15, that floor starts running agents: watsonx Orchestrate turns these tuned models into things that take actions. Before you read it, go look at your own GPU nodes and ask one question: are any of them sitting idle right now?

References

• IBM Documentation, GPU requirements for models
• Red Hat OpenShift AI, Overview of distributed workloads
• NVIDIA, MIG support in OpenShift Container Platform
• Red Hat, Autoscaling NVIDIA GPUs on Red Hat OpenShift

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts