The model server would not start. oc get pods showed it stuck at Pending, and oc describe gave the reason in one line: 0/6 nodes are available, 6 Insufficient nvidia.com/gpu. The cluster had eight GPUs in it. The scheduler could not see a single one of them, because nothing had told OpenShift those cards existed or how to hand one to a pod.

Buying GPUs is the easy part. Getting them to a place where watsonx.ai can actually schedule a model is where teams new to Red Hat OpenShift lose a day. This part is the GPU layer under the software deployment you chose in Part 4, what a GPU has to hold, how OpenShift exposes one, how many cards a model needs, and when splitting a card is smart rather than a trap.

Why watsonx.ai needs its own GPUs

On the as a Service option, IBM ran the GPUs and you never thought about them. On software you own them, and the reason watsonx.ai wants them at all comes down to two jobs. Serving a model, called inference, means loading its weights into memory and running billions of multiplications for every token it returns. Tuning a model means the same math in reverse across a training set. Both are the kind of dense, parallel arithmetic a graphics processing unit, a GPU, was built for, and a CPU does the same work an order of magnitude slower. That is why a watsonx.ai software install that will serve a foundation model expects GPU nodes underneath it.

The catch is that a GPU is a fixed box of memory. A card carries a set amount of high bandwidth memory, and everything the model needs while it runs has to fit inside that memory at once: the weights, the working state for each request, and some overhead. When it does not fit, the model does not run slowly, it does not load at all. So the whole of GPU planning for watsonx.ai is really a memory question wearing a hardware costume, and the sections below build up to answering it with a real number.

How OpenShift hands a GPU to a pod

OpenShift does not expose GPUs to workloads on its own. Out of the box the scheduler knows about CPU and memory, and a card in a server is invisible to it. Two operators fix that, and both must be in place before you install watsonx.ai. The Node Feature Discovery operator, NFD, inspects each node and labels it with the hardware it carries, so a node with an NVIDIA card gets a label the scheduler can match against. The NVIDIA GPU Operator then installs the driver, the container toolkit, and a device plugin that advertises each card to OpenShift as a schedulable resource named nvidia.com/gpu. Only once a pod can request that resource, and a labelled node can satisfy it, will a model land on a card.

That chain is exactly what my Pending pod was missing. The GPUs were physically installed, but the device plugin was not advertising them, so from the scheduler’s point of view there was no nvidia.com/gpu to give out. Install the operators first, confirm the nodes advertise capacity, then install the service. Do it in the other order and the watsonx.ai install looks for schedulable GPUs, finds none, and fails in a way that sends you debugging the service when the real gap is two layers below it.

flowchart TD
  P[Model pod requests nvidia.com/gpu] --> S[OpenShift scheduler]
  S --> N[Node labelled GPU capable by NFD]
  N --> DP[NVIDIA device plugin advertises the card]
  DP --> DR[GPU Operator driver and container toolkit]
  DR --> G[Model weights load and inference runs]
  S -->|No GPU node free| PEND[Pod stays Pending]

Which card belongs under watsonx.ai

Not every GPU qualifies. For serving custom foundation models on watsonx.ai software, IBM calls for datacenter cards with large memory, specifically the NVIDIA A100 80GB or H100 80GB, and the 80 GB is the number that matters more than the model name on the box. Memory sets the ceiling on how large a model and how long a context a single card can hold, so a 40 GB variant of the same card is a different, smaller machine for this purpose. Newer NVIDIA parts such as the H200, which carries more memory again, and accelerators from Intel and AMD are moving into supported configurations through OpenShift AI, but confirm the exact model against the IBM and OpenShift AI support matrix for your release before you buy.

My advice here is boring on purpose. Standardize on one 80 GB card across the GPU nodes rather than mixing generations, because a model that needs several cards has to gather them from one node, and a node stocked with two different cards is a scheduling headache you create for yourself. The table lists the common choices and where each fits. Treat the memory column as the real spec.

Supported cards change by release. A100 80GB and H100 80GB are the cards IBM documents for custom foundation models; the rest are moving targets, so confirm on the IBM GPU requirements page and the OpenShift AI supported configurations article before ordering.

GPU memory is the whole budget

Here is the mental model that makes sizing simple. Three things sit in GPU memory while a model serves. First the weights, which is the model itself. A weight in 16 bit precision is 2 bytes, so a model with 8 billion parameters holds roughly 16 GB of weights, and a 70 billion parameter model holds roughly 140 GB. Second the working state for each in flight request, the key value cache, which grows with how long a context is and how many requests run at once. Third a slice of overhead for activations and memory fragmentation. Add them and compare against the card. An 8B model at 16 GB of weights sits comfortably inside one 80 GB card with plenty left for the cache. A 70B model at 140 GB of weights does not fit on one 80 GB card at all, before you have served a single request.

Because the 70B does not fit on one card, IBM publishes how many it needs, and this is where a hard rule enters. The published minimums move as models change, so read the current table, but two are worth stating: enabling both stock Granite models together calls for a minimum of 2 GPUs, and running llama-3-1-70b-instruct as a default model calls for a minimum of 4. The count is higher than raw weight memory alone because IBM budgets for the cache and for throughput, not just for the model to load. Size around the largest model you intend to serve, not the smallest, or you order hardware twice.

Minimum GPU counts are from the IBM GPU requirements for models page and change between releases. Read the table for the exact model and version you plan to run.

MIG and time slicing, sharing one card

The opposite problem is just as common. You have an 80 GB card and a small model that uses a fraction of it, and the card sits mostly idle. Two features let more than one workload share it. Multi Instance GPU, MIG, partitions the card in hardware into several smaller instances, each with its own slice of memory and its own fault isolation, and each looks to OpenShift like an independent GPU. Time slicing takes a different route: it hands the same physical card to several pods in turn, multiplexing them in software, with no memory or fault isolation between them. MIG gives you clean, isolated fractions. Time slicing gives you cheap oversubscription and trusts the workloads to behave.

Reach for MIG when several small, steady models need to run side by side with predictable memory, which is common for a set of Granite endpoints that each fit in a partition. Reach for time slicing for development and test, or bursty low volume endpoints, where you would rather share than pin a whole card to something used twice an hour. The trap is thinking either one helps a large model. Splitting a card into pieces gives each piece less memory, not more, so a 70B model that already overflows one card is only worse off on a fraction of one. Sharing is a utilization tool for small models, never a workaround for the single node rule.

Sizing a deployment before you buy

You do not have to guess. The script below takes a model size, a precision, a context length, and a concurrency target, adds weights plus a rough key value cache plus overhead, and prints how many 80 GB cards the model needs and whether the single node rule applies. Change the four inputs to your own case and run it. It is a planning floor, not a promise, and I say why under the output.

import math
# watsonx.ai GPU sizing floor: how many 80 GB cards a model needs.
# Planning estimate. Defer to the IBM GPU requirements table for the supported minimum.
CARD_GB = 80              # A100 80GB or H100 80GB
params_billion = 70      # model size in billions of parameters
bytes_per_param = 2      # 16 bit weights; use 1 for INT8, 0.5 for INT4
context_tokens = 8000    # longest context you serve
concurrency = 16         # requests in flight at once
kv_gb_per_1k_per_req = 0.5   # rough KV cache, 70B class, 16 bit

weights_gb = params_billion * bytes_per_param
kv_gb = (context_tokens / 1000) * concurrency * kv_gb_per_1k_per_req
overhead_gb = 0.10 * weights_gb          # activations and fragmentation
total_gb = weights_gb + kv_gb + overhead_gb
cards = math.ceil(total_gb / CARD_GB)

print('Weights GB:', weights_gb)
print('KV cache GB:', round(kv_gb, 1))
print('Total GB:', round(total_gb, 1))
print('80GB cards needed:', cards)
print('Single node required:', cards > 1)

Expected output: Weights GB: 140, KV cache GB: 64.0, Total GB: 218.0, 80GB cards needed: 3, Single node required: True. Failure modes: this floor prints 3 cards, but IBM lists a minimum of 4 for llama-3-1-70b as a default, because the table budgets for throughput on top of memory, so always take the higher supported number; the KV estimate is rough and models with grouped query attention use far less, so treat it as a ceiling; quantizing to INT8 or INT4 shrinks the weights term but can cost quality; and remember all of these cards, however many, must land on one worker node.

Idle GPUs are the real cost

On SaaS you paid per token and IBM absorbed the cost of a quiet hour. On your own OpenShift the meter never stops. A GPU pinned to a serving deployment draws power and depreciates whether it answered a million requests that day or none, so the number that decides whether owning the hardware was worth it is utilization, not peak throughput. A card at 15 percent utilization is not cheaper than SaaS, it is a fixed cost you chose to carry. This is why sharing matters beyond tidiness: MIG and time slicing exist to push small models onto fewer cards so the cards you own are actually busy.

Measure it, do not assume it. The NVIDIA GPU Operator ships a metrics exporter that feeds OpenShift monitoring, so you can watch per card utilization and memory use on a dashboard and see which cards run hot and which idle. Watch that for a week before you buy more hardware, because the usual finding is not that you are short of GPUs, it is that a few models hold whole cards they barely use. The FinOps side of this, chargeback and right sizing across the cluster, gets its own treatment in Part 20; here the point is narrower. A GPU you own is only cheap when it is busy.

If you are weighing whether custom silicon changes any of this, the accelerator choice plays out the same way on other clouds, and the AWS take in AWS Trainium and Inferentia vs GPUs is a useful contrast to the NVIDIA first world watsonx.ai lives in.

Size for the largest model, then chase utilization

Here is the sequence I follow on every watsonx.ai software build. Start from the largest model you intend to serve, read its GPU minimum from the IBM table, and note that all of those cards have to sit on one worker node, so that model sets the shape of at least one node before anything else does. Standardize on one 80 GB card across your GPU nodes. Install NFD and the NVIDIA GPU Operator, confirm the nodes advertise nvidia.com/gpu, and only then install the service. Do that and the Pending pod that opened this part never happens.

Then work the other direction on the small models. Do not give a Granite 8B its own card out of habit; put it and its neighbours behind MIG or time slicing so the cards you paid for run busy. Buy for the biggest model, share for the smallest, and measure everything in between. That is the difference between a GPU estate that earns the capital you sank into it and one that quietly bleeds it while you tell yourself you needed the headroom.

Before Part 7, do one thing. Write down the single largest model this deployment will serve and its GPU minimum from the IBM table, because that number sizes your first node. Part 7 stays on the cluster and takes on regions, private connectivity, and the security controls that decide who and what can reach these GPUs once they are running.

References

IBM docs: GPU requirements for models
IBM docs: Installing operators for services that require GPUs
IBM docs: Installing Red Hat OpenShift AI
NVIDIA: Time slicing GPUs in OpenShift
Red Hat: watsonx.ai with Red Hat OpenShift AI

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts