A data center A100 runs about 3.90 dollars an hour on demand, and the support assistant was holding a whole one to answer one question at a time. Granite 3.1 8B needs roughly 16 GB in bf16, so on a 40 GB card two thirds of the memory sat empty and, on a single request stream, compute idled near 15 percent. That is a card billed at 100 percent and used at a fraction. Part 17 gave the assistant a live KServe endpoint on a full GPU. This part makes one physical card serve more than one model, and shows why the obvious way to do that is usually the wrong one.

From one model per GPU to several

Last part the assistant took over an entire GPU through a KServe InferenceService, answered real traffic, and cost a full card to do it. This part keeps the same Granite 3.1 8B weights and the same endpoint, and changes only what sits underneath: instead of one model owning one physical GPU, we split the card so the assistant takes the share it actually needs and the rest of the GPU goes to other work, a second smaller model, a batch evaluation job, or a data science notebook. Nothing in the client code changes. What changes is how the GPU presents itself to Kubernetes, and that is a scheduling and node decision, not an application one.

Red Hat did not invent GPU sharing. Both mechanisms come from NVIDIA and are driven by the upstream NVIDIA GPU Operator; Red Hat packages that operator, tests it against OpenShift releases, wires it into the OpenShift AI dashboard as hardware profiles, and carries the support line when a node will not partition. Where the raw utilisation economics of GPUs live, the Data Science Series covers them in GPU cost, scale and sizing; this part is the OpenShift specific mechanics of acting on that math.

Two ways to share one GPU

Time slicing and MIG solve the same sentence, more than one pod per GPU, in opposite ways. Time slicing is a software trick: the device plugin advertises a single physical GPU as several logical replicas, and the GPU rapidly context switches between the pods scheduled onto them. Every replica sees the full memory of the card, and there is the catch, because they all draw from that one pool with no limit and no fault boundary. MIG is a hardware feature on the card itself: the GPU is physically carved into up to seven instances, each with a private slice of memory, its own cache and its own compute engines, so a crash or a memory spike in one instance cannot touch another. One is elastic and unsafe, the other is rigid and isolated.

GPU time slicing on OpenShift AI

Time slicing is configured through a ConfigMap the GPU Operator reads, then referenced from the cluster policy. You tell the device plugin how many replicas to expose per physical GPU, and Kubernetes then treats that card as if it had that many. Here is a config for a card exposed as four slices, adapted from the OpenShift AI procedure. The versions below are what I tested against.

# Tested on OpenShift AI 3.4 self managed, NVIDIA GPU Operator 25.3.x [VERIFY exact z-stream]
# ConfigMap must live in the nvidia-gpu-operator namespace
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: nvidia-gpu-operator
data:
  a100-40gb: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4          # one physical GPU now advertises as 4

Point the cluster policy at that ConfigMap with spec.devicePlugin.config set to name time-slicing-config and default a100-40gb, label the machine set with nvidia.com/device-plugin.config: a100-40gb, and the node capacity jumps. That jump is exactly where the danger hides.

$ oc get node gpu-node-1 -o jsonpath='{.status.capacity.nvidia.com/gpu}'
4
# Kubernetes now believes there are 4 GPUs. There is still one 40 GB pool.
# Schedule two Granite 8B pods, each grabbing 16 GB, and the third or fourth pod dies:
$ oc logs support-assistant-b-predictor-5f7c9
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 39.38 GiB of which 1.12 GiB is free.
Process 0 has 16.too GiB memory in use. Process 1 has 16.9 GiB memory in use.

Four logical GPUs, one real 40 GB pool. Two copies of a 16 GB model fit; the moment a third schedules, or one conversation grows its KV cache, the allocator runs dry and a pod that was healthy a second ago is killed. The setting failRequestsGreaterThanOne: false means a pod can also ask for more than one slice and quietly get scheduled anyway. Time slicing raised the node count from 1 to 4 and raised the risk of a 3 a.m. OOM by the same factor. For notebooks running small models, or preprocessing that barely touches GPU memory, that risk is fine and the utilisation win is real. For a memory-heavy language model, it is a loaded gun.

Multi-Instance GPU partitioning with MIG

MIG fixes the isolation problem by moving the split into silicon. A supported card is reconfigured into instances whose names encode what they contain: a profile of 3g.20gb has three of the seven compute slices and 20 GB of memory, and 1g.5gb has one slice and 5 GB. A tenant placed on a 20 GB instance physically cannot allocate the twenty first gigabyte, so it cannot starve a neighbour no matter what it does. The GPU Operator drives this with a mig-manager that watches a node label and applies the requested layout.

Two strategies decide how the slices are advertised. Single strategy makes every GPU on a node use the same profile and advertises them under the plain nvidia.com/gpu resource, simplest when a node does one job. Mixed strategy lets a node hold different profiles at once and advertises each as its own resource, for example nvidia.com/mig-3g.20gb, which is what you want when a card serves the assistant on one half and small jobs on the other. Enabling it is a cluster policy flag plus a node label.

# 1. Tell the operator to run mig-manager in mixed mode (ClusterPolicy spec)
spec:
  mig:
    strategy: mixed
  migManager:
    enabled: true

# 2. Ask a node for a layout by label. all-balanced gives a mix of profiles;
#    a named profile like all-3g.20gb gives two 20 GB halves on an A100-40GB.
$ oc label node gpu-node-1 nvidia.com/mig.config=all-3g.20gb --overwrite
node/gpu-node-1 labeled

# 3. After mig-manager reconfigures the card, the halves show up as a resource
$ oc get node gpu-node-1 -o jsonpath='{.status.capacity}' | tr ',' 'n' | grep mig
"nvidia.com/mig-3g.20gb":"2"
$ oc debug node/gpu-node-1 -- chroot /host nvidia-smi -L
GPU 0: NVIDIA A100-SXM4-40GB
  MIG 3g.20gb Device 0: (UUID: MIG-9c1b...)
  MIG 3g.20gb Device 1: (UUID: MIG-4a7e...)

An InferenceService then requests nvidia.com/mig-3g.20gb: 1 instead of nvidia.com/gpu: 1, and the assistant lands on a hardware isolated 20 GB half of the card. The other half is free for a second tenant that cannot crash it.

Reconfiguring is not free, and this is where MIG bites first-timers. To change a layout, mig-manager cordons the node, evicts the GPU pods, and on many cards needs the GPU reset or the node rebooted before the new geometry takes. If workloads will not drain, the label just sits in a pending state and nothing happens.

$ oc describe node gpu-node-1 | grep mig.config.state
  nvidia.com/mig.config.state=pending
$ oc logs -n nvidia-gpu-operator ds/nvidia-mig-manager -c nvidia-mig-manager
Waiting for pods using GPUs to be deleted before applying MIG config
Unable to set MIG config: at least one GPU is in use by a running workload
# Fix: drain the GPU pods first, or set the reboot annotation so the node cycles
$ oc label node gpu-node-1 nvidia.com/mig.config=all-3g.20gb 
    nvidia.com/mig.config.state- --overwrite   # clear stale state, then drain

MIG profiles and how much fits

Profiles are not arbitrary; each card supports a fixed menu, and the memory number is what decides whether a given model fits. Keep this table next to the datasheet when you size a slice. A 16 GB model like Granite 3.1 8B needs a 20 GB instance with room for the KV cache, so 3g.20gb is the smallest honest home for it; a 1g.5gb slice at 5 GB will not load it at all.

Choosing time slicing, MIG or a whole card

Here is the matrix I actually use when someone asks how to place a new GPU workload. It weighs the three real options against the concerns that decide the answer.

Cost is the argument that usually settles it. Divide the roughly 3.90 dollars an hour of an A100 across whatever the card can safely serve, and the picture is stark. A whole card behind one 8B model carries the full rate. Two 3g.20gb halves each carry half. Seven tiny 1g slices carry a seventh apiece, but only fit models that do not need the memory. This chart is the reason the finance conversation about self hosted GenAI, which the AI Engineering Series frames in cost control and model routing, so often ends at MIG.

Node management and applying a share

Both mechanisms are applied per node, through labels the GPU Operator reconciles, which means the unit of decision is a machine set and not an individual pod. A common and clean pattern is to keep pools distinct: one machine set of MIG nodes for steady inference, another of whole-card nodes for training or large-model serving, and if you use time slicing at all, a small pool of sliced nodes reserved for notebooks. Mixing all three geometries on one machine set invites a reconfigure to evict production traffic. The flow below is what actually happens when you change a MIG label, and knowing it stops the twenty minute stare at a pending state.

flowchart LR
  A[admin sets node label mig.config] --> M[mig manager reads label]
  M --> C[cordon node]
  C --> D[evict GPU pods]
  D --> R[reset GPU and apply geometry]
  R --> P[device plugin advertises mig 3g.20gb]
  P --> S[scheduler places pods on slices]
  D -->|pods will not drain| W[label stays pending]

My own worst hour on this came from ignoring the card support row. I labelled our A10G node nvidia.com/mig.config=all-1g.5gb and waited. The label sat pending for twenty minutes, mig-manager logged nothing that pointed at the real cause, and no nvidia.com/mig resources ever appeared. An A10G does not do MIG at all; only A30, A100, H100 and newer data center cards do, and I had skipped the one line in the datasheet that says so. Half a morning gone, I fell back to time slicing with four replicas on the 24 GB card, and the second Granite pod promptly died with CUDA out of memory because two 16 GB models will never share 24 GB. The fix was not a flag. We moved the assistant to a real A100 node and gave it a 3g.20gb slice, and the noisy-neighbour problem disappeared with the hardware wall.

Where GPU sharing breaks

Five symptoms cover nearly every sharing failure I have hit. Keep this lookup by the terminal.

One more that wastes an afternoon: after moving to mixed MIG, existing InferenceServices that request nvidia.com/gpu: 1 go unschedulable, because that plain resource no longer exists on a fully partitioned node. Every workload on a mixed node must ask for a specific MIG profile, and the hardware sizing that tells you which profile is the same KV-cache math from Part 12.

Give the assistant a MIG slice, not a whole card

References

• Red Hat OpenShift AI 3.4, About GPU time slicing
• NVIDIA GPU Operator, MIG support in OpenShift Container Platform
• NVIDIA Multi-Instance GPU User Guide, supported MIG profiles

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts