AI infrastructure · Kubernetes · NVIDIA GPU Operator · MIG · MPS

In the monitoring post, one of the first alerts was a GPU sitting idle while a pod held it. The cause is simple: Kubernetes hands out GPUs in whole numbers. A notebook that uses 4% of an H100 still gets the whole card. This post explains the three ways to share one GPU (time-slicing, MPS and MIG), what each one protects you from, and how to pick. There is a live demo where you can put the same workloads on one GPU four ways and break each setup.

The problemKubernetes counts GPUs in whole numbers

A pod asks for nvidia.com/gpu: 1. There is no 0.25. So every workload gets a full card, however little it uses. Here is a typical mix of four teams, each holding its own H100:

Four GPUs bought, less than one GPU of work done. At $2 to $4 per H100 hour, that is $6 to $12 an hour paid for idle silicon. Sharing fixes it, but each way of sharing gives up something different.

The optionsThree ways to share, plus the default

The testsThree things that go wrong, and which mode survives

Each of these is a button in the demo. Run them in every mode and the table above stops being theory.

MIG in detailThe slice sizes on an H100 80GB

An H100 has seven compute slices and eight memory slices of 10 GB. Each MIG profile is named after both: 3g.40gb means three compute slices and 40 GB. You can mix sizes on one card as long as they fit.

The number on the right is how many of that profile fit on one card if you use only that profile.

The choiceWhich one should you use?

The setupTurn it on with the GPU Operator

1. Write a sharing config

   Time-slicing and MPS both live in a device plugin config. This one turns each GPU into four.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: gpu-sharing-config
     namespace: gpu-operator
   data:
     any: |-
       version: v1
       sharing:
         timeSlicing:          # or: mps:
           renameByDefault: false
           resources:
           - name: nvidia.com/gpu
             replicas: 4

   With renameByDefault: true the shared resource is called nvidia.com/gpu.shared, so pods have to opt in to sharing on purpose. MPS support needs a recent device plugin.

2. Point the GPU Operator at it

   kubectl apply -f gpu-sharing-config.yaml
   kubectl patch clusterpolicies.nvidia.com/cluster-policy --type merge \
     -p '{"spec":{"devicePlugin":{"config":{"name":"gpu-sharing-config","default":"any"}}}}'

   After a minute, kubectl describe node shows nvidia.com/gpu: 4 per physical GPU, and GPU Feature Discovery adds a -SHARED suffix to the product label.

3. Or switch a node to MIG

   # once, in the ClusterPolicy
   kubectl patch clusterpolicies.nvidia.com/cluster-policy --type merge \
     -p '{"spec":{"mig":{"strategy":"mixed"}}}'
   
   # per node, after draining it
   kubectl label node gpu-node-d nvidia.com/mig.config=all-1g.10gb --overwrite

   The MIG manager applies the layout and sets nvidia.com/mig.config.state=success. The lifecycle post shows the drain that has to come first.

4. Ask for a share from a pod

   # time-slicing or MPS: same as before
   resources:
     limits:
       nvidia.com/gpu: 1
   
   # MIG: ask for the slice by name
   resources:
     limits:
       nvidia.com/mig-2g.20gb: 1

The series so far: monitoring, logging, node lifecycle and this post on sharing. Next: scheduling and queues.