AI infrastructure · Kubernetes · Kueue · Quotas · Gang scheduling
Once several teams share a GPU cluster, the question changes from “is the GPU healthy?” to “who gets it?”. The default Kubernetes scheduler was built for web services: it places pods one at a time, first come first served, with no idea that eight pods belong to one training job. This post shows the three ways that goes wrong with GPUs and how Kueue fixes them with queues, quotas, borrowing and preemption. The live demo lets you watch both side by side.
Three teams, 16 GPUs. Submit two 8-GPU training jobs in plain Kubernetes and watch them deadlock. Switch to Kueue and do it again, then let a team reclaim its quota.
The problemThree ways the default scheduler fails GPU work
Two training jobs each need two 4-GPU pods. Each gets one pod placed, neither can start, and 8 GPUs sit held and idle. Nothing in Kubernetes notices.
One busy team can take every GPU. A ResourceQuota stops that, but it is a hard wall: idle quota in one namespace cannot be lent to another.
Pending pods are just retried. Small jobs keep slipping into free gaps, and the big job that needs a whole node can wait forever.
The fixHow Kueue works
Kueue is a Kubernetes project that sits in front of the scheduler. It does not place pods itself. It decides when a whole job may start, then lets the normal scheduler place its pods.
The job’s pods do not exist until Kueue admits it, so a waiting job never holds GPUs.
ResourceFlavorA kind of hardware, such as “h100” or “l40s”, usually matched to node labels.
ClusterQueueA team’s share: how many GPUs of each flavor it is guaranteed, and its rules for queueing and preemption.
LocalQueueThe queue a team submits to from its own namespace. It points at a ClusterQueue.
CohortA group of ClusterQueues that can lend each other unused quota.
Quota you can borrow, and take back
This is the part a ResourceQuota cannot do. Each team has a guaranteed share, called its nominal quota. When a team is quiet, others may borrow its share. When it needs its GPUs back, Kueue preempts the borrowers.
The red line on each bar is the team’s own quota. When Platform submits a 2-GPU job, it is within its quota, so Kueue preempts Research’s borrowed work to give the GPUs back. The demo shows exactly this.
Side by sideSame jobs, two outcomes
| What happens | Plain Kubernetes | Kueue |
|---|---|---|
| Two 8-GPU jobs, room for one | Both half-placed, 8 GPUs idle | One admitted, one waits holding nothing |
| A quiet team’s GPUs | Free for whoever asks first | Lent out, taken back when needed |
| Team over its share | No limit, or a hard wall with ResourceQuota | May borrow, gets preempted first |
| Urgent job | Needs PriorityClass and pod preemption | Workload priority, preempts lower priority in its queue |
| Why is my job waiting? | “Insufficient nvidia.com/gpu” | “Waiting for quota: would use 18 of 12” |
waitForPodsReady so a job that cannot get all its pods running in time is evicted and requeued, and look at its topology-aware scheduling for jobs that need GPUs close together.AlternativesOther ways to queue GPU work
Kubernetes SIG project. Works with the normal scheduler and with Job, JobSet, Kubeflow, Ray and plain pods. Quotas, borrowing, priorities.
A replacement scheduler built for batch and AI. Strong gang scheduling and queues. More to run, since it replaces the default scheduler for those pods.
Hierarchical queues and fair sharing, popular with teams coming from Hadoop and Spark.
GPU-focused scheduling with fractional GPUs, quotas and dashboards. A product you buy, not a project you run.
The setupThree teams, one cohort
-
Install Kueue
# pick the current release from github.com/kubernetes-sigs/kueue/releases kubectl apply --server-side -f \ https://github.com/kubernetes-sigs/kueue/releases/download/<version>/manifests.yaml -
Describe the hardware and each team’s share
apiVersion: kueue.x-k8s.io/v1beta1 kind: ResourceFlavor metadata: name: h100 spec: nodeLabels: nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3 --- apiVersion: kueue.x-k8s.io/v1beta1 kind: ClusterQueue metadata: name: research spec: cohort: gpu-pool namespaceSelector: {} queueingStrategy: BestEffortFIFO preemption: reclaimWithinCohort: Any withinClusterQueue: LowerPriority resourceGroups: - coveredResources: ["nvidia.com/gpu"] flavors: - name: h100 resources: - name: nvidia.com/gpu nominalQuota: 8 # product: 6, platform: 2Repeat the ClusterQueue for each team with its own quota and the same cohort.
-
Give each namespace a queue
apiVersion: kueue.x-k8s.io/v1beta1 kind: LocalQueue metadata: name: research namespace: research spec: clusterQueue: research -
Point jobs at the queue
apiVersion: batch/v1 kind: Job metadata: name: llama-train-1 namespace: research labels: kueue.x-k8s.io/queue-name: research spec: parallelism: 2 completions: 2 template: spec: containers: - name: trainer resources: limits: nvidia.com/gpu: 4Kueue suspends the job until it is admitted.
kubectl get workloads -n researchshows why it is waiting.
- Every GPU namespace submits through a LocalQueue, and GPU pods without a queue are blocked by policy.
- Team quotas add up to the GPUs you actually have.
- Teams are in one cohort so idle GPUs get lent, and they know borrowed work can be preempted.
- Training jobs checkpoint, so preemption costs minutes, not days.
waitForPodsReadyis on, so a half-placed job gets requeued instead of holding GPUs.- Your dashboards show quota used and borrowed per team, next to GPU utilisation.
Open GPU Queue Lab, submit the two training jobs in plain Kubernetes, and watch the idle GPU-seconds climb. Then switch to Kueue and try the Platform job.
The series so far: monitoring, logging, node lifecycle, sharing and this post on scheduling. Next: what all of this costs.
About The Author
DrJha