Our finance lead asked one question in the platform review: if four teams share the same GPU cluster, what stops one of them from eating the whole thing on a Friday and starving the support assistant. It is a fair question, and the honest first answer is nothing. A fresh data science project on OpenShift AI ships with no quota. Last part we cut one A100 into two hardware isolated halves and gave one to the assistant. This part gives those halves accountable owners, a project with a GPU budget it cannot exceed and a fair queue that decides who runs when the budget is full, so the Friday scenario ends in a wait instead of an outage.

Projects, quotas and who controls what

Three mechanisms get lumped together as multi-tenancy, and each answers a different question. Confusing them is the single most common reason a tenancy model leaks. RBAC is about identity and action, a ResourceQuota is about total consumption, a LimitRange is about defaults for a pod that asks for nothing, and Kueue is about ordering work when the pool is contended. Keep this table where you plan a new tenant, because the wrong tool in the wrong slot is where budgets quietly escape.

Read that last column carefully, because it holds the difference that matters most. A ResourceQuota rejects: a pod that would push the namespace over its cap never gets created, and the workload owner sees an error. Kueue queues: a job that cannot fit right now is suspended and admitted later when room appears. For a serving pod you want the hard rejection, since you never want the assistant scaled beyond its budget. For a training run you want the queue, since a tuning job is happy to wait twenty minutes rather than fail outright.

Turning a namespace into a governed project

Start with the floor every project needs, a ResourceQuota. This one caps the assistant project at two GPUs, sixteen CPU cores of requests and 64 GB of memory. Note that a quota tracks both requests and limits separately, and for an extended resource like a GPU, OpenShift requires the request and the limit to be equal, so you cap both. Versions I tested against are in the first comment.

# Tested on OpenShift AI 3.4 self managed, OpenShift 4.16, Red Hat build of Kueue [VERIFY z-stream]
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-support-quota
  namespace: support-assistant
spec:
  hard:
    requests.cpu: "16"
    requests.memory: 64Gi
    requests.nvidia.com/gpu: "2"
    limits.nvidia.com/gpu: "2"
    count/pods: "40"

$ oc apply -f team-quota.yaml
resourcequota/team-support-quota created
$ oc get resourcequota team-support-quota -n support-assistant
NAME                 AGE   REQUEST                                                          LIMIT
team-support-quota   9s    requests.cpu: 3/16, requests.memory: 12Gi/64Gi,
                           requests.nvidia.com/gpu: 1/2                                     limits.nvidia.com/gpu: 1/2

One of the two GPUs is already spoken for by the assistant serving pod on its MIG slice, so the project shows 1 of 2 used. A second team gets its own project with its own quota, and neither can reach into the other. Here is the part the dashboard will not tell you: a data science project created through the OpenShift AI dashboard gets no ResourceQuota at all by default. Multi-tenancy is not the out-of-the-box state, it is something you add on purpose to every namespace, and a cluster that has been growing projects organically almost certainly has several with no cap.

Default requests with a LimitRange

Apply that quota and the next workbench launch fails, which surprises everyone the first time. Once a ResourceQuota tracks a resource, every pod in the namespace must declare a matching request, or admission rejects it outright.

$ oc apply -f granite-notebook.yaml
Error from server (Forbidden): error when creating "granite-notebook.yaml":
pods "granite-nb" is forbidden: failed quota: team-support-quota:
must specify limits.memory,requests.cpu,requests.memory

A workbench started from the dashboard often omits explicit CPU and memory requests, and with a quota in force that is fatal. Rather than forcing every user to hand-write requests, drop a LimitRange into the project. It injects a default request and limit into any container that leaves them blank, so the quota has a number to count and the pod is admitted.

apiVersion: v1
kind: LimitRange
metadata:
  name: team-support-defaults
  namespace: support-assistant
spec:
  limits:
    - type: Container
      default:            # becomes the limit if none is set
        cpu: "2"
        memory: 4Gi
      defaultRequest:     # becomes the request if none is set
        cpu: "1"
        memory: 2Gi
---
$ oc apply -f limitrange.yaml && oc apply -f granite-notebook.yaml
limitrange/team-support-defaults created
pod/granite-nb created

Fair sharing with Kueue queues

A ResourceQuota is a wall, and walls do not queue. Two teams that each want three GPUs from a pool of four will both get hard rejections the moment the pool is full, with no notion of whose job should run first. Kueue fills that gap. It is an upstream Kubernetes project that Red Hat packages, tests and supports as the Red Hat build of Kueue, and it introduces three objects: a ResourceFlavor describing a class of hardware, a ClusterQueue holding a pool of quota across flavors, and a LocalQueue in each project that points at a ClusterQueue. Jobs carry a label naming their LocalQueue, and Kueue suspends them until their share is free.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: default-flavor
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-cluster-queue
spec:
  namespaceSelector: {}          # which namespaces may submit here
  cohort: shared-gpu             # queues in one cohort can borrow idle quota
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: default-flavor
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: "2"
            - name: "cpu"
              nominalQuota: "16"
            - name: "memory"
              nominalQuota: 64Gi
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: team-support-queue
  namespace: support-assistant
spec:
  clusterQueue: gpu-cluster-queue

Submit a training job with the label kueue.x-k8s.io/queue-name: team-support-queue and Kueue takes over its admission. If both GPUs are busy, the job does not fail, it waits.

$ oc get pytorchjob tune-granite -n support-assistant
NAME           STATE
tune-granite   Suspended
$ oc describe workload pytorchjob-tune-granite-9f2c1 -n support-assistant
Status:
  Conditions:
    Reason:   Pending
    Message:  couldn't assign flavors to pod set worker:
              insufficient unused quota for nvidia.com/gpu in flavor
              default-flavor, 1 more needed

Both GPUs in this ClusterQueue are in use, one by the serving pod and one by an earlier job, so the tuning run sits Suspended with a clear reason instead of dying. This is where the cohort earns its place. Give the analytics and research ClusterQueues the same shared-gpu cohort, and a team that is idle lends its unused GPUs to a team that is busy, up to a borrowing limit you set. The assistant can borrow one idle GPU from research at peak, then give it back when research needs it. Fair sharing across teams is the distributed workloads model on OpenShift AI, and it is described end to end in the docs linked below.

flowchart LR
  J[training job with queue name label] --> L[LocalQueue in project]
  L --> Q[ClusterQueue checks nominal quota]
  Q -->|quota free| RQ[ResourceQuota admission in namespace]
  Q -->|quota full| S[job suspended and queued]
  Q -->|cohort has idle GPUs| B[borrow from sibling queue]
  RQ -->|requests within cap| N[pods scheduled on GPU node]
  RQ -->|no requests set| F[rejected, needs LimitRange]

Sharing a project without giving away the cluster

Access is the third leg, and it is orthogonal to the first two. Granting a colleague the edit role on a project lets them run workbenches and deploy models in it, and granting admin lets them manage access within it. Neither role lets them raise the GPU budget, and that separation is the whole design.

$ oc adm policy add-role-to-user edit user2 -n support-assistant
clusterrole.rbac.authorization.k8s.io/edit added: "user2"
$ oc adm policy add-role-to-group admin support-platform-team -n support-assistant
clusterrole.rbac.authorization.k8s.io/admin added: "support-platform-team"

# user2, a project admin, tries to raise their own GPU quota:
$ oc patch resourcequota team-support-quota -n support-assistant 
    --type merge -p '{"spec":{"hard":{"requests.nvidia.com/gpu":"8"}}}'
Error from server (Forbidden): resourcequotas "team-support-quota" is forbidden:
User "user2" cannot patch resource "resourcequotas" in the namespace "support-assistant"

That forbidden line is the good kind of failure. A project admin owns everything inside the project except the size of its budget, because the ResourceQuota is written by a cluster-admin who sits above the namespace. Here is the belief that trips people: many read admin as unlimited. Namespace admin is scoped to the namespace, and the quota is set one level up, so a team can be fully autonomous inside its project and still be unable to spend a GPU it was not given. If you find yourself using RBAC to try to limit consumption, you have reached for the wrong tool. RBAC gates actions, ResourceQuota gates amounts.

Where multi-tenancy leaks

Even a project with a quota, a LimitRange and a queue can leak, usually through the gap between what the quota names and what the node advertises. Keep this lookup by the terminal.

That Friday the finance lead worried about did arrive, just not the way anyone guessed. We had put a ResourceQuota of two GPUs on every project and felt covered. Then we moved the inference nodes to mixed MIG from Part 18, and the quota went blind, because the pods now requested nvidia.com/mig-3g.20gb while our quota only named nvidia.com/gpu. An analytics batch job spun up nine MIG slices across three nodes over about twenty five minutes, the assistant tuning job went Pending, and the dashboard cheerfully reported 0 of 2 GPUs used in every project. The quota was real and completely blind. The fix was two lines per project, adding requests.nvidia.com/mig-3g.20gb and its limit, and a rule I have kept since: write the quota against whatever resource the node actually advertises, not the one you assume it does. The economics behind why a stray nine slices matters are in the Data Science Series on GPU cost, scale and sizing, and the broader self hosted bill in the GenAI Series cost breakdown.

Put a quota on every project before Monday

References

• Red Hat OpenShift AI, Managing distributed workloads with Kueue
• Red Hat Developer, Implement GPU-as-a-Service with Kueue and NVIDIA MIG
• OpenShift Container Platform, Resource quotas per project

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts