AI infrastructure · Kubernetes · NVIDIA GPU Operator · Prometheus · Observability

Kubernetes knows that a pod asked for one GPU. It does not know if that GPU is busy, overheating, out of memory or dead. That information sits inside the card, and you have to bring it out yourself. This post shows how the pieces fit, which numbers matter, and the four failures you will see first. I also built a live demo so you can break a cluster and watch the monitoring react.

The problemWhy Kubernetes alone is not enough

When you install the NVIDIA device plugin, each GPU node advertises a resource called nvidia.com/gpu. A pod asks for one, two or four of them, and the scheduler finds a node with that many free. That is the whole contract. To Kubernetes a GPU is a counter, like a slot in a car park.

So Kubernetes can tell you how many GPUs are handed out. It cannot tell you what happens to them after that. A pod can hold an H100 for a week and use none of it. A card can run at 88 °C and quietly slow down your training job. A GPU can drop off the PCIe bus, and the only sign is a line in the kernel log. You need three layers of information, and they come from different places.

The pipelineHow the numbers get from the card to your dashboard

The standard setup has four parts. The NVIDIA GPU Operator installs most of them for you: the driver, the container toolkit, the device plugin and the DCGM Exporter. You add Prometheus and Grafana, usually with the kube-prometheus-stack Helm chart.

Two details are easy to miss. First, the exporter runs as a DaemonSet, one pod per GPU node, and only on nodes that have GPUs. Second, Prometheus only scrapes it if a ServiceMonitor exists and the Prometheus instance is allowed to pick it up. That second point is the usual reason people install everything and see no GPU metrics at all.

The numbersSix metrics worth a panel each

The DCGM Exporter publishes dozens of fields. These six answer most questions. Every name below is the real Prometheus series name, and every one is shown live in the demo.

The utilisation trap

GPU_UTIL answers “was anything running?”, not “how much of the card was used?”. One small kernel running all the time scores 100%, even if it touches a tiny part of the chip. For a truer picture, the exporter also publishes profiling fields on data-centre GPUs, such as DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE.

The failuresFour things you will see in your first month

Each of these is a button in the demo. Read the card, then open the demo and trigger it.

Which Xid errors mean broken hardware

Xid codes come from the NVIDIA driver and also appear in dmesg. Some point at your code, some at the card. A short list that covers most cases:

The setupBuild it on your own cluster

This assumes a Kubernetes cluster with at least one NVIDIA GPU node and Helm installed. The steps are in order: each one needs the one before.

1. Install Prometheus and Grafana

   The last flag lets Prometheus pick up ServiceMonitors from other Helm releases, including the GPU Operator’s. Without it you get no GPU metrics.

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install kps prometheus-community/kube-prometheus-stack \
     -n monitoring --create-namespace \
     --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

2. Install the GPU Operator with its ServiceMonitor on

   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
   helm repo update
   helm install gpu-operator nvidia/gpu-operator \
     -n gpu-operator --create-namespace --wait \
     --set dcgmExporter.serviceMonitor.enabled=true

   If your nodes already have the NVIDIA driver installed, add --set driver.enabled=false.

3. Check that the exporter is publishing

   kubectl -n gpu-operator port-forward svc/nvidia-dcgm-exporter 9400:9400 &
   curl -s localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL

   You should see one line per GPU, with gpu, UUID, Hostname and, for GPUs in use, pod and namespace labels.

4. Add the alert rules

   These are the four rules the demo uses. Apply them with kubectl apply -f gpu-alerts.yaml. The release: kps label is what the Prometheus from step 1 looks for.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: gpu-alerts
     namespace: monitoring
     labels:
       release: kps
   spec:
     groups:
     - name: gpu
       rules:
       - alert: GPUXidError
         expr: DCGM_FI_DEV_XID_ERRORS > 0
         labels: {severity: critical}
         annotations:
           summary: "Xid {{ $value }} on {{ $labels.Hostname }} GPU {{ $labels.gpu }}"
       - alert: GPUThermalThrottle
         expr: DCGM_FI_DEV_GPU_TEMP > 85
         for: 5m
         labels: {severity: warning}
       - alert: GPUAllocatedButIdle
         expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[30m]) < 5
         for: 30m
         labels: {severity: info}
       - alert: GPUPodUnschedulable
         expr: kube_pod_status_unschedulable == 1
         for: 10m
         labels: {severity: warning}

5. Import a dashboard

   In Grafana, go to Dashboards → New → Import and enter ID 12239, the NVIDIA DCGM Exporter dashboard. It is a good start. Add a panel per team using sum by (namespace) (DCGM_FI_DEV_GPU_UTIL) so people can see their own usage.

Related on this site: AI Infrastructure Lab Workbook (section F covers observability hands-on) and sizing GPUs for Kubernetes.