← Back to the article: How to monitor GPUs in Kubernetes

GPU Cluster Watch

Kubernetes GPU monitoring demo · DCGM Exporter → Prometheus → Grafana
t+0s

A simulated Kubernetes cluster with 12 GPUs on 3 nodes. Every number on this page is what the NVIDIA DCGM Exporter would publish to Prometheus. Trigger a scenario below and watch the metrics, alerts and events react.

How the data gets here

GPU + driverNVML counters on each card
→
DCGM ExporterDaemonSet on every GPU node, :9400/metrics
→
PrometheusServiceMonitor scrape, plus kube-state-metrics
→
Grafana · AlertmanagerDashboards and alert rules (this page)

Installed together by the NVIDIA GPU Operator. The exporter reads the kubelet pod-resources API, so each GPU series carries pod and namespace labels. Demo clock: 1 tick = 1 scrape; real scrapes are usually every 15–30 s.

GPU map · click a card

GPU utilisation0%100% THROTTLEXID
—

Scenarios

Alerts

kubectl get events