In a private AI environment the GPU is the part that costs the most, runs the hottest, and gets fought over the most. A handful of accelerators end up shared across training jobs, inference endpoints, notebooks, and embedding services. When something slows down or a job dies, the first question is almost always the same: what is the GPU actually doing right now? This post answers two practical questions that come up constantly. Which metrics tell you the truth about GPU health, and which tools you should reach for to see them.
Short answer
Track GPU monitoring at four levels: compute utilization, memory, thermal and power, and capacity. Watch compute utilization percent, memory usage percent, temperature, power draw, and the count of GPU-backed VMs as your front-line signals. For tools, pair a fleet-level operations dashboard (fed by NVIDIA DCGM) for the big picture with node-level utilities like nvidia-smi and nvitop for fast, close-up checks.
Why GPU monitoring deserves its own playbook
It is tempting to treat a GPU like any other resource and reuse the CPU and memory habits you already have. That gets you into trouble fast, for three reasons.
First, GPUs are shared in ways CPUs usually are not. One physical card gets carved into slices through vGPU profiles, time-slicing, or MIG, and several virtual machines or pods ride on the same silicon. A number that looks fine for the host can hide a single tenant starving the others.
Second, GPU memory is a hard wall. When a model does not fit, the job does not slow down gracefully, it dies with an out-of-memory error. So memory headroom is something you watch before it bites, not after.
Third, an idle GPU is rarely free. A model can be loaded into GPU memory and sit there answering nothing, holding the card hostage while compute utilization reads close to zero. On a CPU that pattern is cheap. On a GPU it is one of the most common forms of waste in the building.
Think in layers
The cleanest way to reason about GPU telemetry is to follow the resource from the metal up to the workload. Each layer answers a different question, and a healthy monitoring setup gives you all of them.
What tool do you use to monitor GPU resources?
There is no single tool that does everything well, and you should not pretend otherwise. The right setup is a fleet-level operations console for the wide view, with node-level utilities underneath it for close inspection. Here is how the pieces fit.
The fleet view: an operations dashboard backed by DCGM
The center of gravity is a private cloud operations console with a set of GPU-focused dashboards. Underneath those dashboards sits NVIDIA Data Center GPU Manager, usually shipped as the dcgm-exporter that the GPU operator installs on every GPU node. DCGM is the part that actually reads the hardware counters. The console collects that telemetry across every host and presents it as dashboards you can sort, filter, and trend over hours or days.
In practice the GPU story is split across four dashboards, each answering a different question.
| Dashboard | The question it answers | What you read off it |
|---|---|---|
| GPU Overview | How is the whole fleet of GPUs doing right now? | Total GPU count, a temperature heat grid, memory usage and compute utilization tiles per card, colored by threshold. |
| GPU Equipped Clusters | Which clusters and hosts have GPUs, and how loaded are they? | Per-cluster GPU count, compute and memory usage, host-by-host breakdown, and the vGPU profile in use. |
| GPU Consumers | Who is actually using the GPUs? | Every VM with a GPU attached, its profile, and its compute usage, plus totals like VMs-with-GPU versus total VMs. |
| GPU Providers | What is the supply side doing over time? | Average GPU and memory utilization trends, so you can see demand build across the day. |
The split matters. The Overview tells you if anything is on fire. Equipped Clusters tells you where capacity sits. Consumers tells you who to talk to when a card is pinned. Providers tells you whether you are running out of room. You will jump between them constantly.
The close-up view: node and pod tools
The fleet console is the wrong tool when you want to know what a single GPU is doing this second, or what a specific training job is doing to the card it landed on. For that you drop down to the node.
| Tool | Scope | Reach for it when |
|---|---|---|
| nvidia-smi | One host or one pod, point in time | You want a quick snapshot: which processes hold memory, current utilization, driver and CUDA version. The first command you run inside a GPU pod. |
| nvitop | One host or pod, live | You want a moving picture during a job. It refreshes continuously and is excellent to watch while a training run climbs. |
| dcgm-exporter | Per GPU, continuous | You are the data source, not the consumer. It feeds the fleet console and can also feed Prometheus. |
| Prometheus + Grafana | Fleet, custom | You want your own dashboards and alert rules scraped straight from dcgm-exporter, independent of the operations console. |
| TensorBoard | One workload | You care about the job, not the hardware: training loss, learning rate, throughput per step. |
A simple rule that keeps you sane: use the operations console to decide where to look, and a node tool to decide what is wrong. The console finds the hot or idle card across hundreds of VMs. nvidia-smi or nvitop tells you which process is responsible once you are on the box.
What metrics do you use to monitor GPU resources?
Metrics fall into four buckets: utilization, memory, thermal and power, and capacity. Read across all four and you rarely get surprised. Here is the working set, with what each one is really telling you.
Utilization
| Metric | Unit | What it tells you | What to watch for |
|---|---|---|---|
| Compute Utilization | % | How much of the GPU’s compute is busy right now. | Sustained near 100% means the card is the bottleneck. Sustained low single digits with high memory means an idle, loaded model. |
| GPU Compute Usage | MHz | Actual clock work being done. | Compare against the limit to see how much room is left. |
| GPU Compute Demand | MHz | How much work is being requested. | Demand above usage is a sign of contention: more is wanted than the card can give. |
| GPU Compute Limit | MHz | The ceiling for this GPU or slice. | The reference line for everything above. |
Memory
| Metric | Unit | What it tells you | What to watch for |
|---|---|---|---|
| Memory Usage | % | How full the GPU framebuffer is. | Above 90% you are one large batch away from an out-of-memory failure. |
| Memory Used / Reserved | GB | How much memory a workload is holding. | Memory held with no compute is the classic idle-but-loaded model. Reclaim it. |
| GPU Memory Demand | KB | How much memory is being requested. | Rising demand near total memory predicts an OOM before it happens. |
| Total Memory | GB | The card or slice capacity. | Your denominator for sizing models and profiles. |
Thermal and power
| Metric | Unit | What it tells you | What to watch for |
|---|---|---|---|
| Temperature | °C | Thermal health of the card. | Past the mid-80s many GPUs throttle, so a hot card quietly gets slower. Treat high temperature as an alert, not a trend. |
| Power Used | Watt | How hard the card is working in electrical terms. | Sitting near the power cap for long stretches is normal under load but matters for cooling and capacity planning. |
Capacity and inventory
| Metric | Unit | What it tells you | What to watch for |
|---|---|---|---|
| Number of GPUs | count | How many cards a host or cluster has. | Your supply. The denominator for all utilization math. |
| Number of vGPU Configured VMs | count | How many VMs currently claim a GPU slice. | A step up means a new workload landed. A high count against few cards means heavy sharing. |
| Configured Type / Active Type | profile | The vGPU sharing mode in play, such as a shared-direct or time-sliced profile. | A mismatch between configured and active type points at a placement or driver issue. |
Reading the dashboards: real situations
Numbers are only useful when you know what story they tell. Here are patterns that come up again and again, drawn from a small estate of seven GPUs spread across four hosts in one GPU cluster, with roughly two dozen GPU-backed VMs sharing them.
The idle but loaded GPU
A cluster shows GPU memory usage around 15% while compute utilization sits near zero. That is not a quiet cluster, it is a wasteful one. Models are loaded into memory and holding their slots while serving almost nothing. On the Consumers dashboard you find the VMs holding memory with flat compute, and you either consolidate them or unload the models. This single pattern is the most common GPU saving you will ever find.
The uneven fleet
Four hosts, and their GPU memory usage reads 54%, 0%, 29%, and 51%. One host is doing nothing while others carry the load. That points at placement: new GPU workloads keep landing on the same boxes. The fix lives in scheduling and affinity, but you only see the problem because the host-level breakdown exists. A cluster average would have hidden it at around 33%.
The training run, live
Kick off a fine-tuning job and watch a Kubernetes worker node appear in the Consumers list with its compute usage climbing into the double digits, while the cluster utilization line turns into a sawtooth of repeated bursts. That sawtooth is the training loop itself, each tooth a wave of work. If you want the close-up, open a terminal in the notebook and run nvitop to see memory and utilization move in real time. This is the moment where the fleet view and the node view agree, and it is satisfying to watch.
The hot card
The temperature grid shows one tile in red at the mid-80s while the rest sit in the 50s and 60s. Average temperature looks fine. That one card is throttling, so any job on it is quietly slower than it should be. You would never catch this from a single cluster number. You catch it because the Overview paints each card separately.
The capacity creep
The count of GPU-configured VMs steps up from three to four, and the Providers trend shows average utilization drifting higher each day. Nothing is broken, but the supply is filling up. This is the early signal to plan another GPU host before the next big workload arrives and there is nowhere to put it.
Turning metrics into alerts
Trends are for humans browsing dashboards. Alerts are for the things you cannot afford to miss. A reasonable starting point looks like this, then you tune it to your hardware.
| Signal | Healthy | Watch | Act |
|---|---|---|---|
| Compute Utilization | 40 to 95% | under 10% sustained (waste) | ~100% sustained with a growing queue |
| Memory Usage | under 80% | 80 to 90% | over 90%, OOM risk |
| Temperature | under 75°C | 75 to 83°C | over 84°C, throttling |
| Power Used | varies with load | near cap for long stretches | at cap with rising temperature |
Notice that two of these fire on the low side. An underused GPU does not page anyone at 3am, but over a month it is just as expensive as one that is broken. Build a weekly review for the waste signals and a pager for the failure signals.
Practical habits that pay off
- Separate reserved from used. A card can look busy because memory is held while doing no compute. That gap is your reclaimable capacity.
- Always read per card, not just per cluster. Averages hide the one hot or idle GPU that matters.
- Keep both views ready. The console finds the box, the node tool finds the process. Neither replaces the other.
- Alert on memory and temperature, trend on utilization. Memory and heat cause failures, utilization mostly causes waste.
- Right-size the vGPU profile. A workload on a slice far larger than it needs is capacity nobody else can use.
- Watch the consumer count as a capacity gauge. When GPU-backed VMs climb against a fixed card count, plan hardware before you are forced to.
So, the two answers in one breath. The metrics that matter are compute utilization, memory usage, temperature, power, and the count of GPU-backed workloads, read together across the physical card, the host, the slice, and the consumer. The tools are a fleet operations console fed by DCGM for the wide view, with nvidia-smi and nvitop at the node for the close-up, and TensorBoard when the question is really about the workload and not the hardware. Watch all four metric families, keep both the wide and the close view within reach, and the most expensive resource in your private AI estate stops being a mystery.
Part of the Private AI series. Vendor and product names belong to their respective owners. Hostnames and identifiers in examples are illustrative.
About The Author
DrJha