Model
Model
Weight format
KV cache format
Hardware
GPU
GPUs per copy
Engine memory utilisation
Traffic
Conversations at once
Tokens per conversation
What you are looking at: one GPU's memory while it serves a model, built from the bottom up: the CUDA context, this GPU's share of the weights, activations and CUDA graphs, then the KV cache that every conversation needs, all under the engine's utilisation line. If the model is split, each GPU holds an equal slice and pays its own overheads.
Try this: load 70B FP8 on one H100. The weights (70.6 GB) are smaller than the card, yet nothing is left for conversations. Now set GPUs per copy to 2 and watch the KV space appear. Press Play the load to see the order an engine fills memory in.
Per GPU: 1 GB CUDA context, 2.5 GB activation peak and 0.5 GB CUDA graphs (illustrative; measure your own). Weights: BF16 2 bytes, FP8 1 byte, 4-bit 0.5625 bytes per parameter (gpt-oss-120b 4-bit counted as MXFP4 at about 4.25 bits). KV per token in 16-bit: Llama 3.1 8B 128 KiB, 70B 320 KiB, 405B about 504 KiB, gpt-oss-120b about 72 KiB, DeepSeek-V3 about 69 KiB. Decimal GB throughout.
What you are looking at: the four jobs a GPU can do with a model. Running it needs the weights and the KV cache. Full training needs about 16 bytes per parameter plus activations. LoRA freezes the base and trains small adapters; QLoRA also stores the frozen base in 4-bit.
Try this: pick 70B and switch from Train fully to LoRA to QLoRA with the H100 selected: the GPU count falls from 19 to 3 to 1. Then pick Run it and raise users to 512 on the 8B model to see the KV cache take over.
Assumptions. Running: weights at the chosen precision (4-bit counted as 0.56 bytes with its scales), KV cache in 16-bit for Llama/Qwen-style models with 8 KV heads of 128, plus 5% working memory and 1 GB runtime per GPU. Full training: 16 bytes per parameter (16-bit weights and gradients plus 32-bit master weights and two Adam states) plus about 20% for activations with checkpointing. LoRA: frozen 16-bit base; QLoRA: frozen 4-bit base; both with small adapters and about 20% for activations. GPUs are filled to 92%. Estimates for planning, not guarantees.
What you are looking at: the fewest GPUs that hold each model with the chosen workload, each GPU filled to 90% with 4 GB of overhead, in splits of 1, 2, 4 or 8. Green fits with room, amber fits with under 5% spare, grey needs more than 8 GPUs (two servers or more). Click any cell to open it in the memory builder.
Try this: switch the format from BF16 to FP8 and watch most cells halve. Then choose the long documents workload: the 8B model, tiny in weights, now needs as much as a 70B model did, because the KV cache dominates.
What you are looking at: what happens when the weights do not fit and the rest is kept in CPU memory. Every token needs every weight, so the offloaded part crosses the CPU-GPU link on every token. The bars compare one GPU with offload against enough GPUs to hold the whole model in their own memory.
Try this: pick Llama 3.1 70B, BF16, one H100, then switch the link from PCIe 5.0 to NVLink-C2C. Then raise the batch: offload gets more total tokens per second because one trip over the link serves every request in the batch, but each user still waits just as long.
Ideal ceilings, illustrative: the GPU keeps as much of the model as fits in 90% of its memory after 4 GB of overhead and 2 GB of KV cache; time per token = GPU part ÷ HBM bandwidth + offloaded part ÷ link speed. Real offload is slower because copies and compute do not overlap perfectly. Link speeds: PCIe 4.0 x16 about 32 GB/s, PCIe 5.0 x16 about 64 GB/s, NVLink-C2C 900 GB/s as quoted by NVIDIA.