AI Infrastructure series · Part 4 of 24
GPU makers advertise trillions of operations per second, but when a chatbot answers one person, an H100 uses less than 1% of that. The rest of the time it is waiting for memory. This post explains when a GPU is limited by memory and when by maths, why serving many users at once changes everything, and what that means when you choose hardware.
Change the number of users, conversation length, model and GPU. See whether the GPU is waiting for memory or for maths, how many words each user gets, and when the KV cache runs out of VRAM.
What are the two limits on GPU speed?
Every piece of GPU work needs two things: numbers brought from memory, and maths done on them. Whichever takes longer sets the speed.
The maths is quick but the numbers arrive slowly. The cores finish and wait. Speed depends on memory bandwidth: 3.35 TB/s on an H100.
The numbers arrive fast enough and the cores are busy all the time. Speed depends on maths speed: 989 TFLOPS in BF16 on an H100.
To tell which one you are in, count how many operations you do for each byte you read. This is called arithmetic intensity. Divide the GPU’s maths speed by its memory speed and you get the break-even point, often called the ridge:
Below about 295 operations per byte, an H100 waits for memory. Above it, it waits for maths. The demo draws this as a “roofline”: a sloped line where memory is the limit, then a flat roof where maths is.
Why is serving one user so wasteful?
When a model writes one word for one user, it reads every weight once and does two operations with each: a multiply and an add. Each weight is 2 bytes in 16-bit precision, so that is about one operation per byte. The ridge is at 295. The GPU is nearly 300 times below the point where its Tensor Cores would be fully used.
Part 3 put numbers on it for an 8B model: about 0.016 ms of maths and 4.8 ms of reading memory for each word. The world’s fastest AI chips spend over 99% of a single-user conversation waiting.
How does batching fix it?
The fix is to serve many users in the same step. The weights are read once, and every user in the batch uses them. Reading cost stays almost the same while useful work grows with the number of users. Serving engines such as vLLM do this constantly, adding and removing users from the batch as they come and go.
Here is an 8B model on one H100, with conversations of 2,000 tokens, as the number of users grows. These are upper limits from memory and maths speed:
| Users at once | Total output | Words a second, each user | Tensor Cores busy | VRAM needed |
|---|---|---|---|---|
| 1 | 206 tokens/s | 154 | under 1% | 16 GB |
| 8 | 1,481 tokens/s | 139 | 2% | 18 GB |
| 32 | 4,395 tokens/s | 103 | 7% | 24 GB |
| 128 | 8,653 tokens/s | 51 | 14% | 50 GB |
| 512 | 11,418 tokens/s | 17 | 18% | 150 GB: does not fit |
Total output rises 42 times from 1 user to 128, on the same GPU. Each user gets words more slowly, but 51 words a second is still far faster than anyone reads. That trade, a little speed for each person in exchange for many more people served, is the basic economics of running AI.
Notice two other things in the table. The Tensor Cores never get anywhere near fully busy. And at 512 users the model no longer fits in VRAM. Both come from the same cause.
Where does the KV cache come in?
Every user in the batch has their own KV cache: the stored working memory of their conversation so far. For this 8B model it is 128 KB for every token. A 2,000-token conversation needs 256 MB of it, and the GPU has to read each user’s cache on every step.
Weights are shared, so batching makes them cheap per user. KV cache is not shared, so it grows with every user and every token of conversation. At 128 users, the KV cache is already about twice the size of the weights. That is why long conversations hurt throughput so much, and why serving engines work hard to manage KV cache memory. Part 14 of this series works out its size in detail.
The weights cost the same however many users there are. The KV cache grows with every user, so with many users it becomes most of what the GPU reads.
Which AI work is compute-bound?
Not everything waits on memory. Work that handles many tokens at once does lots of maths for each weight it reads:
- Reading the prompt (prefill). A 2,000-token prompt is processed in one go, so each weight read is used about 2,000 times. That is well above the ridge, so prefill keeps the Tensor Cores busy. Part 13 covers prefill and decode.
- Training. Training pushes large batches of long sequences through the model, so it is mostly compute-bound. This is why training benchmarks quote TFLOPS and serving benchmarks care about bandwidth.
- Writing the answer (decode). One new token per user per step. Memory-bound unless the batch is very large, and the KV cache usually runs out of room first.
How do newer GPUs change the picture?
Compare GPUs on both numbers, not just TFLOPS:
| GPU | BF16 Tensor speed | Memory bandwidth | VRAM | What it means for serving |
|---|---|---|---|---|
| H100 SXM | 989 TFLOPS | 3.35 TB/s | 80 GB | The baseline |
| H200 | 989 TFLOPS | 4.8 TB/s | 141 GB | Same maths, 43% more bandwidth and much more room for KV cache. Writing answers gets faster with no change to the cores. |
| B200 | about 2,250 TFLOPS | 8 TB/s | 192 GB | More of both. Bandwidth grows 2.4 times over the H100. |
The H200 is the clearest proof that bandwidth matters. It has the same compute as the H100 but more and faster memory, and it is marketed mainly as an inference upgrade. For serving chat, a GPU with more bandwidth and VRAM often beats one with more TFLOPS.
How do you make better use of bandwidth?
Every technique for faster AI serving does one of three things: reads fewer bytes, does more useful work per byte, or avoids reading the same thing twice.
Batch users together. Read the weights once for many users. This is the single biggest lever, and serving engines such as vLLM do it automatically.
Use fewer bytes per weight. 8-bit or 4-bit quantization halves or quarters the read for every word. Part 18 covers it.
Keep the KV cache small. Shorter context where possible, models with fewer KV heads, and smart cache memory management. Parts 14 and 17 cover these.
Reuse cached prompt text. If many requests start the same way, compute that start once and reuse it. Cache-aware routing, covered in the llm-d post, makes this work across many GPUs.
Choose hardware on bandwidth. For serving, compare TB/s and GB of VRAM before TFLOPS.
- A GPU is either waiting for memory or waiting for maths. The ridge, about 295 operations per byte on an H100, divides the two.
- Writing an answer for one user is about one operation per byte, so the GPU mostly waits for memory.
- Batching many users reads the weights once for all of them, multiplying total output on the same GPU.
- The KV cache grows with users and conversation length, and eventually limits batching by both bandwidth and VRAM.
- Prefill and training are compute-bound; decode is memory-bound. For serving, bandwidth and VRAM often matter more than TFLOPS.
GPU figures are NVIDIA’s published peaks (dense, no sparsity). Throughput figures are upper limits for a Llama-style 8B model in 16-bit with 8 KV heads, ignoring attention maths and software overheads; real systems reach less.
About The Author
DrJha