GPU Bandwidth Lab

AI Infrastructure series · part 4 · memory-bound or compute-bound?

For every word it writes, a model reads its weights from memory and does a little maths with them. With one user, the maths is tiny and the GPU spends almost all its time waiting for memory. Serve many users together and the same weights are used for all of them, so the maths grows while the memory read barely changes. Move the sliders and watch which one sets the speed.

Setup

GPU

Model (16-bit)

Users served at once

Conversation length per user

Try this

Time for one step (one new word for every user)

Roofline: work done per byte read

limit set by memory bandwidthlimit set by Tensor Core speedyour setup

Model: Llama-style 8B (32 layers) and 70B (80 layers), 16-bit weights, 8 KV heads of 128. Each step reads all weights once plus every user's KV cache, and does 2 operations per weight per user. Peak speeds from NVIDIA: H100 989 TFLOPS BF16 and 3.35 TB/s; H200 989 TFLOPS and 4.8 TB/s; B200 2,250 TFLOPS and 8 TB/s. Attention maths and software overheads are left out, so real numbers are lower.