Workload presets
Model and memory
Model
KV cache precision
GPUs
Conversation length
Users at once
What you are looking at: for every token in every conversation, each layer of the model stores a key and a value so they never have to be recomputed. That store is the KV cache. It grows with every token and every user, and on a busy server it can need more GPU memory than the model itself. Pick a workload preset or set your own model, GPUs, conversation length and users.
Try this: start with the Chatbot preset, then switch to Long-document agent and watch users that fit collapse. Then pick "8B without GQA" to see what sharing key-value heads saves, and switch the cache to 8-bit to see the users double.
Shapes from the makers' published configurations. Llama 3 uses grouped-query attention with 8 key-value heads of 128. "8B without GQA" shows the same model if every one of its 32 heads kept its own keys and values. DeepSeek-V3 uses multi-head latent attention, storing a compressed 576-number latent per layer per token (about 69 KB in 16-bit). Weights are 16-bit except 405B and DeepSeek-V3 (8-bit); 10% of memory is kept free and activation memory is ignored, so real engines fit somewhat fewer users. Pre-reserved assumes each user is given room for the full 128K context up front. Presets are illustrative workloads.
What you are looking at: a few conversations with the model you pick. Each small tile is one token's key (dark half) and value (red half) in one layer. Prefill writes the whole prompt into every layer at once. Each decode step first reads every stored tile in every layer (outlined), then appends one new tile per layer for the token it just wrote. The counters show the real bytes for that model.
Try this: press Prefill, then Decode a few times and watch the read outline cover more tiles every step. Add users: each one gets its own cache, so bytes written per step multiply. Switch to DeepSeek-V3 and see the per-token number drop below the 8B model's.
Only a few layers are drawn; the counters use every layer. Real prompts are hundreds or thousands of tokens, so the scale-up line converts the same per-token cost to an 8,000-token conversation. Reads assume plain attention over the whole history in every layer, as in Llama 3; sliding-window layers would read at most their window.
What you are looking at: how attention design sets the cache. The diagram shows query heads (blue) sharing key-value heads for a Llama 3 8B shape: 32 layers, 32 query heads of 128 numbers. The bars compare one conversation's cache across real designs at the context length you choose: plain multi-head, grouped-query, multi-query, DeepSeek's latent attention, and gpt-oss's mix of full and sliding-window layers.
Try this: drag KV heads from 32 down to 1 and watch the cache fall in step: that is MHA, then GQA, then MQA. Then raise the context to 128K and widen the sliding window: the hybrid model's advantage holds at long context because its sliding layers stop growing.
Configurations: Llama 3 8B (32 layers, 8 KV heads of 128); DeepSeek-V3 (61 layers, 128 heads, a 512 + 64 number latent per layer; as plain MHA it would store 192-number keys and 128-number values for all 128 heads); gpt-oss-120b (36 layers, 8 KV heads of 64, alternating full and 128-token sliding-window layers). Window sizes other than 128 are what-if values. Bars longer than the scale are hatched.
What you are looking at: one decode step. The GPU reads all the weights once for the whole batch, then every request's KV cache. Both are limited by memory bandwidth, so step time is roughly bytes read ÷ bandwidth. As history grows, the cache becomes a bigger share of every step and each user's tokens arrive more slowly.
Try this: with one user, slide the context from 1K to 128K and watch per-user speed halve on an H100. Then set 16 users at 8K: the step reads the same bytes as one user at 128K. Switch to FP8 cache or to an H200 and see how much speed comes back.
Ideal ceilings: step time = (weights + KV cache read) ÷ memory bandwidth, ignoring compute, kernel overheads and activations, so real engines are slower; the ratios are what matter. H100 SXM 80 GB at 3.35 TB/s, H200 141 GB at 4.8 TB/s, B200 192 GB at 8 TB/s. 90% of memory is treated as usable for the fit check.