Prefill and Decode Lab

What you are looking at: 30 users are already streaming answers, each with 2,000 tokens of history. Each mark is one token written for one of them. Then a new user's long prompt arrives. Without chunking its whole prefill runs as one step and everyone freezes; with chunked prefill one chunk rides along in each step; with separate prefill GPUs the decode GPUs never see it.

Try this: set the incoming prompt to 32,000 tokens and compare the freeze with the chunked lane. Then change the chunk size from 256 to 8,192: small chunks keep each step short but take more steps; big chunks get the newcomer to its first token sooner but slow everyone else more per step.

What you are looking at: a roofline for the chosen GPU. The x axis is arithmetic intensity: FLOPs done per byte read. Left of the ridge, memory bandwidth sets the speed; right of it, compute does. Prefill (red) does about as many FLOPs per byte as it has tokens. Decode (black) does about one per sequence in the batch, minus what the KV cache reads cost.

Try this: move the batch from 1 to 256 and watch the decode point climb the slope, then stall. Now cut the history to 200 tokens: with less KV cache to read, the same batch climbs much higher. Finally push the batch past the memory limit and see the warning.

What you are looking at: requests that start with the same text (system prompt, tool definitions, a shared document, or an agent's earlier conversation). With prefix caching, the KV blocks for that shared beginning are kept and reused, so only the new tail is prefilled. The hit rate is the share of requests that land on a GPU holding the prefix.

Try this: pick Coding agent: almost the whole prompt is shared, so a hit removes nearly all of the prefill. Then drop the hit rate to 10% (what round-robin across ten replicas gives you) and watch the saving vanish. Set the shared part to 0 to see a workload caching cannot help.

What you are looking at: speculative decoding for one user. A small draft model guesses the next few tokens; the big model checks them all in one pass, which costs about the same as one normal decode step because decode is limited by reading the weights, not by maths. Accepted guesses are free tokens; the first wrong guess is replaced by the big model's own token.

Try this: start at 70% acceptance and find the number of drafted tokens with the best speed-up. Then drop acceptance to 40% (creative writing) and see more drafting start to hurt. Switch the draft model to a larger one: better guesses cost more time per guess.