Scheduler
Speed
All three, same requests
Batch slots on the GPU
prompttokens generatedreserved but unused
What you are looking at: 60 requests are waiting, each with a different answer length. The GPU has room for 160 blocks of KV cache. Every step, each running request produces its next token, and the scheduler decides who runs. Pick a scheduler and press Run.
Try this: run Static batching and watch short requests leave empty, hatched slots behind. Then run Continuous + PagedAttention and compare the steps to finish in the table.
A simplified simulation. One step stands for 16 tokens per running request, and one block holds 16 tokens of KV cache. Prompts are 2 to 6 blocks and answers 2 to 24 steps, drawn from the same fixed random list for every scheduler. "Reserved" schedulers keep room for the longest possible request (30 blocks) from the start; the paged scheduler hands out blocks as they are used and stops admitting new requests above 90% use. At most 16 requests run at once. The idea follows the Orca paper (iteration-level scheduling) and the vLLM paper (PagedAttention); real engines also mix in prefill chunks, preempt requests and cache shared prompts.
What you are looking at: a tiny GPU with 24 KV blocks of 4 tokens each (vLLM uses 16; 4 keeps it readable). Five chat requests wait. A, B, C and E start with the same 8-token system prompt; D has a different one. Each running request has a block table listing the physical blocks that hold its tokens. Admit requests, then generate tokens, and watch blocks get allocated, shared and freed.
Try this: admit A, then B with prefix caching on: B's first two blocks are shared (striped, ref 2) and its prefill skips 8 tokens. Reset, turn caching off and do the same: B gets its own copies. With caching off, alternate Admit and Generate 1 token until all five are in: the pool runs out, a request is preempted, and the log says which. Count how many blocks caching saved.
Teaching model of vLLM's KV cache manager: full blocks get a hash chained from the previous block's hash and their own tokens; reference counts track sharing; freed blocks keep their hash until reused, and the oldest cached block is evicted first. When a running request needs a block and none is free, the newest running request is preempted and recomputed later, the default in vLLM's V1 engine. A preempted request keeps the tokens it already wrote; on readmission they are prefilled again with its prompt. Answer lengths are fixed per request.
What you are looking at: how vLLM fills each step's token budget (--max-num-batched-tokens). Running requests get 1 decode token each, first. Whatever is left goes to waiting prompts, in order; a prompt that does not fit is cut into chunks across steps (chunked prefill). A step with more prefill tokens takes longer, and every streaming user waits for that step.
Try this: pick "Long document" and press Run all steps with a budget of 2,048, then 16,384. Watch the trade: a small budget keeps every step short for the users already streaming, a large one gets the new prompt to its first token in fewer, longer steps.
Illustrative step times for Llama 3.1 8B on one H100: about 5 ms to read the weights plus 0.02 ms per decode request for its KV cache, plus about 0.035 ms per prefill token (2 × 8 billion operations at roughly half of 989 TFLOPS). Real numbers vary with model, kernels and context length. Decode-first ordering follows the vLLM documentation for its V1 scheduler.
What you are looking at: the trade behind --max-num-seqs. Each decode step reads all the weights once plus every running request's KV cache, so a bigger batch gives more total tokens but a slower stream for each user. Pick a GPU, a model and how long conversations are, then set the speed each user must see.
Try this: start with the Chatbot preset, then switch to Coding agent: long contexts make every request read more KV cache, so the batch that meets the target shrinks. Then try FP8 on the same GPU and see the KV pool double.
Ideal ceilings from memory bandwidth and dense compute, illustrative: step time = max(bytes read ÷ bandwidth, 2 × parameters × batch ÷ peak FLOPS). Real engines reach a fraction of these and add prefill work, so use the shape and the comparison, not the absolute numbers. KV pool = memory × 0.9 − weights − 4 GB overhead. GPU figures: H100 SXM 80 GB, 3.35 TB/s; H200 141 GB, 4.8 TB/s, same compute as H100; L40S 48 GB, 864 GB/s. KV per token for Llama 3.1 8B is 128 KB in 16-bit.