Sixty people are chatting with the same model at once. Every message carries the conversation so far plus the documents attached at the start. Both clusters below get exactly the same messages. The left one hands them to pods in turn. The right one uses the llm-d router, which knows what each pod has cached and how busy it is. A pod that has seen the conversation before only reads the new text; any other pod reads everything again.
Model: 8 pods, each reads prompts at about 8,000 tokens a second and keeps about 150,000 tokens of cache. Each turn adds about 400 tokens of history. A green flash is a message that found its conversation cached, red means the pod read it all again. Numbers are illustrative; the behaviour matches what llm-d's prefix-cache routing does.
Three apps share one GPU cluster that can answer about 100 requests a second. Customer chat matters most, internal tools come next, and nightly batch summaries can wait. Press Start a batch job to send a big burst of low-priority work. Without flow control every request joins one queue, so customers wait behind the batch. With llm-d 1.0 flow control, requests queue by priority and the lowest priority is dropped first when the wait gets too long.
Charts show the average wait over the last 2 minutes (0 to 30 s). Requests waiting more than 30 s time out. With flow control, batch requests are shed after 8 s and internal tools after 20 s. Real llm-d also shares fairly between tenants of the same priority. Numbers are illustrative.
Every answer has two stages. Prefill reads the prompt in one heavy burst of maths. Decode writes the answer one word at a time and is mostly limited by memory speed. When the same GPUs do both, each new prompt briefly pauses the answers already being written, so text arrives in stutters. llm-d can give each stage its own GPUs and pass the cache between them over the network. Compare both on the same 8 GPUs.
Model: a 70B-class model, each GPU reads prompts at about 10,000 tokens a second and writes about 2,500 tokens a second across all its answers. The cache for one token is about 0.32 MB. Numbers are illustrative; real gains depend on the model, hardware and how well the split matches the workload.
Answer five questions about your setup. The result tells you whether plain vLLM is enough, or which llm-d setup to start with. The project calls its tested setups "well-lit paths", and the names below match its guides.