AI Infrastructure series · Part 17 of 24

vLLM is the open-source serving engine you will meet most often when running open models on your own GPUs. It is the model server inside llm-d, inside many Kubernetes AI platforms, and behind a good share of hosted model APIs. Its speed does not come from faster maths. It comes from scheduling: it adds and removes requests from the running batch on every step, and it hands out KV cache memory in small blocks instead of reserving it up front. This part walks through both ideas with numbers, then covers the settings that matter, what the start-up log tells you, and how vLLM fits under llm-d.

What is vLLM?

vLLM started in 2023 at UC Berkeley’s Sky Computing Lab, alongside the research paper that introduced PagedAttention (Kwon et al., SOSP 2023). It is now a PyTorch Foundation project with contributors from many companies. It loads a model from Hugging Face or local storage, runs it on one or many GPUs, and serves it through an OpenAI-compatible API on port 8000. Part 16 showed the one-line start command and what a model server does in general.

It runs most popular open models, including Llama, Qwen, Mistral, DeepSeek and gpt-oss, on NVIDIA and AMD GPUs, Google TPUs and several other accelerators. When it launched, its authors measured up to 24 times the throughput of plain Hugging Face Transformers and up to 3.5 times that of TGI, the engine most people used at the time. The GPU does the same arithmetic in every engine. vLLM wins by keeping more requests on the GPU at once and wasting less memory on each one. To see why that matters, start with what one decode step costs.

Why can one GPU step serve many users?

During decode, a model writes one token per request per step. To write that token it must run every layer, which means reading every weight from GPU memory once (part 4, part 13). The maths per weight is tiny: two operations per parameter per token. So a single-user decode step is limited by how fast the GPU can read memory, not by how fast it can calculate.

Batching changes this. If eight requests are in the batch, the GPU still reads each weight only once per step and uses it eight times. The weights are the expensive part of the read, so the step takes only a little longer, and you get eight tokens instead of one. What grows with the batch is the KV cache each request must also read: its own history of keys and values (part 14).

How much does a bigger batch buy? A worked example

Take Llama 3.1 8B in 16-bit on one H100 SXM. The numbers are ideal ceilings (illustrative), real engines reach a fraction of them, but the shape is right.

Total tokens per second from one H100, Llama 3.1 8B in 16-bit, 1,000 tokens of history per request. Ideal bandwidth ceilings, illustrative.

Two lessons fall out of this table, and every vLLM design choice follows from them. First, a serving engine should keep the batch as full as possible on every step, because an empty slot is throughput thrown away. Second, the batch is limited by how many requests’ KV caches fit in memory at once. Continuous batching solves the first problem. PagedAttention solves the second.

How do static, dynamic and continuous batching differ?

All three engines batch. The difference is when a request may join or leave the batch.

Static batching collects a fixed number of requests, runs them together until the last one finishes, then takes the next group. Answers vary in length, from a yes or no to a 2,000-word report, so the short answer finishes early and its slot sits idle until the long one is done, and new arrivals wait for the whole group.

Dynamic batching, common in general inference servers, closes a batch when it is full or when the oldest request has waited a set time. That cuts waiting when traffic is light, but once a batch starts it still runs to completion, so the idle-slot problem stays.

Continuous batching (iteration-level scheduling, from the Orca paper, Yu et al., OSDI 2022) makes the decision on every step. When a request writes its last token, it leaves, and a waiting request takes its slot on the very next step. Nobody waits for a slow neighbour and the batch stays full. The timelines below run the same 16 requests, arriving over time, through all three.

The cost is bookkeeping: every step is a different mix of requests, rebuilt every few milliseconds. That is the scheduler’s job.

What does the scheduler do on every step?

The scheduler keeps two lists. The running list holds requests that already have KV cache on the GPU. The waiting list holds new requests and any that were pushed out earlier for lack of memory (preempted), which go back to the front. Each step has a token budget, set by --max-num-batched-tokens, and a cap on requests, set by --max-num-seqs.

Recent vLLM versions treat prompt tokens and answer tokens the same way: the scheduler’s output for a step is simply a list of requests and how many tokens each will process. A decoding request needs 1. A new request needs its whole prompt, or a slice of it. This is what lets one forward pass mix prefill and decode, called chunked prefill, which is on by default.

What happens, step by step, every few milliseconds:

1. Serve the running requests. Each decoding request gets its next token. If its last KV block is full, the cache manager must find it one new block first.
2. Admit from the waiting list, in order. For each request the manager looks up prompt blocks already in the prefix cache, then allocates blocks for the rest. If free blocks run short, admission stops for this step.
3. Chunk what does not fit. A prompt bigger than the remaining budget gets a slice now and the rest later.
4. Run and clean up. One forward pass covers all of it. Tokens stream back, finished requests leave and their blocks return to the free pool.

The budget is the main dial between two latency numbers from part 13. A small budget (the vLLM docs use 2,048 as an example) keeps time between tokens steady because each step carries little prefill. A larger one, above 8,192, gets prompts through faster and raises throughput, at the cost of slightly longer steps for everyone decoding.

How does PagedAttention store the KV cache?

Continuous batching keeps slots full, but each running request needs its KV cache in GPU memory, and that is where older engines lost most of their capacity. They gave each request one contiguous region sized for the longest answer it might give, because they could not know in advance how long it would be. A request that wrote 200 tokens into a region reserved for 2,048 left 90% of it empty, and nobody else could use that space. Memory also fragmented as regions of different sizes came and went. The vLLM paper measured 60 to 80% of KV memory wasted this way.

PagedAttention borrows the operating system’s answer to the same problem, virtual memory. The KV cache is split into fixed-size blocks, 16 tokens each by default. A request sees its cache as a list of logical blocks, 0, 1, 2 and so on. A per-request block table maps each logical block to any free physical block in GPU memory. The attention kernel follows the table to find each token’s keys and values, so the physical blocks never need to be next to each other.

What happens as a request grows and ends?

Blocks are handed out one at a time, as the request needs them. Follow request A from the diagram above:

Because a request only ever holds blocks it has filled plus one partial block, the paper measured waste under 4%. The memory that older engines left empty now holds more requests, which, from the bandwidth table, means more tokens per step from the same GPU.

How do prefix caching and copy-on-write share blocks?

A block table is just a list of block numbers, so two requests can point at the same physical block. vLLM uses this in two ways.

Prefix caching. When a block is full, vLLM computes a hash from the hash of the block before it, the block’s own tokens, and extras such as a LoRA adapter ID. Because each hash includes its parent, a hash identifies the whole prompt up to that point, not just 16 tokens. When a new request arrives, the scheduler hashes its prompt block by block and looks each hash up. Every hit is a block whose keys and values are already in memory, so prefill skips those tokens. Only full blocks are cached. Each block carries a reference count; when it reaches zero the block goes to the free queue but keeps its contents and hash, so a later request can still hit it. Blocks are evicted least recently used first. Prefix caching has been on by default since vLLM’s V1 engine, because its overhead is close to zero.

Copy-on-write. When you ask for several answers to one prompt (parallel sampling, n greater than 1, or beam search), all answers share the prompt’s blocks. The paper’s design lets them share even the last part-full block, and copies it only when one answer writes a different token into it. Current vLLM versions get the same saving by running each extra sample as its own request that shares the prompt’s full blocks through the prefix cache.

Prefix caching is what makes chat and agents cheap to serve. A chat resends its whole history every turn and an agent resends the same instructions and tool list every step; with those already in blocks, only the new text needs prefill. The cache only helps if the follow-up lands on the server that holds it, which is where llm-d comes in.

How much KV cache does an H100 give Llama 3.1 8B?

vLLM decides the size of the block pool at start-up. It loads the weights, runs a test forward pass at the largest batch it will schedule to measure peak activation memory, captures CUDA graphs, and gives everything that is left, within --gpu-memory-utilization, to KV blocks. Worked through for Llama 3.1 8B in 16-bit on one 80 GB H100 (overhead figure illustrative):

Illustrative: 4 GB overhead assumed. Request counts above --max-num-seqs are capped by that setting, and long prompts need budget room too.

The first two rows show the whole PagedAttention story in numbers. The reserved engine and the paged engine have the same memory and the same worst case. They differ when real requests are shorter than the worst case, which is almost always: paged memory lets five times as many conversations run. The same arithmetic for a bigger model is in part 14.

What happens when the blocks run out?

vLLM admits requests based on what they need now, not what they might need later, so the pool can fill while answers are still growing. When a running request needs a new block and none is free, the scheduler preempts: it takes the most recently admitted request off the GPU, frees its blocks, and puts it back at the front of the waiting list. The log shows a warning that names the preemption mode.

Nothing fails, but the user whose request was preempted sees their answer stall for a while. Occasional preemption under a burst is fine. Constant preemption means you are trying to run more concurrent work than the pool holds. The vLLM docs suggest raising --gpu-memory-utilization, lowering --max-num-seqs or --max-num-batched-tokens, or adding GPUs with tensor parallelism.

What is inside a running vLLM server?

A vLLM server is several processes, split so that CPU work (HTTP, tokenising, streaming) never makes the GPU wait.

• API server. Accepts OpenAI-style requests, applies the chat template, turns text into token IDs and streams tokens back. It runs in its own process and talks to the engine over ZeroMQ, so heavy tokenising does not hold up the GPU loop.
• Engine core. A tight loop around the scheduler and the KV cache manager described above. Each step it decides who runs and how many tokens each gets, and sends only the changes since the last step to the workers.
• Workers. One process per GPU. Each holds its share of the weights and of every KV block, and runs the forward pass. With tensor parallelism (part 5) every layer is split across GPUs, which exchange partial results with NCCL over NVLink.
• CUDA graphs and compilation. A decode step for an 8B model is hundreds of small GPU kernels taking a few milliseconds in total, so launching each one from Python would waste a large share of the step. vLLM compiles the model with torch.compile and records the kernel sequence as CUDA graphs for common batch sizes, then replays a whole graph with one launch. --enforce-eager turns this off: start-up gets faster, decode gets slower.

Quantization (part 18), multi-LoRA (part 19), speculative decoding and expert parallelism for models like those in part 12 all plug into this same loop.

Which settings should you start with?

Most of vLLM’s tuning comes down to a handful of flags. These are starting points, not answers; measure on your own traffic.

A sensible first command for Llama 3.1 8B on one H100:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192

What does the start-up log tell you?

Three lines in the start-up log answer most sizing questions: how much memory the weights took, how many tokens of KV cache fit, and how many full-length requests that is. An example of their shape (illustrative numbers, and exact wording varies between versions):

# illustrative excerpt, not a real capture
INFO  Model loading took 14.99 GiB memory
INFO  GPU KV cache size: 396,272 tokens
INFO  Maximum concurrency for 16,384 tokens per request: 24.19x

Read the last line with care. “24x” does not mean vLLM will run only 24 requests; it means 24 requests could each use the full 16,384 tokens at once. Real requests are shorter, so more run, up to --max-num-seqs. If the number is below 1, vLLM refuses to start, because not even one maximum-length request would fit. Lower --max-model-len, raise the utilisation, quantize the KV cache or add a GPU.

How do you choose max-num-seqs?

From the bandwidth table, a bigger batch always gives more total tokens and always gives each user fewer. The right batch size is the largest one where every user is still fast enough. Plot both curves and draw your target on the left one:

In practice you run a load test at rising concurrency, watch time per output token at the 90th or 99th percentile, and stop raising --max-num-seqs just before it crosses your target. The demo’s tuning tab does this arithmetic for different GPUs, context lengths and targets.

How do llm-d and vLLM work together?

Everything so far happens inside one vLLM server. vLLM does not decide which of twenty copies of a model a request should go to, and the default Kubernetes answer, a Service that spreads requests in turn, throws away much of what vLLM built. A follow-up message in a chat has a one-in-eight chance of reaching the pod with its history cached when there are eight replicas. The other seven times a GPU prefills the whole conversation again. llm-d is the layer that fixes this on Kubernetes, and it uses vLLM as its default engine.

The link between the two is the block. vLLM stores the KV cache as fixed-size hashed blocks; llm-d uses the same hashes to know where every prefix lives across the fleet, and moves and tiers the same blocks. A cache-aware router can only be as good as the cache it routes to.

When is vLLM alone enough?

What do people get wrong about vLLM?

• Leaving max-model-len at the model’s maximum. Llama 3.1 offers 128K tokens of context. If your app never sends more than 8K, the default makes vLLM report tiny concurrency, and on smaller GPUs refuse to start at all.
• Reading “maximum concurrency” as a hard limit. It counts full-length requests. Real ones are shorter, so far more run at once.
• Expecting gpu-memory-utilization to cap the KV cache. It sets how much vLLM takes in total. Whatever the weights and activations leave inside that share all becomes KV blocks, so vLLM always looks as if it has filled the GPU. That is by design, not a leak.
• Raising max-num-seqs until throughput stops growing. Throughput keeps rising long after users notice slow streams. Tune against time per output token, not total tokens.
• Round-robin across replicas with long shared prompts. Each replica’s prefix cache only helps requests that reach it. Use session affinity at minimum, or a cache-aware router.

Short answers to common questions

Does vLLM make a single request faster?

Not much. One request is limited by memory bandwidth in any engine. vLLM’s gains are in how many requests one GPU can serve at a good speed. Speculative decoding and quantization are the features that help a single stream.

Why does vLLM take 90% of GPU memory as soon as it starts?

It allocates the whole KV block pool up front so it never has to ask the GPU for memory mid-request. Lower --gpu-memory-utilization if something else must share the GPU.

Does prefix caching change the answers?

No. A hit reuses keys and values that would have been computed the same way. It changes how fast the first token arrives, not what is written.

Do I need llm-d to use vLLM on Kubernetes?

No. A vLLM Deployment behind a Service works. llm-d becomes useful when you run several replicas of a model and your prompts share long beginnings.

PagedAttention, block size and memory waste figures are from the vLLM paper (Kwon et al., SOSP 2023); launch throughput figures are from the vLLM project’s announcement. Continuous batching follows Orca (Yu et al., OSDI 2022). Scheduler behaviour, chunked prefill, preemption modes, prefix-cache hashing and eviction, CUDA graph modes and process layout are from the vLLM documentation (optimization and prefix caching design pages) and the vLLM V1 engine announcement; setting names and defaults can change between versions. llm-d details are from the llm-d project documentation. Bandwidth, memory and step figures are worked ideal ceilings, and the log excerpt and timelines are illustrative.