Before PagedAttention, a served model threw away 60 to 80 percent of the GPU memory it set aside for its key value cache. Not to a bug, to bookkeeping: the cache for each request was reserved as one contiguous slab sized for the longest answer the model might ever give, and most requests never came close to filling it. Last part we measured the support assistant at 8,775 output tokens per second on a single A100 and never asked where that number comes from. This part opens the engine underneath the Red Hat AI Inference Server, which is the same upstream vLLM, and shows how two ideas, paged memory for the cache and batching that reshuffles on every step, turn a memory bound problem into a throughput machine. A KV cache is the running store of attention keys and values vLLM keeps for every token already in a request, and it grows with the context window.

Why KV cache memory decides throughput

A served language model is bound by memory long before it is bound by arithmetic. Every token already in a request leaves behind a key and a value vector at every layer, and vLLM must keep all of them, the KV cache, to generate the next token. Weights load once and are shared by every request, but the cache is per request and grows with each token, so on a serving box the cache, not the weight matrix, is what runs out first. That is why batch size, and therefore throughput, is a memory question rather than a compute one. Serving shapes like real time versus batch endpoints carry over directly from the Data Science Series piece on serving machine learning models.

Older servers made that memory problem worse than it needed to be. Each request reserved one contiguous slab of cache sized for the largest answer it might produce, often the model maximum, and held it for the request’s whole life. A request that asked for 40 tokens still sat on a reservation for thousands. Measured across a real workload, 60 to 80 percent of KV cache memory went unused, lost to that over reservation and to gaps too small for the next request to slot into. Fewer requests fit at once, so the GPU sat half idle while users queued behind a card that was mostly empty.

How PagedAttention pages the KV cache

PagedAttention, introduced by Kwon and colleagues at SOSP 2023, fixes this with an idea taken straight from operating systems: paging. Instead of one contiguous slab, each request’s cache is cut into fixed blocks of 16 tokens, and a per request block table maps the logical position of each block to a physical block that can live anywhere in GPU memory. Blocks need not be adjacent, so there are no large unusable gaps, and a request only ever wastes part of its last, partly filled block. Internal fragmentation falls from most of the cache to at most one block per request.

Those blocks change the numbers that matter. Measured KV cache utilisation rises from roughly 20 to 40 percent to above 95 percent, and the paper reports 2 to 4 times the throughput at the same latency against contiguous systems. Because every block is reached through a table, two requests that share an identical prefix can point at the same physical blocks instead of each storing a copy, so a shared system prompt is cached once. Cache cost scales with the context window, so the longer your prompts, the more this paging discipline is worth.

You can size that cache before you rent the card. This snippet reads a model’s architecture and prints the cache cost per token and per sequence, so you plug in the numbers from any config.json and know your batch ceiling in advance.

# Tested with Python 3.11. Numbers below assume a 40 layer model with
# 8 key value heads of dimension 128 (grouped query attention), FP16 cache.
n_layers, n_kv_heads, head_dim, dtype_bytes = 40, 8, 128, 2
per_token = 2 * n_layers * n_kv_heads * head_dim * dtype_bytes  # 2 for K and V
print(f"{per_token/1024:.0f} KiB per token")
for ctx in (4096, 8192, 32768):
    gib = per_token * ctx / 1024**3
    print(f"{ctx:>6} tokens: {gib:.2f} GiB for one sequence")

# output
160 KiB per token
  4096 tokens: 0.62 GiB for one sequence
  8192 tokens: 1.25 GiB for one sequence
 32768 tokens: 5.00 GiB for one sequence

At 160 KiB per token, one 8k sequence needs 1.25 GiB of cache, and forty of them at once need 50 GiB, which on an 80 GiB card is most of what is left after the weights load. That one calculation predicts your batch ceiling better than any single benchmark run, and it is the artifact I reach for first when sizing a serving node.

Continuous batching at the iteration level

Paging decides how many requests fit. Continuous batching decides how well they share the GPU. A naive server groups requests into a fixed batch, runs that batch until every member finishes, then starts the next; one slow 500 token answer holds a batch of fast 20 token answers hostage until it is done. vLLM, following the Orca line of work upstream, schedules at the granularity of a single token step instead, and Red Hat ships that scheduler unchanged inside the Inference Server.

After every forward pass the scheduler looks at three queues, waiting, running and swapped, and rebuilds the batch. A finished sequence leaves at once and returns its blocks to the free pool; a waiting request joins on the very next step if free blocks exist. Sequences enter and leave the running batch continuously, which is where the throughput comes from, and it is why raw tokens per second on the Inference Server matches upstream vLLM on identical hardware. That trade between throughput and latency is the same one the AI Engineering Series works through in caching, batching and latency engineering.

flowchart TD
  W[waiting queue] --> SCH[scheduler builds a batch]
  R[running queue] --> SCH
  SCH --> FP[one forward pass, all sequences step together]
  FP --> DONE{sequence finished}
  DONE -->|yes| FREE[free its blocks to the pool]
  DONE -->|no| KV{KV cache full}
  KV -->|no| R
  KV -->|yes| PRE[preempt, recompute or swap out]
  FREE --> W
  PRE --> W

One branch in that loop is the one people forget. When the cache fills, a running sequence can be preempted, its blocks freed and its tokens recomputed later, and that path costs real time. It is the difference between a server that degrades gracefully and one that collapses, and you meet it the moment you admit more requests than the cache can hold.

Watching the vLLM scheduler under load

Enough theory. Serve Granite with the flags that matter and read what the engine prints on the way up. Your Hugging Face token comes from the environment, never the command line or an image layer.

# Tested on RHAIIS 3.2.5 (vLLM core v0.11.2), RHEL 9.4, NVIDIA A100 80GB, Podman 5.2
$ podman run --rm -it --device nvidia.com/gpu=all --security-opt=label=disable 
    --shm-size=16g -p 8000:8000 --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" 
    registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.5 
    --model RedHatAI/granite-3.1-8b-instruct-FP8 
    --max-model-len 8192 --max-num-seqs 64 --gpu-memory-utilization 0.92 --block-size 16

INFO   Automatically detected platform cuda.
INFO   GPU KV cache size: 151,552 tokens
INFO   Maximum concurrency for 8,192 tokens per request: 18.5x
INFO   Application startup complete.

Two banner lines tell you the capacity of this deployment. GPU KV cache size: 151,552 tokens is the total the card holds after weights, and Maximum concurrency for 8,192 tokens per request: 18.5x means about eighteen full length requests fit at once before the cache becomes the limit. Under load, vLLM logs the scheduler state every few seconds, and those lines are the truest gauge you have.

INFO   Engine 000: Avg prompt throughput: 2510.3 tokens/s, Avg generation throughput: 8721.4 tokens/s,
       Running: 64 reqs, Waiting: 12 reqs, GPU KV cache usage: 61.4%, Prefix cache hit rate: 0.0%

Running: 64 reqs with cache usage near 61 percent is a server working hard with headroom to spare. Push concurrency higher and that usage climbs toward 100 percent, at which point the scheduler starts evicting. Here is the failure you will actually hit, and it does not look like a crash, it looks like the server quietly getting slower.

# admitting 512 sequences at 8k context on a card whose cache holds ~18 of them
WARNING   Sequence group 0x3f2a is preempted by PreemptionMode.RECOMPUTE because there is
          not enough KV cache space. This can affect the end to end performance. Increase
          gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory.

# fix: admit fewer, give vLLM more of the card for cache
$ podman run ... --model RedHatAI/granite-3.1-8b-instruct-FP8 
    --max-model-len 8192 --max-num-seqs 64 --gpu-memory-utilization 0.95
INFO   Avg generation throughput: 8703.1 tokens/s, Running: 64 reqs, GPU KV cache usage: 78.9%

Recompute means the evicted request throws away its cache and regenerates it from scratch when it resumes, so that work is paid for twice. This is the part that contradicts the obvious move. Raising --max-num-seqs to admit more requests feels like more throughput, and up to a point it is, but past the cache ceiling it triggers preemption and total tokens per second falls while tail latency climbs. A default of 1024 in the V1 engine is a ceiling the hardware rarely reaches at real context lengths, not a target to aim for.

V1 engine and what changed underneath

Everything above runs on vLLM’s V1 engine, which became the default in 2025 and, as of vLLM v0.11 and Red Hat AI Inference Server 3.2.4, is the only engine; the older V0 path was removed entirely. V1 rebuilt the scheduler and execution loop to cut per step overhead, and three of its changes matter the moment you tune serving.

Chunked prefill splits a long prompt’s prefill into pieces so it interleaves with other requests’ decode steps instead of stalling them, and it is on by default in V1. Disaggregated prefill and decode goes further, running the two phases in separate scheduling domains so a large prefill never blocks the decode steps of in flight streams. An experimental --async-scheduling flag overlaps the scheduler with the GPU runner for extra throughput. Defaults moved too: V1 raised --max-num-seqs from 256 to 1024 and keeps --gpu-memory-utilization at 0.90. A supported build pins all of this, so RHAIIS 3.2.5 fixes the scheduler behaviour and these defaults for that exact tag, which is why Part 20 argued for pinning the z-stream rather than chasing latest.

Keep this within reach; it is the serving knob reference I check before touching a running deployment.

Tune two knobs before buying a bigger GPU

Next part measures where the cache pressure actually comes from and cuts it at the source: model compression and quantization with LLM Compressor, and how the format you pick changes both the memory you just learned to size and the accuracy you serve.

References

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023
• vLLM blog, vLLM V1, a major upgrade to the core architecture
• Red Hat AI Inference Server 3.2, Release notes

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts