A repeated question reached first token in about 140 milliseconds on the assistant’s llm-d cluster, against roughly 900 on the single node vLLM box it replaced. Model weights, prompt and sampling settings were identical. All that changed was routing: the gateway sent the second request to the pod that already held its key value cache, so the expensive prefill work never ran twice. That behaviour, not raw GPU count, is why distributed inference earns the extra moving parts.

Where single node serving stops scaling

Last part left the assistant as one quantized Granite model on a single A100, holding about 18.5x concurrency at 8k context after the FP8 pass. That is a fine place to stop if traffic fits one card. Ours did not. Ticket volume climbed, prompts grew as we packed more product docs into context, and a single pod could not hold enough concurrent requests without queueing behind the ones already in flight.

An obvious move is to add replicas behind a Kubernetes Service and let round robin spread the load. For stateless web traffic that works. LLM serving is not stateless. Each request carries a large KV cache, the intermediate state vLLM builds during prefill and reuses during decode, and that cache is the most expensive object on the GPU. Round robin ignores it. Two requests that share a long document prefix land on different pods, each rebuilds the same cache from scratch, and you pay for the identical prefill twice.

Here is the part the scaling tutorials skip: more replicas behind a naive Service can serve shared prefix traffic slower per GPU than one well batched pod, because you have multiplied the wasted prefill rather than removed it. Adding cards does not fix a routing problem. This is the failure llm-d exists to correct, and it extends the same caching and latency argument the AI Engineering Series makes in caching, batching and latency engineering, now at cluster scale rather than inside one process.

How llm-d is built

llm-d is an open source CNCF sandbox project under Apache 2.0, started by contributors from Red Hat, Google, IBM, NVIDIA and AMD. It does not replace vLLM, it orchestrates many vLLM engines across a cluster. Red Hat’s contribution is the productisation, the OpenShift AI integration and the support path; the engine underneath is the same vLLM you met in Part 21, and llm-d runs on any Kubernetes, with NVIDIA and AMD GPUs or Google TPUs.

Four pieces matter. vLLM engines run inside pods as the workers that actually generate tokens. An Inference Gateway (IGW), the official Kubernetes project that extends the Gateway API with inference routing, sits in front; llm-d ships kgateway as its implementation, and Red Hat helps drive that upstream extension rather than owning it. Inside the gateway path, an Endpoint Picker (EPP) scores candidate pods by cache locality and current load, then picks one. When prefill and decode run on separate pods, NIXL, the NVIDIA Inference Xfer Library, moves the KV cache between them over RDMA or InfiniBand. Everything above vLLM is scheduling and transport; the model math is unchanged.

flowchart LR
  U[Client request] --> GW[Inference Gateway]
  GW --> EPP[Endpoint Picker]
  EPP -->|prefill route| PRE[Prefill pod vLLM]
  EPP -->|cache aware score| DEC[Decode pod vLLM]
  PRE -->|KV cache via NIXL| DEC
  DEC --> R[Tokens to client]

Three well-lit paths and when each applies

Red Hat documents llm-d as a set of well-lit paths, meaning deployment patterns tested end to end rather than left as loose config. Three carry most production use. Cache aware inference scheduling makes the gateway route by prefix and KV locality instead of round robin. Prefill decode disaggregation splits the two phases onto separate pools so each runs on hardware suited to it, and even lets prefill fall to CPU when GPUs are scarce. Wide expert parallelism spreads a Mixture of Experts model, one whose weights are divided into specialised experts, across nodes using expert parallelism (EP) and data parallelism (DP), so models like DeepSeek or GPT-OSS run on several smaller GPUs instead of one very large one. Release 0.5 added a fourth worth knowing, hierarchical KV offload that tiers cache across GPU, CPU and a shared filesystem so it survives beyond one pod.

Keep this selector next to your cluster. It is the artifact I reach for when someone asks which path a workload belongs on.

Deploying Granite behind the inference gateway

llm-d installs through a quickstart of Helmfile bundles, not a single operator toggle. The precise prefix cache aware example is the right first target, because it exercises the scheduling path on modest hardware, two GPUs rather than a rack. It ships pointed at Qwen3-0.6B so the demo runs cheaply; pointing it at the FP8 Granite from Part 22 is a change in the model values file, not a different procedure. Read the Hugging Face token from the environment and store it as a secret, never in the manifest.

# Tested against llm-d v0.8 via the llm-d-infra quickstart, vLLM v0.11 engine,
# OpenShift 4.17, NVIDIA GPU Operator 25.3, Node Feature Discovery 4.18,
# 2x NVIDIA L40S, Granite 3.1 8B FP8 from Part 22. HF_TOKEN comes from the
# environment. Repo layout shifts between releases; v0.8 guides now live in
# github.com/llm-d/llm-d, so confirm directory names for the tag you deploy.
git clone https://github.com/llm-d-incubation/llm-d-infra.git
cd llm-d-infra/quickstart
./dependencies/install-deps.sh              # installs helm and helmfile

cd gateway-control-plane-providers
./install-gateway-provider-dependencies.sh
helmfile apply -f istio.helmfile.yaml       # gateway provider

cd ../examples/precise-prefix-cache-aware
export NAMESPACE=llm-d-precise
oc new-project ${NAMESPACE}

export HF_TOKEN_NAME=${HF_TOKEN_NAME:-llm-d-hf-token}
oc create secret generic ${HF_TOKEN_NAME} 
  --from-literal="HF_TOKEN=${HF_TOKEN}" 
  --namespace "${NAMESPACE}" --dry-run=client -o yaml | oc apply -f -

helmfile apply -n ${NAMESPACE}

After a few minutes the namespace should hold the endpoint picker, the gateway, and the model service decode pods.

$ oc get pods -n llm-d-precise
NAME                                                       READY   STATUS    RESTARTS   AGE
gaie-kv-events-epp-5d4f98d6b6-sxf9w                        1/1     Running   0          25m
infra-kv-events-inference-gateway-istio-5f68d4f854-qpnq4   1/1     Running   0          25m
ms-kv-events-llm-d-modelservice-decode-648464d84b-2r58r    2/2     Running   0          22m
ms-kv-events-llm-d-modelservice-decode-648464d84b-lclzf    2/2     Running   0          18m

My first apply did not reach that state. Both decode pods sat in Pending, and the reason was not GPU capacity but a taint on the GPU nodes that the model pods did not tolerate.

$ oc get pods -n llm-d-precise
NAME                                                      READY   STATUS    RESTARTS   AGE
ms-kv-events-llm-d-modelservice-decode-648464d84b-2r58r   0/2     Pending   0          6m

$ oc describe pod ms-kv-events-llm-d-modelservice-decode-648464d84b-2r58r | tail -3
Events:
  Warning  FailedScheduling  default-scheduler  0/8 nodes are available:
  8 node(s) had untolerated taint {nvidia.com/gpu: NVIDIA-L40S-PRIVATE}.

# Cause: GPU nodes carry a taint the pods do not tolerate, so the scheduler
# will not place them. Fix: patch the deployment with the matching toleration.
$ oc patch deployment ms-kv-events-llm-d-modelservice-decode -n llm-d-precise 
  -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"nvidia.com/gpu","operator":"Equal","value":"NVIDIA-L40S-PRIVATE","effect":"NoSchedule"}]}}}}'
deployment.apps/ms-kv-events-llm-d-modelservice-decode patched

Four stalls cover most first installs. Keep this failure to cause table with the selector above.

Watching cache aware routing choose a pod

Confirm the gateway is serving, then prove the routing does what it claims. Ask for the model list first.

$ export SVC_EP=$(oc get svc infra-kv-events-inference-gateway-istio 
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ curl http://$SVC_EP/v1/models
{"object":"list","data":[{"id":"Qwen/Qwen3-0.6B","max_model_len":40960,"owned_by":"vllm"}]}

Now send a prompt over 200 tokens twice, from a file so the shell does not mangle it, and read the endpoint picker log after each call. First time, the picker has no cache to match and scores null. Second time, the same prefix is warm on one pod, and the picker returns that pod’s address with a positive score.

# first call with a long, reused prompt
$ curl -s http://$SVC_EP/v1/completions -H 'Content-Type: application/json' 
    -d @long_prompt.json | jq -r '.choices[0].text' | head -c 40
$ oc logs deployment/gaie-kv-events-epp -n llm-d-precise | grep 'Got pod scores' | tail -1
Got pod scores {"model":"Qwen/Qwen3-0.6B","criticality":"Sheddable","scores":null}

# identical prompt again
$ curl -s http://$SVC_EP/v1/completions -H 'Content-Type: application/json' 
    -d @long_prompt.json | jq -r '.choices[0].text' | head -c 40
$ oc logs deployment/gaie-kv-events-epp -n llm-d-precise | grep 'Got pod scores' | tail -1
Got pod scores {"model":"Qwen/Qwen3-0.6B","criticality":"Sheddable","scores":{"10.131.2.23":2}}

A score of null then a pod address is the whole thesis in two log lines: the second request skipped prefill because the picker found where its cache already lived. That is the metric that maps to user experience, and it is the kind of signal the AI Engineering Series argues you should trace in observability for LLM applications. Cache hit rate, TTFT and inter token latency are the three llm-d exposes, and they matter more here than CPU or request count ever did.

Gains from disaggregation and scheduling

Numbers, not adjectives, should drive the decision. Red Hat reports that disaggregation alone gives about a 25 percent throughput lift at default settings, from letting Kubernetes schedule prefill and decode separately. Cache aware scheduling goes much further: on a Qwen3-32B deployment across 8 vLLM pods and 16 NVIDIA H100 GPUs, the llm-d team measured up to 109 percent higher throughput and 99 percent lower TTFT against a baseline Kubernetes Service, with P50 TTFT held at 136 to 157 milliseconds under load. These are published, reproducible figures, and they are workload dependent, so treat them as the shape of the win rather than a promise for your tickets.

Two more figures set the ceiling. On a throughput oriented Wide EP topology of 16 prefill and 16 decode NVIDIA B200 GPUs, llm-d sustained about 50,000 output tokens per second, near 3,100 per decode GPU. And the v0.5 hierarchical KV offload, tiering cache to a shared filesystem, held roughly 185,000 tokens per second on a Llama-3.1-70B run as concurrency climbed to 250 users, a 13.9x improvement over the point where a GPU only setup collapsed once HBM filled. Whether any of this holds for your model is a question for your own eval set, the same discipline the Data Science Series applies to serving machine learning models across batch and real time paths.

Reach for llm-d when one GPU is not enough

Adopt llm-d when a single card can no longer hold your concurrency and your traffic shares prefixes, because that is exactly the case a plain Service handles worst and cache aware routing handles best. Start with the scheduling path, since it is the largest and cheapest win and needs no extra hardware. Add disaggregation only for large models or long prompts, and measure it, because on small models it can cost more than it returns. Save Wide EP for genuine Mixture of Experts models. On OpenShift, llm-d also slots under KServe from Part 17 as the serving runtime, so you keep the model registry and InferenceService workflow you already built rather than replacing it.

Next part turns the throughput and latency numbers you just saw into money, working through token economics, batching and the latency knobs that decide how few GPUs the whole deployment needs.

References

• Red Hat Developer, Introduction to distributed inference with llm-d
• Red Hat Developer, Getting started with llm-d for distributed AI inference
• llm-d 0.5, Sustaining Performance at Scale
• llm-d project on GitHub

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts