Output token throughput of 8,775 tokens per second from a single A100, on a container I pulled and ran in under ten minutes. That number is the whole argument for this part, and also its trap, because it is the exact number upstream vLLM would give you for free. What Red Hat sells around that container is not speed, it is the paperwork that lets you run it in production and phone someone when it breaks. Last part we gave every data science project a GPU budget and a fair queue on OpenShift AI. This part replaces the ad hoc vLLM process the support assistant has been running on with the Red Hat AI Inference Server, a supported, hardened build of the same vLLM engine, shipped as one container image.

Where the Inference Server sits above vLLM

vLLM is an open source inference engine from a project that started at UC Berkeley, and it is the piece doing the actual work: paged attention for the key value cache, continuous batching so requests join a running batch instead of waiting, and tensor parallelism to split a model across GPUs. Red Hat did not write vLLM, and says so. What Red Hat did was acquire Neural Magic, the company behind much of vLLM’s optimization work, and then wrap the engine in the things an enterprise cannot ship without: a hardened base image, a support contract, a security response process, and a version matrix that says exactly which engine build a given product release contains.

Hardened is a specific word here, not a slogan. It means a Red Hat build of vLLM on a UBI9 base with a known, scanned package set, rebuilt when a CVE lands in any layer, and shipped with a support life cycle that states how long a given stream keeps receiving fixes. Community vLLM offers none of that guarantee: you inherit whatever base image and transitive dependencies the wheel happened to pull in, and you own every CVE in them yourself.

One image, three homes. That same vllm-cuda-rhel9 container runs standalone under Podman on a single box, ships as the serving core inside RHEL AI, and drops into OpenShift AI as a KServe serving runtime. We will run it standalone here so nothing hides behind an operator, then reuse the identical model and flags on OpenShift AI in Part 17’s KServe setup. Serving concepts like batch versus real time endpoints carry over directly from the Data Science Series piece on serving machine learning models.

flowchart TD
  U[client, OpenAI compatible request] --> API[HTTP API on port 8000]
  subgraph RHAIIS [Red Hat AI Inference Server container]
    API --> V[vLLM engine, paged attention and batching]
    V --> C[LLM Compressor, quantized checkpoints]
  end
  V --> G[NVIDIA, AMD, TPU or Spyre accelerator]
  RHAIIS -.same image.-> S1[standalone Podman]
  RHAIIS -.same image.-> S2[inside RHEL AI]
  RHAIIS -.same image.-> S3[inside OpenShift AI via KServe]

What Red Hat adds to upstream vLLM

Here is the claim that gets oversold and the honest version of it. RHAIIS is marketed as accelerating inference two to four times, and community readers hear that as a faster engine than plain vLLM. It is not. That two to four times figure is the gain from running a compressed model, an FP8 or INT4 checkpoint, against a full precision one, and you can do exactly that with community vLLM. Its engine is the same code. What the supported build actually buys you is on the right side of this table, and it is the reference I hand anyone deciding whether to pay for RHAIIS or run community vLLM.

One row in that table hides real work: the optimized model set. Red Hat maintains a collection of pre-compressed checkpoints under the RedHatAI organization on Hugging Face, models already quantized to FP8 or INT4 with LLM Compressor and validated to load and serve on this exact engine. Pulling RedHatAI/granite-3.1-8b-instruct-FP8 hands you a checkpoint someone has tested against the vLLM build in the container, which is the difference between a model that serves on Monday and a weekend spent chasing a quantization mismatch. You can also ship weights as a modelcar, an OCI image that carries the model, so the same registry, signing and scanning that govern your application images govern your models too.

Serving the assistant from the container

Log in to the Red Hat registry, pull an exact tag, and let SELinux pass the GPU through. Never pull a floating tag in production, and the reason is the version matrix two sections down. Your token is read from the environment, never written into the command or an image layer.

# Tested on RHAIIS 3.2.5 (vLLM core v0.11.2, LLM Compressor v0.8.1), RHEL 9.4, NVIDIA A100 80GB, Podman 5.2
$ podman login registry.redhat.io
$ podman pull registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.5

# let the container reach the GPU when SELinux is enforcing
$ sudo setsebool -P container_use_devices 1
$ echo "export HF_TOKEN=<your_token>" > private.env && source private.env
$ mkdir -p rhaiis-cache && chmod g+rwX rhaiis-cache

Now start the server. Each flag earns its place, so read them before you copy. Our assistant runs a Granite model in production, but the benchmark below uses a small validated checkpoint so the numbers are reproducible on one card.

$ podman run --rm -it 
    --device nvidia.com/gpu=all 
    --security-opt=label=disable 
    --shm-size=16g -p 8000:8000 
    --userns=keep-id:uid=1001 
    --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" 
    -v ./rhaiis-cache:/opt/app-root/src/.cache:Z 
    registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.5 
    --model RedHatAI/Llama-3.2-1B-Instruct-FP8 
    --tensor-parallel-size 1

INFO   Automatically detected platform cuda.
INFO   Starting vLLM API server on http://0.0.0.0:8000
INFO   Route: /v1/completions, Methods: POST
INFO   Application startup complete.

--security-opt=label=disable stops SELinux relabeling the mounts, without which the container often will not start. --userns=keep-id:uid=1001 maps the host user to the vLLM process rather than running it as root; the docs offer --user=0 as an alternative and I avoid it, because root inside the serving container is a needless risk. --shm-size defaults to 4g in the docs, and that is the value I would change first, covered in the war story below. That -v ./rhaiis-cache:...:Z mount keeps downloaded weights on the host, so a restart does not re-pull sixteen gigabytes from Hugging Face, and the :Z suffix applies the SELinux label the mount needs; drop it only on a host without SELinux. Once it is up, the endpoint is OpenAI compatible, so any client from the AI Engineering Series works unchanged against it.

$ curl -s -X POST -H "Content-Type: application/json" -d '{
    "prompt": "How do I reset my support portal password?",
    "max_tokens": 50
}' http://localhost:8000/v1/completions | jq '.choices[0].text, .usage'

" Visit the portal, choose Forgot password, and follow the emailed link."
{
  "prompt_tokens": 11,
  "total_tokens": 33,
  "completion_tokens": 22
}

Because that endpoint speaks the OpenAI schema, a client you already wrote points at it by changing one base URL. Swap https://api.openai.com/v1 for http://localhost:8000/v1, keep the same request body, and your application never learns it changed engines. That is exactly the portability the AI Engineering Series argues for in choosing and switching providers without a rewrite, and it is what lets a team move off a hosted API onto self hosted Granite without touching application code.

Measuring TTFT, throughput and latency

One request tells you nothing about a serving layer. Four numbers do. Time to first token (TTFT) is how long a user waits before anything appears. Time per output token (TPOT) is the pace of the stream after that. Latency is the total for a full response, and throughput is how many tokens the server produces per second across every concurrent user. vLLM’s benchmark script drives real load and reports all four. It ships in the vLLM repository, which the supported container mirrors.

$ pip install vllm pandas datasets
$ git clone https://github.com/vllm-project/vllm.git
$ python vllm/benchmarks/benchmark_serving.py 
    --backend vllm --model RedHatAI/Llama-3.2-1B-Instruct-FP8 
    --num-prompts 100 --dataset-name random 
    --random-input 1024 --random-output 512 --port 8000

============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  4.61
Output token throughput (tok/s):         8775.85
Total Token throughput (tok/s):          30946.83
Mean TTFT (ms):                          193.61
P99 TTFT (ms):                           303.90
Mean TPOT (ms):                          9.06
P99 TPOT (ms):                           13.57
==================================================

Read those numbers as a shape, not a score. A mean TTFT of 194 ms with a P99 of 304 ms tells you the tail is close to the middle, which is what continuous batching buys you, and 8,775 output tokens per second on one A100 is the ceiling this model and card reach at this load. Change one dimension, longer prompts, more concurrency, a bigger model, and every number moves. That is why you benchmark your own model at your own concurrency before you size hardware, exactly as the AI Engineering Series argues in caching, batching and latency engineering.

Throughput and latency pull against each other, and continuous batching is how vLLM refuses to fully choose between them. Pack more concurrent requests into a running batch and total tokens per second climbs while any single user’s TPOT drifts up; run one request alone and its latency is best but the GPU sits mostly idle. At 100 concurrent prompts this card held P99 TTFT under 304 ms while producing 30,947 total tokens per second, a healthy point on that curve. Your own point depends on prompt length and how bursty real traffic is, and replaying traffic that looks like production is the only way to find it.

Version discipline and where serving breaks

A supported build only helps if you know which engine it contains. Each RHAIIS z-stream pins an exact vLLM core and an exact LLM Compressor, and they move fast. Keep this table, and match your compressed checkpoints to the LLM Compressor version that produced them.

Between 3.2.0 and 3.2.5 the engine crossed from vLLM v0.9.2 to v0.11.2, two minor versions, and LLM Compressor appeared where there had been none. A floating tag walks you across that gap without asking. A failure more common than versioning, though, is memory: a model that loads fine leaves too little room for the key value cache at your context length, and vLLM refuses to start rather than crash mid request.

# asking for the full 131072 context on a card that cannot hold its KV cache
$ podman run ... --model RedHatAI/granite-3.1-8b-instruct-FP8 --max-model-len 131072

ValueError: To serve at least one request with the model's max seq len (131072),
8.44 GiB KV cache is needed, which is larger than the available KV cache memory (3.62 GiB).
Based on the available memory, the estimated maximum model length is 56304.
Try increasing gpu_memory_utilization or decreasing max_model_len.

# fix: cap the context to what the assistant actually needs, and give vLLM more of the card
$ podman run ... --model RedHatAI/granite-3.1-8b-instruct-FP8 
    --max-model-len 8192 --gpu-memory-utilization 0.92
INFO   Maximum concurrency for 8192 tokens per request: 18.4x
INFO   Application startup complete.

That message is doing you a favor. It tells you the exact sequence length the card can hold, 56,304 tokens here, so you can right size --max-model-len instead of guessing. Our support assistant needs 8k of context at most, so capping it there frees cache for concurrency and lifts the batch size the server will accept.

Confirm what you are actually running before you trust it. Inside the container, python -c 'import vllm; print(vllm.__version__)' prints the engine version, and it should match the row above for your tag. This matters most where you cannot reach the registry at all: an air gapped site mirrors one specific image digest and lives with it for months, so choosing the right z-stream up front stops being a convenience and becomes the deployment itself, a case Part 29 takes up in full.

Move the assistant onto the supported container this week

References

• Red Hat AI Inference Server 3.2, Serving with Podman on NVIDIA CUDA
• Red Hat AI Inference Server 3.2, Product and version compatibility
• Red Hat blog, Introducing Red Hat AI Inference Server

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts