Finance asked me a fair question about the assistant: what does one answer cost? My honest reply was that it depends almost entirely on how busy the GPU is at the moment the answer is generated. A card sitting at 5 percent load still draws its full hourly rate, so an idle accelerator is the most expensive way to serve a token. Everything in this part follows from that one fact.

Cost of a GPU you already paid for

Last part moved the assistant onto llm-d so shared prefixes stopped paying for prefill twice. This part turns the throughput and latency numbers that change produced into a cost per answer, and tunes the handful of flags that decide how few GPUs the deployment needs.

Start with the number that dominates everything else: the hourly cost of the accelerator. An owned H100, amortised over three years with power and hosting, lands near 1.80 dollars an hour. Rent the same card on demand in a cloud and you pay closer to 3.00 dollars an hour. That rate is fixed. It does not fall when traffic is light. So the only lever that moves cost per token is how many tokens the card generates in each of those hours, and that is a function of two things you control, peak throughput and average utilisation.

Peak throughput is what the model can produce when fully loaded. Utilisation is what fraction of the day it actually runs that hot. Most internal assistants sit near 20 to 30 percent because traffic follows office hours, and that idle time is pure waste billed at the full rate. This is the same argument the AI Engineering Series makes about cost control and model routing, only here the meter is a card you own rather than a per token API bill, and it never pauses.

Three numbers that set serving cost

Red Hat documents four metrics for judging an inference deployment, and three of them decide the money. Output token throughput, in tokens per second across all requests, sets how quickly the card empties its work queue. Time to first token measures the wait a user feels before anything appears, and it is dominated by the prefill pass that builds the KV cache from the prompt. Time per output token is the steady decode speed once generation starts. Throughput is the cost axis; TTFT and TPOT are the experience axis. You cannot read one from the other, which is why a single tokens per second figure quoted with no latency is close to meaningless.

To ground these, here is a real serving benchmark from the Red Hat AI Inference Server documentation running a small FP8 model on one card: about 21.7 requests per second, 8,775 output tokens per second, mean TTFT 193 milliseconds and mean TPOT 9 milliseconds. Those are the shape of the numbers. An 8B model like our Granite serves fewer tokens per second and a higher TTFT than that 1B example, which is exactly what the next section measures.

Measuring where the assistant stands

Serve the quantised Granite from Part 22 with an explicit set of tuning flags, so nothing is left to a default you did not choose. Read the token and the pull secret from the environment, never the command file.

# Tested against Red Hat AI Inference Server 3.2 (vLLM 0.10.x upstream),
# Granite 3.3 8B Instruct quantised to FP8 in Part 22, one NVIDIA H100 80GB,
# CUDA 12. HF_TOKEN comes from the environment, not the manifest.
podman run --rm -it --device nvidia.com/gpu=all --shm-size=8GB -p 8000:8000 
  --env "HF_TOKEN=${HF_TOKEN}" 
  -v ./rhaiis-cache:/opt/app-root/src/.cache 
  --security-opt=label=disable 
  registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.0 
  --model RedHatAI/granite-3.3-8b-instruct-FP8 
  --gpu-memory-utilization 0.90 
  --max-num-seqs 64 
  --max-num-batched-tokens 2048 
  --max-model-len 8192

Watch the startup banner, because it prints the two numbers that decide how much concurrency the card can hold. GPU KV cache size is how many tokens of cache fit after weights are loaded, and maximum concurrency is how many full length requests that supports at once.

INFO ... Using Flash Attention backend.
INFO ... GPU KV cache size: 244,912 tokens
INFO ... Maximum concurrency for 8,192 tokens per request: 29.90x
INFO ... Starting vLLM API server on http://0.0.0.0:8000

Now drive load at the server with the vLLM benchmark script and read the result. Running 200 prompts capped at 64 concurrent requests, each 1,024 tokens in and 512 out, gives the picture at our chosen batch ceiling.

$ python vllm/benchmarks/benchmark_serving.py --backend vllm 
    --model RedHatAI/granite-3.3-8b-instruct-FP8 
    --dataset-name random --random-input 1024 --random-output 512 
    --num-prompts 200 --max-concurrency 64 --port 8000

============ Serving Benchmark Result ============
Successful requests:                     200
Benchmark duration (s):                  34.71
Output token throughput (tok/s):         2902.44
Total Token throughput (tok/s):          8802.13
Request throughput (req/s):              5.76
---------------Time to First Token----------------
Median TTFT (ms):                        243.10
P99 TTFT (ms):                           471.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.83
==================================================

Read the P99 TTFT of 472 milliseconds carefully. If your target is a first token inside 300 milliseconds, this config already fails it at the tail even though median looks fine. That gap between median and P99 is where most latency complaints actually live, and it widens as batch size grows.

Throughput and latency pull against each other

Drop the concurrency cap to 32 and rerun, and the trade becomes visible in two lines. Throughput falls but the tail latency comes back under target.

$ python vllm/benchmarks/benchmark_serving.py ... --max-concurrency 32
Output token throughput (tok/s):         2118.90
Median TTFT (ms):                        188.20
P99 TTFT (ms):                           278.65
Mean TPOT (ms):                          13.40

So 32 concurrent meets a 300 millisecond tail at 2,119 tokens per second, while 64 concurrent breaks it for only 37 percent more throughput. Push concurrency higher and throughput barely climbs while TTFT runs away, because the card saturates and new requests queue behind the ones already decoding. Plotting the whole sweep shows a clear knee, and the cheapest safe config sits at it, not past it.

Tuning knobs that move the numbers

Four flags do almost all of the work. Raising gpu-memory-utilization gives vLLM more room for KV cache, which raises the concurrency the card can hold, until a long prompt runs it out of memory. Raising max-num-seqs lifts throughput and pushes TTFT up. Max-num-batched-tokens governs how many tokens a scheduler step processes, so higher values favour prefill and first token latency while lower values favour smoother inter token latency, and it must stay at or above max-num-seqs. Max-model-len sets the context ceiling, and larger values reserve more cache per sequence, which quietly cuts how many sequences fit. Keep this reference next to the serve command.

Here is the failure that lands most often when you get greedy with the first flag. Setting gpu-memory-utilization to 0.97 alongside a 16k context left no room for a real long prompt.

$ podman run ... --gpu-memory-utilization 0.97 --max-model-len 16384 ...
ERROR ... EngineCore encountered a fatal error.
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.94 GiB.
GPU 0 has a total capacity of 79.15 GiB of which 1.07 GiB is free.

# Cause: at 0.97 the KV cache plus a 12k token prompt activation peak left
# no headroom. Fix: give the card room back, and cap context if you can.
$ podman run ... --gpu-memory-utilization 0.90 --max-model-len 8192 ...

Chunked prefill, on by default in the vLLM V1 engine you read about in Part 21, softens this by slicing large prefills into pieces and prioritising decode, which steadies inter token latency under load. It is one of the few settings where the default is also the right answer for most traffic.

Where the cost model misleads you

Vendor tuning guides and the vLLM docs both push you toward maximum throughput: raise max-num-seqs, raise max-num-batched-tokens, fill the card. That advice is wrong the moment a latency target binds. Maximum throughput and minimum cost per token are not the same point once you refuse to ship a first token slower than, say, 300 milliseconds. Past the knee in the chart above you buy a few percent more tokens per second at the price of a doubled tail latency, which either violates the target or forces you to shed load, and shed load is just idle capacity wearing a different hat. The cheapest config is the highest throughput that still clears your latency target, and nothing beyond it.

Translate throughput into money and the utilisation point becomes concrete. At roughly 3,000 output tokens per second peak for Granite 8B FP8 on one H100, the cost per million output tokens depends far more on how full you keep the card than on which card it is. This table is the artifact I bring to the budget conversation.

Figures illustrative, at about 3,000 output tokens per second peak on one H100. Cost per million output tokens equals the hourly rate divided by tokens generated per hour, then divided by utilisation.

Two readings jump out. Moving from 30 to 60 percent utilisation halves the unit cost, which is a bigger win than most model or hardware swaps deliver, and it costs nothing but scheduling. And a self hosted card only beats a hosted API on price once utilisation is genuinely high; at 15 percent you are paying more per token than many managed endpoints charge, with none of their elasticity. The decision to self host should rest on data residency and control, the reason this assistant cannot call a hosted API at all, not on a cost claim that only holds at utilisation you have not yet reached. The GenAI cost breakdown frames the same trade from the buyer side, and the Data Science Series works the raw hardware maths in GPU cost, scale and sizing.

flowchart LR
  G[GPU hourly cost] --> C[Cost per million tokens]
  T[Peak tokens per second] --> C
  U[Average utilisation] --> C
  C --> V[Falls fast as utilisation rises]

Cutting the assistant GPU bill

For this assistant I would fix the latency target first, then find the highest max-num-seqs that clears it, which on our traffic was 32, not the 64 I started with. I would hold gpu-memory-utilization at 0.90 and resist the urge to chase the last few percent, because a Saturday page taught me what that space is for. And I would attack utilisation, not throughput, by routing overnight batch jobs and document reindexing onto the same card so it stops idling at the office hours rate. Higher utilisation is the cheapest performance win on the table, and it needs no new hardware.

Next part builds the benchmark harness properly, turning these one off measurements into a repeatable load test you can run before every model or config change, so a silent regression never reaches production unmeasured.

References

• Red Hat AI Inference Server, Validating benefits using key metrics
• vLLM, Optimization and Tuning
• Red Hat, Efficient and reproducible LLM inference, MLPerf Inference v5.1 results

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts