I once shipped a serving config that benchmarked six percent faster on the one test I ran by hand, pushed it on a Friday, and spent Monday morning explaining why chat felt slow. Mean latency really had improved. Tail latency had not, and a single manual run had no way to show me the difference.

Limits of a single manual run

Last part turned the serving numbers into a cost per answer and tuned three flags against a latency knee. This part makes those measurements repeatable, so the next model bump or flag change cannot ship a regression nobody caught. A curl against the endpoint tells you it is alive. One run of a benchmark script tells you how it behaved for a few seconds, at whatever load happened to be in flight at that moment. Neither is something you can trust to compare two builds, because neither is repeatable and neither reports the tail.

What a user feels as a slow assistant is almost never the median request. It is the P99, the one request in a hundred that waited behind a full batch while the card was busy decoding for everyone else. Measuring speed under load is a different job from measuring answer quality, which the GenAI Series covers under evaluating GenAI output. Here the question is latency and cost when the endpoint is busy, not whether the answer is correct, and the AI Engineering Series frames the same latency work from the application side in caching, batching and latency engineering.

Installing GuideLLM and pinning a workload

GuideLLM fixes the two things a hand run gets wrong. It ships its own corpus and drives the endpoint with a named, repeatable load pattern, and it uses the model’s own tokenizer. That last detail matters more than it sounds: tokens per second and time to first token both depend on how a prompt splits into tokens, so a benchmark that counts with the wrong tokenizer reports the wrong workload. Install it, confirm the endpoint is ready, then take a clean latency floor with one request at a time. Pin a workload profile that matches the assistant, roughly 512 prompt tokens of retrieval grounded question and 256 output tokens of answer.

# Tested against GuideLLM 0.3.1, Red Hat AI Inference Server 3.2 (vLLM 0.10.x
# upstream), Granite 3.3 8B Instruct FP8 from Part 22, one NVIDIA H100 80GB.
# HF_TOKEN is read from the environment, never written into a command file.
pip install guidellm[recommended]   # Python 3.9 to 3.12

# The endpoint from Part 20 is already serving on :8000. Confirm it is ready
# before benchmarking, or every request fails (proven in the next section).
curl -sf http://localhost:8000/health && echo OK

# One request at a time: a latency floor with no queueing.
guidellm benchmark 
  --target http://localhost:8000 
  --rate-type synchronous 
  --max-seconds 30 
  --data "prompt_tokens=512,output_tokens=256"

OK
===== Benchmarks Stats =====
 Type        | Req/s | TTFT med (ms) | TTFT p99 (ms) | ITL (ms) | Out tok/s
 synchronous | 1.90  | 42.3          | 61.0          | 11.9     | 84.2
Results saved to benchmarks.json

Running a sweep against the assistant

Before the numbers mean anything, one failure is worth causing on purpose, because it is the one that quietly wastes an afternoon. Point GuideLLM at the endpoint before vLLM has finished loading the FP8 weights, and it runs to completion against nothing at all.

$ guidellm benchmark --target http://localhost:8000 --rate-type sweep 
    --max-seconds 30 --data "prompt_tokens=512,output_tokens=256"
...
Benchmarks Info:
  Successful requests: 0
  Errored requests:    418
httpx.ConnectError: All connection attempts failed

# Cause: the server was still loading weights and was not yet accepting
# requests on :8000. GuideLLM does not wait for readiness on its own.
# Fix: block on the health endpoint first, then benchmark.
$ until curl -sf http://localhost:8000/health >/dev/null; do sleep 2; done

With the endpoint ready, a sweep ramps the load for you: one request at a time first, then all requests in parallel to find peak throughput, then intermediate rates to fill in the curve between. Save the result to a named JSON, because this run becomes the baseline everything else is measured against.

$ guidellm benchmark --target http://localhost:8000 
    --rate-type sweep --max-seconds 45 
    --data "prompt_tokens=512,output_tokens=256" 
    --output granite-33-fp8-baseline.json

Benchmarks Metadata: model=granite-3.3-8b-instruct-FP8  data=512/256
Benchmarks Stats (per rate):
  synchronous  req/s=1.90   TTFT p99=61ms    ITL=11.9ms  out tok/s=84
  concurrent5  req/s=8.10   TTFT p99=121ms   ITL=12.4ms  out tok/s=690
  concurrent10 req/s=12.30  TTFT p99=214ms   ITL=13.1ms  out tok/s=1180
  concurrent25 req/s=15.00  TTFT p99=402ms   ITL=15.8ms  out tok/s=1760
  throughput   req/s=16.10  TTFT p99=1090ms  ITL=22.0ms  out tok/s=2090
Saved 5 benchmarks to granite-33-fp8-baseline.json

Laid out as a table, the shape of the deployment is clear. Throughput keeps climbing as concurrency rises, but P99 time to first token climbs faster, and it crosses a 300 millisecond target somewhere between 10 and 25 concurrent requests. Keep this table next to the serve command; it is the reference for where the assistant still meets its latency budget.

Granite 8B FP8 on one H100, 512 prompt and 256 output tokens, figures illustrative. The shaded row is the operating point this assistant runs at, concurrency 10, where P99 TTFT still clears a 300 millisecond budget.

Tail latency versus throughput

Here is where common practice sends you wrong. GuideLLM guides and vendor tuning docs point you at a sweep and tell you to read the peak requests per second, and that is right for capacity planning. It is the wrong input for a regression gate. A sweep drives the card to saturation, and at saturation queueing is chaotic, so the peak throughput and its P99 latency wobble several percent between otherwise identical runs. Build a gate on those numbers and it fails on noise. For a gate you want a fixed concurrency held steady at your real operating point, run identically every time, reporting the tail your users actually feel.

The picture below is the whole argument for gating on the tail. Two builds at the same concurrency 10 operating point, same workload, same card. Mean latency barely moved between them. P99 time to first token jumped from 214 to 391 milliseconds and pushed the candidate over the target, a change a mean or a throughput figure would have hidden completely.

Catching a regression before it ships

A repeatable measurement is only useful if something compares it to the last one and refuses to promote a build that got worse. Save the sweep JSON from a known good build once as the baseline, benchmark each candidate at the operating concurrency, and let a small script decide. This is the artifact worth keeping: a regression gate that reads two GuideLLM JSON files and exits nonzero when the tail latency or throughput crosses the budget you set.

# compare_baseline.py  -  fail a build when serving got worse.
import json, sys

OP = 10                    # gate at the operating concurrency, not saturation
MAX_TTFT_P99_REGRESS = 0.15   # fail if P99 TTFT is more than 15 percent worse
MIN_TOK_S_RATIO      = 0.90   # fail if output throughput drops below 90 percent

def _scalar(x):            # a metric is a distribution summary or a plain number
    return x['mean'] if isinstance(x, dict) and 'mean' in x else x

def _p99(x):               # p99 lives at the top level or under percentiles
    if isinstance(x, dict):
        return x['p99'] if 'p99' in x else x['percentiles']['p99']
    return x

def pick(path, concurrency):
    data = json.load(open(path))
    for b in data['benchmarks']:
        m = b['metrics']
        if round(_scalar(m['request_concurrency'])) == concurrency:
            return _p99(m['time_to_first_token_ms']), _scalar(m['output_tokens_per_second'])
    sys.exit(f'no benchmark at concurrency {concurrency} in {path}')

base_ttft, base_tok = pick(sys.argv[1], OP)
cand_ttft, cand_tok = pick(sys.argv[2], OP)
ttft_delta = cand_ttft / base_ttft - 1
tok_ratio  = cand_tok / base_tok

print(f'P99 TTFT  base={base_ttft:.0f}ms cand={cand_ttft:.0f}ms delta={ttft_delta*100:+.0f}%')
print(f'Out tok/s base={base_tok:.0f}   cand={cand_tok:.0f}   ratio={tok_ratio:.2f}')

fail = ttft_delta > MAX_TTFT_P99_REGRESS or tok_ratio < MIN_TOK_S_RATIO
print('REGRESSION, failing the build' if fail else 'within budget, passing')
sys.exit(1 if fail else 0)

$ python compare_baseline.py granite-33-fp8-baseline.json granite-33-fp8-candidate.json
P99 TTFT  base=214ms cand=391ms delta=+83%
Out tok/s base=1180  cand=1290  ratio=1.09
REGRESSION, failing the build
$ echo $?
1

Throughput went up and the build still fails, which is the point. The candidate is faster in aggregate and slower for the individual user, and the gate encodes that you care about the user. Tune the two thresholds to your own budget, but keep them explicit and in version control, so a future you cannot quietly relax them under deadline. This is the serving twin of the regression testing the AI Engineering Series does on prompts and model upgrades in regression testing prompts and model upgrades.

Wiring the gate into CI

A gate a human remembers to run is a gate that eventually gets skipped. Put it in the pipeline that builds a serving change so it runs on every candidate without anyone deciding to. The flow is short: serve the candidate, wait for health, drive a fixed concurrency load, compare to the baseline, and let a nonzero exit stop the promotion.

flowchart LR
  A[Change model or flag] --> B[Serve candidate]
  B --> C[Wait for health]
  C --> D[GuideLLM concurrency 10]
  D --> E[Compare to baseline]
  E --> F[Fail build on regression]
  E --> G[Promote if within budget]

#!/usr/bin/env bash
set -euo pipefail          # any step failing stops the pipeline

# HF_TOKEN and the registry pull secret come from CI secrets, not the repo.
podman run -d --name cand --device nvidia.com/gpu=all -p 8000:8000 
  --env "HF_TOKEN=${HF_TOKEN}" 
  registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.0 
  --model RedHatAI/granite-3.3-8b-instruct-FP8 --max-num-seqs 48

until curl -sf http://localhost:8000/health >/dev/null; do sleep 2; done

guidellm benchmark --target http://localhost:8000 
  --rate-type concurrent --rate 10 --max-seconds 45 
  --data "prompt_tokens=512,output_tokens=256" 
  --output candidate.json

python compare_baseline.py baseline.json candidate.json   # nonzero fails CI

P99 TTFT  base=214ms cand=391ms delta=+83%
REGRESSION, failing the build
# step exits 1, the candidate never promotes

Because GuideLLM ships its own corpus and uses the local tokenizer, this whole gate runs with no outbound network call, which is the only way it can live in the disconnected environment this assistant is built for. Benchmarking catches the regression you cause on purpose; it does not catch the model slowly getting worse as traffic shifts, which is a monitoring job the Data Science Series covers in monitoring machine learning models, and the pipeline plumbing to run gates like this belongs with CI and CD for machine learning pipelines.

Benchmark gate to add this week

Install GuideLLM, run one sweep against your current good build, and read off the concurrency where P99 TTFT still clears your latency target. Save that sweep JSON as the baseline and commit it. Then add the two step gate to whatever ships a serving change: benchmark the candidate with –rate-type concurrent at that concurrency, and run compare_baseline.py so a regression exits nonzero. My verdict, use concurrent at your operating point and gate on P99 TTFT, and avoid gating on a sweep peak or on mean latency, the two numbers that look reassuring while a real regression walks straight past them.

Next part turns the safety question outward, adding input and output guardrails on OpenShift AI so the assistant refuses the prompts and redacts the answers it should never return, without wrecking the latency budget you just learned to defend.

References

• Red Hat, GuideLLM, Evaluate LLM deployments for real-world inference
• Red Hat, Benchmarking with GuideLLM in air-gapped OpenShift clusters
• GuideLLM, project repository and benchmark CLI

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts