A colleague pinged me on day one of the assistant with a fair question: the curl test returns a clean answer, so why is the chat window blank? Both were pointed at the same box. One was serving granite-3.1-8b-instruct, the other was chatting to a slightly different model path, and vLLM answered the request it recognised and quietly dropped the one it did not. That mismatch, not a crash, is the most common way a first local serve looks broken.
Last part the assistant went from a bootable image to granite-3.1-8b-instruct sitting on disk. This part serves that model with ilab and vLLM, calls it from Python, then locks the endpoint down, all on the single L40S box we sized in Part 6. Serving is one command; making that command reproducible and safe is the rest of the work.
Serving through ilab versus raw vllm serve
vLLM is the open source inference server that does the real work here, with paged attention and continuous batching, and we take its internals apart in Part 21. Red Hat does not replace it; it ships a supported vLLM build inside the RHEL AI image and wraps it with ilab so that serving reads from a config file. In Part 6 we ran vllm serve by hand on purpose, to watch the KV cache refuse to allocate, and a bare command is fine for a probe like that.
It is a poor way to run anything you care about, though. A hand typed vllm serve dies when the SSH session drops, and it forgets every flag the moment you restart it. ilab model serve reads its arguments from config.yaml, so the box comes back the same way every time, and you can later wrap it in a service that restarts on crash. For anything past a one off test, drive vLLM through ilab, and keep the raw command only for debugging.
Starting the server with ilab model serve
With granite-3.1-8b-instruct already downloaded, serving it is a single command. Tested against RHEL AI 3.5 with the bundled vLLM and Granite 3.1 8B Instruct:
# RHEL AI 3.5, bundled vLLM, Granite 3.1 8B Instruct
$ ilab model serve --model-path ~/.cache/instructlab/models/granite-3.1-8b-instruct
INFO Using model granite-3.1-8b-instruct with vLLM backend
INFO Automatically detected dtype: torch.bfloat16
INFO Model weights loaded on GPU: 15.9 GiB
INFO Started server process
After application startup complete see http://127.0.0.1:8000/docs for API.
Press CTRL+C to shut down the server.Run with no arguments, ilab model serve loads whatever model config.yaml names as the default. That last line is the signal you want: an OpenAI compatible server is live on port 8000, with interactive docs at /docs and the chat route at /v1/chat/completions. Nothing has been sent to it yet, but the endpoint exists.
Passing vLLM arguments through config.yaml
Every vLLM flag you would type on a bare command belongs instead in the serve section of config.yaml, under vllm_args, where ilab passes it straight through. Open the file with ilab config edit and set the arguments that matter for one card.
$ ilab config edit
# ~/.config/instructlab/config.yaml
serve:
model_path: ~/.cache/instructlab/models/granite-3.1-8b-instruct
vllm:
vllm_args:
- --served-model-name
- granite-3.1-8b-instruct
- --tensor-parallel-size
- '1'
- --gpu-memory-utilization
- '0.9'
- --max-model-len
- '16384'Keep the table below near your config; it is the vllm_args reference for this box, the six arguments that decide whether a single card serves cleanly.
| Argument | What it does | Value on the L40S box |
|---|---|---|
| –served-model-name | Name clients pass in the model field | granite-3.1-8b-instruct |
| –tensor-parallel-size | Shards the model across N host GPUs | 1, one card only |
| –gpu-memory-utilization | Fraction of the card vLLM may claim for weights plus KV cache | 0.9 |
| –max-model-len | Longest prompt plus output accepted | 16384, the real ceiling |
| –dtype | Weight precision vLLM loads | auto, resolves to bfloat16 |
| –api-key | Require a bearer token on every request | read from VLLM_API_KEY |
Talking to the endpoint from Python and the chat CLI
With the server up in one terminal, chat from another with the built in client. ilab model chat opens a REPL against the same endpoint.
$ ilab model chat --model ~/.cache/instructlab/models/granite-3.1-8b-instruct
╭─────────────────────────── system ───────────────────────────╮
│ Welcome to InstructLab Chat w/ GRANITE-3.1-8B-INSTRUCT │
╰───────────────────────────────────────────────────────────────╯
>>> In one sentence, what does a support assistant need from its docs?
Clear, current answers tied to a source the reader can open,
so a reply can be trusted and checked.For the application itself you want the HTTP endpoint, not the REPL. Because it speaks the OpenAI protocol, the standard client works unchanged; point base_url at the local server and read any key from the environment rather than pasting it in. Tokens and how they are counted are covered in the GenAI series under tokens and embeddings.
import os
from openai import OpenAI
client = OpenAI(
base_url='http://localhost:8000/v1',
api_key=os.environ.get('VLLM_API_KEY', 'EMPTY'), # local serve needs no key
)
resp = client.chat.completions.create(
model='granite-3.1-8b-instruct', # must match --served-model-name
messages=[{'role': 'user', 'content': 'Reply with the single word ready.'}],
max_tokens=5,
)
print(resp.choices[0].message.content)
# readyHere is the blank chat window from the opening, reproduced on purpose. Point the model field at a name the server is not serving and vLLM does not guess; it returns a 404 that a thin client can swallow into an empty reply.
>>> model='granite-8b-code-instruct' # not the served name
openai.NotFoundError: Error code: 404
message: the model granite-8b-code-instruct does not exist
# fix: pass the exact --served-model-name, granite-3.1-8b-instructPin one served name in config.yaml and use that same string everywhere, in ilab model chat, in the Python client, in whatever ships to production. Batch versus real time serving, and the trade offs behind each, the Data Science series covers in general terms under serving machine learning models.
Where tensor parallel serving breaks
Tensor parallelism splits one model across several GPUs so a model too large for one card can still serve, and you turn it on by setting –tensor-parallel-size to the number of GPUs. On the single L40S the assistant runs on, that value is 1 and there is nothing to think about. On the four card box I keep for tuning, I got greedy and reused its config to serve, with –tensor-parallel-size set to 4.
Serving would not start. A stuck vllm process from an earlier run still held one card, so vLLM saw three GPUs, not four, and refused outright:
$ ilab model serve
AssertionError: please set tensor_parallel_size to less than max local gpu count
$ nvidia-smi --query-gpu=index,memory.used --format=csv
index, memory.used [MiB]
0, 41216 MiB # orphaned vllm process still holding this card
1, 3 MiB
2, 3 MiB
3, 3 MiBTwo things fix it, and I wasted about twenty minutes before checking the obvious one. Kill the orphan holding GPU 0 so all four cards are free, or drop –tensor-parallel-size to match what vLLM can actually see. The rule is plain once the panic passes: tensor-parallel-size can never exceed the GPUs visible to the server, and a leftover process or a narrowed CUDA_VISIBLE_DEVICES quietly narrows that count. On the L40S serving box, tensor-parallel-size stays at 1 and this failure never appears.
Throughput on a single L40S
Continuous batching is why one card serves more than one user well. Rather than finishing one request before starting the next, vLLM interleaves many in flight, so aggregate output climbs steeply with concurrency until the KV cache fills. Measured on our single L40S with the four bit build of Granite 3.1 8B, aggregate output went from about 90 tokens per second for one caller to roughly 1,300 for thirty two concurrent ones, approximate and workload dependent.
Read the throughput and the latency together. Higher concurrency lifts total tokens per second but stretches the wait for the first token, from about 40 ms with the card idle to about 280 ms with thirty two requests batched, so a chat UI feels slower under load even as the box does more work overall. Where this behaviour comes from, and how to tune it with batching and caching, the AI Engineering series covers under caching, batching and latency. To reproduce these figures on your own box, drive a fixed request mix at the endpoint and read the tokens per second vLLM prints in its logs, which is the honest way to size concurrency before launch, and full load benchmarking is Part 25.
Running serve as a systemd service with an API key
A serve command tied to your SSH session dies with it. Wrap it in a user systemd service so it starts on boot and restarts on crash, which is the whole reason for keeping arguments in config.yaml rather than the command line.
$ cat $HOME/.config/systemd/user/ilab-serve.service
[Unit]
Description=ilab model serve service
[Service]
ExecStart=ilab model serve --model-family granite
Restart=always
[Install]
WantedBy=multi-user.target default.target
$ systemctl --user daemon-reload
$ systemctl --user start ilab-serve.service
$ sudo loginctl enable-linger # keep it running after logoutBy default the endpoint has no authentication, which is fine bound to localhost and dangerous the moment it is reachable off box. Generate a key into an environment variable, add –api-key to vllm_args, and never write the key itself into config.yaml or a script; read it from the environment or a mounted secret. Miss the header and vLLM answers with a plain 401.
$ export VLLM_API_KEY=$(python -c 'import secrets; print(secrets.token_urlsafe())')
# client without the key
openai.AuthenticationError: Error code: 401
error: Unauthorized
# client reading the key from the environment: succeeds
$ ilab model chat -m granite-3.1-8b-instruct
--endpoint-url http://localhost:8000/v1 --api-key $VLLM_API_KEYWith a restart safe service and a key on the endpoint, the assistant is served in a way you can hand to a teammate without a page of caveats. How this same model gets tuned on the company docs comes next, and how vLLM achieves this throughput underneath is Part 21.
Local serving pipeline at a glance
Recommended local serving setup for the support assistant
For the assistant I would serve granite-3.1-8b-instruct through ilab model serve, with every argument in config.yaml, tensor-parallel-size at 1 on the single L40S, gpu-memory-utilization at 0.9, and max-model-len at 16384, then run it as a user systemd service with an API key read from the environment. Avoid two habits: hand typing vllm serve for anything meant to persist, because it forgets its flags and dies with your shell, and setting tensor-parallel-size above the GPUs the server can see, which is the assertion that stops most first serves cold.
Next in the series we stop serving the stock model and start making it ours, building a taxonomy and generating synthetic data with InstructLab so the assistant learns the company docs.
References
- Red Hat Enterprise Linux AI 1.5, serving and chatting with the models
- Red Hat AI Inference Server 3.1, vLLM server arguments
- Red Hat Customer Portal, tensor_parallel_size assertion on ilab model serve
- vLLM, vllm serve command reference
- Red Hat Developer, download serve and interact with LLMs on RHEL AI
About The Author
