A freshly deployed vLLM endpoint on OpenShift AI will let 100 concurrent requests pile onto a single pod before Knative even considers adding a second one. That number is the platform default, and it is wrong for every large language model I have served. Part 16 left the support assistant with a Granite version marked production in the registry and no way to talk to it, just a row of metadata pointing at some weights in object storage. This part gives that pointer a network address and an API, then spends most of its length on the two settings, the deployment mode and the scaling target, that decide whether the endpoint survives contact with real traffic.

From a registered version to a live endpoint

Last part the registry got a Granite 3.1 8B checkpoint marked production, sitting in S3 at a URI nobody could call. This part gives that URI a network address and an API. Same tuned model, same S3 bucket, same MT-Bench gate from earlier in the series; what we add is a KServe InferenceService that pulls those weights onto a GPU and exposes them on the OpenAI chat completions API. By the end the assistant answers a real HTTP request instead of living as a row in a metadata store.

Red Hat did not write this serving layer. Underneath sits KServe, an upstream Kubeflow project that turns a model into a Kubernetes custom resource, and vLLM, the upstream inference engine that actually runs the model. Red Hat contributes the single model serving platform packaging, a set of supported and pre built serving runtimes, dashboard wiring, and integration with OpenShift Serverless and Service Mesh, plus the support line when a pod will not start at three in the morning. Serving concepts in general, batch against real time, are covered generically in the Data Science Series piece on serving machine learning models; what changes here is that the mechanics are Kubernetes native and the model is an 8B language model rather than a scikit-learn classifier.

KServe and the two objects that serve a model

Serving one model on this platform means creating two objects, and keeping them straight saves a lot of confusion. A ServingRuntime describes the container that runs models of a given format, its image, its ports, its default arguments. An InferenceService describes one deployment, which runtime to use, which model to load, where the weights live, how many replicas and how much GPU. Red Hat ships the runtime you want already built: the one named vLLM NVIDIA GPU ServingRuntime for KServe, with an Intel Gaudi variant for Gaudi accelerators. That naming matters because only the vLLM runtime exposes the OpenAI REST API, so a client written against a hosted provider keeps working against this endpoint, the same portability argument the AI Engineering Series makes in choosing and switching providers.

Weights can come from three places, and the choice has consequences later. KServe pulls from S3, from a PVC, or from an OCI image, the last one packaged as a ModelCar container so the model ships like any other image. For the assistant the S3 URI recorded in the registry is the natural source, since Part 16 already stored it and the data connection already holds the credentials. When this series reaches air gapped serving, the same InferenceService swaps that S3 URI for an OCI reference and nothing else changes. Picture the request path before the YAML, because the path is where latency and the cold start problem both live.

flowchart LR
  U[client with OpenAI SDK] --> R[OpenShift route]
  R --> K[KServe InferenceService]
  K --> P[vLLM predictor pod]
  P --> G[GPU loads Granite from S3]
  P -->|streams tokens| U
  K -->|idle past retention| Z[scaled to zero]
  Z -->|next request| C[cold start, reload weights]

Deploying Granite with a vLLM InferenceService

Here is the InferenceService for the assistant, in RawDeployment mode, pulling the production weights from the S3 URI the registry recorded. No key is typed into the file; the credentials come from the data connection secret the platform mounts for you, the same assistant-s3-conn connection the pipeline already uses.

# Tested on Red Hat OpenShift AI 3.4 self managed, single model serving platform
# Runtime: vLLM NVIDIA GPU ServingRuntime for KServe. Storage: S3 URI from the registry
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: support-assistant
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment   # Standard mode, no Knative
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 3
    model:
      modelFormat:
        name: vLLM
      runtime: vllm-granite-runtime          # the ServingRuntime you created from the template
      storageUri: s3://assistant-models/granite-retrain/epoch-1
      args:
        - --max-model-len=8192               # cap KV cache so it fits the GPU
      resources:
        requests:
          cpu: '4'
          memory: 8Gi
          nvidia.com/gpu: 1
        limits:
          cpu: '8'
          memory: 12Gi
          nvidia.com/gpu: 1

$ oc apply -f support-assistant-isvc.yaml
inferenceservice.serving.kserve.io/support-assistant created
$ oc get inferenceservice support-assistant
NAME                URL                                                     READY
support-assistant   http://support-assistant.assistant.svc.cluster.local    True
$ oc get pods -l serving.kserve.io/inferenceservice=support-assistant
NAME                                           READY   STATUS
support-assistant-predictor-7c9d8f6b4-r4k2p    1/1     Running

Once READY is True and a route is exposed, the endpoint speaks the OpenAI API. Read the URL from the environment, never from a literal in code, and the same call you made against a hosted provider works unchanged.

# endpoint comes from the environment, not a hardcoded string
$ export URL="https://support-assistant-assistant.apps.example.com"
$ curl -s $URL/v1/models | python -m json.tool
{
    "object": "list",
    "data": [
        {"id": "support-assistant", "object": "model", "owned_by": "kserve"}
    ]
}
$ curl -s $URL/v1/chat/completions -H "Content-Type: application/json" 
    -d '{"model":"support-assistant","messages":[{"role":"user","content":"how do I reset my VPN token"}],"max_tokens":64}'
# {"choices":[{"message":{"role":"assistant",
#   "content":"Open the self service portal, choose Security, then Reset VPN token ..."}}]}

My first deploy did not get that far. I left max-model-len unset, trusting Granite 3.1 to advertise its full context, and the predictor pod crashed on start. Granite 3.1 8B claims a 128k token context, but the weights alone are roughly 16 GB in bf16, and on a single 24 GB A10G that leaves only a few gigabytes for the KV cache, the memory vLLM reserves to hold the running conversation. Nowhere near 128k tokens fit, so vLLM refused to boot rather than lie about capacity.

$ oc logs support-assistant-predictor-7c9d8f6b4-r4k2p
...
ValueError: The model's max seq len (131072) is larger than the maximum
number of tokens that can be stored in KV cache (48016). Try increasing
gpu_memory_utilization or decreasing max_model_len when initializing the engine.

Setting --max-model-len=8192 fixed it in one redeploy. The assistant answers support questions, not novels, so an 8k window is generous and the freed memory goes to serving more concurrent requests. That trade, context length against concurrency on a fixed GPU, is the single knob you will turn most often on this platform, and the hardware sizing in Part 12 is where the KV cache math comes from.

Standard and Advanced deployment modes

KServe can run an InferenceService two ways, and the dashboard labels them in words that hide what they are. Standard is RawDeployment, plain Kubernetes Deployments and Services with nothing in the request path. Advanced is Serverless, which pulls in OpenShift Serverless, the Red Hat build of Knative, and OpenShift Service Mesh, the Red Hat build of Istio, to add scale to zero and request based autoscaling. More capability, more moving parts, and a cold start you pay for. This table is the one I keep open when someone asks which mode a new endpoint belongs in.

My verdict for the assistant is RawDeployment, and it is the mode I reach for by default. A user facing support endpoint sees traffic all day, so scale to zero saves nothing and the Knative and mesh hops only add latency and two more failure domains. Reach for Serverless when traffic genuinely comes in bursts with long idle gaps, or on a shared dev cluster where idle GPUs are pure waste. One honest caveat on Standard mode: until recently its only autoscaling signals were CPU and memory, useless for a GPU bound model, so a busy endpoint could not scale out on load. The KEDA based request autoscaling that fixes this landed in the 3.x line, which narrows the old reason to choose Serverless [VERIFY exact GA release].

Autoscaling and the cold start tax

Serverless mode brings two levers that look like free money and each hides a bill. Scale to zero drops replicas to none when nobody is calling, and request based scaling adds replicas as a queue builds. Both are governed by Knative annotations on the predictor, and both ship with defaults tuned for stateless web services, not for a model that takes tens of seconds to load onto a GPU.

# Serverless (Advanced) only. Annotations on the predictor tune Knative scaling.
spec:
  predictor:
    annotations:
      autoscaling.knative.dev/target: '5'                              # queue depth per pod, not 100
      autoscaling.knative.dev/scale-down-delay: '10m'                  # wait before removing a pod
      autoscaling.knative.dev/scale-to-zero-pod-retention-period: '2m' # keep a warm pod after last call

Notice the target of 5. Knative defaults that value to 100 concurrent requests per pod, a sane number for a lightweight web handler and a disaster for an 8B model that saturates well before then. Left at the default, requests queue behind a maxed out pod for seconds while Knative sees no reason to scale, since it is still under target. Set the target to a small number, five to eight as a start, then confirm it with a real load test rather than a guess. GuideLLM, the same benchmarking tool the vLLM project ships, is how you find the right value, and the latency engineering behind these numbers is covered in the AI Engineering Series piece on caching, batching and latency.

The cold start bill came due on my second night with scale to zero on. I switched it on to stop paying for an idle A10G overnight, felt clever, and the next morning the first support query timed out at the client’s 30 second limit while the pod cold started, roughly 40 seconds to pull the image, load 16 GB of weights and warm the KV cache. Every first request after an idle gap returned a 503 until the pod was ready. I set minReplicas back to 1 for the interactive endpoint and kept scale to zero only on a separate batch evaluation service, where a pipeline waits for readiness anyway. This chart is the tax I was ignoring.

Where KServe serving breaks

Keep this lookup next to the terminal. Five symptoms cover almost every serving failure I have hit on this platform, and each maps to a one line cause and fix.

One more that costs an afternoon if you meet it cold: a model that pulls fine and starts, then answers with an endpoint that reports the wrong model name. That happens when the served name defaults from the InferenceService rather than the value your client sends in the model field, so a client passing a stale name gets a 404 on chat completions. Send the exact name from /v1/models, or set the served model name explicitly, and the mismatch disappears.

Serve one version, then benchmark before you trust it

References

• Red Hat OpenShift AI, serving models on the single model serving platform
• Red Hat Developer, autoscaling vLLM with OpenShift AI, Standard and Advanced modes
• KServe, the upstream model serving project on Kubernetes

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts