Our finance lead asked one thing about the support assistant that I could not answer on the spot: what does it cost us per answer. I had a GPU invoice and a request count, and no honest number joining the two. This part builds that number, then uses it to decide when a self hosted deployment is worth its bill and when a managed API quietly wins.

Unit cost of a self hosted token

A hosted API hands you a unit price already: so many dollars per million input and output tokens. Self hosting hides that number inside a fixed hourly rental, and you have to reconstruct it. It is one division. Take the fully paid cost of the GPU for an hour, divide by the number of tokens it actually produced in that hour, and you have your cost per token. Last part grounded the assistant with RAG on a self hosted vector store; this part puts a price on every GPU hour that grounding now burns.

Concrete version. An A100 80GB on a one year reserved rate lands near 1.60 dollars per GPU hour. Running Granite 3.1 8B quantized to FP8, vLLM sustains roughly 5,500 output tokens per second when the batch is full. That is about 19.8 million tokens an hour, so the raw cost is 1.60 divided by 19.8, near 0.08 dollars per million output tokens. Set beside a hosted model at 0.20 to 0.90 dollars, self hosting looks like a bargain. It only looks that way because the second sentence assumed a full batch. Drop the load and the whole picture inverts, which is the point most build-versus-buy spreadsheets miss. The wider cost model behind this sits in the generative AI cost breakdown, and the throughput half of the equation is Part 24 on token economics and latency tuning.

GPU utilisation and what it does to unit cost

Your hourly rental does not care whether the card is flat out or asleep. You pay the same 1.60 dollars either way. So unit cost is just the raw rate divided by utilisation: at half load a token costs twice as much, at one tenth load it costs ten times as much. Continuous batching, the vLLM mechanism from Part 21, is what lets a card reach 80 to 95 percent busy instead of the 30 to 50 percent a naive one-request-at-a-time server settles at. Quantizing the weights, as Part 22 covered, frees memory for a bigger batch and pushes the same lever. Everything you do to serving is ultimately a move on this one ratio.

Here is the same A100 8B card at different loads. Nothing changes except how much of it you are using, and the effective cost per million output tokens climbs from small change to eye watering as the card empties out.

Those same points as a lookup you can price a scenario against. Raw figures, so you can add your own loaded multiplier on top.

Measuring real utilisation on OpenShift AI

You cannot manage this ratio without measuring it, and this is where the default metric betrays you. OpenShift AI exposes GPU telemetry through the upstream NVIDIA DCGM exporter feeding Prometheus, with Red Hat wiring it into the console dashboards; the platform integrates the plumbing, it did not invent DCGM. Most dashboards lead with DCGM_FI_DEV_GPU_UTIL, and that number is a trap for costing. It reports the card busy whenever any kernel is resident, so a single request decoding one token at a time can read 100 percent. It answers whether a kernel ran, not whether the card was full.

# Tested on OpenShift AI 2.x (RHOAI) with the NVIDIA GPU Operator and
# dcgm-exporter, vLLM as shipped in Red Hat AI Inference Server 3.x,
# Granite 3.1 8B. Queries run in the console under Observe, Metrics.

# What most dashboards show, and why it lies for cost:
DCGM_FI_DEV_GPU_UTIL{exported_pod=~"granite.*"}
# -> 100
# Reads 100 whenever a kernel is on the GPU, even one request decoding a
# single token. Says the card is busy, not that it is full.

# What to bill against instead, SM activity:
DCGM_FI_PROF_GR_ENGINE_ACTIVE{exported_pod=~"granite.*"}
# -> (empty)
# In the dcgm-exporter pod log:
#   Failed to watch fields group: Profiling is not supported for this group
#   of GPUs, or the profiling metrics are not in the exporter config.
# Why: DCP profiling fields are not in the default dcgm-exporter ConfigMap,
# and on MIG slices they are not reported per instance.
# Fix: add DCGM_FI_PROF_GR_ENGINE_ACTIVE to the exporter metrics ConfigMap,
# or read utilisation from vLLM itself, which always exposes it:
rate(vllm:generation_tokens_total[5m])   # output tokens per second
vllm:gpu_cache_usage_perc                # KV cache in use, 0 to 1
vllm:num_requests_running                # requests in the batch right now

# Seven day average for the assistant on its dedicated A100:
# rate(vllm:generation_tokens_total[5m]) -> 46.2   tokens/s
# vllm:gpu_cache_usage_perc              -> 0.03    3 percent of KV cache
# a card sized for 5,500 tok/s is doing 46

Read that against the coarse metric and the gap is the whole story. DCGM_FI_DEV_GPU_UTIL would have told the dashboard the card was 100 percent used all week. SM activity and the KV cache usage from vLLM say it is doing about 1 percent of what it could. Bill against the first and you conclude the GPU is maxed and you need another. Bill against the second and you realise you are paying full rate for a card that is asleep. This is the metric that contradicts the default dashboard, and it is worth wiring in before any cost review.

Self hosted versus a managed API, where the lines cross

Now the build-versus-buy question, answered honestly. A managed API is pure variable cost: zero traffic, zero bill, and every token priced. A self hosted card is fixed cost: you pay for it idle or flat out. Plot both against monthly volume and they cross. Below the crossover the API is cheaper because you are not paying for empty GPU; above it the card wins because its fixed cost spreads over enough tokens. For the assistant on one loaded A100 near 3,000 dollars a month, and a blended API near 0.50 dollars per million tokens, that crossover sits around 6 billion tokens a month.

This is the claim that contradicts most self hosting advice: for a workload the size of this assistant, self hosting is not cheaper, it is about fifty times more expensive per token. It earns its place only because the support tickets and customer records cannot be sent to a hosted endpoint, which was the founding constraint of the whole series. When residency does not force your hand, the honest move is often a hosted model plus routing, exactly the pattern the AI Engineering Series lays out in cost control and model routing.

Costs the unit price leaves out

Raw GPU rate is the cost people quote and the smaller half of the bill. Industry estimates put the card at only 30 to 40 percent of true infrastructure cost, so a realistic loaded multiplier is 2.5 to 3 times the sticker rate. What fills the gap is not exotic: the CPU host the card sits in, fast local storage for weights and the KV cache, network to move requests, the OpenShift AI control plane, and a fraction of an engineer who keeps drivers, operators and model upgrades from drifting. On the assistant that turns a 1,170 dollar card into a nearer 3,000 dollar line, and turns the tidy 0.08 dollar raw token into something closer to 0.20 loaded even at good utilisation, or the ugly numbers from the table once the card is idle.

Two structural costs deserve their own line. Idle time is the biggest, since inference traffic follows a business day and a reserved card is paid for the nights and weekends it serves almost nothing; roughly 55 to 80 percent of enterprise AI GPU spend goes to inference, and a good slice of that is cards waiting for requests. Reserved versus on demand is the other: committing a year drops the hourly rate by a third or more, but only if utilisation justifies owning the card at all. The fix for both is consolidation, and OpenShift AI gives you the mechanism directly through time slicing and MIG from Part 18 on GPU sharing: put the guardrail model, the RAG embedding model and several small assistants on one card so it is busy across the day. This is the same discipline the Data Science Series applies to training hardware in GPU cost, scale and sizing decisions.

Raise utilisation before you chase a cheaper token

Next part takes the assistant off the comfortable network entirely, into security and air gapped, disconnected deployments, where the cost trade offs here collide with the reality that nothing can pull an image or a model from the internet.

References

• NVIDIA GPU Operator on OpenShift, enabling the GPU monitoring dashboard and DCGM metrics
• Red Hat Developer, boost AI efficiency with GPU autoscaling on OpenShift
• vLLM production metrics reference

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts