I put the tuned Granite on a single L40S with 48 GB, started vLLM, and watched concurrency top out at a handful of requests while the card sat two thirds empty. Weights took 16 GB, and the server had quietly reserved 21 GB of KV cache to guarantee one request at the model 128k context. One flag, –max-model-len 8192, took concurrency from single digits to twenty one. Sizing a box for inference is mostly a KV cache problem, and the model weights are the part almost everyone gets right.

Where the assistant stands before sizing

Last part scored the tuned checkpoint, samples_1945, and it earned its place, plus 0.11 on product knowledge and nearly a full point on real support questions. This part answers the question that decides the bill, what to run it on. Two boxes are in play and they are not the same box. The one that tuned the model, back in Parts 9 and 10, ran synthetic data generation and multi phase training and needed several GPUs at once. A serving box needs a single card, but a different card depending on how many people ask questions at the same moment.

One term before any numbers. KV cache is the stored keys and values for every token already in a request, held in GPU memory so the model does not recompute them for each new token it generates. Every token inside the context window costs KV memory for as long as the request is live, and that single fact is why a sizing chapter exists at all. Weights you pay for once when the server starts. KV cache you pay for again on every concurrent request, which is why two engineers running the identical model on the identical card can serve wildly different numbers of users.

Two hardware jobs, training and serving

Red Hat publishes two separate hardware charts for RHEL AI, and reading the wrong one is the first sizing mistake. The end to end workflow chart, covering synthetic data generation, multi phase training and evaluation, starts at two A100 class GPUs and recommends four to eight. A separate inference serving chart lists single cards, from a 24 GB L4 up to a 192 GB MI300X. Same model, two very different bills, because tuning holds optimizer state and gradients in memory while serving holds only weights and KV cache. Keep this table beside your cloud console before you click launch.

The trap is renting the training box to do the serving. We needed 4x A100 for roughly nine hours to tune the model in Part 10, and we do not need it for a single minute to serve the result. Choosing that first serving card was framed back in Part 6, and now the model is real and the numbers get exact. Give the expensive training node back the moment training ends, and size the serving card on its own math.

GPU memory math for a served model

Two numbers set the whole plan. Weights are simple, parameters times bytes per parameter, so a bf16 8B model is 8.1 billion times two, 16.2 GB, and that figure never moves with load. KV cache is the number that moves. For each token you store one key and one value in every layer, across the model KV heads, at the head dimension, in the serving dtype. Granite 3.1 8B has 40 layers, 8 KV heads after grouped query attention, and a head dimension of 128, so one token costs 2 times 40 times 8 times 128 times 2 bytes, which lands at 160 KB. Grouped query attention, the reason there are 8 KV heads and not 32, shares one set of keys and values across several query heads.

# Tested on RHEL AI 1.5 bundled vLLM, Granite 3.1 8B Instruct, bf16
# kv_plan.py, size a served model before you rent the card
def kv_per_token_kb(layers, kv_heads, head_dim, dtype_bytes=2):
    # 2 counts the key and the value
    return 2 * layers * kv_heads * head_dim * dtype_bytes / 1024

def plan(vram_gb, weight_gb, layers, kv_heads, head_dim, ctx, util=0.9):
    kv_tok_kb = kv_per_token_kb(layers, kv_heads, head_dim)
    kv_budget_gb = vram_gb * util - weight_gb
    per_seq_gb = kv_tok_kb * ctx / 1024 / 1024
    seqs = int(kv_budget_gb / per_seq_gb)
    print(f'KV per token    : {kv_tok_kb:.0f} KB')
    print(f'KV per {ctx} ctx : {per_seq_gb*1024:.0f} MB per sequence')
    print(f'KV budget       : {kv_budget_gb:.1f} GB after weights')
    print(f'Max concurrent  : {seqs} sequences at {ctx} tokens')

# Granite 3.1 8B on one L40S: 40 layers, 8 KV heads, head_dim 128, 16.2 GB weights
plan(vram_gb=48, weight_gb=16.2, layers=40, kv_heads=8, head_dim=128, ctx=4096)

KV per token    : 160 KB
KV per 4096 ctx : 640 MB per sequence
KV budget       : 27.0 GB after weights
Max concurrent  : 43 sequences at 4096 tokens

Read the output and the sizing rule falls out. After 16 GB of weights an L40S has 27 GB left, and at 640 MB per 4k session that is about 43 concurrent requests. Not the 48 GB printed on the box, the 27 that survive the weights at a 0.9 memory utilization. Halve the context to 2k and the same card holds 86 sessions, double it to 8k and it holds 21. Context length is a dial you control, and it moves concurrency more than any hardware choice on the shelf.

Accelerator options compared

Run that same math across every card RHEL AI supports for inference and the result is a straight line, concurrency tracks spare memory after weights and almost nothing else. Two tables carry this section. First, what each card is good for, then the actual concurrency numbers behind those judgements.

Where sizing goes wrong

The most common sizing failure is not a slow server, it is a server that refuses to start. Point vLLM at the cheapest card, a 24 GB L4, leave the context at the model default of 128k, and vLLM does the KV math for you and quits, because a single 128k request needs about 21 GB of KV and only 5.4 GB survives the weights.

# tempting choice, cheapest card, one L4 at 24 GB, full context
$ vllm serve ~/models/granite-3.1-8b-support --max-model-len 131072

ValueError: To serve at least one request with the model's max seq len
(131072), 21.0 GiB KV cache is needed, which is larger than the available
KV cache memory (5.4 GiB). Based on the available memory, the estimated
maximum model length is 35376. Try increasing gpu_memory_utilization or
decreasing max_model_len when initializing the engine.

# fix: cap context at what you actually serve
$ vllm serve ~/models/granite-3.1-8b-support --max-model-len 8192
INFO ... GPU KV cache size: 35,376 tokens
INFO ... Maximum concurrency for 8192 tokens per request: 4.31x
INFO ... Application startup complete.

Read the error, it hands you the answer. It even estimates the maximum context the card can hold, 35376 tokens, so you know 8192 is safe with headroom to spare. A second failure hides on a shared card, where gpu-memory-utilization defaults high, another process already holds a few gigabytes, weights load, and then CUDA out of memory lands mid warmup rather than at request time. Keep this lookup beside the deploy and a red traceback becomes a one line change. Tensor parallelism, the row in that table that splits one model across several GPUs so each holds a shard of the weights, is a scale lever an 8B does not need, because 16 GB of weights fit whole on every card here, and it arrives when Part 13 moves to multi GPU serving.

Choosing the accelerator for this assistant

For this assistant, one tuned Granite 3.1 8B answering an internal queue with tens of people active at once, the pick is a single L40S at 48 GB. It serves the model at about 43 concurrent 4k sessions, leaves KV headroom for spikes, and costs a fraction of an 80 GB card. I would avoid renting an 8x H100 node to serve one 8B model, because you would light up 16 GB of 640 and pay for the rest to idle. For the periodic retune the honest answer is a different box entirely, four A100s rented for the hours the job runs and released the moment it finishes. Match the card to the job, not to the logo on the model.

flowchart TD
  A[Job to run] --> B{Serve or tune}
  B -->|Serve one 8B| C{How much concurrency}
  C -->|Under 40 at 4k| D[One L40S 48 GB]
  C -->|Near 90 at 4k| E[One A100 or H100 80 GB]
  C -->|Long context or big batch| F[One H200 or MI300X]
  B -->|Tune the 8B| G[Rent 4x A100, give it back]

Serving cost per token, batching and latency are their own engineering discipline, laid out for hosted models in the AI Engineering series under caching, batching and latency, and the batch against real time trade off is covered in the Data Science series under serving machine learning models. Making this single card faster and cheaper per token, then serving many copies at scale on OpenShift AI, is where Part 13 goes next.

References

• Red Hat Enterprise Linux AI 1.5, hardware requirements
• vLLM, optimization and conserving memory
• Granite 3.1 8B Instruct model card, architecture

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts