One line from the server ended a planning meeting faster than any slide could: ValueError: No available memory for the cache blocks. We had a 48 GB card, an eight billion parameter model that used 16 GB of it, and we still could not serve a single 128k token request. Nobody in the room had done anything wrong on paper. We had sized the box the way you size a laptop, by the number printed on the model card, and that number is the one that lies to you.

Choosing a first model and accelerator is not a hunt for the biggest model a card can hold. It is a memory budget you cannot read off a spec sheet, plus a second machine most teams forget to plan for. This part works through both using the support assistant we have been building. Last part we wrote the taxonomy that will tune it; this part we choose the Granite size and the GPU it will actually run on, because that one decision constrains Parts 7 through 12.

Where the memory actually goes

Three things sit in GPU memory when you serve a model, and only one of them is on the model card. Model weights come first. Granite 3.1 8B in bfloat16 uses two bytes per parameter, so eight billion parameters is about 16 GB before you serve anyone. Quantize to eight bit and that roughly halves to 8 GB; quantize to four bit weights with the w4a16 build and it drops to about 5 GB. Weights are the easy part, because they are fixed and you can look them up.

KV cache is the part that surprises people. Every token already in the context leaves a set of attention keys and values on the card, so the next token can be generated without recomputing the whole sequence. That store grows linearly with context length and with the number of concurrent requests. For Granite 3.1 8B, with 40 layers and eight grouped query attention key value heads at a head dimension of 128, each token costs about 160 KiB of cache. Multiply that by a 128k context and you are near 20 GB for one request, sitting on top of the 16 GB of weights. Runtime overhead, CUDA graphs and activations then add another gigabyte or two.

A short script makes the trade visible. Tested against RHEL AI 3.5 and Granite 3.1 8B Instruct in bf16:

# fit_estimator.py  (RHEL AI 3.5, Granite 3.1 8B Instruct, bf16)
PARAMS_B = 8.0
BYTES_PER_PARAM = 2       # bf16
LAYERS = 40
KV_HEADS = 8             # grouped query attention
HEAD_DIM = 128
KV_BYTES = 2             # fp16 key plus value, per element

def gib(x):
    return round(x / (1024 ** 3), 1)

weights = PARAMS_B * 1e9 * BYTES_PER_PARAM
per_token_kv = 2 * LAYERS * KV_HEADS * HEAD_DIM * KV_BYTES   # bytes per token

for ctx, users in [(4096, 1), (16384, 4), (131072, 1)]:
    kv = per_token_kv * ctx * users
    total = weights + kv + 1.5e9        # 1.5 GB runtime overhead
    print('ctx', ctx, 'users', users,
          'weights', gib(weights),
          'kv', gib(kv),
          'total', gib(total), 'GiB')

ctx 4096 users 1 weights 14.9 kv 0.6 total 16.9 GiB
ctx 16384 users 4 weights 14.9 kv 10.0 total 26.3 GiB
ctx 131072 users 1 weights 14.9 kv 20.0 total 36.3 GiB

Read the last row again. Weights are 14.9 GiB, but a single 128k request pushes the total past 36 GiB. That is the number that decides your card, and it is nowhere on the model card. Layer and head counts here come from the Granite 3.1 8B config; confirm them in config.json for the exact build you serve [VERIFY per build].

Granite sizes worth starting with

Granite 3.1 ships in two dense instruct sizes that matter for a first deployment, 2B and 8B, both with a 128k context and both under Apache 2.0, which we unpacked in Part 2. There are mixture of experts variants and a separate Granite Guardian model for safety checks, but for a support assistant the real choice is 2B versus 8B, and then bf16 versus a quantized build of the 8B. Upstream has moved on to Granite 3.3, yet the model Red Hat validates and ships as the RHEL AI starter is Granite 3.1 8B, so that is what I would build against first and swap later only if the eval set asks for it.

Download is one command. RHEL AI pulls the starter model from the Red Hat registry, and you authenticate once through ilab config and your registry credentials rather than pasting a token into a script.

# bf16 starter, verified tag
$ ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3-1-8b-instruct:1.5

# four bit build for a smaller card (RedHatAI on Hugging Face)
$ ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w4a16:1.5   # [VERIFY tag]

$ ilab model list
+-----------------------------------------+----------+
| Model Name                              | Size     |
+-----------------------------------------+----------+
| granite-3.1-8b-instruct                 | 15.9 GB  |
| granite-3.1-8b-instruct-quantized.w4a16 | 5.2 GB   |
+-----------------------------------------+----------+

Grab both the bf16 and the w4a16 build in the same session. You want them side by side so you can A/B them on the eval set from Part 5 before you commit a card to either one.

One number keeps the four bit choice honest: the w4a16 build of Granite 3.1 8B typically recovers around 99 percent of the bf16 model score on standard benchmarks, per the RedHatAI model card, which is usually inside the noise of your own eval set, yet it frees about 11 GB of memory. If you also run Granite Guardian to screen inputs and outputs, remember it is a second model that wants its own slice of the card, so size for both before you decide one GPU is enough.

Accelerators Red Hat AI supports for serving

Red Hat publishes a supported accelerator list, and straying off it means you own the drivers and the support burden yourself. For inference serving on RHEL AI the current list runs from a 24 GB L4 up to a 141 GB H200, with AMD MI300X at 192 GB and Intel Gaudi 3 in technology preview. Check what you actually have first, because the reported figure is a little under the marketing number.

$ nvidia-smi --query-gpu=name,memory.total --format=csv
name, memory.total [MiB]
NVIDIA L40S, 46068 MiB

Keep the table below; it is the accelerator sizing table for this whole series. It maps each supported serving card to what it can and cannot do with Granite 3.1 8B, using the memory budget from the first section. For the broader economics of GPU choice, the Data Science series covers GPU cost, scale and sizing in general terms.

Serving Granite on a single GPU

With a model downloaded and a card chosen, serving is one command. RHEL AI wraps vLLM, the open source inference server we take apart in Part 21, and starts an OpenAI compatible endpoint on port 8000. Here is where the memory budget stops being theoretical.

On the assistant, I sized a single L40S with 48 GB, served granite-3.1-8b-instruct in bf16 at a 4k context, and it was quick and clean. Then I set the context to 128k to match the model card, restarted, and every request longer than a few thousand tokens died. Weights were 16 GB, the 128k KV cache wanted about 20 GB more than the card had left, and vLLM refused to allocate the cache at all.

$ vllm serve RedHatAI/granite-3.1-8b-instruct --max-model-len 131072
INFO   Automatically detected dtype: torch.bfloat16
INFO   Model weights loaded on GPU: 15.9 GiB
ERROR  ValueError: No available memory for the cache blocks. Try increasing
       gpu_memory_utilization or decreasing max_model_len when initializing
       the engine.

Two fixes brought it back. Capping max model length to 16384, the longest ticket plus document we actually saw, kept the cache inside 48 GB. Switching to the w4a16 build freed another 11 GB by shrinking weights from 16 GB to 5 GB, which left real headroom for concurrent users. We shipped with both changes, and served the friendly model name so nothing downstream had to know it was quantized.

$ vllm serve RedHatAI/granite-3.1-8b-instruct-quantized.w4a16 
    --served-model-name granite-3.1-8b-instruct 
    --max-model-len 16384
INFO   Automatically detected dtype: torch.float16
INFO   GPU KV cache size: 6144 blocks
INFO   Uvicorn running on http://0.0.0.0:8000

Confirm it serves with a plain client call. Read any real endpoint key from the environment; a local serve needs none.

import os
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:8000/v1',
    api_key=os.environ.get('VLLM_API_KEY', 'EMPTY'),   # local serve needs no key
)
resp = client.chat.completions.create(
    model='granite-3.1-8b-instruct',
    messages=[{'role': 'user', 'content': 'In one sentence, what is a KV cache?'}],
    max_tokens=60,
)
print(resp.choices[0].message.content)

# A KV cache stores the attention keys and values for tokens already processed,
# so the model can generate the next token without recomputing the sequence.

That whole detour cost about a day, and it would have cost nothing if I had run the estimator first. How this endpoint behaves under load, batching and latency, is its own subject; the AI Engineering series covers caching, batching and latency, and the Data Science series covers batch versus real time serving in general terms.

Serving hardware will not run training

Here is the part the starter tutorials gloss over. That L40S will serve the assistant all day, but it cannot tune it. Red Hat’s full InstructLab workflow, synthetic data generation then multi phase training then evaluation, carries a hard floor in the hardware requirements: at least two data center GPUs and 160 GB of aggregate memory for A100 or H100 class cards, or four L40S at 192 GB, with 3 TB of scratch disk instead of the 1 TB serving asks for. One L4 or one L40S does not qualify, full stop.

So the honest bill of materials for the support assistant is two machines, not one: a modest serving card that runs the model in production, and a short lived multi GPU node you rent only while tuning the taxonomy from Part 5. Buying a single box because it served your first demo is the mistake that surfaces three weeks later, when you try to run training and the job will not even start. We tune it properly in Parts 9 and 10; for now, budget the second node.

Renting that tuning node is not only about cash. An 8x H100 box costs the same whether it runs for three hours a month or sits idle for the other 717, so owning one to tune a model you retrain every few weeks means paying for hundreds of dark GPU hours. Rent it for the run, release it after evaluation, and the tuning line on your bill tracks real work rather than fear of the next run.

Matching model size to accelerator

Putting the pieces together gives a short decision path. Start from the model, check whether weights plus KV cache fit one card at your real context, and only then decide whether to quantize, cap context, or add a GPU. Tuning is a separate question with a separate, larger machine.

flowchart TD
  A[Pick smallest Granite that clears eval] --> B{Weights plus KV fit one card at your context}
  B -->|Yes| C[Serve on one GPU]
  B -->|No| D{Can you shorten context or quantize}
  D -->|Yes| E[Use w4a16 or cap max model len]
  D -->|No| F[Add a GPU or move to a larger card]
  C --> G{Need to tune with InstructLab}
  E --> G
  F --> G
  G -->|Yes| H[Provision 2 plus data center GPUs, 160 GB floor]
  G -->|No| I[Serving box is enough]

Recommended first setup for the support assistant

For the assistant, I would serve granite-3.1-8b-instruct in the w4a16 build on a single L40S with 48 GB, capped at a 16k context, which fits with room for a handful of concurrent users and keeps the option to move back to bf16 later. For the periodic InstructLab tuning I would rent an 8x H100 node with 640 GB for the few hours a run takes, rather than buying idle GPUs that sit dark between runs. Avoid two things: a single L4 as your only machine, because it serves a quantized 8B but can never tune one, and any plan that sets max model length to 128k on a card with less than about 40 GB free. On Monday, run the estimator against your real context length and concurrency, download both the bf16 and w4a16 builds, and A/B them on your eval set before you commit to any hardware. Next in the series we install RHEL AI on that first box and take it from a bootable image to a running endpoint.

References

• Red Hat documentation, Red Hat Enterprise Linux AI 3.5
• Red Hat Enterprise Linux AI hardware requirements, serving and end to end workflow
• Red Hat AI 3, supported product and hardware configurations
• RedHatAI, granite-3.1-8b-instruct-quantized.w4a16 on Hugging Face
• IBM, Granite 3.1 performance and 128k context announcement

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts