Granite Model Family and Choosing a Size Under Apache 2.0 (Red Hat Gen AI Series, Part 2)

Granite 4.0 comes in four practical sizes under Apache 2.0. Here is how total versus active parameters decide GPU memory, and why H-Tiny, not H-Small, is the right first model for a self hosted support assistant.

Dr. Pranay Jha

July 28, 2026

No comments

14 minutes

Read Time

Red Hat Gen AI Series · Part 2 of 30

My first attempt at serving the largest open Granite model on a single card never returned a token. It failed at load with CUDA out of memory, because I had read 9B active parameters and sized the box for a 9B model. Granite 4.0 H-Small is 9B active but 32B total, and every one of those 32B parameters sits in GPU memory whether it fires on a given token or not. That gap between active and total decides which Granite you can run, what it costs to run, and how many first day deployments die at startup instead of answering a question.

Part 1 settled where the assistant runs: a single RHEL AI server, not a cluster. This part settles what runs on it. Granite is the open weight model family that Red Hat ships and supports, and picking a size is the first decision that has a wrong answer you pay for in hardware. Get it right and a modest card serves the whole thing. Get it wrong and you either overspend on GPUs or watch the server refuse to start.

Who this is for: the same engineer from Part 1, who has called a hosted large language model API but has not run self hosted inference. You need a Linux box with one NVIDIA or AMD GPU, or a cloud instance with one, to run the serving commands. This part builds on Part 1 and does not assume Kubernetes. For token and context ideas it leans on the GenAI explainers on tokens and embeddings and the context window rather than re-teaching them.

Key takeaways

Granite 4.0 is a hybrid Mamba-2 and transformer family under Apache 2.0, in four practical sizes: Micro at 3B dense, H-Micro at 3B, H-Tiny at 7B with about 1B active, and H-Small at 32B with about 9B active, plus a Nano pair at 350M and 1B.

Total parameters decide memory, active parameters decide compute. A mixture of experts model like H-Small loads all 32B into GPU memory but computes like a 9B, so it is memory heavy and compute light, not cheap on every axis.

Apache 2.0 is the feature a hosted model cannot match. You may tune, quantize, redistribute and ship Granite commercially with no gate and no call home.

If you tune on RHEL AI with InstructLab, the supported starter is still Granite 3.1 8B, not Granite 4.0. Newest is not automatically the one the tuning pipeline accepts.

For the support assistant, start with H-Tiny at FP8, not H-Small. It fits a 24 GB card and answers grounded questions well.

Granite 4.0 and what changed with the hybrid models

Granite is IBM and Red Hat open weight model family, and the 4.0 line that landed in October 2025 changed the architecture underneath. Earlier Granite models were conventional transformers, where every layer runs self attention over the whole sequence, which is accurate but grows expensive as context gets long. Granite 4.0 interleaves a small fraction of attention layers with a majority of Mamba-2 layers, a state space design that carries sequence information forward without paying the full attention cost on every token, at roughly a nine to one ratio. A state space layer, in one clause, is a block that summarises the sequence so far into a fixed size running state instead of re-reading all of it. In practice IBM reports more than 70 percent lower memory and about twice the inference speed against comparable models, most of it showing up in long context and many session workloads.

Four sizes matter for a serving decision, with a fifth pair for the edge. Micro is a 3B dense transformer kept deliberately conventional for maximum compatibility, including CPU and older runtimes. H-Micro is the 3B hybrid. H-Tiny is a 7B hybrid mixture of experts, meaning it holds 7B parameters but routes each token through only about 1B of them. H-Small is the 32B hybrid mixture of experts with about 9B active. Below all of these sit the Nano models at 350M and 1B, small enough to run in a browser tab. Every one is Apache 2.0, cryptographically signed, and covered by an ISO 42001 AI management certification, which matters when an auditor asks where your model came from. Granite 4.0 trains on sequences up to 512K tokens and is evaluated to 128K, so context length is not the constraint here, memory is.

Model	Total params	Active params	Architecture	Best first fit
granite-4.0-micro	3B	3B	Dense transformer	CPU, edge, maximum compatibility
granite-4.0-h-micro	3B	3B	Hybrid	Small footprint with hybrid efficiency
granite-4.0-h-tiny	7B	~1B	Hybrid MoE	Fast cheap serving, RAG grounded assistants
granite-4.0-h-small	32B	~9B	Hybrid MoE	Hardest tasks, agents, tool use
granite-4.0-nano	350M and 1B	same	Dense and hybrid variants	On-device, browser, tight edge

Granite 4.0 at a glance. The active column is why H-Tiny and H-Small cost far less compute than their total size suggests.

Apache 2.0 and why the licence is the real feature

Open weights and open source are not the same thing, and the difference is exactly what makes Granite useful for the company in our running project. Plenty of freely downloadable models carry a community licence with an acceptable use policy, a monthly active user ceiling, or a clause that reserves rights to the vendor. Granite 4.0 ships under Apache 2.0, a permissive licence that grants you the right to use, modify, redistribute and commercialise the model, with an explicit patent grant, and asks nothing back beyond preserving the notice. There is no gate to click, no user cap, and no phone home. For a firm whose contracts forbid sending customer data to a third party, that is not a nicety, it is the requirement, because tuning Granite on your own resolved tickets and then shipping the result is only legal if the licence lets you.

This is the practical edge over a hosted API that no amount of throughput can close. A rented model can change under you between one week and the next, its terms can shift, and your data leaves your network on every call. An Apache 2.0 model you have downloaded, quantized and tuned is frozen the way you left it, audited by you, and portable to any hardware you choose, which is the same portability argument the AI Engineering series makes about retrieval versus fine tuning for real business problems, only now applied to the weights themselves. Cryptographic signing and the ISO 42001 certification exist so that when governance asks you to prove provenance, you can.

Sizing a Granite model to a GPU

Model memory has two parts, and only one of them is the weights. Weights cost roughly the parameter count times the bytes per parameter: 2 bytes at bf16, 1 byte at FP8, about half a byte at INT4. A second part, the KV cache, is the running memory of the conversation, and it grows with context length and with how many requests you serve at once. For a mixture of experts model the trap is that all parameters load into memory even though only the active subset computes, so H-Small needs room for 32B parameters while running at the compute cost of 9B. The table below is the number I keep pinned, because it turns a model name into a card requirement before anyone provisions hardware.

Model	bf16 weights	FP8 weights	INT4 weights	Fits comfortably on
Micro 3B	~6 GB	~3 GB	~1.8 GB	Any 8 GB card, or CPU
H-Tiny 7B	~14 GB	~7 GB	~4 GB	16 GB card, 24 GB for long context
H-Small 32B	~64 GB	~32 GB	~18 GB	Two 48 GB cards at bf16, one 48 GB at FP8 is tight
Nano 1B	~2 GB	~1 GB	~0.5 GB	CPU or any small GPU

Weights only. Add KV cache on top, which at 128K context and real concurrency can be several more GB. This is the reference artifact for this part.

Weights scale with total parameters, not active. That is why H-Small at bf16 needs two cards while H-Tiny fits one.

flowchart TD Start[Pick a Granite size] --> Q1{Tuning with InstructLab on RHEL AI} Q1 -->|Yes| T[granite 3.1 8b starter] Q1 -->|No, serving only| Q2{Hardest tasks or agents} Q2 -->|Yes| HS[granite 4.0 h small 32B] Q2 -->|No, RAG grounded QA| HT[granite 4.0 h tiny 7B] Q2 -->|Edge or CPU| MC[granite 4.0 micro 3B]

Choosing a Granite size. The left branch is the one the docs bury: the InstructLab starter is still 3.1 8B.

Serving a Granite model with vLLM

With a size chosen, serving is one command, but the Granite hybrid architecture has a hard version floor. Support for the Mamba-2 hybrid landed in vLLM 0.10.2, so anything older will refuse to load the model with an architecture error, not a warning. Standing up this endpoint is the large language model case of the same problem the Data Science series covers in serving machine learning models, specialised to a token generating server.

# Tested against: RHEL AI 1.5, Granite 4.0 (October 2025 release),
# vLLM 0.10.2 (the floor for Granite hybrid), Python 3.11
pip show vllm | grep -i version
# Version: 0.10.2

# Granite is Apache 2.0 and public, so no token is needed here.
# If a repo were gated, vLLM reads HUGGING_FACE_HUB_TOKEN from the environment.
vllm serve ibm-granite/granite-4.0-h-tiny --max-model-len 8192

# Representative startup on a single 24 GB card:
# INFO  Automatically detected architecture: GraniteMoeHybridForCausalLM
# INFO  Using Mamba2 plus attention hybrid layers
# INFO  Started server process
# INFO  Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
# INFO  Avg generation throughput: 78.4 tokens/s, Running: 1 reqs

H-Tiny serving on one card. The architecture line confirms the hybrid model loaded correctly.

Two failures are worth showing because most people hit at least one. Run the command on an older vLLM and it stops cold, and if you point the same command at H-Small on a single 48 GB card at bf16 it dies at load, not at inference, which is the exact mistake from the opening paragraph.

# Failure 1, vLLM older than 0.10.2:
ValueError: Model architecture 'GraniteMoeHybridForCausalLM' is not supported.
# Fix: pip install -U vllm  (get to 0.10.2 or newer)

# Failure 2, H-Small at bf16 on one 48 GB L40S:
vllm serve ibm-granite/granite-4.0-h-small
...
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 44.35 GiB of which 1.02 GiB is free.
# Cause: 32B at bf16 is about 64 GB of weights, before any KV cache.
# Fix: serve the FP8 build (RedHatAI/granite-4.0-h-small-FP8-dynamic,
#      about 32 GB) or drop to H-Tiny.

The two errors that stop a first serve. Neither is a bug, both are a sizing or version mismatch.

War story: I lost most of an afternoon, about three hours, to exactly this. I put H-Small on a single L40S with 48 GB because the benchmark charts made it the obvious pick, reasoning that 9B active meant it would behave like a 9B model. It did not load. The 32B of weights at bf16 needed roughly 64 GB and the card had 44 usable, so it threw CUDA out of memory before the first token. I switched to the FP8 build, which brought weights to about 32 GB and did start, but at our 128K context target the KV cache left almost no headroom and long requests still failed. I dropped the first cut of the assistant to H-Tiny at FP8, which sat under 10 GB with room to spare, and it answered grounded support questions well enough that nobody noticed it was a seventh the size. The lesson cost three hours and one wrong assumption: active parameters buy you compute, never memory.

Once the server is up it speaks the OpenAI API, so the client is boringly familiar. Read the key from the environment even though a local vLLM accepts any value, because the same code then works unchanged against a secured endpoint later.

import os
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:8000/v1',
    api_key=os.environ.get('VLLM_API_KEY', 'EMPTY'),
)

resp = client.chat.completions.create(
    model='ibm-granite/granite-4.0-h-tiny',
    messages=[{'role': 'user', 'content': 'In one sentence, what is a changelog?'}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

# Output:
# A changelog is a curated, chronological record of the notable
# changes, fixes and additions made to a project across its releases.

The same OpenAI client you already use, pointed at your own box. Nothing about the calling code is Red Hat specific.

One more failure is worth naming because it is silent, which makes it the nastiest. On some vLLM and FlashInfer combinations the Granite hybrid model loads without error and then generates fluent nonsense, because the attention backend mishandles the hybrid layers. If your output degrades into garbage the moment you switch models, pin the attention backend rather than the model:

Where the H-Small default goes wrong

Two pieces of common advice will steer you wrong on this exact project. First, the benchmark tables and the marketing both point at the 32B flagship as the enterprise choice, and for an open ended assistant with no grounding that is defensible. Our assistant is grounded: it answers from product docs, a ticket archive and a changelog, so retrieval carries the knowledge and the model mostly has to read and phrase. In that setup H-Tiny closes most of the quality gap at a seventh of the memory, which is the whole reason the GenAI explainer on what RAG is matters before you reach for a bigger model. Retrieval, not parameter count, is usually the cheaper lever.

Second, the instinct to grab the newest Granite collides with the tuning pipeline. RHEL AI 1.5 ships its InstructLab workflow around granite-3.1-8b-starter as the student model and granite-3.1-8b-lab as the inference model, not Granite 4.0. If your plan for Phase 2 is to tune on the company docs with InstructLab, the supported and tested path still runs on the 3.1 8B line, and reaching for a 4.0 hybrid there is unsupported territory today. So the honest split is this: tune on granite-3.1-8b-starter where the LAB method is validated, and serve Granite 4.0 hybrid where you only need inference. Treating the newest model as automatically correct is how you spend a week fighting a tuning pipeline that was never going to accept it.

Picking H-Tiny for the support assistant

For the assistant this series builds, the first served model is granite-4.0-h-tiny at FP8, on the single RHEL AI box from Part 1, on one 24 GB card. It leaves headroom for the KV cache at a working context length, it answers grounded questions well, and it costs a fraction of what H-Small would in memory and hardware. Reserve H-Small for the day an evaluation set shows H-Tiny genuinely failing on the hardest tickets, and avoid the dense Micro unless you are targeting CPU or an edge box, since the hybrid models give you the same footprint with better long context behaviour. If your Phase 2 plan includes InstructLab tuning, keep granite-3.1-8b-starter in your pocket for that step specifically.

Your action for Monday is to run the memory table against the card you actually have, pick the largest Granite that leaves real KV headroom, and serve it with the vLLM command above until you get one real answer back, not just a clean load. Confirm the architecture line reads GraniteMoeHybridForCausalLM and that the reply is coherent. Part 3 takes the next decision head on: RHEL AI, plain RHEL, or OpenShift AI, and which one this project actually belongs in, now that you have a model to put on it.

Granite 4.0 and what changed with the hybrid models

Where the H-Small default goes wrong

Picking H-Tiny for the support assistant

References

About The Author

Dr. Pranay Jha

Discover more from Journal of Intelligent Infrastructure

Leave a Reply Cancel reply

Architect’s Toolkit

PJ’s Tools

VMware Cloud Foundation

Nutanix

AI & Cloud-Native Platform

Architecture & Design

About the Author

Dr Pranay Jha

You May Have Missed

Choosing a First Model and Accelerator for Red Hat AI (Red Hat Gen AI Series, Part 6)

InstructLab and the LAB Method for Taxonomy Driven Alignment (Red Hat Gen AI Series, Part 5)

Red Hat AI Hybrid Cloud Deployment, and Where It Actually Runs (Red Hat Gen AI Series, Part 4)

RHEL AI vs RHEL vs OpenShift AI, and Where a Project Belongs (Red Hat Gen AI Series, Part 3)

Granite Model Family and Choosing a Size Under Apache 2.0 (Red Hat Gen AI Series, Part 2)