granite-3.1-8b-instruct is 16 GB in bfloat16. Quantized to four bit weights it is 4.9 GB, and it still recovers about 99 percent of its benchmark score. That one trade, a quarter of the memory for roughly a point of accuracy, is what stands between a model that fits your GPU and one that does not. Last part opened the vLLM engine and showed the key value cache, not the weight matrix, is what runs out of room first. This part cuts the weights at the source so more of the card is left for that cache, and works through which format to cut them with, because the wrong choice can leave you serving slower than you started.

Why quantization is the cheapest memory you will ever buy

Weight memory is just a multiplication: parameter count times bytes per parameter. An 8 billion parameter model at bfloat16, which is 2 bytes, needs about 16 GB before a single request arrives. Drop each weight to 8 bits and that halves to 8 GB; drop it to 4 bits and it falls near 4 GB. Nothing about the architecture changes, only how many bits each number is stored in, and post training quantization does this after the model is trained with no retraining and no gradients.

Freed weight memory does not vanish, it becomes KV cache. On the assistant’s A100, halving the weights let vLLM hold roughly half again as many concurrent requests at the same context length, which is the number Part 21 taught you to read off the startup banner. So quantization buys two things at once: a model that fits a smaller or cheaper card, and more room to batch on the card you already own. Set against the alternative, renting a second GPU, it is the cheapest capacity you can add, and it feeds straight into the unit economics the GenAI Series lays out in the generative AI cost breakdown.

What you pay for it is a little accuracy, and usually less than people fear. Modern PTQ methods fit per channel or per group scales that absorb most of the rounding error, and Red Hat’s published Granite checkpoints report recovering around 99 percent of the original benchmark score. The catch is that recovery is measured on a benchmark, not on your support tickets, so the number is a starting estimate and not a promise for your workload.

Four formats worth knowing

Format names in this world read as W then A: weight bits, then activation bits. W8A8 quantizes both weights and activations to 8 bits, so the matrix multiply itself runs in low precision. W4A16 quantizes only the weights to 4 bits and leaves activations at 16, so weights are dequantized back to 16 bits inside the kernel before the multiply. That difference decides almost everything about speed, and it is the point most tutorials skip.

Four families cover almost every real deployment. FP8 (W8A8, an 8 bit floating format) keeps a wide dynamic range and applies cleanly with round to nearest, so no calibration data is needed. INT8 (W8A8 integer) needs calibration, typically SmoothQuant to tame activation outliers plus GPTQ for the weights, and runs on older cards that lack FP8 silicon. INT4 weight only (W4A16) uses GPTQ or AWQ to fit 4 bit weight scales from a few hundred calibration samples. NVFP4 and MXFP4 are 4 bit microscale floating formats that quantize weights and activations together with tiny per block scales, built for the newest hardware. One older option is gone: LLM Compressor dropped 2 to 4 structured sparsity support for lack of hardware traction, so advice that still recommends SparseGPT pruning for serving is out of date.

Quantizing Granite with LLM Compressor

LLM Compressor is the Apache 2.0 library, part of the vLLM project and shipped inside the Red Hat AI Inference Server, that produces these checkpoints. It writes them in the compressed-tensors format that vLLM loads directly, and its whole surface is one call, oneshot, driven by a recipe. Start with FP8 dynamic, because it needs no calibration data and finishes in minutes. Your Hugging Face token comes from the environment, never a hardcoded string.

# Tested with llmcompressor 0.10.0.2, compressed-tensors, transformers 4.x,
# Python 3.11, vLLM core v0.11.2 as shipped in RHAIIS 3.2.5, NVIDIA A100 80GB.
# export HF_TOKEN=... first; the loader reads it from the environment.
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "ibm-granite/granite-3.1-8b-instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Weights to FP8 by round to nearest; activations quantized per token at
# inference. No calibration set, so this is the fastest path to a smaller model.
recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
oneshot(model=model, recipe=recipe)

SAVE_DIR = "granite-3.1-8b-instruct-FP8-dynamic"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

# output (trimmed)
# Applying QuantizationModifier: targets=['Linear'] scheme=FP8_DYNAMIC
# Compressing model: 100%|##########| 322/322 [00:44<00:00,  7.3it/s]
# Saved compressed-tensors checkpoint to granite-3.1-8b-instruct-FP8-dynamic (8.1 GB)

INT4 weight only is a different job, because 4 bit scales have to be fit against real activations. Swap the recipe for GPTQModifier and hand it a few hundred calibration samples. This is also where the first honest failure lives.

from llmcompressor.modifiers.quantization import GPTQModifier

# INT4 weight only via GPTQ. group_size 128 shares one scale per 128 weights.
recipe = GPTQModifier(targets="Linear", scheme="W4A16", group_size=128, ignore=["lm_head"])
oneshot(
    model=model,
    recipe=recipe,
    dataset="open_platypus",
    num_calibration_samples=512,
    max_seq_length=2048,
)

# On a 24 GB card, this dies part way through calibration:
# torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
#   GPU 0 has a total capacity of 23.68 GiB of which 1.44 GiB is free.
#
# Why: GPTQ holds the layer being quantized plus its Hessian on the GPU while
# it fits scales, and the 16 GB base model plus activations will not fit in 24 GB.
# Fix: run the quantization on the 80 GB A100, or let LLM Compressor onload one
# layer at a time so only a single decoder block sits on the GPU during the pass.

Quantization is a memory heavy job in its own right, so the machine you compress on is not always the machine you serve on. Once the checkpoint exists it is an ordinary model directory, and it slots into the same serving and evaluation flow the Data Science Series covers for serving machine learning models. If you would rather not run the pass at all, Red Hat publishes pre-quantized Granite checkpoints on Hugging Face under the RedHatAI namespace, and loading one is identical to loading the base model.

Memory and accuracy on granite-3.1-8b

Here is the table I keep pinned when sizing a serving node. Numbers are for granite-3.1-8b-instruct; disk and GPU weight figures are close because the compressed-tensors weights load almost as they sit on disk. Accuracy recovery is Red Hat’s published range on standard leaderboards, and the throughput column is a rough multiple against bfloat16 at a serving batch, not a single request.

Watch what that does to the serving banner from Part 21. Same A100, same 8k context, same 0.92 memory utilisation, only the weights change.

$ vllm serve ibm-granite/granite-3.1-8b-instruct --max-model-len 8192 --gpu-memory-utilization 0.92
INFO   Model weights take 15.98 GiB
INFO   GPU KV cache size: 99,072 tokens
INFO   Maximum concurrency for 8,192 tokens per request: 12.1x

$ vllm serve ./granite-3.1-8b-instruct-FP8-dynamic --max-model-len 8192 --gpu-memory-utilization 0.92
INFO   Model weights take 8.10 GiB
INFO   GPU KV cache size: 151,552 tokens
INFO   Maximum concurrency for 8,192 tokens per request: 18.5x

Concurrency rose from 12.1x to 18.5x, a 53 percent lift in how many full length requests fit at once, from one oneshot call and no new hardware. Whether that accuracy trade is safe is a question you answer with your own eval set, exactly as the Data Science Series argues in model evaluation, cross validation and leakage; run the tuned and the quantized model over the same held out tickets before you trust the recovery figure.

Matching a format to your GPU

Every format has a floor set by GPU compute capability, the NVIDIA version number for a card generation. Turing is 7.5, Ampere 8.0, Ada 8.9, Hopper 9.0, Blackwell 10.0. FP8 needs native 8 bit float units, so it wants Ada or Hopper; INT8 runs back to Turing; NVFP4 needs Blackwell. Pick below the floor and vLLM refuses at load rather than silently emulating, which is the good behaviour.

flowchart LR
  G{GPU generation} --> BL[Blackwell]
  G --> HA[Hopper or Ada]
  G --> AM[Ampere]
  G --> TU[Turing]
  BL --> NV[NVFP4 W4A4]
  HA --> Q{memory bound and low batch}
  Q -->|yes| W4[INT4 W4A16]
  Q -->|no| FP[FP8 dynamic]
  AM --> AF{fit a model or push batch}
  AF -->|fit| W4
  AF -->|batch| I8[INT8 W8A8]
  TU --> I8

The obvious move, grab the checkpoint with the fewest bits, is the one that bites. Load a NVFP4 checkpoint on the A100 that serves the assistant today and vLLM stops cold.

$ vllm serve RedHatAI/granite-3.1-8b-instruct-NVFP4 --max-model-len 8192
ValueError: The quantization method nvfp4 is not supported on your GPU.
  NVFP4 requires compute capability 10.0 (NVIDIA Blackwell); device 0 is an
  NVIDIA A100-SXM4-80GB with compute capability 8.0.

# fix on Ampere: use INT4 W4A16 to fit, or FP8 dynamic for throughput
$ vllm serve ./granite-3.1-8b-instruct-FP8-dynamic --max-model-len 8192

This is the piece the format leaderboards omit. Fewer bits is only faster when your GPU has the units to run them and when your bottleneck is the resource those bits cut. On an Ampere card with FP8 not native and NVFP4 out of reach, INT4 W4A16 is the way to shrink weights, and INT8 W8A8 is the way to push batch throughput. The bit count alone tells you almost nothing.

Default to FP8, drop to INT4 only when memory bound

Next part takes this compressed model off a single card and spreads inference across many with llm-d, the distributed serving layer on Kubernetes, where quantization decides not just what fits one GPU but how few GPUs the whole deployment needs.

References

• LLM Compressor, vllm-project/llm-compressor on GitHub
• Red Hat AI Inference Server 3.1, LLM Compressor documentation
• Red Hat Developer, Optimizing generative AI models with quantization
• RedHatAI, granite-3.1-8b-instruct-quantized.w4a16 model card

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts