optimum library does two jobs: it exports Transformers models to ONNX, and it quantizes them for a target runtime or accelerator. For 4-bit weight-only quantization of large language models, GPTQ and AWQ are the calibration-based methods that hold accuracy; bitsandbytes and HQQ skip calibration but trade speed or precision. A 4-bit build cuts weight memory by roughly 4x versus fp16. Choose the method by hardware and accuracy budget, benchmark on your own task and card, and never assume a smaller file means faster inference.You pull a 70B model, call from_pretrained, and the box dies with torch.OutOfMemoryError: CUDA out of memory. The weights alone want about 140 GB in fp16, and your card has 80. Nothing is broken in your code. The model simply does not fit. That wall is the most common one an infra team hits the first time it tries to self-host, and quantization is the standard way through it.
Quantization is a capacity decision, not a science project
Strip away the math and quantization is one idea: store each weight in fewer bits. A model trained in 16-bit floating point can have its weights packed into 8-bit or 4-bit integers, which shrinks the bytes you have to hold in VRAM and move across memory. The concept is covered in the GenAI series; this part is about doing it with Hugging Face tools and living with the result in production.
For the person who owns the hardware, the framing that matters is capacity. A weight count times a bytes-per-weight number gives you a floor on VRAM. Drop the bytes-per-weight and the floor drops with it. That is the whole game: not a smarter model, just a smaller one on the same silicon. The trade you are managing is bits against accuracy, and the question is never "is quantization good" but "how few bits can I run before quality on my task falls below what users accept."
What optimum does
The optimum library is the bridge between a Transformers model and the runtimes and accelerators you deploy on. It supports ONNX Runtime, Intel and AMD hardware paths, Furiosa NPUs, GPTQ, and lower level PyTorch quantization. Think of it as the layer that takes a model the data scientist handed you and produces an artifact your serving stack can run on your chips.
Job one: export to ONNX and quantize for ONNX Runtime
ONNX is a portable graph format. Once a model is in ONNX, ONNX Runtime can execute it on CPU, on GPU, and across hardware backends without dragging the full PyTorch stack along. The optimum-onnx package exports the model, then applies dynamic or static quantization and graph optimization through the ORTQuantizer and ORTOptimizer classes. Dynamic quantization needs no calibration data and is the fast path for CPU-bound encoder models; static quantization feeds a small calibration dataset through the model to set activation ranges and usually holds accuracy better.
# Export a Transformers model to ONNX, then quantize it
pip install 'optimum-onnx[onnxruntime]'
from optimum.onnxruntime import ORTModelForSequenceClassification, ORTQuantizer
from optimum.onnxruntime.configuration import AutoQuantizationConfig
ckpt = 'distilbert-base-uncased-finetuned-sst-2-english'
ort_model = ORTModelForSequenceClassification.from_pretrained(ckpt, export=True)
qconfig = AutoQuantizationConfig.arm64(is_static=False, per_channel=False)
quantizer = ORTQuantizer.from_pretrained(ort_model)
quantizer.quantize(save_dir='onnx-int8/', quantization_config=qconfig)
# Expected: writes model_quantized.onnx in onnx-int8/, roughly 4x smaller than fp32
# Failure mode: forget export=True and you load PyTorch weights, not an ONNX graph,
# and ORTQuantizer.from_pretrained will not find a graph to quantize.Job two: hardware backends
The second job is targeting specific silicon. If you run Intel CPUs, AMD GPUs, or specialized NPUs, optimum exposes the vendor toolchains behind one interface so the same model export can be pointed at the backend you have. When the destination is an NVIDIA GPU and you want the high-throughput inference server path, that is a different stack: see the NVIDIA AI series for TensorRT-LLM and Triton, and Part 12 of this series for Text Generation Inference, which loads 4-bit quantized weights directly.
GPTQ vs AWQ: the two calibration-based 4-bit methods
For large language models, 4-bit weight-only quantization is the workhorse. Two methods dominate when you care about accuracy: GPTQ and AWQ. Both quantize the weights to 4-bit and restore them to 16-bit on the fly inside a fused kernel during inference, so you get the memory savings without holding full-precision weights. Both need a calibration step, a short pass over a sample dataset that tells the algorithm which weights matter most. GPTQ quantizes each row of the weight matrix to minimize error; AWQ scales weights based on which activations are most sensitive.
| Factor | GPTQ (GPT-QModel) | AWQ |
|---|---|---|
| Bit width | 4-bit (int4), restored to fp16 at inference | 4-bit, restored to fp16 at inference |
| Calibration time (8B, one A100) | about 20 minutes | about 10 minutes |
| Accuracy | High; risk of overfitting the calibration set | High at 4-bit, sometimes ahead of GPTQ on a task |
| Kernels | Marlin for A100-class inference; ROCm, Apple, CPU support | Widely supported in serving stacks |
| If you skip calibration | Not an option; calibration is required | Not an option; load a pre-quantized model instead |
The detail that saves a day of work: for popular models you almost never quantize yourself. The Hub already hosts thousands of GPTQ and AWQ builds. Check first, and only run calibration when you have a private fine-tune or a model nobody has published a quantized version of.
# GPTQ 4-bit quantization via Transformers + GPT-QModel
pip install --upgrade accelerate optimum transformers
pip install gptqmodel --no-build-isolation
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
tok = AutoTokenizer.from_pretrained('facebook/opt-125m')
gptq = GPTQConfig(bits=4, dataset='c4', tokenizer=tok)
model = AutoModelForCausalLM.from_pretrained(
'facebook/opt-125m', device_map='auto', quantization_config=gptq)
# Expected: a 4-bit model, roughly 4x smaller weight memory than fp16
# Failure mode: out of memory during calibration on a large model.
# Fix: cap memory per device instead of letting it grab everything ->
# from_pretrained(..., max_memory={0: '30GiB', 'cpu': '30GiB'})
# Note: AutoGPTQ is no longer supported; install gptqmodel.The memory math, worked through
Rough sizing is a two-number calculation: parameter count times bytes per weight. fp16 is 2 bytes, int8 is 1 byte, int4 is half a byte. Add headroom for the KV cache, which scales with context length and batch size, plus a bit for activations and CUDA overhead.
Worked example
You have one 24 GB GPU and want to serve an 8B model. In fp16 the weights alone are about 16 GB, leaving roughly 6 to 7 GB for KV cache after overhead, which caps you at small batches and short context. Quantize to 4-bit and weights drop to about 4 GB. Now you have around 18 GB for KV cache, so you can raise batch size, run longer context, or both. Same card, several times the serving capacity. The cost is a measurable accuracy dip you have to verify is acceptable on your task, not a free lunch.
| Precision | Memory vs fp16 | Accuracy | When I reach for it |
|---|---|---|---|
| 8-bit | about 2x smaller | very close to fp16 | Accuracy-sensitive work with some VRAM to spare |
| 4-bit (AWQ / GPTQ) | about 4x smaller | relatively high | The default for fitting LLMs on tight VRAM |
| Sub-4-bit | more than 4x smaller | noticeable drop, worse at 2-bit | Only when nothing else fits and you have tested it |
Hugging Face publishes its own benchmarks on Llama 3.1 8B and 70B, measured on A100 and H100 cards. Read them for the relative picture, but treat the absolute throughput numbers as a starting point. They were measured at batch size 1 generating 64 tokens, which is not your traffic.
The calibration dataset is a hidden dependency
GPTQ and AWQ both run a calibration pass, and the dataset you feed it is not a detail you can wave off. The algorithm tunes the quantized weights to the distribution it sees during calibration, so a set that looks nothing like your production traffic gives you a model that benchmarks fine and falls short on your real inputs. The Transformers docs default GPTQ to the c4 dataset, a reasonable general choice, but if your model serves a narrow domain, calibrating on a sample of that domain is the difference between an acceptable 4-bit build and a disappointing one. You can pass your own list of strings as the calibration data.
For the platform owner this is a reproducibility and provenance question, the kind you already manage for build inputs. Pin the calibration dataset, version it, and record which set produced which quantized artifact, the same way you track the base image and dependencies behind a container build. When someone asks six months later why the 4-bit model drifted, you want the answer in your records, not a re-run. And because calibration on a large model can take twenty minutes or more on an A100, treat it as a scheduled job with its own GPU budget, not something you fire off interactively and hope finishes. Cache the result; you should never calibrate the same model twice.
What I would validate first
Three checks before a quantized build ships. First, accuracy on your task, not a generic leaderboard. Run your real eval set against the fp16 baseline and the quantized version and look at the gap. A drop that is invisible on a benchmark can be obvious on your domain. Second, throughput on your card. A smaller model is not automatically faster; if your hardware lacks the right kernel, dequantization overhead can eat the win. Third, the kernel and hardware match. Marlin is tuned for A100-class GPUs; on other cards you want a different backend, and an unsupported pairing falls back to a slow path.
My take
For most teams self-hosting an LLM on fixed GPUs, 4-bit is the default and AWQ is where I start: the calibration is shorter than GPTQ, accuracy at 4-bit is strong, and serving stacks support it widely. I reach for GPTQ when I want the last bit of accuracy and can spend the calibration time, and I lean on the Marlin kernel when the target is A100-class hardware. For CPU-bound encoder models and portability across chips, the ONNX Runtime path through optimum-onnx is the cleaner answer than forcing everything onto a GPU. The one rule that does not bend: check the Hub for a pre-quantized build before you calibrate anything, and benchmark accuracy and throughput on your own task and card before you call it done. Smaller is only better if it still answers correctly and runs fast on the silicon you have.
For the on-prem and air-gapped version of this, where you mirror and serve quantized models on your own GPUs, the Private AI series covers the deployment side. Next in this series we move from fitting models to demoing them.
Your move: pick one model you serve in fp16 today, pull a 4-bit build from the Hub, and run your eval set against both. The capacity you free up is usually larger than people expect.
References
Transformers: Selecting a quantization method
Transformers: GPTQ quantization
Optimum ONNX: ONNX Runtime quickstart
About The Author
DrJha