Model, format, GPUs
Model
Number format for weights
GPU
How many
KV cache format
Conversation length
What you are looking at: quantization stores each weight in fewer bits. Going from 16-bit to 8-bit halves the memory; 4-bit quarters it. Less memory per weight means a bigger model fits, more room is left for the KV cache, and decode gets faster because fewer bytes are read per token. The cost is rounding error, shown at the bottom.
Try this: pick Llama 3.1 70B on one H100 and switch from BF16 to FP8 to INT4. Watch "Fits?" turn green and the user count jump. Then switch the KV cache to FP8 and see the users double again.
Illustrative. Weight sizes use the makers' published parameter counts and include the scale factors stored with 4-bit formats (NVFP4 about 4.5 bits per weight, MXFP4 about 4.25, INT4 with group scales about 4.25). Real checkpoints keep a few layers, such as embeddings, in higher precision, so they are slightly larger. 10% of GPU memory is kept free. Users assume conversations of the chosen length; 16-bit KV cache per token: Llama 3.1 8B 128 KB, 70B 320 KB, 405B 504 KB, DeepSeek-V3 about 69 KB; gpt-oss-120b about 72 KB counting every layer as full attention. An FP8 KV cache halves these. Decode speed is an upper bound for one user: memory bandwidth × 70% ÷ bytes of active weights per token, ignoring the KV cache and multi-GPU overheads. FP8 tensor cores need Ada, Hopper or newer GPUs; FP4 tensor cores need Blackwell. The rounding chart uses 24 sample values with one shared scale (absolute maximum), which is the simplest method; GPTQ, AWQ and block-scaled formats lose less.
What you are looking at: how one number is stored in each floating-point format. The sign bit says plus or minus, the exponent bits set the range (which power of two), and the mantissa bits set the precision (where between that power of two and the next). Fewer bits means the stored value is further from the one you typed, or out of range altogether.
Try this: type 0.1 and compare BF16 with FP8 and FP4. Then try 300 and 70000: FP4 and FP8 E4M3 run out of range, which is why low-bit formats always divide by a scale first. Try 0.0007 to see small numbers vanish.
Rounding is to the nearest representable value. FP16 and FP8 E5M2 follow IEEE-style rules and overflow to infinity. FP8 E4M3 (max 448) and FP4 E2M1 (max 6) have no infinity; quantization kernels clip values to the maximum, shown here. Values smaller than half the smallest step become zero.
What you are looking at: absmax quantization done by hand on eight weights, one step at a time: find the largest absolute value, work out the scale, divide, round, store, and multiply back. The last column is the error each weight picks up. This is the same worked example as in the post.
Try this: step through INT8 first, then switch to INT4 and FP4 and compare the average error. Then load "with an outlier" and see how one big weight ruins the small ones.
Symmetric absmax quantization with one scale for the eight weights. INT8 codes run from -127 to 127, INT4 from -7 to 7, FP4 E2M1 stores 0, 0.5, 1, 1.5, 2, 3, 4 and 6 with a sign. Real tools store the scale in FP16, FP32 or FP8.
What you are looking at: 128 weights from one illustrative layer, quantized with one scale for all of them or a separate scale per block. Grey bars are the originals, red dots the stored values, dashed lines the block borders. Smaller blocks protect the small weights from outliers but store more scales.
Try this: choose "one outlier" and INT4, then go from one scale to blocks of 16. Watch the share of weights rounded to zero fall. Then pick "clean layer" and notice that block size matters much less when there are no outliers.
Illustrative weights from a bell curve (standard deviation 0.1) with added outliers. Each block uses absmax scaling; the scale is counted as 8 bits (an FP8 scale, as in NVFP4). Average error is the mean absolute difference between original and stored values. Real layers have millions of weights and the outlier pattern varies by model.