AI Infrastructure series · Part 7 of 24

Before anyone asks how fast a model will run, someone has to ask where it will fit. A model’s weights, every user’s conversation memory, the GPU software and the scratch space for each calculation all have to sit in GPU memory at the same time, and training adds several times more on top. This part adds it all up with real numbers: what lives in an H100’s 80 GB while it serves Llama 3.1 8B, how to size the weights of any model, why a model that fits on paper can still fail to load, which GPUs hold which models, and why spilling into CPU memory is not a way out.

How big is a model in memory?

A model is mostly a very long list of numbers called parameters, or weights (part 9 explains what they do). To answer a single question, the GPU reads every one of them, so all of them must be in GPU memory, not on disk and not in the server’s RAM. The size of the weights is simply the number of parameters times the bytes used to store each one:

The bytes per parameter depend on the number format. BF16 and FP16 use 2 bytes. FP8 uses 1. A 4-bit format uses half a byte for the value, plus a little for the scale factors that sit beside each small block of values (about 4.25 to 4.5 bits per value in total), so this post counts 4-bit as about 0.56 bytes. Part 18 explains how those formats work and what they do to quality. For memory, the rule of thumb is: billions of parameters × 2 = GB in 16-bit, halve it for 8-bit, and take a bit more than a quarter for 4-bit.

Weights worked out for five real models

Step by step for Llama 3.1 70B, which has about 70.6 billion parameters: in BF16, 70.6 × 2 = 141.2 GB. In FP8, 70.6 × 1 = 70.6 GB. In 4-bit, 70.6 × 0.5625 ≈ 39.7 GB. The same arithmetic for the rest of the table:

The two mixture-of-experts rows are the ones people get wrong. An MoE model (part 12) uses only a few of its experts for each token, so DeepSeek-V3 does the maths of a 37B model per token. But any expert can be picked for the next token, so all 671 billion parameters must stay in memory. Active parameters set the compute and bandwidth per token. Total parameters set the memory.

Weights are the floor, never the answer. A model whose weights just fit has no space left to serve anyone, which is what the rest of this post is about.

Why do vendors, Hugging Face and nvidia-smi show different numbers?

Three tools can describe the same GPU and the same model with three different numbers, and a 7% gap is enough to turn “fits” into “does not fit”. The cause is two units that share a name. A GB (gigabyte) is 1,000,000,000 bytes. A GiB (gibibyte) is 1,073,741,824 bytes (1,024³), about 7.4% bigger. Model hubs and file listings usually show GB. PyTorch, vLLM and nvidia-smi report GiB or MiB, often labelled loosely.

The rule: pick one unit and convert everything into it before you subtract. Mixing a GB weight size with a GiB GPU size can hide 5 GB on an 80 GB card, enough for tens of thousands of tokens of KV cache.

What sits in GPU memory while a model serves?

Here is the full list for one GPU running an inference server such as vLLM. Some items are fixed, some grow with traffic, and one only appears for a moment.

Put numbers on it for Llama 3.1 8B in BF16 on one H100, using vLLM’s default of taking 90% of the card (the overhead split is illustrative; the method matches part 17). Beside it is a model that looks as if it should fit and does not: Llama 3.1 70B in FP8, whose 70.6 GB of weights are less than 80.

Two lessons come out of the picture. On a well-sized serving GPU, the weights are often the minority: the KV cache is the biggest tenant, and the number of users you can serve depends on it. And “weights smaller than the card” is not a test of anything. The 70B FP8 model would start on one H100 only with utilisation pushed to about 0.95 and a short context, which leaves room for a handful of conversations. In practice it runs on two H100s or one H200.

How much does the KV cache add?

Every token a conversation has seen leaves a key and a value in every layer, so the model does not recompute them on the next token. Per token, that costs 2 (key and value) × layers × KV heads × head size × bytes. For Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB. Llama 3.1 70B, with 80 layers, needs 320 KiB per token. Multiply by tokens per conversation and conversations at once: 32 users with 4,000 tokens each on the 70B model is 32 × 4,000 × 327,680 bytes ≈ 42 GB, a third of the weights again. Part 14 covers the cache fully, including why grouped-query attention keeps it this small and how FP8 KV halves it.

KV cache only, worst case with every conversation at full length. Weights for comparison: 141 GB in BF16.

What happens when a model loads, and why can it still fail?

Loading is where most memory problems show up, because the server does its sizing then. Here is what a vLLM-style server does between the start command and the first request, for Llama 3.1 8B on one H100.

Engines that do not pre-allocate, such as a plain Hugging Face Transformers generate() loop, skip steps 3 to 5. They start quickly and then allocate KV cache and activations as requests arrive, so they run out of memory in the middle of serving instead of at start-up. That is the main reason production servers carve memory up front.

Why does a model that fits on paper fail?

Why does training need so much more memory?

Running a model reads the weights. Training changes them, and to change a weight well you have to keep several more numbers for it. The standard recipe, mixed precision with the Adam optimizer, keeps about 16 bytes for every parameter (from the ZeRO paper), which part 15 walks through step by step:

On top of that come activations: the outputs of each layer from the forward pass, kept so the backward pass can work out the gradients. They grow with batch size and sequence length. With activation checkpointing (recomputing some of them instead of storing them), budget about 20% on top. So full training of Llama 3.1 70B is 70.6 × 16 = 1,130 GB, plus 20% ≈ 1.36 TB, nearly ten times the 141 GB needed to run it.

Fine-tuning changes this. LoRA freezes the base model and trains small adapter matrices beside it, so the base needs only its 16-bit weights: no gradients, no optimizer state. QLoRA also stores the frozen base in 4-bit (NF4); its authors fine-tuned a 65B model on a single 48 GB GPU. Part 19 covers both in depth. The memory, for the same 70B model:

Full training never fits on one GPU above a few billion parameters, so it is always spread: PyTorch FSDP and DeepSpeed ZeRO shard the weights, gradients and optimizer state so each GPU holds a slice and fetches the rest just in time. That fetching is the traffic NVLink exists for (part 6).

Which GPUs hold which models?

The table below puts it together. Each cell is the fewest GPUs that hold the model with a modest working load: 8 conversations of 8,192 tokens (16-bit KV in the BF16 rows, FP8 KV in the others), about 4 GB of overhead per GPU, and each GPU filled to 90%. Splits come in 1, 2, 4 or 8, because the model’s attention heads have to divide evenly (part 5). Amber cells fit with under 5% to spare.

Memory only, for planning. A cell that says 1 still needs checking for speed (part 20). RTX 4090 and L40S have no NVLink, so splits on them run over PCIe.

Read across a row and the pattern is plain: each step up in GPU memory halves the number of GPUs a model needs, and each step down in precision does the same. Read down a column and you see why bigger cards matter for MoE models: DeepSeek-V3 in FP8 needs 671 GB for weights alone, more than the 640 GB of a full 8 × H100 server, so it needs eight H200s, eight B200s or two servers.

What if it does not fit on one GPU?

When the model is bigger than one card, the usual answer is tensor parallelism within one server: every layer’s weight matrices are cut into equal slices, each GPU holds one slice of every layer, and the GPUs swap partial results over NVLink twice per layer. The KV cache splits the same way, because each GPU handles its share of the KV heads. Here is Llama 3.1 70B in BF16 across four H100s:

Splitting adds memory as well as capacity, which is the part people expect least. On two H100s, 70B BF16 has 144 GB of budget against 141 GB of weights and 8 GB of overheads: the sum is already negative before any KV cache, so it does not start at the default setting. On four it has 131 GB of KV pool, room for about 50 conversations of 8,000 tokens at once. Beyond one server, the next layer is pipeline parallelism across servers, where each server holds a run of layers; part 5 explains both and what each costs in traffic.

Why is offloading to CPU memory so slow?

If the GPU is too small, why not keep part of the model in the server’s RAM, which is cheap and large? Many tools allow it: Hugging Face Accelerate with device_map="auto", llama.cpp with a partial --n-gpu-layers, vLLM with --cpu-offload-gb, and CUDA unified memory, which lets the GPU address more than it has and pages data in on demand. They all work. The problem is the pipe.

Every token needs every weight once. Weights in GPU memory are read at HBM speed. Weights in CPU memory have to cross PCIe for every single token, and PCIe is roughly fifty times narrower.

Worked through: the H100 holds about 60 GB of the 141 GB model after leaving room for overhead and a small cache. Reading those 60 GB at 3,350 GB/s takes about 18 ms. The other 81 GB must come over PCIe 5.0 x16 at about 64 GB/s: 1.27 seconds. Every token waits for both, so the GPU produces under one token per second and sits idle for 98% of the time. Four H100s, which is what this model needs to serve properly (see the split above), read their 35 GB slices in parallel in about 10.5 ms: roughly 95 tokens per second for one user, over a hundred times faster.

Batching helps a little, because one trip of the weights over PCIe serves every request in the batch, which is why offload is acceptable for overnight batch jobs. NVIDIA’s Grace Hopper superchips narrow the gap by joining CPU and GPU with NVLink-C2C (NVIDIA quotes 900 GB/s), but that is still far below HBM. For interactive serving, offloading weights is a way to try a model, not to run it. Offloading parts of the KV cache for idle conversations is different and useful, because that data is not needed on every token; part 14 covers it.

How do you read nvidia-smi and the vLLM memory lines?

Two places tell you what is actually in memory. The first is nvidia-smi, which shows the whole GPU and each process. For scripts, ask it for the numbers directly:

nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv
# illustrative output
index, name, memory.used [MiB], memory.total [MiB]
0, NVIDIA H100 80GB HBM3, 73420 MiB, 81559 MiB

A vLLM server shows about 90% used the moment it is ready, with no traffic at all. That is not a leak: the KV pool was carved at start-up and stays reserved whether it holds conversations or not. So nvidia-smi tells you how much the engine took, not how busy it is. For how full the cache is, use the engine’s own metric (vLLM exports KV cache usage as a percentage on its metrics endpoint).

The second place is the engine’s start-up log. Its shape (illustrative numbers, and wording varies between versions):

# illustrative excerpt, not a real capture
INFO  Model loading took 14.99 GiB memory
INFO  Memory profiling: peak activation 2.31 GiB, non-torch 0.62 GiB
INFO  GPU KV cache size: 396,272 tokens
INFO  Maximum concurrency for 16,384 tokens per request: 24.19x
INFO  Graph capturing finished, took 0.51 GiB

Each line maps to a band in the stacked diagram: the weights (in GiB, so 15 rather than 16), the activation peak from profiling, the non-torch memory (CUDA context and libraries), the KV pool in tokens, and the graphs. If the model is split, the loading line is per GPU: 70B in BF16 across four GPUs should show about 33 GiB each. A number twice what you expected means the weights were loaded at the wrong precision.

When you debug PyTorch code directly, torch.cuda.memory_allocated() is what tensors use and torch.cuda.memory_reserved() is what PyTorch’s caching allocator holds from the driver. A large gap between them is cached or fragmented memory: nvidia-smi counts it as used, but your tensors cannot always use it.

How do you check whether a model will fit, step by step?

Here is the method in full, worked for one request: Llama 3.1 70B for 40 conversations at once of up to 8,192 tokens each, on H100s.

1. Size the weights

   70.6 billion × 2 bytes = 141.2 GB in BF16, or 70.6 GB in FP8. Use total parameters, even for MoE models.

2. Size the KV cache for the worst case

   40 × 8,192 tokens × 327,680 bytes = 107.4 GB in 16-bit, or 53.7 GB in FP8. Use the context your app allows, not the model’s maximum.

3. Find the usable memory per GPU

   80 GB × 0.9 = 72 GB, minus about 4 GB per GPU for context, activations and graphs (illustrative; measure yours) = 68 GB for weights and cache.

4. Divide and round up to 1, 2, 4 or 8

   BF16 weights and 16-bit KV: 248.6 ÷ 68 = 3.7, so 4 H100s, with 23 GB spare. FP8 weights and FP8 KV: 124.3 ÷ 68 = 1.8, so 2 H100s, with 11.7 GB spare.

5. Check the spare and the alternatives

   Under about 10% spare, look again. The FP8 plan keeps 11.7 of 136 GB, about 9%: acceptable only because real conversations are rarely all at full length together. One H200 instead: 141 × 0.9 − 4 = 122.9 GB, against 124.3 needed, just short. Two H200s leave plenty of room to grow.

6. Confirm on the GPU

   Start the engine with the real settings and read the KV-tokens and concurrency lines. If the KV tokens divided by your context are below your users at once, the plan is short.

Memory decides whether it runs. It does not decide whether each user gets words fast enough; that is the bandwidth and compute check in part 20, which uses this method as its first step.

What do people get wrong about GPU memory?

• Sizing by the weights alone. The weights are the floor. On a busy serving GPU the KV cache is usually bigger, and overheads add several GB per GPU.
• Sizing MoE models by active parameters. DeepSeek-V3 computes like a 37B model and stores like a 671B one.
• Planning to 100% of the card. Context, activation peaks and the memory other processes use all need room. Plan to about 90% and check the busiest GPU in a split.
• Mixing GB and GiB. A 7% unit gap on an 80 GB card is 5 to 6 GB, which is tens of thousands of tokens of cache.
• Setting the context to the model’s maximum. Llama 3.1 supports 128K tokens; one such request on the 8B model needs about 17 GB of KV cache. Set what the application needs.
• Treating 4-bit as exactly a quarter. Scale factors add about 6 to 13% to the 4-bit values, and some layers are often kept at 16-bit.
• Assuming two small cards equal one big card. Two 24 GB cards pay overhead twice and talk over PCIe, so they hold less and run slower than one 48 GB card.
• Using CPU offload for live traffic. It runs, at about one token per second for a 70B model. It is for trying models, not serving them.

Short answers to common questions

Can I run a 70B model on one RTX 4090?

Not in GPU memory. Even in 4-bit the weights are about 40 GB against 24 GB. It runs with CPU offload at a few tokens per second or less, or across two to four cards in 4-bit with a small cache.

Why does a longer context not make the model bigger?

It does not change the weights at all. Longer context adds KV cache, which grows with every token of every conversation, and slightly raises the activation peak during prefill.

Why does vLLM take 90% of the GPU before any request arrives?

It reserves the KV cache pool up front so it can hand out blocks quickly and never fragment. Lower --gpu-memory-utilization if something else must share the GPU.

Do MoE models need less memory?

No. They need less compute and bandwidth per token, because each token uses only a few experts. All experts must still be loaded.

Is FP8 or 4-bit good enough?

FP8 is close to 16-bit quality for most models and halves the weights. 4-bit quarters them with a larger, model-dependent quality cost. Part 18 covers how to test it.

Sources: Meta Llama 3.1 model cards and configuration (parameters and layer shapes); OpenAI gpt-oss-120b model card; DeepSeek-V3 technical report; NVIDIA H100, H200, B200, L40S and RTX 4090 specifications; Rajbhandari et al., ZeRO (16 bytes per parameter); Hu et al., LoRA; Dettmers et al., QLoRA; vLLM documentation. nvidia-smi totals from published H100 HBM3 output. Overhead, activation and speed figures are worked estimates and illustrative; log and command output excerpts are illustrative.