What you are looking at: the memory on one GPU while training a Llama 3.1 model, split into 16-bit weights, gradients, Adam optimizer state and activations, next to what serving the same model needs. Training state costs 16 bytes per parameter; activations grow with sequence length and batch; sharding (ZeRO / FSDP), tensor and pipeline parallelism divide parts of it.
Try this: pick the 8B model with DDP on 8 GPUs and see it overflow 80 GB. Switch to ZeRO-3, then raise the sequence length to 32K and watch activations take over. Turn on checkpointing to bring them back.
Estimates are illustrative. Training state uses the ZeRO paper's formulas (2 + 2 + 12 bytes per parameter). Activations use 34 × sequence × batch × hidden bytes per layer (Korthikanti et al., 2022) without the attention-score term, which FlashAttention-style kernels do not store, divided by tensor parallelism (assuming sequence parallelism). Pipeline parallelism divides the model state; with a 1F1B schedule the first stage still holds about one full set of activations, so they are not divided. Checkpointing keeps each layer's input (2 × s × b × h) plus one layer being recomputed. Serving: 16-bit weights plus KV cache for 32 users at the chosen sequence length (8B 128 KB, 70B 320 KB, 405B about 504 KB per token), split over the tensor-parallel GPUs. Real frameworks add buffers and fragmentation.
What you are looking at: the size of a training run. Operations come from 6 × parameters × tokens; GPU hours follow from the H100's dense BF16 peak of 989 TFLOPS times the share of peak the job really achieves (MFU). Meta measured 38% to 43% for Llama 3.1 405B and reports 30.84 million H100 hours.
Try this: leave 405B at 15.6 trillion tokens and compare the calculated hours with Meta's figure. Then turn on activation recomputation and lower MFU to 30% to see how quickly a run gets longer.
Meta reports 30.84 million H100 GPU hours for Llama 3.1 405B on about 15.6 trillion tokens, and 1.46 million and 7.0 million for 8B and 70B. Its reported hours include restarts, checkpointing and lower-MFU phases, so they sit above the pure calculation.
What you are looking at: cumulative compute after launch. Training is paid once; every served token costs 2 × parameters operations, a third of a training token, so serving matches training after about three times the training tokens. Measured in GPU hours it happens sooner, because serving runs at a lower share of peak.
Try this: click through the traffic presets and watch the crossing day move. Then switch the chart to GPU hours and lower serving MFU to 5%: the crossing comes much earlier.
Traffic levels and serving MFU are illustrative. Decode is limited by memory bandwidth, so serving engines often reach a small share of peak operations; real values depend on batch size, quantization and hardware.
What you are looking at: one GPU cluster over 24 hours. Inference demand follows its users; training runs flat out. With separate pools, the serving pool is sized for the peak and sits partly idle. With a shared cluster and preemption, training fills whatever inference is not using and gives GPUs back as traffic rises.
Try this: start with the office assistant on separate pools and note the idle hours. Switch to shared and see how many GPU hours move to training and how many preemptions that costs. Then try the global app, which leaves less to borrow.
Profiles are illustrative. Separate pools: the serving pool is the inference peak plus 10% spare for failures, and training runs on the rest of the cluster. Shared: inference always gets what it needs plus 10% spare, and training uses everything else; each hour inference grows, that many training GPUs are preempted and must resume from a checkpoint.