Model and GPU
Model
Precision
GPU
How to split it
Tensor parallelslices of every layer
Pipeline parallelblocks of whole layers
Replicasfull copies for more users
Link between GPUs
What you are looking at: a model that does not fit on one GPU has to be split. Tensor parallel cuts every layer into slices, one per GPU, and the GPUs swap results after each layer. Pipeline parallel gives each GPU a block of whole layers, and the work passes from one to the next. Replicas run complete copies side by side to serve more users. Each GPU card shows its weights (dark) and its room for KV cache and working memory (blue).
Try this: pick 70B in 16-bit, then try pipeline 2× and tensor parallel 2×: both fit, but only tensor parallel raises the speed. Then set tensor parallel 8× and switch the link to PCIe, and watch the time spent talking jump.
Memory per GPU = its share of the weights + 10% for KV cache and working memory + 2 GB runtime. Speed is for one user: each GPU reads its share of the weights at full bandwidth, plus about 8 µs per tensor-parallel exchange on NVLink and 30 µs on PCIe (two exchanges per layer), plus a small hop between pipeline stages. Exchange times are illustrative. NVLink on H100 and H200: 900 GB/s per GPU, both directions; PCIe 5.0 x16: about 64 GB/s each way (128 GB/s both directions).
What you are looking at: a tiny MLP block with real numbers (hidden size 4, MLP size 8) split over 2 or 4 GPUs exactly the way Megatron-style tensor parallelism splits a real layer. The first matrix W1 is cut by columns, the second W2 by rows. Each GPU computes on its own slice with no talking, until the end, when one all-reduce adds the partial results. ReLU is used so the numbers stay whole; Llama uses SiLU, which works the same way here.
Try this: press Next step through all five steps and check that the all-reduce sum equals the one-GPU answer. Then switch to 4 GPUs: each slice gets narrower, and there is still exactly one exchange. Below, raise the tokens per step and switch the link to see when the exchanges start to cost more than the GPU work.
What you are looking at: the timeline of pipeline parallelism. Each row is one GPU holding a block of layers. In training, dark boxes are forward passes of a micro-batch and red boxes backward passes (twice as long). In serving, the boxes are different requests passing through. The hatched gaps are the bubble: time a GPU waits because work has not reached it yet or has already left.
Try this: set 4 stages and 1 micro-batch: three quarters of the time is idle. Slide micro-batches up to 32 and watch the bubble fall below 10%. Then compare the two schedules: same bubble, but 1F1B keeps far fewer micro-batches' activations in memory.
What you are looking at: how much faster a job gets as you add GPUs, against the ideal straight line. The gap is time spent communicating, which does not shrink as the GPU work does. The bars split each setup's time into GPU work (dark) and communication (red). All numbers come from a simple illustrative model, not benchmarks.
Try this: in 70B serving, switch NVLink to PCIe and see efficiency at 8 GPUs fall. In 405B serving, compare pipeline across servers with tensor parallel across servers. In Training run, push the GPU count to 1,024 and then raise the overlap: hiding the all-reduce behind the backward pass is what keeps big clusters efficient.