What you are looking at: AI work makes GPUs swap data constantly: partial results after every layer, gradients after every training step, conversation memory between servers. Pick a job and see how long it takes over each kind of link. Click a link to see how the GPUs are wired.
Try this: pick "Training: sync gradients" and compare PCIe with NVLink 4. Then pick "tensor-parallel exchanges" and notice that NVLink 5 is no faster than NVLink 4: small messages are limited by delay, not bandwidth.
Per-GPU bandwidth in each direction: PCIe 5.0 x16 64 GB/s, NVLink 4 (H100) 450 GB/s, NVLink 5 (B200) 900 GB/s, one 400 Gb/s InfiniBand port 50 GB/s. Large transfers use a ring all-reduce: each GPU sends and receives 2 × 7/8 of the data. Small tensor-parallel exchanges are set by delay, not bandwidth: about 8 µs each over NVLink with NVSwitch and 30 µs over PCIe or InfiniBand (illustrative). Real libraries such as NCCL pick smarter methods, so treat these as relative.
What you are looking at: a ring all-reduce. Each GPU starts with its own buffer, cut into as many chunks as there are GPUs. In the reduce-scatter half, each GPU sends one chunk to its right-hand neighbour, which adds it to its own; in the all-gather half, finished chunks travel round until every GPU has the full sum. Darker red means a chunk holds more GPUs' data. Outlined chunks are being sent this step.
Try this: press Play with 4 GPUs and count the steps (6). Then set 8 GPUs: the steps nearly double, but "sent per GPU" only grows from 1.5× to 1.75× the buffer. Switch the buffer to 16 KB and compare PCIe with NVLink: at that size delay, not bandwidth, sets the time.
Time = 2 × (n − 1)/n × buffer ÷ per-direction bandwidth, plus 2 × (n − 1) steps × per-step delay (2 µs on NVLink, 8 µs on PCIe or InfiniBand, illustrative). NCCL may choose a tree, or on NVSwitch systems let the switch do the adding (NVLS), which changes these numbers.
What you are looking at: three ways to put GPUs in a box or a rack. Pick a design, then click any two GPUs to see the path a message takes between them, how many hops it crosses, the code nvidia-smi topo -m would show, and the bandwidth each way.
Try this: in the PCIe server, pick GPU 0 and GPU 1 (same switch), then GPU 0 and GPU 6 (other CPU) and watch the path climb through both CPUs. Switch to HGX: every pair is the same. Switch to NVL72 and pick GPUs in different trays: still NVLink.
Simplified layouts. PCIe servers vary in how many GPUs share a switch. NVL72 GPUs are drawn as 18 groups of 4 to keep the picture readable, not as a tray count. Bandwidths are per GPU, each direction.
What you are looking at: one forward pass of a model split with tensor parallelism. Each layer ends with two all-reduces over the hidden vector of every token in the step. The calculator compares that talking time with the useful work: reading this GPU's share of the weights, or doing the maths when the step holds many tokens. GPU fixed at H100-class: 3.35 TB/s memory, 989 TFLOPS BF16.
Try this: start with "Chat, 70B, 1 user" and switch the link from NVLink 4 to PCIe: talking jumps from about a fifth of the step to nearly half. Then try the long-prompt preset, where bandwidth rather than delay sets the cost, and the 8B preset at TP 1, where nothing is sent at all.
Estimates (illustrative). Useful work = the larger of (weights ÷ TP ÷ 3.35 TB/s) and (2 × parameters × tokens ÷ TP ÷ 989 TFLOPS), ignoring KV cache reads and real efficiency. Communication = 2 × layers × (bytes each GPU sends ÷ bandwidth + per-message delay), with bytes = 2 × (TP − 1)/TP × tokens × hidden × 2. Per-message delay 8 µs on NVLink with NVSwitch, 30 µs on PCIe or InfiniBand. Engines overlap some communication with compute, so real numbers are lower.