Setup
Network between nodes
Nodes (8 GPUs each)
Model
Break something
One training step
computenetwork, hidden behind computenetwork, GPUs waiting
A training job runs on several 8-GPU H100 nodes. After every step, all GPUs must share their gradients (an all-reduce) before anyone can start the next step. Inside a node that goes over NVLink and is fast. Between nodes it goes over the network, and the slowest link sets the pace for everyone. Change the network, add nodes, then break something. Timings are illustrative, but the behaviour is how NCCL and data-parallel training work.