A single H100 will train a 7 billion parameter model. It will just take a week you do not have. So at some point you stop asking one GPU to do the whole job and spread a single training run across many of them, and the moment you do, three things you could ignore before start to matter at once: the InfiniBand fabric between the machines, the NCCL library that moves gradients over it, and the GPU quota you never checked. The failure that teaches this is quiet. Your job launches, every process reports in, and then it sits at the first gradient sync doing nothing, because the GPUs cannot find each other on the fast network.
Key takeaways
- Distributed training splits one run three ways: data parallel copies the whole model to every GPU and splits the batch, tensor parallel splits a single layer across GPUs, and pipeline parallel puts different layers on different GPUs. Most jobs start with data parallel.
- On Azure ML you get this by pointing a command job at a multi-node GPU cluster and setting a pytorch distribution with a process per GPU. Azure ML launches the ranks and sets the network environment for you.
- The ND H100 v5 SKU gives each GPU a dedicated 400 Gb/s InfiniBand link, 3.2 Tbps per VM. That fabric, not the GPUs, is what lets a run scale past one node without the gradient sync eating your speedup.
- Adding nodes cuts wall-clock time but never in a straight line. Scaling efficiency falls as you add GPUs, so more hardware can cost more money for less marginal speed. Size the cluster to a deadline, not to a wish.
Why one GPU stops being enough
Two limits push a training run off a single GPU. The first is memory. A model in training holds far more than its weights: it holds the optimizer states, the gradients, and the activations saved for the backward pass, and for a mixed-precision Adam run that overhead is several times the size of the weights alone. A 7 billion parameter model that looks like 14 GB of weights can need well over 80 GB once you add all of that, and a 70 billion parameter model does not fit on any single card made. The second limit is time. Even when a model fits, one GPU grinding through hundreds of billions of tokens can take days or weeks, and a run you cannot finish before the requirement changes is a run you did not really do.
Distributed training answers both, but with different tools. To beat the time wall you add more GPUs and give each one a slice of the work, which is data parallelism. To beat the memory wall you split the model itself across GPUs so no single card holds all of it, which is tensor or pipeline parallelism, or a sharded optimizer. Real large-model runs combine them. The reason the topic feels hard is that the moment work spans two machines, correctness depends on those machines exchanging data quickly and constantly, and that turns the network into part of your training loop.
What a distributed job actually splits
Data parallelism is the workhorse, so understand it first. Every GPU holds a complete copy of the model. You split each training batch into shards, one per GPU, and each GPU runs a forward and backward pass on its shard to produce gradients. Before the weights update, every GPU has to agree on the average of all those gradients, and that agreement is a collective operation called an all-reduce. PyTorch implements this as DistributedDataParallel, DDP for short, which wraps your model and runs the all-reduce for you after each backward pass. The cost is obvious once you see it: every step, every GPU sends and receives a full set of gradients. On one machine that traffic rides NVLink between GPUs. Across machines it rides the network, which is why the fabric matters.
When the model itself will not fit, you shard it. A sharded optimizer, the approach DeepSpeed calls ZeRO and PyTorch calls Fully Sharded Data Parallel, or FSDP, splits the optimizer states, gradients, and optionally the weights across GPUs so no card holds the whole thing, while still training on split batches. Tensor parallelism cuts individual layers, so a matrix multiply is done in pieces on several GPUs and stitched back together, which means very heavy communication and so it stays inside one node where NVLink is fast. Pipeline parallelism assigns whole blocks of layers to different GPUs and streams micro-batches through them like an assembly line, trading a start-up and drain cost, the pipeline bubble, for the ability to hold a model too tall for one device. Azure ML supports the machinery for all of these because it runs your training script as is; it schedules the cluster and wires the network, and the parallelism lives in your PyTorch or DeepSpeed code.
| Strategy | What it splits | Reach for it when | Communication cost |
|---|---|---|---|
| Data parallel (DDP) | The batch, model copied whole | Model fits on one GPU, you want it faster | One gradient all-reduce per step |
| Sharded (ZeRO or FSDP) | Optimizer, gradients, maybe weights | Model plus states will not fit | More collectives, gather on demand |
| Tensor parallel | Inside a single layer | A layer is too wide for one GPU | Very high, keep it in one node |
| Pipeline parallel | Blocks of layers across GPUs | Model is too deep for one GPU | Lower, but adds a pipeline bubble |
The ND family and the InfiniBand fabric
The reason multi-node training works on Azure at all is a network most cloud VMs do not have. The ND-series virtual machines carry InfiniBand, a low-latency remote direct memory access fabric that lets one GPU write into another GPU memory across machines without going through the operating system network stack the way ordinary Ethernet does. On the ND H100 v5 SKU, Standard_ND96isr_H100_v5, each of the eight H100 GPUs gets its own dedicated 400 Gb/s NVIDIA Quantum-2 InfiniBand link, which adds up to 3.2 Tbps of cross-node bandwidth per VM. That is the number that decides whether your all-reduce is free or fatal. Inside a VM the eight GPUs talk over NVLink; between VMs they talk over InfiniBand; and NCCL, the NVIDIA Collective Communications Library, is the software that picks the fast path automatically when everything is configured right.
This is also where a silent performance cliff hides. NCCL will use InfiniBand when it can see it and fall back to ordinary TCP sockets when it cannot, and a run on TCP still produces correct results, just many times slower, so nothing errors and you simply pay for a cluster that crawls. That happens when the cluster nodes are not placed close enough on the fabric, when the InfiniBand driver is missing from your container image, or when an environment variable disables it. The defense is to prove the fast path is live before you trust a long run, which I come back to below. The older ND A100 v4 generation gave each GPU a 200 Gb/s link, and the newer ND H200 v5 keeps the 400 Gb/s per-GPU fabric while roughly doubling GPU memory to fit larger models per node. Newer Blackwell-based ND generations extend this further. [VERIFY: current ND GB200 v6 SKU name, per-GPU InfiniBand rate, and general availability]
| SKU | GPUs | Per-GPU InfiniBand | Fabric per VM |
|---|---|---|---|
| Standard_ND96amsr_A100_v4 | 8x A100 80GB | 200 Gb/s | 1.6 Tbps |
| Standard_ND96isr_H100_v5 | 8x H100 80GB | 400 Gb/s | 3.2 Tbps |
| Standard_ND96isr_H200_v5 | 8x H200 141GB | 400 Gb/s | 3.2 Tbps |
The isr in the H100 and H200 names is the marker to look for: it signals the InfiniBand-equipped variant. A same-generation SKU without it may not carry the fabric. [VERIFY: exact A100 v4 InfiniBand SKU suffix and H200 v5 memory per GPU]
In practice
Create the GPU cluster with a minimum node count of zero so it scales to nothing between jobs and stops billing, and set a maximum that matches the quota you actually hold. Use an Azure curated ACPT training image rather than a bare PyTorch container, because the curated images ship the InfiniBand user-space libraries and a matching NCCL, which is exactly the piece a hand-built image tends to miss. That one choice removes the most common cause of a silent TCP fallback.
Running a multi-node job on Azure ML
Here is the whole thing in the Azure ML v2 Python SDK. It submits a command job to a two-node H100 cluster, launches eight processes per node so one process drives each GPU, and lets Azure ML set the distributed environment. Your training script does not parse any of this; it reads the standard variables RANK, WORLD_SIZE, and MASTER_ADDR that Azure ML injects, and calls the normal PyTorch setup.
from azure.ai.ml import MLClient, command
from azure.identity import DefaultAzureCredential
ml_client = MLClient(
credential=DefaultAzureCredential(),
subscription_id='SUB_ID',
resource_group_name='RG_NAME',
workspace_name='WS_NAME',
)
job = command(
code='./src', # folder holding train.py
command='python train.py --epochs 3',
environment='azureml://registries/azureml/environments/acpt-pytorch-2.2-cuda12.1/labels/latest',
compute='nd-h100-cluster', # a multi-node ND H100 v5 cluster
instance_count=2, # 2 nodes
distribution={'type': 'pytorch', 'process_count_per_instance': 8},
display_name='sft-7b-2node',
)
returned = ml_client.jobs.create_or_update(job)
print(returned.studio_url)Expected output: a studio URL printed at once, and in the job the World size reads 16, that is instance_count times process_count_per_instance, with 8 ranks on each node. Failure mode one: the ranks all start but the job hangs at init_process_group or the first all-reduce, which almost always means NCCL cannot reach InfiniBand and is waiting on a path that is not there. Failure mode two: you asked for 2 nodes of a SKU you have quota for only one of, and the job sits in Not started until it times out. Raise quota for the ND H100 family in the region first. [VERIFY: current ACPT curated environment name and tag, and the ND H100 v5 quota family label in your region]
Scaling efficiency and what it really costs
Doubling the GPUs never halves the time, and treating scaling as if it did is how teams overspend. Every extra GPU adds communication that the compute has to hide behind, and past a point the all-reduce stops fitting neatly under the math and starts adding wall-clock time of its own. The number to track is scaling efficiency: the speedup you actually got divided by the speedup a perfect linear scale would give. At 90 percent efficiency, two nodes give you 1.8 times the throughput of one, not 2 times. At 75 percent, four nodes give you 3 times, not 4. The efficiency almost always falls as you add nodes, because communication grows faster than the compute you are hiding it behind.
Worked example
Take a fine-tune that needs about 168 GPU-hours of compute. On one ND H100 v5 node, eight GPUs at near 100 percent efficiency, that is roughly 21 hours of compute, call it 22.8 hours of wall clock once you count startup and checkpointing. Go to two nodes, 16 GPUs, and at an assumed 85 percent efficiency you get 1.7 times the throughput, so about 12.3 hours. Four nodes, 32 GPUs, at an assumed 75 percent efficiency give 3 times, about 7.0 hours. Now the cost. At an assumed 98 dollars per node-hour, one node for 22.8 hours is about 2,234 dollars, two nodes for 12.3 hours is about 2,411 dollars, and four nodes for 7.0 hours is about 2,744 dollars. You bought speed and paid for it: the four-node run finishes in under a third of the time and costs about 23 percent more. Whether that trade is worth it depends only on your deadline. [VERIFY: current Standard_ND96isr_H100_v5 hourly rate in your region and real measured efficiency for your model]
The practical takeaway is to size the cluster to a deadline. If the one-node run finishes inside your window, run it on one node and save the money. If it does not, add nodes only until the wall-clock time drops under the deadline, and stop, because the next doubling costs more and buys less. Measuring your real efficiency on a short run at one and two nodes tells you the slope of your own curve, which is worth far more than any rule of thumb. AWS runs the same arithmetic on its own resilient training service, which I cover in the SageMaker HyperPod part, and the lesson carries across clouds: the fabric and the efficiency curve decide the bill, not the GPU count on the invoice.
Where multi-node runs break
Four failures cover most of the wasted days. The first is the TCP fallback already named: correct output, terrible speed, no error, caught only by reading the NCCL log. The second is a straggler. In data-parallel training every step waits for the slowest GPU, so one node with a hot card, a bad disk, or a noisy neighbor drags the whole cluster to its pace, and the symptom is a step time that is fine most of the time and spikes for no visible reason. The third is data loading. Sixteen GPUs eat input far faster than eight, and a dataset pipeline that fed one node comfortably can starve two, leaving expensive GPUs idle while they wait on storage, so you move the data to a fast mounted store and give the loader enough worker processes.
The fourth is the one that hurts most: losing a long run to a single node failure. The more nodes and the longer the run, the higher the odds that one machine drops during training, and without checkpoints a failure at hour 20 of a 22 hour run costs you all 20 hours. Checkpoint to a durable store on a schedule and make the script resume from the latest checkpoint, so a lost node costs minutes, not the whole job. This is exactly the resilience that purpose-built training clusters automate, and it is why for very large multi-week runs the industry moved toward managed resilient training rather than raw command jobs. For the two-node, one-day jobs most teams actually run, disciplined checkpointing on a standard Azure ML cluster is enough. Later in this series, the reference-architecture part pulls the training cluster together with serving and evaluation into one production picture.
Start on one node, add nodes only when the clock demands
My rule is boring and it saves money. Get the run correct and fast on a single ND node first, with DDP or FSDP working, checkpoints writing and resuming, and the data loader keeping the GPUs fed. Only then reach for a second node, and reach for it because a deadline demands it, not because more GPUs feel faster. When you do scale, measure your real efficiency at one and two nodes before committing to four or eight, because your curve, not a benchmark from a blog, tells you where the next node stops being worth it. And whatever the node count, prove InfiniBand is carrying the traffic before you trust a long run, because the difference between the fast fabric and a TCP fallback is the difference between a training bill you planned and one you did not.
So here is where I would start this week. Take a training job you already run on one GPU, move it to a single eight-GPU ND node with a pytorch distribution and DDP, confirm the World size and the NCCL InfiniBand lines in the log, and time it. That single number, your one-node wall clock, is the baseline every scaling decision after it depends on. Get that solid and the jump to two nodes is a change of one integer, not a leap into the unknown.
References
- Distributed GPU training guide, Azure Machine Learning SDK v2
- ND H100 v5 size series, Azure Virtual Machines
- ND family virtual machine size series
About The Author
DrJha