You wrote a training script that runs fine on one GPU. You dropped a second card into the box, ran the exact same script, and nothing got faster. Both GPUs show up in nvidia-smi, but one sits at 0 percent while the other does all the work. The script never asked for the second card, so it never used it. Single-process PyTorch does not spread itself across GPUs on its own.

This is a scheduling and placement problem before it is a deep learning problem. You have N accelerators and one workload, and something has to launch one process per GPU, give each its own slice of data, and keep the copies in sync. That is exactly what accelerate handles. It wraps torch.distributed so your loop stays readable while the launcher deals with ranks, processes, and the collective communication underneath.

What accelerate changes in your code

accelerate does not replace your training loop and it does not hide PyTorch from you. It inserts one object, the Accelerator, that becomes the single point where device placement and distributed coordination happen. You stop writing .to(device) and .cuda() by hand. You hand your model, optimizer, and dataloader to accelerator.prepare(), and you swap loss.backward() for accelerator.backward(loss) so gradient synchronization fires at the right moment. The same script then runs unchanged on a laptop CPU, one GPU, or eight.

Here is the whole diff in practice. The left is single-GPU PyTorch you have probably seen. The right is the accelerate version.

pip install accelerate

# before: single GPU
model = model.to('cuda')
for batch in dataloader:
    batch = {k: v.to('cuda') for k, v in batch.items()}
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step(); optimizer.zero_grad()

# after: one GPU, eight GPUs, or many machines, same code
from accelerate import Accelerator
accelerator = Accelerator()
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
for batch in dataloader:
    outputs = model(**batch)        # no manual .to(device)
    loss = outputs.loss
    accelerator.backward(loss)      # not loss.backward()
    optimizer.step(); optimizer.zero_grad()

That is the entire change. prepare() moves tensors to the right device and shards the dataloader so each process sees a different slice. accelerator.backward() handles gradient sync.

Setting it up: config once, launch every time

Editing the script is half the job. The other half is launching one process per GPU, which python train.py will never do. You answer a short questionnaire once with accelerate config, which writes a YAML file describing the machine, then you run the script through accelerate launch.

# interactive setup, asks about GPUs, mixed precision, FSDP/DeepSpeed
accelerate config

# launch using that saved config
accelerate launch train.py

# or skip the file and pass flags directly
accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 train.py

--num_processes is your GPU count on a single node. A saved config example for two GPUs holds distributed_type: MULTI_GPU, num_processes: 2, and mixed_precision: fp16. In CI you commit one YAML per machine class and point at it with --config_file so a run is reproducible.

DDP, FSDP, DeepSpeed: pick the backend on memory

accelerate exposes three strategies behind the same code. The choice between them is not about speed first, it is about whether the model, its gradients, and the optimizer state fit in one GPU. That total is bigger than people expect. A model trained with the Adam optimizer carries optimizer state roughly twice the size of the parameters, plus the gradients, plus activations. The weights you downloaded are only part of the bill.

DDP: the default when the model fits

Distributed Data Parallel puts a full copy of the model, gradients, and optimizer state on every GPU. Each card processes a different batch, then the gradients are averaged across cards every step. This is the fast path and the one to reach for first. It scales throughput close to linearly when the interconnect can keep up, and it adds no memory headroom. If the model did not fit on one card, DDP will not save you, because every card still holds the whole thing.

FSDP and DeepSpeed: when one card is not enough

Fully Sharded Data Parallel and DeepSpeed ZeRO solve the same problem in nearly the same way: they shard the parameters, gradients, and optimizer state across the data-parallel workers so no single GPU holds the full set. DeepSpeed expresses this as ZeRO stages, where stage 1 shards optimizer state, stage 2 adds gradients, and stage 3 adds the parameters themselves. FSDP at full sharding is the equivalent of ZeRO-3. The cost is communication: shards have to be gathered before a layer runs and released after, so you trade interconnect bandwidth for the ability to train a model that no single card could hold.

A real two-GPU run

Say the model fits on one card and you want to halve wall-clock time on two. You configure DDP, then launch. The thing to watch is the effective batch size, because it multiplies by the process count and that quietly changes your training dynamics.

# two GPUs, mixed precision, plain data parallel
accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 train.py

# expected: two processes start, both cards light up in nvidia-smi
# Detected 2 processes, distributed across 2 GPUs
# only rank 0 prints progress if you guard logging

# for a model that will not fit on one card, switch to FSDP
accelerate launch --use_fsdp --num_processes=2 
  --fsdp_auto_wrap_policy=TRANSFORMER_BASED_WRAP 
  --fsdp_backward_prefetch=BACKWARD_PRE train.py

Failure mode: if you see CUDA out of memory the moment FSDP starts, the wrap policy is usually the cause. Without an --fsdp_auto_wrap_policy, FSDP cannot decide where to split the model and behaves like it is holding the whole thing. Set a transformer wrap policy so each block is sharded. If logs print twice for every line, you forgot to guard printing behind accelerator.is_main_process and every rank is writing.

Where this fits with the rest of the stack

accelerate is the training-time half of the multi-GPU story. The serving half, running an already-trained model across cards, is a different tool, and we get to that with Text Generation Inference later in this series. If you are doing a parameter-efficient fine-tune, the previous part on LoRA and PEFT pairs directly with this: LoRA shrinks the trainable footprint so much that a fine-tune which would have needed sharding often fits in plain DDP, or even one card. For the concept of data parallelism itself, the GenAI series covers the idea vendor-neutral. When the next question is which physical GPUs to buy and how to wire them, that is the NVIDIA series, and for running all of this on VMware Cloud Foundation, the Private AI series.

My recommendation

Reach for accelerate as the default way to take a single-GPU script to many GPUs. Why: it changes about four lines, keeps the loop readable, and gives you DDP, FSDP, and DeepSpeed behind one launcher, so you do not rewrite anything when requirements grow. When it is not the right tool: if you are purely serving an already-trained model, this is the wrong layer, use an inference server. And if your bottleneck is a slow interconnect rather than GPU count, more cards through accelerate will not help, fix the wiring first. What to validate before you trust a run: confirm you launched with accelerate launch and not python, check the effective batch size after the process-count multiplier, and watch real GPU utilization across all cards rather than assuming the second one is busy. Do those three and accelerate will hold up from one card to a full node.

Your move: take one training script you already run on a single GPU, add the Accelerator, and launch it on two. Then watch both cards in nvidia-smi and check that the effective batch size is still what you intended.

Saving from a sharded run without breaking the checkpoint

The moment you go multi-GPU, saving a model stops being a one-liner. With FSDP the weights are split across cards, so a naive save writes a shard, not a whole model, and every process tries to write at once. accelerate gives you the primitives to do this correctly: call accelerator.wait_for_everyone() so the processes line up, accelerator.unwrap_model(model) to get the underlying model back out of the distributed wrapper, and accelerator.save_state() to write a checkpoint that can resume. Guard any plain logging or single-file writes behind accelerator.is_main_process so only rank 0 touches the disk.

This matters to whoever runs the storage as much as whoever runs the GPUs. A long training job checkpoints repeatedly, and a sharded checkpoint of a large model is not small. Multiply checkpoint size by how often you save by how many runs are in flight, and you have a real capacity number for shared storage, plus the write bandwidth to land those checkpoints without stalling the GPUs. Plan that the way you would plan backup storage, because a run that cannot write its checkpoint has wasted every GPU hour up to that point.

References

• accelerate docs: launching scripts
• accelerate docs: FSDP vs DeepSpeed
• huggingface/accelerate on GitHub

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts