Model
GPUs in the job
Where checkpoints go
Save a checkpoint every
GPU time wasted vs checkpoint interval
Where 24 hours of GPU time goes
trainingwaiting for checkpoint savesredoing work lost in crashesrestarting
Long training jobs crash, so they save checkpoints: a full copy of the model and optimizer state. Save too rarely and a crash throws away hours of work. Save too often and the GPUs spend their time waiting for storage. Pick a model, a cluster size and a storage system, and the calculator finds the interval that wastes the least GPU time. Failure rates follow Meta's published Llama 3 numbers; storage speeds are typical, not measured.
Weights only, 16-bit, read by one node (one node cannot use a whole storage system's bandwidth). This is the "load weights" part of the cold start in the autoscaling post. The selected storage is in bold.