Nine percent. That was the mean GPU utilisation across a training cluster I inherited two years ago, measured over a fortnight with the vendor tooling that had been installed and never opened. Eight accelerators were on the invoice every hour of every day. Slightly less than one of them was doing arithmetic at any given moment. Nobody on that team was careless. They had simply never been told that the number existed, and the finance report they were being judged against showed instance hours, which were exactly as expected.

Compute cost for machine learning is decided far less by which instance you pick than by how much of the hour that instance spends waiting. An architect who understands that spends their budget conversations arguing about pipelines and batch sizes rather than about hourly rates, and gets a better answer. This part is the arithmetic behind those conversations: how to size a training run, which accelerator class fits which job, how the purchase models actually price out, and where scaling out stops paying for itself.

Utilisation, not price per hour

Where the churn project stands: Part 27 gave us four planes and a set of contracts between them, but said nothing about the hardware underneath. This part sizes that hardware, and our churn model is a useful test case precisely because it is unglamorous. Roughly seven thousand customers, twenty or so engineered features from Part 6, a gradient boosted model from Part 12, retrained nightly. Sizing compute for that honestly is a more common architectural problem than sizing a language model cluster, and it is the one people get wrong most often in the expensive direction.

One formula governs everything that follows. Cost of a training run equals hourly rate multiplied by wall clock hours, and wall clock hours equal useful compute time divided by utilisation. Halving utilisation doubles the bill just as surely as doubling the rate does, and utilisation is usually the variable you can move without asking anyone for money. On the cluster I opened with, moving utilisation from 9 percent to 78 percent was worth more than every negotiated discount that team had ever obtained, and it took an afternoon.

Low utilisation almost always has a mundane cause. Data loading on a single worker thread while the accelerator waits. Preprocessing that belongs in the warehouse running inside the training loop. Small batches that leave the tensor cores half empty. Synchronous logging of a metric to a remote server between steps. None of these are interesting, and all of them are cheaper to fix than to buy around. Profile before you procure is the whole discipline, stated in four words.

GPU classes and what each one is for

Accelerators differ along three axes that matter to a modeller: memory capacity, memory bandwidth, and which numeric formats the tensor cores handle natively. Capacity decides whether your model and its optimiser state fit at all. Bandwidth decides how fast large batches move. Format support decides whether you can halve your memory footprint with mixed precision, which means computing most operations in a sixteen bit or eight bit format while keeping the master weights in thirty two bit. PyTorch exposes that through torch.amp, and it is the single highest leverage change available on most training jobs.

Reading the table below, notice that per accelerator rates span roughly eight times from an A100 to a B200, while memory capacity spans under three times. Paying for a newer part buys throughput and format support far more than it buys room. If your problem is that the model will not fit, more memory per device or sharding across devices is the answer. If your problem is that a run takes fourteen hours, throughput is the answer. Confusing the two is how teams end up renting a B200 to hold a model that fits comfortably in 24 gigabytes.

My verdict for a team in the churn project’s position: default to CPU for gradient boosted models on tabular data, and buy a single L4 when you start doing anything with text or embeddings. What I would avoid is standing up an eight GPU H100 instance for exploratory work, because the utilisation pattern of exploration is bursts of activity around long human thinking pauses, and that pattern is the worst possible match for a machine that costs 55 dollars an hour whether or not anyone is looking at it.

Rightsizing a training run

Memory sizing is arithmetic anyone can do on a whiteboard, and doing it in the room saves a procurement cycle. Take the parameter count, multiply by bytes per parameter, then account for what training adds. Weights in thirty two bit floating point cost four bytes each. Gradients cost the same again. A momentum based optimiser such as Adam keeps two further state tensors, so another eight bytes. That is sixteen bytes per parameter before a single activation is stored, which means a one billion parameter model needs roughly sixteen gigabytes just to exist in a training state.

Activations are the part people forget, and they scale with batch size rather than with parameters. Doubling the batch roughly doubles activation memory, which is why a job that trains happily at batch 16 dies at batch 64 with a message about being unable to allocate. Mixed precision helps on both fronts by storing activations in a sixteen bit format, and bfloat16 in particular keeps the same exponent range as thirty two bit floating point, which is why it converges without the loss scaling gymnastics that half precision needs.

Purchase models and what each one really costs

Four ways exist to pay for accelerator time, and they differ in price by more than most people assume. On demand is the list rate, currently 55.04 dollars an hour for a p5.48xlarge with eight H100 GPUs in North Virginia. Reserved capacity blocks price the same instance at 34.608 in that region and 31.464 in Mumbai, Tokyo, London and several others, a discount of roughly 37 to 43 percent in exchange for committing to a fixed window scheduled in advance. Spot capacity is cheaper again but can be reclaimed with short notice, which is survivable for a job that checkpoints and fatal for one that does not. Commitment plans discount steady baseline usage over one or three years.

Operating system charges are a small trap worth naming. Linux carries no additional charge on capacity blocks, while Red Hat Enterprise Linux adds 1.8432 dollars per instance hour on the P5 family. Across a 192 instance hour reservation that is 354 dollars of pure licensing on top of the compute, which is the kind of line item that appears in month two and surprises somebody. AWS publishes a worked example on exactly this: four p5.48xlarge instances reserved for 48 hours at 31.464 dollars comes to 6,041.09 in reservation fee, plus 346.52 for Red Hat usage across 188 instance hours, for 6,387.61 total.

Reading that chart, my verdict is that reserved capacity on the previous generation is the right default for scheduled retraining, and I would avoid on demand H100 for anything with a predictable cadence. Spot deserves a separate note. It suits hyperparameter sweeps beautifully, because each trial is independent and losing one costs you one trial. It suits a single long distributed run poorly, because reclamation of any one node halts everything. Checkpoint every few minutes and spot becomes viable even for long runs, and that engineering effort is normally the cheapest discount available to you.

Scaling out, and where returns stop

Adding accelerators to a job does not divide its duration proportionally. Data parallel training synchronises gradients across devices after each step, and that exchange grows with device count. Inside a single instance the interconnect is fast, and P5 instances give 900 GB/s of NVSwitch bandwidth between the eight GPUs, so intra node scaling stays efficient. Crossing to a second instance drops you onto the network, where even 3,200 Gbps of Elastic Fabric Adapter bandwidth is an order of magnitude slower than the internal fabric. That discontinuity is the single most important shape in distributed training economics.

Two practical rules follow. Fill one instance before reaching for a second, because the interconnect inside the box is free relative to the network between boxes. And measure scaling efficiency on your actual model before committing to a cluster size, because efficiency depends on the ratio of computation to parameters, and a model with many small layers behaves quite differently from one with few large ones. I have seen efficiency at eight GPUs range from 0.51 to 0.94 on models of similar parameter count, and no rule of thumb would have predicted either.

Inference economics on the churn platform

Training is a burst. Inference is a tenancy. A model that trains for forty hours once a month and then serves predictions continuously will spend the large majority of its lifetime cost on serving, and yet almost every sizing discussion I sit in spends its energy on the training side. Part 22 covered the serving patterns; here is what they cost.

Sit with the gap between rows one and three for a moment. Nearly four hundred times the cost, for a freshness improvement that a retention team contacting customers on a weekly cycle cannot use. That is the most expensive unexamined assumption in applied machine learning, and it is usually made in the first design meeting by someone saying that predictions should obviously be real time. Ask what happens to the prediction after it is made. If a human reads it tomorrow, batch.

Where an accelerator is genuinely needed at serving time, which for us means the text classification work on support tickets rather than churn itself, throughput per dollar is the metric and batching is the lever. Grouping requests raises latency slightly and raises throughput a great deal, because the fixed overhead of a forward pass is amortised across the batch. An L4 at 24 gigabytes and 72 watts serves quantised models at a fraction of the cost of a training class part, and quantisation to eight bit integers commonly halves memory again with accuracy loss small enough to be lost in the noise of a business metric. One caution worth stating: measure that accuracy loss on your own data rather than trusting a published benchmark, because the tolerance depends entirely on how the prediction is used. Note also that the heavy preprocessing an analyst does, of the kind covered in cleaning messy data for analysts, belongs on cheap CPU capacity upstream and never inside a GPU serving path.

Compute plan I would fund for the churn platform

Concretely, for the platform designed in Part 27 running the churn workload and a small amount of text work alongside it: no accelerator for the churn model at all, since gradient boosting on seven thousand rows trains in under a minute on four CPU cores. One L4 for the text and embedding work, sized by memory rather than by throughput. Batch scoring by default, with a real time path only for the cancellation page where a session bound action genuinely consumes the score. Capacity blocks on previous generation hardware for any occasional larger training, booked to a window we have enough experiments to fill. Spot for hyperparameter sweeps, because losing a trial costs a trial.

What I would avoid, stated plainly: an always on GPU endpoint for a tabular model, an eight GPU instance held for exploratory work, current generation silicon bought for a job whose bottleneck has not been profiled, and any reservation window longer than the queue of experiments waiting to fill it. Each of those looks like capability in a budget request and reads as waste in a quarterly review.

Underneath all of it sits one governance habit that costs nothing. Publish utilisation alongside spend, on the same dashboard, refreshed at the same cadence. Spend without utilisation produces pressure to buy cheaper hardware. Utilisation without spend produces pressure to buy more hardware. Together they produce the only question worth asking, which is whether the money already committed is doing work. Part 29 takes the same platform into responsible AI, model risk and governance that survives an audit, where the constraint stops being money and starts being defensibility. Before you read it, go and look up the mean GPU utilisation on whatever your organisation is currently paying for. If you cannot find the number in ten minutes, that is the finding.

References

• Amazon EC2 Capacity Blocks for ML pricing, source of every reserved per accelerator rate and the worked reservation example quoted above
• Amazon EC2 P5 instance specifications, including 640 GB HBM3 per p5.48xlarge, 900 GB/s NVSwitch and 3,200 Gbps EFA networking
• PyTorch torch.amp documentation, on autocasting to float16 and bfloat16 and the memory footprint it saves
• NVIDIA L4 Tensor Core GPU product page, 24 GB memory, 300 GB/s bandwidth and a 72 watt thermal design power
• Cleaning messy data for analysts, Data Analyst Series, assumed background for the preprocessing that belongs on CPU
• IBM Telco Customer Churn dataset, the running project across this series

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts