Business inputs
People using it on a busy day
Requests per person per day
Busy hours per day
Peak hour vs average hour
Answer length
Speed each user needs
Wait for first token
What you are looking at: a business gives you daily users, but a GPU only cares how many requests are being generated at the same moment. Little's law links them: users at once = requests per second at peak × seconds each request takes. The timeline below draws simulated requests at that rate; count how many bars overlap at any moment.
Try this: raise answer length or lower speed and watch the bars get longer and the overlap grow, with the same number of daily users. Then press Use these numbers in the calculator.
What you are looking at: the sizing method. How big are the weights? How much KV cache does each user hold? How fast must each user get words? Is there time left for prefill? How many copies for peak and failures? The calculator tries 1, 2, 4 and 8 GPUs per copy and picks the option with the fewest GPUs.
Try this: load the RAG over documents preset, then untick Count prefill load. The GPU count falls, because 15,000 new prompt tokens per request is real work that a decode-only estimate ignores.
What you are looking at: one copy of the model from the calculator. Every decode step reads all the weights plus every user's KV cache, so each extra user makes every step a little slower. The red line is the speed each user gets; the dark line is the total tokens the copy writes per second. The copy can take users until it hits the memory wall (KV cache no longer fits) or the speed target, whichever comes first.
Try this: switch GPUs per copy below, then go back to the calculator and change the speed target from 20 to 50 tokens/s. Watch the limit move from the memory wall to the speed line.
What you are looking at: the same workload from the calculator sized on four GPUs, each at its own illustrative hourly price. More memory per GPU means fewer GPUs per copy and more users per copy; more bandwidth means more users before the speed target bites; more compute only matters for prefill and very large batches.
Try this: compare the worked example with the RAG preset. H200 has more memory than H100 but the same compute, so it gains less when prefill is the bottleneck.
Estimates are illustrative. Memory: weights at 2, 1 or about 0.55 bytes per parameter; KV cache per token from the models' published shapes (Llama 3.1 8B 128 KB, 70B 320 KB, 405B 504 KB, gpt-oss-120b about 72 KB, DeepSeek-V3 about 69 KB in 16-bit; half in FP8), with every user at the full conversation length; 10% of memory kept free. Speed: each decode step reads all weights and every user's KV cache once, at 60% of the GPUs' combined memory bandwidth (L40S 864 GB/s, H100 3.35 TB/s, H200 4.8 TB/s, B200 8 TB/s); compute is checked at 50% of dense FP8 or BF16 peak. Prefill: 2 × parameters × new prompt tokens at that same 50% of compute, sharing each copy's time with decode (attention cost of very long prompts not included). For MoE models, all weights are counted as read each step, which overstates the cost at small batches. Tensor-parallel communication (slower over PCIe on L40S), multi-node copies and real prices are not modelled; benchmark your own setup before buying.