What you are looking at: running an open model yourself means paying for GPUs whether or not they are busy. An API charges only for the tokens you use. Which is cheaper depends mostly on how many tokens you need, how busy you can keep the GPUs and the other monthly costs of running them. Pick a preset or set your own numbers.
Try this: pick "Internal chatbot" and note the verdict. Then drag Other monthly costs to $0: the break-even falls a lot, which shows how much of self-hosting is people, not GPUs. Then try "High-volume classification" and change the API price from $0.40 to $6.
Throughput per GPU group is a rough sustained figure for mixed chat traffic with a serving engine such as vLLM, about 40% of the memory-bandwidth limits from part 4; real numbers vary widely with prompt and answer length. GPUs are rented around the clock. Other monthly costs cover staff time, spare capacity, monitoring and storage. API prices vary by provider and model; set the blended price (input and output mixed) you are comparing. All figures are illustrative.
What you are looking at: GPU memory for one model on one hardware choice. The weights take parameters × bytes per parameter: 2 bytes in 16-bit, 1 in FP8, about 0.53 in 4-bit. Every user then needs KV cache for their context. The grid shows the same model in all three formats on every hardware option: green fits with your users, amber fits the weights but not that many users, grey does not fit.
Try this: pick Llama 3.1 70B on one H100 in FP8: the weights fit but almost no users do. Switch to 4-bit and watch the user count jump. Then pick gpt-oss-120b in 4-bit (close to how it ships) and compare its KV cache per user with the 70B model.
Usable memory is 90% of the total (vLLM's default --gpu-memory-utilization) minus about 1 GB per GPU for the runtime. KV cache per token in 16-bit: Llama 3.1 8B 128 KB, 70B 320 KB, 405B about 504 KB, DeepSeek-V3 (MLA) about 69 KB; gpt-oss counts only its full-attention layers (half its layers use a 128-token sliding window), about 24 KB (20b) and 36 KB (120b). 4-bit assumes about 4.25 bits per value; gpt-oss ships its MoE weights in MXFP4 with other layers in 16-bit, so its real size is a little above the 4-bit figure. Multi-node splits the model over the network and needs a fast fabric.
What you are looking at: one week of traffic for the model and GPU price chosen on tab 1. You size the GPUs for the busiest hour plus headroom, then pay for that capacity every hour. Utilisation is the tokens you serve divided by the tokens you could serve. Everything hatched is paid for and unused.
Try this: start with Office hours and see utilisation land near 30%. Switch to Office + overnight batch: the same GPUs now fill the night with batch work and the cost per million falls. Then go back to Office hours and turn on night scale-down, which only works if your GPUs are rented by the hour.
Hourly shapes are illustrative. Weekly utilisation = (5 weekdays + 2 weekend days at the weekend share) ÷ 7, times the planned peak share. Night scale-down halves paid capacity in any hour below 40% of peak demand. Cost uses the model, throughput and GPU price from tab 1, GPUs only.
What you are looking at: a router sends a share of requests to an open model on your GPUs and the rest to a frontier API. The open model runs on whole GPU groups, so its cost rises in steps; the API part shrinks smoothly. The chart shows total monthly cost for every routing share from 0 to 100%.
Try this: at 2B tokens a month, move the routed share from 0% to 80% and watch where the total crosses below all-API. Then lower the open model's utilisation to 15%: the saving shrinks or disappears, because idle GPUs now carry the open share.
GPU price comes from tab 1. Self-hosted cost counts GPUs only; add people and spares from tab 1 before deciding. The router itself and the quality checks it needs are not costed. Prices illustrative.