The most expensive AI infrastructure I have reviewed was not the one with the highest hourly rate. It was a set of eight H100 GPUs, reserved for a year, running a research team’s experiments about four hours a working day. On paper the team had negotiated a good rate. In practice they were paying roughly forty US dollars for every hour of GPU work they actually got, because the meter does not stop when nobody submits a job. Nobody in that room had done anything careless. They had simply carried over an instinct from ordinary compute, where an idle server is a small waste, into a place where an idle accelerator is the single largest line on the invoice.

Two meters, one bill

Definitions first, because AI cost conversations collapse when finance and engineering use the same words for different things. Training is the process of building or adjusting a model’s weights from data, and it runs as a job with a start and an end. Inference is running an already built model to answer a request, and it runs continuously in production. A token is the unit models read and write, roughly three quarters of an English word, so a 400 word answer is around 500 tokens. The context window is how much text the model can consider at once, and everything you stuff into it is charged as input. A GPU hour is one graphics processing unit held for one hour, and it is charged whether that GPU is computing or waiting. Utilisation is the fraction of that time the GPU is doing useful work.

Those definitions split into two billing worlds, and the split is the most useful thing in this part. If your organisation runs models on infrastructure it rents, whether that is fine tuning an open weights model or serving one behind your own endpoint, you are on the GPU hour meter. If your organisation calls a hosted model through an API, you are on the token meter. Many organisations are on both at once and report them as a single AI number, which is how the two get confused.

The FinOps Foundation places AI in its own technology category for exactly this reason, noting that allocation is harder because fast moving teams often do not tag their spend, that forecasting needs shorter windows because variance is high, and that rate optimisation is genuinely difficult because bursty early projects argue against commitments while capacity scarcity argues for them. That last tension is real and it does not resolve cleanly. It is worth saying plainly that the framework does not hand you an answer here, it hands you the correct question.

flowchart LR
  W[AI workload] --> T{Do you run the model yourself}
  T -->|Yes| G[Rent GPUs by the hour]
  T -->|No| K[Buy tokens by the million]
  G --> U[Cost set by utilisation]
  K --> P[Cost set by prompt and response length]
  U --> B[One AI line on the bill]
  P --> B
  B --> X[Two problems reported as one number]

Why a GPU hour is not a VM hour

Everything that made ordinary compute forgiving is absent here. A normal virtual machine is cheap enough that idling one for a weekend is an annoyance rather than an incident, it scales out in seconds when you need more, and it scales back to nothing when you do not. Accelerated instances break all three assumptions at once.

Start with the hardware, because the shapes explain the prices. An Amazon EC2 p5.48xlarge carries eight NVIDIA H100 GPUs with 640 GB of HBM3 memory, 192 vCPUs and 2 TiB of system memory, and up to 3,200 Gbps of Elastic Fabric Adapter networking. The p5e and p5en variants move to eight H200 GPUs with 1,128 GB of HBM3e. These are not instances you nudge up and down. You take the whole shape or you take none of it, and the smallest H100 option, the p5.4xlarge with a single GPU, does not support the GPUDirect remote memory access that multi node training needs.

Then there is availability. For most of the last three years the constraint on accelerated capacity has been supply rather than budget, which inverts the usual FinOps posture. In ordinary cloud you buy commitments to lower a rate you could otherwise get on demand. In accelerated cloud you frequently buy commitments, capacity blocks or reservations to guarantee the hardware exists at all when your training run is scheduled. The discount is a side effect. That changes how you should evaluate the decision, because the question stops being whether the discount beats the flexibility and becomes whether a failed training window costs more than the idle hours you will certainly accumulate.

My position on this is firmer than the framework’s. Do not buy a GPU commitment on the strength of a roadmap. Buy it on the strength of a measured utilisation curve from at least one quarter of on demand or spot usage, because a commitment on a workload you have never metered converts an uncertain bill into a certain one at the wrong number. The commitment mechanics from Part 12 all still apply, but the sequencing matters far more here because the amounts are larger and the terms are longer.

Utilisation decides your real GPU rate

Here is the arithmetic that should govern every accelerated capacity decision, and it fits in one line. Your effective cost is the hourly rate divided by your utilisation. Nothing else in GPU FinOps moves the number as far, and almost nobody reports it.

Work it on a concrete shape. Take an eight GPU H100 instance at an illustrative on demand rate of 55.04 US dollars an hour, which is 6.88 US dollars per GPU hour [VERIFY current rate for your region before quoting it]. Over a 730 hour month that instance costs about 40,179 US dollars if you leave it running. What you get for that money depends entirely on a number that does not appear anywhere on the invoice.

Arithmetic on one eight GPU instance over a 730 hour month at an illustrative 6.88 US dollars per GPU hour. Useful hours are 5,840 total GPU hours multiplied by utilisation. The idle column is what you pay for capacity that computed nothing.

Read the chart and the procurement implication is uncomfortable. Teams spend months negotiating five or ten percent off a GPU rate while running at a utilisation that costs them three hundred percent. If you have a research cluster sitting under 40 percent, no discount available anywhere in the market will catch up with simply queueing the work properly. Shared scheduling, a job queue that keeps the hardware fed overnight, and consolidating three half used clusters into one busy one are the interventions that matter, and they are engineering work rather than finance work. This is the same shared capacity argument made for containers in Part 15, with an order of magnitude more money attached to it.

Token pricing has four rates, not one

Switch meters. On the token side the published price per million tokens looks like a single fixed rate, and it is not. Amazon Bedrock, to take a documented example, exposes several service tiers for the same model. Standard is the default rate. Batch inference, where you submit requests as a file and collect results asynchronously, is priced at a 50 percent discount to standard for supported models. The Flex tier is also a 50 percent discount, and the Priority tier is a 75 percent premium. Separately, prompt caching stores the repeated part of your input so that reading it back costs a fraction of the normal input rate.

Those mechanisms are not exclusive, which is where the money is. Take the published Bedrock rates for Anthropic Claude 3.5 Sonnet v2 as they stand in July 2026: 6.00 US dollars per million input tokens, 30.00 per million output tokens, 3.00 and 15.00 respectively under batch, 7.50 per million to write to the cache and 0.60 per million to read from it. That cache read rate is one tenth of the standard input rate, and it applies to the part of your prompt that never changes.

Most production retrieval applications send a long fixed system prompt and a large block of retrieved context on every single request, then a short user question. That shape is close to the worst case at standard rates and close to the best case with caching, and the two differ by a factor of ten on the portion of the bill that is usually largest.

Arithmetic on published Amazon Bedrock rates for Claude 3.5 Sonnet v2 in July 2026, at 40 billion input and 5 billion output tokens a month with 3,500 of every 4,000 input tokens cacheable. Cache write cost is excluded because a stable system prompt is written once and read millions of times. Batch requires the workload to tolerate asynchronous results.

The output token is the one you cannot discount away

Look again at the bottom row of that table, because it contains the least obvious lesson in this part. After caching and batching have done everything they can, input has fallen from 240,000 US dollars to 25,500, a reduction of nearly 90 percent. Output has only halved, and it now makes up roughly three quarters of the remaining bill. Every optimisation people talk about at conferences attacks the input side, and the input side is the part that optimises itself.

The reason is structural. Input tokens are known before the call, so they can be cached, deduplicated, compressed and reused. Output tokens are generated one at a time by the model, cannot be cached, and cost five times as much per million on the rates above. There is exactly one lever on output cost and it is length. How many tokens does the model produce, and do you need all of them.

In practice this is the most reliably ignored cost control in AI applications. Teams leave the maximum output length at whatever the SDK default is, write prompts that invite the model to explain its reasoning at length, and then render only the first two sentences in the user interface. I have seen a support assistant cut its output cost by more than a third by adding one instruction telling the model to answer in three sentences unless asked to expand, with no measurable drop in resolution rate. Before you evaluate a cheaper model, cap the response length and measure again, because a shorter answer from a strong model is frequently cheaper and better than a long answer from a weak one.

Who owns the cost when everyone shares one endpoint?

Allocation is where AI spend most often defeats an otherwise healthy FinOps practice, and the reason is the same one that broke Kubernetes allocation in Part 15. The billing boundary and the ownership boundary do not line up. One model endpoint serves six product teams, the provider bills you for tokens against an account, and the tag strategy from Part 5 has nothing to attach itself to because there is no per team resource to tag.

The FinOps Foundation is candid that this is unsolved at the framework level, noting the absence of generally accepted allocation approaches for multi agent workloads and the inconsistent billing detail vendors provide. That candour is useful, because it means you should stop waiting for the provider to fix it and instrument the application layer yourself.

The practical answer is to record token usage at the point of the call, with the calling team, feature and customer attached, and to reconcile that log against the provider invoice monthly. Every major API returns input and output token counts in its response. Logging those three fields alongside your own identifiers is a small piece of work that gives you allocation, unit economics and anomaly detection at once. Teams that skip it end up dividing the AI bill by headcount, which satisfies nobody and changes no behaviour.

flowchart TD
  A{Does the workload need an answer within seconds} -->|No| BA[Move it to batch, half price]
  A -->|Yes| C{Does every request repeat a long fixed context}
  C -->|Yes| CA[Turn on prompt caching first]
  C -->|No| D{Are you serving on your own GPUs}
  D -->|No| E{Is maximum output length capped}
  E -->|No| F[Cap the response before changing model]
  E -->|Yes| H[Route easy traffic to a smaller model]
  D -->|Yes| I{Is GPU utilisation above 60 percent}
  I -->|No| J[Consolidate and queue before committing]
  I -->|Yes| L[Now price reservations and capacity blocks]
  BA --> E
  CA --> E

Measure cost per outcome before you buy capacity

The recommendation for this part is a sequence, and the order is the whole point. Instrument first: log input tokens, output tokens, GPU utilisation and a business outcome count for every AI workload you run, and get one month of history. Optimise second, using the levers that cost nothing: cap output length, turn on prompt caching where context repeats, move anything asynchronous onto batch pricing, and consolidate half used GPU clusters onto shared queues. Commit last, and only against a utilisation curve you have actually measured.

Teams reliably run that sequence backwards. They commit to hardware or a provider tier during a strategy cycle, then discover their utilisation in a cost review six months later, then start optimising the thing they have already paid for. The financial damage of that ordering is larger in AI than anywhere else in this series, because the commitments are longer, the hardware is scarcer and the per unit numbers are bigger. If you take one thing from this part, take the sequence rather than any individual figure, because the figures will move and the sequence will not.

Part 18 turns all of this into something that runs without you, covering guardrails, policy and infrastructure as code, which is where the manual habits described here become automated controls. The unit economics framing came from Part 10, the anomaly detection you will need for token spikes was covered in Part 9, and every part is indexed on the Cloud FinOps guide.

One thing to do this week: open your highest volume AI application, find the maximum output length setting, and check whether anyone chose it deliberately. In most codebases nobody did, and that single default is quietly setting three quarters of your inference bill.

References

• FinOps for AI, FinOps Framework technology category, allocation and unit economics guidance
• GenAI FinOps, how token pricing really works, FinOps Foundation paper
• Amazon Bedrock pricing, service tiers, batch rates and prompt caching rates
• Amazon EC2 P5, P5e and P5en instances, GPU counts, memory and network specifications
• FinOps for AI overview paper, FinOps Foundation

Bedrock token rates and service tier discounts were read from the Amazon Bedrock pricing page in July 2026 and vary by model and region. The 6.88 US dollars per GPU hour figure used in the utilisation arithmetic is illustrative and should be re-checked against current on demand rates for your region before it is used in a business case [VERIFY]. All figures in this part are arithmetic worked from published list rates rather than from a live account.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts