Two teams run the same Gemini 2.5 Flash assistant, same traffic, same prompt. One pays about 320 dollars a day. The other pays under 60. Nothing about the model changed. What changed is how they bought the throughput and whether they cached the part of the prompt that never moves. Pricing on Vertex AI is a design decision, and most of the savings sit in three levers that are easy to miss.
Key numbers
Vertex charges input and output tokens separately, and output is where the bill lives. On Gemini 2.5 Flash the list rates are 0.30 dollars per million input tokens and 2.50 per million output. Provisioned Throughput sells reserved capacity in Generative AI Scale Units (GSUs) on a fixed term. Context caching cuts repeated input to 10 percent of the standard rate. Batch runs async work at half price. Get these four right and a five figure monthly bill can drop by more than half.
I have signed off on Vertex spend for production assistants and batch pipelines, and the same mistake shows up every time: teams pick a model, read the per token rate, multiply by expected volume, and stop. That number is the ceiling, not the price you should pay. The rate is the starting point. Everything after it is about buying the same tokens more cheaply.
What you pay for, before any discount
On-demand (also called pay-as-you-go) is the default. You send a request, Vertex counts the tokens in and out, and you pay the model’s published rate with no commitment. A token is roughly four characters of English, so a page of text is about 500 tokens. Input covers everything you send, including the system instruction, the retrieved context, and the chat history. Output covers what the model generates, and for reasoning models the internal thinking tokens count as output too. That last point catches people. A short answer from a thinking model can still bill hundreds of output tokens.
Output almost always costs several times more than input. On Gemini 2.5 Pro the gap is eight to one. So the cheapest optimization is rarely a smaller model. It is a shorter answer. If you have not read Part 3 on choosing between Flash and Pro, the short version is that Flash is roughly a quarter the input cost and a quarter the output cost of Pro, and for most retrieval and extraction work it is the right default.
Here are the on-demand list rates for the three models I reach for most. Prices are per million tokens and reflect the standard text tier. The higher context tier on Pro applies once a single prompt crosses 200,000 tokens, and it roughly doubles the rate, which is another reason to keep prompts tight.
| Model | Input per 1M | Output per 1M | Where it fits |
|---|---|---|---|
| Gemini 2.5 Flash-Lite | 0.10 dollars | 0.40 dollars | High volume classification, routing |
| Gemini 2.5 Flash | 0.30 dollars | 2.50 dollars | Default for RAG and assistants |
| Gemini 2.5 Pro | 1.25 dollars | 10.00 dollars | Hard reasoning, long context |
Standard text tier, per million tokens. Confirm live figures on the Vertex AI pricing page before you commit a budget.
Provisioned Throughput and the GSU
On-demand is elastic but it competes with every other Google customer for shared capacity, so under load you can hit quota limits and 429 errors. Provisioned Throughput (PT) fixes that. It is a reservation. You buy a fixed amount of capacity for a fixed term, and within that reservation your requests get priority and predictable latency, billed at a flat price whether you use the capacity or not.
Capacity is sold in Generative AI Scale Units. A GSU is a unit of throughput. The dollar price per GSU is fixed, but how much throughput one GSU buys depends on the model, because a heavy model burns capacity faster than a light one. Vertex converts your input and output through a burndown rate into a common measure of input tokens per second, so a GSU of Flash and a GSU of Pro cost the same but deliver very different token rates. Commitment terms run one week, one month, three months, or one year, and the longer terms discount the hourly price.
Google publishes a concrete example worth memorizing. Buy 25 GSUs of gemini-2.5-flash and you get an average of about 67,250 input tokens per second of sustained throughput. Enforcement uses a dynamic window rather than a hard per second cap, so a spiky workload can burst as long as the running average holds. That works out to roughly 2,690 tokens per second per GSU for Flash. The line below plots how that scales.
How many GSUs do you actually need?
Do not guess. The Google Cloud console has an estimation tool where you enter your expected queries per second and the typical input and output sizes, and it returns a GSU recommendation for the model you picked. Feed it real numbers from your on-demand logs, not a launch day fantasy. The honest way to size PT is to run on-demand first for a week or two, read the actual token throughput at your busy hour from the metrics, and reserve for the baseline that sits under your daily peak.
My rule of thumb from the field: PT pays off when a single model runs a steady, predictable load most hours of the day. If your traffic is bursty, occasional, or spread thin across many models, on-demand plus caching almost always wins, because you are not paying for idle reserved capacity. A one week commitment is the cheapest way to test whether your load is really as steady as you think before you sign a longer term.
Context caching, implicit and explicit
Most production prompts carry a large fixed block: a system instruction, a policy document, a schema, a few retrieved chapters. You send that same block on every request and pay full input price for it every time. Context caching stops that. The model stores the fixed block once and bills the cached portion at 10 percent of the normal input rate on repeat calls, a 90 percent discount on Gemini 2.5 and later models.
There are two flavors. Implicit caching is on by default with no code and no storage fee. When Vertex notices a repeated prefix it passes the discount through automatically on a cache hit, and you see the saving in the response metadata. You do not control it, and a hit is not guaranteed. Explicit caching is the one you manage. You declare the content to cache, get back a cache handle, and reference it from later requests. You control the time to live, you are guaranteed the discount on referenced tokens, and you pay a storage fee for how long the cache lives. Caching kicks in from a minimum of 2,048 tokens up to the model context window.
The response returns a cachedContentTokenCount field so you can see exactly how many tokens were served from cache and confirm the discount landed. My take: rely on implicit caching for chat where the same session repeats a system prompt, and reach for explicit caching when a large document is reused by many users across sessions, because that is where the guaranteed hit and the controlled TTL earn their storage fee.
When does batch make sense?
Batch prediction runs a large set of requests offline and returns results within a service window rather than in real time. In exchange for giving up interactivity you pay half the on-demand token rate. Nightly summarization, bulk classification, backfilling embeddings, evaluation runs over a test set: all of these are batch work masquerading as real-time work because someone wired them to the synchronous endpoint. Move them to batch and the token bill halves with zero quality loss.
Batch and caching stack cleanly. Batch and Provisioned Throughput usually do not, because PT is about guaranteeing low latency you have deliberately chosen not to need in batch. Here is how the three modes line up.
| Mode | Commitment | Price vs on-demand | Best for |
|---|---|---|---|
| On-demand | None | Baseline | Spiky or low volume traffic |
| Provisioned Throughput | 1 week to 1 year | Flat GSU rate, lower per token at steady load | Predictable high volume, latency guarantees |
| Batch | None | About 50 percent off | Async work with no latency need |
A worked cost comparison
Take a support assistant on Gemini 2.5 Flash. It serves 10,000 requests a day. Every request carries the same 100,000 token knowledge and policy prompt, plus about 500 tokens of fresh user text, and returns about 800 tokens. Watch what the fixed prompt does to the bill.
Worked example
On-demand, no cache. Input is 10,000 times 100,500 tokens, which is 1,005 million tokens a day. At 0.30 dollars per million that is 301.50 dollars. Output is 10,000 times 800, which is 8 million tokens, at 2.50 dollars per million equals 20.00 dollars. Daily total is 321.50 dollars, about 9,645 dollars a month.
Explicit cache on the 100,000 token block. The cached input bills at 10 percent, so 1,000 million cached tokens cost 30.00 dollars instead of 300. The 500 fresh tokens per call add 5 million tokens at 1.50 dollars. Output is still 20.00 dollars. Daily total is 51.50 dollars plus a small cache storage fee, roughly 1,545 dollars a month.
The swing is 270 dollars a day, about 84 percent, before storage. The model did not change. The prompt did not change. Only how the fixed block is billed changed.
Batch sits between the two: run the same volume async and the 321.50 dollars becomes about 160.75 dollars, because both input and output halve. The three modes side by side, in dollars per day, look like this.
Do not trust my arithmetic, run it. This short script has no dependencies and prints the two daily totals so you can drop in your own token counts.
IN_RATE = 0.30 / 1_000_000 # USD per input token, Gemini 2.5 Flash
OUT_RATE = 2.50 / 1_000_000 # USD per output token
CACHE_DISCOUNT = 0.90 # 90 percent off cached input on 2.5 and later
reqs = 10_000 # requests per day
shared = 100_000 # cached system plus knowledge tokens
fresh = 500 # fresh user tokens per request
out = 800 # output tokens per request
def daily(cached):
if cached:
billed_in = reqs * shared * IN_RATE * (1 - CACHE_DISCOUNT)
billed_in += reqs * fresh * IN_RATE
else:
billed_in = reqs * (shared + fresh) * IN_RATE
return billed_in + reqs * out * OUT_RATE
od = daily(False)
wc = daily(True)
print(f'on demand : {od:8.2f} USD/day')
print(f'with cache: {wc:8.2f} USD/day')
print(f'saved : {od - wc:8.2f} USD/day')Expected output: on demand : 321.50 USD/day, with cache : 51.50 USD/day, saved : 270.00 USD/day.
Failure mode: this assumes a 100 percent cache hit and ignores explicit cache storage. If reuse is low or the TTL expires between calls, you pay full input again plus storage, and the cache can cost more than it saves. Measure cachedContentTokenCount in production before you trust the model.
Where I would start on Vertex pricing
Start on on-demand and instrument it. You cannot size anything until you know your real token throughput at the busy hour, so run a week or two and read the numbers. Then pull the cheap levers first: shorten outputs, drop to Flash where Pro is overkill, and cache any fixed block over a couple of thousand tokens that many requests share. Those cost nothing to try and, as the worked example shows, they can carry most of the savings on their own.
Reach for Provisioned Throughput last, and only for a single model carrying a steady, high baseline where you also need latency you can promise. Buy the shortest term that proves the load is real, size it to the baseline under your peak, and let spikes spill to on-demand. Move anything async to batch and take the half price. That order, cheap levers first and reservations last, is how I keep a Vertex bill honest.
Pull one metric this week: for your top model, what fraction of input tokens are the same fixed block on every call? If it is high, caching is your first move, and Part 7 on Cloud TPUs versus GPUs is where the conversation shifts from buying tokens to owning the silicon under them.
Compare the same three levers on another cloud: the AWS build covers Amazon Bedrock pricing across on-demand, provisioned, and batch, where the reserved unit is Provisioned Throughput measured differently.
References
- Vertex AI generative AI pricing
- Calculate Provisioned Throughput requirements
- Provisioned Throughput on Vertex AI, Google Cloud blog
- Context caching overview
About The Author
DrJha