A startup I advise burned eighty percent of a quarterly API budget in the first nineteen days, then spent the rest of the quarter rationing calls and apologising to customers. Almost none of it was the model being expensive. It was one bloated system prompt shipped on every request, no caching, a frontier model answering questions a small one would have nailed, and a retry loop that quietly doubled the bill on every timeout. Each of those is fixable in an afternoon. This guide walks the whole cost surface, from the token math a beginner needs through the self hosting economics an expert argues about, and gives you an order to attack it in so you cut the most money for the least effort first.

Where the money actually goes

Before you cut anything, understand the meter. Almost every hosted LLM bills the same way: cost equals input tokens times the input price plus output tokens times the output price, quoted per million tokens. Two facts about that formula drive every decision below. First, output usually costs several times more than input, often three to five times, so a chatty model that pads answers is more expensive than its input footprint suggests. Second, the API is stateless, which means you resend the full context on every single call. That forty thousand token document you attach is not paid once, it is paid on every turn of the conversation until you remove it.

Two more multipliers hide in production. Retries on timeouts or rate limits can double or triple real spend while looking like normal traffic in your code. And model tier swings the unit price by more than an order of magnitude, so the single biggest lever for many workloads is simply not sending every request to the most expensive model. The chart below shows the spread with round illustrative numbers; your provider will differ, so read it for the shape, not the exact figures.

Measuring and attributing spend

You cannot cut what you cannot see, and the default provider dashboard hides the one number you need most, which is cost per feature. Log input and output token counts on every call, and tag each one with the feature or endpoint that made it, the model used, and whether it hit the cache. Roll that into a plain cost per request and cost per feature view, refreshed daily. Most teams find that a handful of endpoints drive the majority of spend, and that a single background job nobody thought about is the quiet leader. Attribution turns a frightening aggregate into a short, ranked list of things to fix, which is exactly what you want in hand before you touch a single line of code.

Then set budgets and alerts so a runaway loop cannot bill all weekend before anyone notices. A hard monthly cap per environment, a spend rate alert that fires when hourly cost jumps past a threshold, and a per request token ceiling together catch the failure modes that otherwise arrive as a shocking invoice at month end. Treat token usage as a first class metric alongside latency and error rate, because in an LLM system it behaves like one, and the teams that watch it in real time are the ones that never get the nasty surprise.

Prompt and context hygiene, the cheapest win

Start here because it costs you nothing but attention and it touches every call you make. Most prompts carry dead weight: a system message that grew by accretion, half a dozen few shot examples where two would do, an entire chat history resent verbatim when a short summary would carry the same meaning. Trim the system prompt to what changes behaviour. Prune few shot examples to the smallest set that holds quality on your eval. Replace a growing transcript with a rolling summary once it passes a few turns. None of this is glamorous and all of it compounds on every request for the life of the feature.

Output is the other half. Set a real max token limit so a model cannot wander into a thousand token essay when you needed a sentence. Ask for structured output, JSON or a fixed schema, which is both cheaper and easier to parse. One team documented cutting average response length from four hundred tokens to two hundred eighty just by specifying a structured format, part of a seventy one percent monthly cost reduction with no visible drop in quality. Use stop sequences to end generation the moment the useful part is done. These are one line changes with outsized returns.

A few more habits pay off quietly. Compress long instructions into crisp directives, since the model does not reward verbosity in the prompt the way a human reader might. Trim tool and function schemas to the fields you actually use, because those definitions ride along on every call that offers the tool. Avoid repeating the same instruction in both the system and the user message, a common duplication that pays twice for the words that matter least. And prefer references over inlined examples where the model already knows the pattern, so you are not teaching it something it learned in pretraining.

Caching and batching, reuse what you already paid for

If any part of your prompt repeats across calls, and for most applications the system prompt, tool definitions, and reference documents repeat on every single request, you are paying to process the same tokens again and again. Prompt caching fixes exactly that. The provider stores the processed prefix, and subsequent calls that share it read at a steep discount, commonly around ten percent of the normal input rate. Writing to the cache costs a small premium, roughly one and a quarter times input, and the entry lives for a few minutes by default with a longer option available. The rule that makes or breaks it: put stable content first and volatile content last, so the cacheable prefix stays identical call to call.

Two neighbours of caching finish the job. Semantic caching stores answers keyed by meaning, using an embedding match, so a question you have effectively answered before is served from your own store without an LLM call at all, ideal for FAQ style traffic. And batching takes work that does not need an instant answer, evaluation runs, nightly backfills, embeddings, report generation, and submits it asynchronously for a flat fifty percent discount, with completion guaranteed within a day and usually much faster. Caching and batching are independent discounts that stack, which is how effective input spend on repetitive pipelines drops by ninety percent or more.

Model selection and routing, right size every call

Once your prompts are lean and cached, the next big lever is refusing to pay frontier prices for work a cheaper model does correctly. Default to the smallest model that clears your evaluation on a task, and escalate deliberately rather than reaching for the biggest model out of habit. There are three ways to decide per request. A cascade tries the cheap model first and escalates only when confidence is low, which is excellent for asynchronous and throughput work but adds the cheap model latency to every escalated request, so it can be the wrong choice on an interactive path. A classifier router predicts the right model before any call, so it pays no escalation penalty and suits interactive traffic. And distillation fine tunes a small model on a larger model output for one narrow task, giving near frontier quality at small model prices where the task is stable.

The savings are real and well documented. Teams routing in production commonly report forty to seventy percent lower cost at equal quality, and the FrugalGPT research showed a cascade reaching up to ninety eight percent reduction against a frontier only baseline on several benchmarks. The catch is that routing on quality requires measuring quality first, so you need an eval before you can trust a router. Build that eval, then let it pick the cheapest model that passes.

flowchart TD
  U[Incoming request] --> C{Rules or classifier}
  C -->|Simple, factual| S[Small model]
  C -->|Standard task| M[Mid model]
  C -->|Hard or high stakes| F[Frontier model]
  S --> V{Confidence ok}
  V -->|No| M
  V -->|Yes| O[Return answer]
  M --> O
  F --> O

Retrieval, infrastructure and self hosting

The advanced levers change the shape of the workload itself. Retrieval augmented generation is a cost technique as much as a quality one. Instead of pasting an entire knowledge base into the prompt, you index it, retrieve only the handful of chunks relevant to the question, and pass those. One documented setup dropped average input from a stuffed window to about eight thousand tokens by retrieving the ten most relevant chunks, roughly a six times reduction in input cost, with better answers because the model was not distracted by irrelevant text. Deduplicate near identical chunks, and where chunks are long, summarize them with a cheap model before handing them to the expensive one.

Self hosting is the lever everyone reaches for too early. Running an open weight model on your own hardware removes the per token markup, but adds fixed hardware, engineering, and operations cost that people consistently underestimate. The break even depends entirely on volume. Below roughly twenty million tokens a month, managed APIs almost always win once you count the ops time. Between fifty and one hundred million tokens a month is where a seventy billion parameter class model starts to pay back. Above one hundred million tokens a month, self hosting usually wins on unit economics. Quantization tilts the math further: moving to an eight bit format on a modern GPU can roughly double throughput at the same hourly cost, bringing cost per million tokens from around a dollar ninety to near a dollar. Serve with an engine built for it, and benchmark before you commit.

If you cross into self hosting territory, the engine and the compression method decide your real cost per token. I have written the hands on detail elsewhere: how to shrink a model without wrecking quality in model compression and quantization, how to prove your throughput numbers in benchmarking a self hosted inference deployment, and how to scale it in distributed inference on Kubernetes. The wider engineering picture lives in the AI Engineering complete guide.

A pragmatic middle path beats a religious one. Many teams run a hybrid: self host the steady baseline load on their own quantized model where the volume justifies it, and burst to a managed API for spikes and for the hardest requests a small model cannot handle. You get the unit economics of owned hardware on predictable traffic without paying to provision for your worst hour. Route by both difficulty and current load, and let the expensive path absorb only what the cheap path cannot carry.

Agents and multi step workflows, the cost multiplier

An agent that plans, calls tools, reads the results, and tries again can turn one user request into twenty model calls, and each call resends the growing history. This is where bills surprise people, because cost scales with the number of steps, not the number of users. A single stuck loop can spend more in an hour than a thousand ordinary chats do in a day. Cap the number of steps a loop may take and fail gracefully when it hits the ceiling rather than spinning forever. Cache tool results so the same lookup inside one task is not paid for twice. Run independent tool calls in parallel instead of chaining them, since a chain resends the whole context on every hop and the context only grows.

Right sizing matters even more inside an agent than outside it. Use a small model for the routine steps, planning, routing, and formatting, and reserve the frontier model for the one or two steps that genuinely need its judgement. Summarize the working memory between steps instead of carrying the full transcript, so context does not grow without bound across a long task. An agent with a step cap, cached tools, parallel calls, and a small model on the routine work can cost a fraction of the same agent built naively, at the same success rate. If you run agents in production, this single section is often the largest line on the whole bill, so it deserves the most careful engineering.

A cost model you can run

Put a number on it before you optimise, because you cannot manage what you do not measure. Take a concrete workload: two million requests a month, six thousand input tokens and five hundred output tokens each, all hitting a frontier model. That is a baseline you can compute in a spreadsheet from your provider rates. Now apply the levers in order and watch the same workload shrink. Prompt and context hygiene trims input and caps output. Caching collapses the repeated prefix. Routing sends the easy majority to a small model. Batching halves the async share. The waterfall below shows the cumulative effect with illustrative percentages; the point is the shape, and the fact that the cheap first two levers already do most of the work.

Try it on your own numbers

The calculator below runs the same model this section describes. Enter your usage, choose a model tier, then pull the caching and batching sliders and watch the monthly bill move. It also flags when your volume crosses into self hosting territory.

Common cost mistakes and their fixes

Your one week cost cut plan

References

• Anthropic, prompt caching documentation, cached read and write pricing
• OpenAI, Batch API guide, fifty percent asynchronous discount
• Chen, Zaharia, Zou, FrugalGPT, cascades for lower LLM cost
• AI Engineering From Zero to Production, the Complete Guide

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts