Our finance lead asked me a question I could not answer: what does one support question cost us? I had a monthly API invoice, a token dashboard and no number. Splitting that invoice by feature took an afternoon and produced 9.14 dollars per thousand questions, which sounds small until you multiply by the 190,000 questions a month the assistant was on track to serve.

That number, not the invoice, is what makes cost control tractable. An invoice tells you spending went up. Cost per unit of work tells you whether a change helped, and it survives traffic growth. Everything in this part hangs off it.

Cost of one answered question, end to end

Last part we cached the 5,120 token instruction prefix on a one hour TTL and moved nightly re indexing to the Batch API, which dropped p95 from 6,605 ms to 2,215 ms. This part we point the same discipline at the bill: attribute every dollar to a request shape, then decide which model each shape deserves.

A single assistant question is four billable pieces. Cached prefix reads at a tenth of base input price, occasional cache writes when the entry expires, fresh input for the retrieved chunks and the user question, and output tokens. Write that down as code once and stop estimating.

# cost_model.py, Python 3.12.3, anthropic 0.117.0 [VERIFY]
# Prices in USD per million tokens, from platform.claude.com pricing,
# read 20 July 2026. Sonnet 5 is on introductory pricing to 31 Aug 2026.
PRICES = {
    "claude-haiku-4-5":  {"in": 1.00, "out": 5.00,  "read": 0.10, "w1h": 2.00},
    "claude-sonnet-5":   {"in": 2.00, "out": 10.00, "read": 0.20, "w1h": 4.00},
    "claude-opus-4-8":   {"in": 5.00, "out": 25.00, "read": 0.50, "w1h": 10.00},
}

def price_request(model: str, usage, batch: bool = False) -> float:
    """Dollars for one Messages call, from the usage block the API returns."""
    p = PRICES[model]
    cost = (
        usage.input_tokens              * p["in"]
        + usage.output_tokens           * p["out"]
        + usage.cache_read_input_tokens * p["read"]
        + usage.cache_creation_input_tokens * p["w1h"]
    ) / 1_000_000
    return cost * 0.5 if batch else cost

# Typical docs assistant request, measured over 4,102 traced calls
class U:  # stand in for the SDK usage object
    input_tokens = 1920                 # 1,800 retrieved + 120 question
    output_tokens = 350
    cache_read_input_tokens = 4915      # 96 percent of a 5,120 token prefix
    cache_creation_input_tokens = 205   # the other 4 percent

for m in PRICES:
    print(f"{m:20s} {price_request(m, U()) * 1000:8.2f} per 1,000")

claude-haiku-4-5         4.57 per 1,000
claude-sonnet-5          9.14 per 1,000
claude-opus-4-8         22.86 per 1,000

Look at that collapse, because it governs every decision that follows. Sticker prices differ by five times between the cheapest and dearest tier. On our actual request the ratio is 5.0 to 1 for Haiku against Opus but only 2.0 to 1 for Haiku against Sonnet, and a large slice of every request is the same 1,920 fresh input tokens regardless. Routing can only ever attack the part of the bill that varies with model choice.

Counting a whole trace, not a single call

One question does not always mean one API call, and pricing a call rather than a task is how teams underestimate their bill by a factor of three. Once the assistant gained tool calling in Part 14, a ticket status lookup became a minimum of three model turns: decide to call the tool, read the result, compose an answer. Each turn resends the growing conversation, so turn three carries turns one and two as input.

Measured on our traffic, a plain docs question costs 0.0091 dollars on Sonnet 5 while a tool using ticket question costs 0.0263, roughly 2.9 times more, because two extra turns each re read a prompt that keeps growing. Cached prefix reads soften this considerably, but tool results and prior turns are fresh input every time. Attribute cost to the trace id from Part 25, sum every span inside it, and report per completed task. Any other denominator will flatter you.

Routing strategies and what each one actually costs

Six approaches turn up repeatedly. They differ less in cleverness than in what they cost to operate and how they fail when traffic shifts under them.

Note what the third row does differently. Rules and classifiers both try to predict difficulty before the model runs. A cascade does not predict anything; it tries the cheap model, checks the answer, and escalates when the check fails. Prediction is hard and verification is easy, so build the thing that verifies.

One more reason to prefer verification: it degrades safely. A drifted classifier sends hard questions to a weak model and you find out from a user. A cascade with a drifted threshold sends everything to the expensive model and you find out from a cost alert, which is a much better place to find out.

Cascade routing measured against a single model baseline

We ran five policies over the 420 question eval set built in Part 20, scoring answer accuracy the way Part 13 separates retrieval quality from answer quality. Cost figures come from the model above, applied to real usage blocks rather than estimates.

Read the last two rows together and the uncomfortable result appears. Routing bought 22 percent. Reducing retrieval from eight chunks to five, which took one line of configuration and cost 0.01 accuracy, bought another 15 percent of the original baseline. Two weeks of router work was worth less than a config change we had not bothered to test.

I am deliberately not saying Opus everywhere is wrong. For an internal docs assistant, a wrong answer costs a few minutes of a colleague reading something twice, and 0.04 accuracy is not worth 13.72 dollars per thousand. Point that same assistant at customer facing billing questions and the arithmetic inverts. Cost control means knowing the price of being wrong, not minimising spend.

Building a router that fails toward the expensive model

Router code should be boring and should have exactly one opinion: when anything is uncertain, spend more money. Silent quality loss is expensive in a way that does not show up on an invoice.

# router.py, anthropic 0.117.0 [VERIFY]
import os, logging
import anthropic
from cost_model import price_request

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
log = logging.getLogger("router")

TIERS = ["claude-haiku-4-5", "claude-sonnet-5", "claude-opus-4-8"]
CONFIDENCE_FLOOR = 0.62      # top1 rerank score, tuned in the next section
SYSTEM = open("prompts/assistant_v7.txt").read()   # 5,120 tokens

def answer(question: str, chunks: list, top_score: float) -> dict:
    start_tier = 0 if top_score >= CONFIDENCE_FLOOR else 1
    spend = 0.0
    for tier in range(start_tier, len(TIERS)):
        model = TIERS[tier]
        resp = client.messages.create(
            model=model,
            max_tokens=800,
            system=[{"type": "text", "text": SYSTEM,
                     "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
            messages=[{"role": "user",
                       "content": format_prompt(question, chunks)}],
        )
        spend += price_request(model, resp.usage)
        text = resp.content[0].text
        if is_grounded(text, chunks):        # cheap string level check, no model call
            return {"answer": text, "model": model, "spend": round(spend, 6),
                    "escalations": tier - start_tier}
        log.warning("ungrounded answer from %s, escalating", model)
    # Exhausted every tier. Never fabricate; hand back to a human.
    return {"answer": None, "model": None, "spend": round(spend, 6),
            "escalations": len(TIERS) - start_tier}

{'answer': 'Rotate the key from Settings, API keys...', 'model':
 'claude-haiku-4-5', 'spend': 0.004571, 'escalations': 0}
{'answer': 'Sandbox workspaces inherit the parent org retention...',
 'model': 'claude-sonnet-5', 'spend': 0.013711, 'escalations': 1}

Sunk escalations are the whole economics of a cascade. Escalating costs you 1.5 times a plain Sonnet call, so a cascade only wins while the escalation rate stays low. Ours runs at 12 percent; above roughly 33 percent the cascade is more expensive than never routing at all, and that break even is worth calculating for your own tier spread before you build anything.

A second trap is louder. Our first router set a one hour TTL on the system block and let automatic caching place its own breakpoint with the default five minute TTL on the same request. That combination is rejected outright:

Traceback (most recent call last):
  File "router.py", line 21, in answer
    resp = client.messages.create(
anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error':
 {'type': 'invalid_request_error', 'message': 'cache_control ttl mismatch'}}
   # exact message string [VERIFY]

Fix is to drop the top level cache_control field and mark the system block explicitly, which is what the code above does. Loud failures like this one are the lucky kind. Budget for the quiet ones.

Budget caps and the switch that turns spend off

Routing shapes normal traffic. Caps handle the abnormal kind, and abnormal traffic is where the frightening invoices come from. A retry loop, a scripted integration, or one enthusiastic team can multiply a daily spend by twenty before anyone opens a dashboard.

# budget.py, redis-py 5.2.1 [VERIFY]
import datetime, redis

r = redis.Redis(host="localhost", decode_responses=True)

DAILY_CAP = {"support-bot": 90.00, "internal-docs": 40.00}
DEGRADE_AT = 0.80    # stop routing up once 80 percent of cap is spent

class BudgetExceeded(Exception):
    pass

def record(tenant: str, dollars: float) -> float:
    key = f"spend:{tenant}:{datetime.date.today().isoformat()}"
    total = float(r.incrbyfloat(key, dollars))
    r.expire(key, 172800)
    return total

def check(tenant: str) -> str:
    key = f"spend:{tenant}:{datetime.date.today().isoformat()}"
    spent = float(r.get(key) or 0.0)
    cap = DAILY_CAP[tenant]
    if spent >= cap:
        raise BudgetExceeded(f"{tenant} spent {spent:.2f} of {cap:.2f}")
    if spent >= cap * DEGRADE_AT:
        return "cheap_only"    # cascade may not escalate past tier 0
    return "normal"

Per tenant keys matter more than the cap value. Without them, one integration exhausts a shared budget and every other user sees a degraded assistant for reasons nobody can explain. With them, the noisy tenant degrades and the rest carry on. Same principle as the drift monitoring in monitoring models in production: aggregate metrics hide the thing you need to see.

Where routing loses money

My worst router was the cleverest one. I built a Sonnet 5 classifier that read each question and picked a tier, on the reasonable theory that a model understands difficulty better than a threshold does. It did, slightly: routing accuracy improved over keyword rules by enough to save 0.95 dollars per thousand questions. Classifier calls cost 1.40 dollars per thousand. Net loss of 0.45 dollars per thousand, plus 310 ms on p95, and it ran for nine days before I plotted classifier spend as its own line on the cost dashboard rather than folding it into a total. Reverting took twenty minutes. Noticing took nine days.

Generalise that into a rule you can apply before writing any router: a router is only viable if its own cost is well under the spread it captures. Our Haiku to Sonnet spread on a real request is 4.57 dollars per thousand. A router that costs 1.40 is spending 31 percent of the prize on the decision, and it only ever captures a fraction of the spread because it is not right every time.

This is also where I part company with the published routing results. RouteLLM reports cost reductions above 85 percent on MT Bench while holding 95 percent of the strong model quality, and the method is sound. Those benchmarks route short standalone prompts, where the model choice determines almost the entire bill. A RAG request is the opposite shape: 1,920 fresh input tokens of retrieved context that every tier is charged for. Routing attacks a smaller share of your bill than any benchmark implies, and quoting 85 percent to a finance lead is how you end up explaining a 22 percent result.

One caveat on the cascade, since a survey without caveats is marketing. Tuning the confidence floor is genuinely fiddly. We swept it from 0.50 to 0.75 in steps of 0.05 across the eval set and watched the escalation rate move from 4 percent to 38 percent; 0.62 was the knee, and it shifted after the corpus grew by a third. Put that sweep in a script, not in your head, and rerun it whenever the corpus or the reranker changes.

Start with a confidence cascade and cut retrieval before you cut models

Order of operations, on Monday morning. Instrument first: write the twenty lines of cost_model.py and log a dollar figure on every request, split by tier and by tenant. Then look at what your context is costing you, because trimming top_k and pruning a bloated system prompt are free wins that no router can match. Only then build the cascade, tune one threshold, and cap spend per tenant with a degrade step before the deny step.

Cost and portability are the same conversation, incidentally. Every price in this part is a vendor decision you do not control, and Sonnet 5 introductory pricing ends on 31 August 2026, which will move our baseline from 9.14 to 12.94 dollars per thousand without a line of code changing. Part 28 covers deploying and rolling back prompts and models safely, and Part 29 covers switching providers without a rewrite, which is the structural answer to a price change you did not choose.

Your turn. Work out your own cost per unit of work this week, whatever the unit is, and tell me the number. Almost nobody I ask can say it, and everybody who works it out changes something within a fortnight.

References

• Anthropic pricing, per model rates, batch discount and cache multipliers, read 20 July 2026.
• Anthropic prompt caching, minimum cacheable lengths per model and TTL mixing rules, read 20 July 2026.
• RouteLLM: Learning to Route LLMs with Preference Data, Ong et al., arXiv 2406.18665.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts