Six percent. That is how much of our documentation assistant p95 latency was spent generating tokens the user actually read. Everything else was setup: embedding the question, searching, reranking, and waiting for a model to re read a 5,120 token instruction block it had already read four hundred times that hour.

Once you see the request that way, cost and latency stop being two projects. They are the same project, and prompt caching is where both of them start.

Where latency actually goes in one assistant request

Last part we wrapped the assistant in OpenTelemetry spans and found a CPU reranker eating 2,310 ms of a 6,605 ms p95. This part we spend those traces: cache the prefix, move the offline work to a batch, and stream what is left. Nothing here is guesswork, because the spans already told us which component to attack.

Before optimising anything, separate time to first token from total time. Users perceive the first one. Your cost report reflects the second. A change that improves one and worsens the other is common enough that measuring both is not optional.

# measure_ttft.py, Python 3.12.3
# anthropic 0.117.0, verified 20 July 2026
import os, time
import anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

SYSTEM = open("prompts/assistant_v7.txt").read()   # 5,120 tokens

def timed_stream(question: str) -> dict:
    start = time.perf_counter()
    ttft = None
    chunks = 0
    with client.messages.stream(
        model="claude-sonnet-5",
        max_tokens=800,
        system=[{"type": "text", "text": SYSTEM}],
        messages=[{"role": "user", "content": question}],
    ) as stream:
        for _ in stream.text_stream:
            if ttft is None:
                ttft = time.perf_counter() - start
            chunks += 1
        usage = stream.get_final_message().usage
    return {
        "ttft_ms": round(ttft * 1000),
        "total_ms": round((time.perf_counter() - start) * 1000),
        "chunks": chunks,
        "cache_write": usage.cache_creation_input_tokens,
        "cache_read": usage.cache_read_input_tokens,
        "fresh_input": usage.input_tokens,
    }

print(timed_stream("How do I rotate an API key for a sandbox workspace?"))

{'ttft_ms': 3204, 'total_ms': 4009, 'chunks': 118,
 'cache_write': 0, 'cache_read': 0, 'fresh_input': 5231}

3,204 ms of that 4,009 ms elapsed before a single character reached the user, and 805 ms produced 118 chunks of readable answer. Reading the prompt is the expensive part, not writing the reply. That ratio is what makes caching the highest leverage change available to you, and it is why I reach for it before touching model choice, which is the next part of this series.

Prompt caching and what a breakpoint keys on

Prompt caching stores the model internal representation of a prompt prefix so a later request with an identical prefix skips reprocessing it. Anthropic charges cache writes at 1.25 times base input price for a five minute entry, 2 times for a one hour entry, and cache reads at 0.1 times base input. OpenAI applies caching automatically once a prefix passes 1,024 tokens. Both providers key on an exact prefix match.

Here is where almost everyone, including me, gets it wrong. Intuition says cache the biggest block of tokens, and in a retrieval system that is the retrieved chunks. Retrieved chunks change with every question. A cache write happens only at your breakpoint, so putting the breakpoint after content that varies writes a fresh entry every single request and reads one never.

Rule to hold onto: place the breakpoint on the last block whose prefix is byte identical across the requests you want sharing a cache. For a retrieval assistant that is the end of system instructions, tool definitions and few shot examples, before anything query dependent. Smaller than you expected, and it is the one that pays.

# cached_call.py, anthropic 0.117.0
STATIC_PREFIX = [
    {"type": "text", "text": SYSTEM},          # 5,120 tokens, identical every request
    {"type": "text", "text": FEW_SHOT_EXAMPLES,
     "cache_control": {"type": "ephemeral", "ttl": "1h"}},   # breakpoint HERE
]

def answer(question: str, chunks: list[str]):
    return client.messages.create(
        model="claude-sonnet-5",
        max_tokens=800,
        system=STATIC_PREFIX,
        messages=[{"role": "user", "content":
            "nn".join(chunks) + "nnQuestion: " + question}],
    )

for q in ["How do I rotate an API key?", "Where do I set a webhook secret?"]:
    u = answer(q, retrieve(q)).usage
    print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

6784 0 1841      # first call: prefix written to cache
0 6784 1903      # second call: prefix read, 0.1x price

Now the failure you will actually hit. Caching below the model minimum produces no error at all. Minimums differ per model, and they are not intuitive: 1,024 tokens for Claude Sonnet 5 and Opus 4.8, but 4,096 for Haiku 4.5. Route a request to the cheap model to save money and your caching silently stops working.

# same code, model swapped to claude-haiku-4-5, prefix 2,940 tokens
0 0 4781
0 0 4795
0 0 4802

# no exception, no warning, no cache. Assert instead:
assert u.cache_creation_input_tokens or u.cache_read_input_tokens, (
    f"prefix not cached on {model}; check the per model minimum"
)
# AssertionError: prefix not cached on claude-haiku-4-5; check the per model minimum

Two more constraints worth memorising. You get four explicit breakpoints, and the lookback window is 20 blocks, so a long conversation can push your breakpoint past the last write and lose every hit. And on a growing chat, automatic caching falls into exactly the same trap as my timestamp block: it places the breakpoint on the last cacheable block, which in a retrieval prompt is the one that changes.

Cost arithmetic on a cached prefix

Vendor guidance says stay on the default five minute cache and only reach for the one hour entry in unusual cases. For a steady public API that is right. For an internal assistant it is wrong, and the reason is traffic shape. Our staff traffic is bursty: a cluster of questions at 09:40, silence until 11:15. Five minute entries expire in the gaps, so we paid write prices on 29 percent of requests. One hour entries cost twice as much to write and we write them 25 times less often.

Figures below are for a 5,120 token prefix on Claude Sonnet 5 at introductory pricing of 2 dollars per million input tokens, cache reads at 0.20 and five minute writes at 2.50. Prefix only; retrieved chunks and output tokens are extra and unaffected.

Cache discounts and the batch discount stack, which is the last row. Verdict: if your traffic arrives in bursts with gaps longer than five minutes, move to the one hour TTL and stop reading the default as a recommendation. If your traffic is genuinely continuous, stay on five minutes, because a five minute entry refreshes on every hit at no charge and you will never pay the 2 times write.

Batching work that nobody is waiting for

Not every model call has a human attached. Our assistant regenerates changelog summaries nightly, re scores the eval set from Part 20, and back fills ticket categories. All of that ran through the synchronous endpoint for months at full price, competing with live traffic for the same rate limit. Batch endpoints halve the price and take that pressure off, which is the same batch versus real time split covered for classical models in serving machine learning models.

# nightly_summaries.py, anthropic 0.117.0
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

reqs = [
    Request(
        custom_id=f"changelog-{entry.id}",
        params=MessageCreateParamsNonStreaming(
            model="claude-sonnet-5",
            max_tokens=400,
            system=STATIC_PREFIX,          # identical in every request, 1h TTL
            messages=[{"role": "user", "content": entry.body}],
        ),
    )
    for entry in changelog_entries          # 1,880 entries
]

batch = client.messages.batches.create(requests=reqs)
print(batch.id, batch.processing_status)

# later, stream results back rather than downloading in one go
for r in client.messages.batches.results(batch.id):
    if r.result.type == "succeeded":
        store(r.custom_id, r.result.message.content[0].text)
    elif r.result.type == "expired":
        requeue(r.custom_id)

msgbatch_01HkcTjaV5uDC8jWR4ZsDV8d in_progress
# 1,880 requests, ended after 34 minutes
# succeeded 1,871  errored 0  expired 9  canceled 0
# cache hit rate across the batch: 88 percent

Batch caching is best effort, because requests run concurrently and a cache entry only exists after the first response begins. Anthropic quotes hit rates from 30 to 98 percent; we saw 88. Use the one hour TTL inside a batch, since a batch commonly runs longer than five minutes. And a limit that caught me: cache pre warming with max_tokens set to 0 is rejected inside a batch, so warm the cache with a synchronous call first, then submit.

Streaming and perceived latency

Streaming changes no cost and no total duration. What it changes is when the user stops waiting, and that is worth a great deal. Our support team rated the assistant usable at a 4,009 ms non streamed response and unusable at 6,605 ms, but rated a 6,605 ms streamed response as fine, because text began arriving at 3,204 ms.

Streaming has a cost elsewhere though, and it is one people discover late. Output validation cannot run on a partial response. If you enforce a JSON schema as in Part 4, or a guardrail as in Part 23, you have already streamed unvalidated text to the browser by the time the check fails. We stream free form answers and buffer anything structured. Splitting those two paths is a five line change and it removes an entire category of retraction bug.

One practical warning about timeouts. A cache miss on a large prefix is much slower than a hit, so a timeout tuned against warm traffic will fire on the first request after every deploy. Ours was set at 8 seconds and the cold path took 11.

Traceback (most recent call last):
  File "/srv/assistant/answer.py", line 88, in answer
    return client.messages.create(**kwargs)
  File "/usr/lib/python3.12/site-packages/anthropic/_base_client.py", line 1052, in request
    raise APITimeoutError(request=request) from err
anthropic.APITimeoutError: Request timed out.

# Fix: raise the client timeout above the cold path, and pre warm on deploy.
client = anthropic.Anthropic(timeout=30.0)
client.messages.create(model="claude-sonnet-5", max_tokens=0,
                       system=STATIC_PREFIX,
                       messages=[{"role": "user", "content": "warmup"}])

Response caching above the model

Prompt caching is provider side and only skips prefix reprocessing. A response cache in your own service skips the call entirely. For our assistant, 12 percent of questions were byte identical repeats within an hour, mostly during onboarding weeks when six new hires asked the same VPN question. Those cost nothing now.

Key the response cache on the question text and a corpus version stamp, so republishing the docs invalidates every stored answer at once. Without that stamp you will serve last month answer about a setting that moved, and nobody will be able to explain why.

Semantic response caching, where you embed the incoming question and serve the stored answer for any question above a similarity threshold, is the one I would avoid. We trialled it at cosine 0.95 and it served the answer for how to rotate a production API key in response to how to rotate a sandbox API key. Two questions can sit at 0.97 similarity and have opposite correct answers. You buy a few percent extra hit rate and pay for it in wrong answers that look confident, which is precisely the failure the retrieval evaluation in Part 13 exists to prevent.

Cache the stable prefix before you tune anything else

On Monday, print cache_creation_input_tokens and cache_read_input_tokens for one hundred production requests and look at the ratio. If reads are near zero while writes are not, your breakpoint is on a block that changes, and moving it is a four minute fix worth more than any model swap you are considering. If both are zero, your prefix is under the minimum for the model you routed to. That single check found more savings for us than a quarter of prompt tuning did.

Order matters after that: prefix caching first, batch anything without a human waiting, stream what remains, and add an exact match response cache only if you have a version stamp to invalidate it with. Skip semantic caching. Model routing, which is where most teams start and where the real remaining money sits once the mechanical wins are taken, is Part 27. Costs also drift as traffic patterns change, so keep the hit rate on a dashboard next to the drift metrics from monitoring models in production, and read the generative AI cost breakdown if you need to explain the bill to somebody who does not write code.

Your turn. Run that ratio check on your own service today and tell me what you found. I have yet to see a first measurement come back healthy.

References

• Anthropic prompt caching, for the pricing multipliers, per model minimums, the 20 block lookback and cache pre warming with max_tokens set to 0
• Anthropic batch processing, for the 50 percent pricing, the 24 hour expiry, result streaming and the best effort cache hit range
• OpenAI prompt caching, for the automatic 1,024 token prefix threshold
• OpenAI Batch API, for the 24 hour turnaround and batch rate discount
• OpenAI flex processing, for the service_tier parameter and the 429 resource unavailable behaviour

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts