Our head of support asked me one question in a review last March. When the assistant gets an answer wrong, can you tell me which sentence in the prompt caused it. I could not. My prompt was a 900 word string in a Python module, edited by four people over two months, with no version, no structure and no record of which edit had helped. That question changed how I write prompts more than any benchmark ever has.

Where the assistant stands

Last part the assistant could call a model, read back a finish status and a token count, and stream a reply through one provider neutral wrapper. It answered from three paragraphs of documentation pasted into a single hardcoded string. This part turns that string into a layered artifact you can cache, version and debug, and it does so before we add retrieval, because a badly shaped prompt makes retrieval look broken when it is not.

To follow along with a real corpus rather than a toy one, clone the kubernetes/website repository and point the assistant at its content directory. Several thousand Markdown files, a genuine changelog and a public issue tracker standing in for a support archive. It is close enough to a SaaS product corpus that every failure mode in this series shows up in it, and you can reproduce my numbers without an NDA.

Anatomy of a production prompt

Stop thinking of a prompt as prose and start thinking of it as a stack of layers, each with its own change frequency. Role and rules change monthly. Examples change weekly. Retrieved documents change per request. Conversation history grows every turn. User input changes every single time. Order those layers from slowest changing to fastest changing and almost every production concern resolves itself, because the cache, the model attention pattern and your debugging story all reward the same layout.

Anthropic documents the prefix order explicitly as tools, then system, then messages, and a change at any level invalidates that level and everything after it. OpenAI expresses the same idea differently, with a separate instructions parameter that outranks anything in the input array, and guidance to pin a model snapshot so behaviour does not shift under you. Different surface, same discipline: put the stable material where it can be reused and keep the volatile material at the end.

Instruction placement and what a model actually obeys

Numbered rules beat paragraphs. Rules stated as behaviour beat rules stated as prohibition. And rules that name an exact output token, rather than describing a desired response, are the only ones you can test automatically. My abstention rule says reply exactly NOT_IN_DOCS, not be honest when unsure, because a grep for one token is a test and a vibe is not.

# tested against anthropic 0.117.0 and openai 2.46.0 on Python 3.12
import os
import anthropic

client = anthropic.Anthropic(api_key=os.environ['ANTHROPIC_API_KEY'])

ROLE = (
    'You are the internal documentation assistant for a SaaS platform. '
    'You answer staff and customer questions from product docs, the '
    'support ticket archive and the changelog.'
)

RULES = (
    'Rules:n'
    '1. Answer only from material inside the docs section below.n'
    '2. If that material does not contain the answer, reply exactly: NOT_IN_DOCSn'
    '3. Cite the source file for every claim, in square brackets.n'
    '4. Keep answers under 120 words unless asked to expand.'
)

def build_system(examples, docs):
    return [
        {'type': 'text', 'text': ROLE},
        {'type': 'text', 'text': RULES},
        {'type': 'text', 'text': examples},
        {'type': 'text', 'text': docs,
         'cache_control': {'type': 'ephemeral'}},
    ]

msg = client.messages.create(
    model='claude-sonnet-5',
    max_tokens=512,
    system=build_system(EXAMPLES, DOCS),
    messages=[{'role': 'user', 'content': question}],
)
print(msg.usage.model_dump())

# first request
{'input_tokens': 41, 'cache_creation_input_tokens': 1187,
 'cache_read_input_tokens': 0, 'output_tokens': 88}

# second request, same prefix, 20 seconds later
{'input_tokens': 39, 'cache_creation_input_tokens': 0,
 'cache_read_input_tokens': 1187, 'output_tokens': 91}

Note where input_tokens sits in that output. It counts only what comes after the breakpoint, not the total you sent. I have watched two separate teams celebrate a token reduction that was really just a cache read, so read all three fields or read none of them. On OpenAI the equivalent structural move is to put stable material in the instructions parameter and the volatile material in input, and to place long documents near the top rather than after the question, which their guidance and Anthropic guidance agree on.

Examples, and how many earn their tokens

Few shot prompting, meaning worked input and output pairs shown inside the prompt, is still the cheapest quality lever available. Anthropic suggests three to five examples as a starting point and recommends wrapping each in tags so the model can tell an example from an instruction. That is sound advice for a prompt you pay for every time. It is far too conservative for a prompt you cache.

EXAMPLES = '''<examples>
<example>
  <question>How do I roll back a failed deployment?</question>
  <answer>Set the deployment back to a previous revision with the
  rollout undo command. [content/en/docs/concepts/workloads/controllers/deployment.md]</answer>
</example>
<example>
  <question>What is our refund policy for annual plans?</question>
  <answer>NOT_IN_DOCS</answer>
</example>
<example>
  <question>Is the scheduler down right now?</question>
  <answer>NOT_IN_DOCS</answer>
</example>
</examples>'''

Here is the measurement that convinced me. I built a held out set of 200 questions, 47 of which had no answer anywhere in the corpus, and scored format compliance as citing a source file, staying under 120 words and returning NOT_IN_DOCS when appropriate. With no examples, compliance sat at 71 percent and the model correctly abstained on 12 of the 47 unanswerable questions. It invented a refund policy, twice. With four examples, two of which were refusals, compliance rose to 96 percent and correct abstention went to 44 of 47.

Refusal examples do the heavy lifting. A rule that says decline when unsure competes with the model tendency to be helpful; an example that shows declining demonstrates that declining is the helpful answer. If you add only one thing to your prompt this week, add two examples where the correct output is your abstention token.

Prompt templates that do not break the cache

Now the war story. I wrote what looked like a small convenience: an f-string that stamped the current date and the requesting team name into the top of the system prompt, so answers could reference today and address the right audience. It shipped in a Friday release and nobody reviewed it twice. Six weeks and roughly 214,000 requests later our finance partner flagged that the model line was running about three times forecast.

# the version that quietly cost us money for six weeks
from datetime import date

def build_system_broken(examples, docs, user_team):
    header = (
        ROLE
        + 'nToday is ' + date.today().isoformat()
        + '. The user is on the ' + user_team + ' team.'
    )
    return [
        {'type': 'text', 'text': header},
        {'type': 'text', 'text': RULES},
        {'type': 'text', 'text': examples},
        {'type': 'text', 'text': docs,
         'cache_control': {'type': 'ephemeral'}},
    ]

# three consecutive requests, different teams
{'input_tokens': 41, 'cache_creation_input_tokens': 1194, 'cache_read_input_tokens': 0, ...}
{'input_tokens': 38, 'cache_creation_input_tokens': 1191, 'cache_read_input_tokens': 0, ...}
{'input_tokens': 44, 'cache_creation_input_tokens': 1193, 'cache_read_input_tokens': 0, ...}

Worse than getting no benefit, we were paying a penalty. A five minute cache write is billed at 1.25 times base input price, so writing a cache entry on every request and never reading one costs about 25 percent more than not caching at all. Moving those two interpolated values out of the system block and into the user message took eleven minutes. Monthly prompt layer spend fell from $612 to $96.

A colleague tried a different fix first, marking every block as cacheable on the theory that more breakpoints meant more chances to hit. That one at least fails loudly:

Traceback (most recent call last):
  File 'assistant/client.py', line 63, in ask
    msg = client.messages.create(
anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error':
{'type': 'invalid_request_error', 'message': 'A maximum of 4 blocks with
cache_control may be provided. Found 5.'}}

There is a second silent failure worth knowing about, and it inverts advice you have heard everywhere. Minimum cacheable prompt length is 1,024 tokens on Claude Sonnet 5 and Opus 4.8, 4,096 on Haiku 4.5 and 512 on Fable 5. Below that threshold your cache_control marker is ignored, both usage counters read zero, and no error is returned. Our first prompt was 890 tokens. Everyone advises trimming prompts to save money; here, trimming was the expensive move.

Inputs that are not questions

Real users do not send well formed questions. Within a fortnight of internal launch our assistant had received a pasted stack trace with no question attached, the word thanks, an entire customer email including a signature block, and someone testing whether it would write a resignation letter. A prompt tuned on clean questions degrades badly on all four, and the retrieval step we build in Part 8 will waste an embedding call on every one of them.

ROUTE_RULES = (
    'Classify the user message. Reply with exactly one word, no punctuation, '
    'chosen from: QUESTION GREETING FEEDBACK OTHER'
)

VALID = {'QUESTION', 'GREETING', 'FEEDBACK', 'OTHER'}

def route(text):
    r = client.messages.create(
        model='claude-haiku-4-5-20251001',
        max_tokens=5,
        system=ROUTE_RULES,
        messages=[{'role': 'user', 'content': text}],
    )
    raw = r.content[0].text.strip().strip('.').upper()
    return raw if raw in VALID else 'OTHER'

Those last two lines are scar tissue. My first version compared the reply directly against a dictionary key, and it worked for about nine hours:

Traceback (most recent call last):
  File 'assistant/router.py', line 41, in handle
    return HANDLERS[route(text)](text)
           ~~~~~~~~^^^^^^^^^^^^^
KeyError: 'QUESTION.'

Prefilling the assistant turn is the classic trick for pinning output format, and it still works well on many models, but it is no longer supported on several of the newest ones including Claude Opus 4.8 and Sonnet 4.6. So treat normalisation and a default branch as the durable answer, and treat any prompt level formatting instruction as advisory. Part 4 replaces this hand rolled parsing with real schemas and validation that fails loudly rather than silently, which is where this pattern properly belongs.

Version prompts like code

Back to the question that opened this piece. To answer which sentence caused a bad answer, you need to know which prompt produced it, and a Git commit hash is not enough because the prompt is assembled at runtime from several blocks. Hash the assembled blocks and log that hash beside every request.

import hashlib
import json

def prompt_id(blocks):
    stable = [b['text'] for b in blocks if 'cache_control' not in b]
    payload = json.dumps(stable, sort_keys=True).encode()
    return hashlib.sha256(payload).hexdigest()[:12]

blocks = build_system(EXAMPLES, DOCS)
print(prompt_id(blocks))

With that hash on every log line, a Monday morning report of the assistant is worse this week becomes a filter rather than an argument. We caught a regression in nine minutes once because complaints clustered entirely on one prompt_id, introduced by a rules edit that had reordered two numbered instructions. Package the prompt with the code that calls it rather than storing it in a database, using the layout from notebook to Python package, so a prompt change goes through review and rollback like any other change. Prompts stored in an admin panel get edited at 2am by whoever is on call, and nobody can tell you what changed.

One caveat on growing conversations. Once history accumulates, your stable prefix is a shrinking fraction of a request that keeps expanding toward the model limit, and the constraint stops being cost and starts being the context window. Part 5 takes that on directly with token budgets.

Prompt layout I would ship on Monday

Four ordered system blocks: role, numbered rules with an exact abstention token, four to eight examples of which at least two are refusals, and retrieved material last. One explicit cache breakpoint on the final stable block, never automatic caching for single turn question answering. Every per request value in the user message, without exception. A twelve character prompt hash on every log line. That layout is not clever and it is not novel, and it survives real users, which cleverer layouts of mine did not.

References

• Claude Platform docs, Prompt caching, for the tools then system then messages prefix order, the four breakpoint ceiling, the 20 block lookback window, the per model minimum cacheable lengths and the 1.25x write and 0.1x read multipliers used throughout this part.
• Claude Platform docs, Prompting best practices, for tagged examples, the three to five example starting point, the long context guidance on placing documents above the query, and the note that prefilling is unsupported on several current models.
• OpenAI, Text generation, for the instructions parameter and its priority over input, message roles, and the recommendation to pin production applications to a model snapshot.
• Wei et al., Chain of Thought Prompting Elicits Reasoning in Large Language Models, arXiv 2201.11903, the paper that established worked exemplars in the prompt as a reasoning lever rather than a formatting trick.
• kubernetes/website, the public Markdown corpus used as a stand in for a product documentation set, so the measurements in this part can be reproduced without private data.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts