At 09:12 on 16 June 2026 our nightly ticket summariser produced nothing at all. It was calling claude-sonnet-4-20250514, which had been retired the previous day, and the API was answering with a 404 that no alert covered.

Fixing it took four minutes once somebody looked. Finding it took 41. Anthropic had sent the notice in April with the 60 days of warning their policy promises, to a shared inbox that nobody reads. Our deployment story for prompts and models was, at that point, a string literal in three files and a hope.

Deployable units in an LLM application

Last part the assistant learned to route between Haiku 4.5, Sonnet 5 and Opus 4.8, which brought cost per thousand questions from 9.14 dollars to 7.13. This part makes each of those decisions deployable and reversible on its own, because right now the routing thresholds, the prompt text and the model pins all ride inside one container image and move only when that image moves.

Five things change in a system like this, at wildly different rates. Treating them as one artifact is what makes a bad week bad: when a quality regression appears and three of them moved in the same release, you cannot attribute it, and reverting one means reverting all.

Keep that table honest by measuring your own numbers rather than copying mine. Rollback time is the only figure in it that matters during an incident, and almost nobody knows theirs.

Model version pinning and alias drift

Anthropic model IDs carry a dated snapshot suffix, claude-opus-4-5-20251101 for instance, and a family alias such as claude-opus-4-5 resolves to the latest snapshot. OpenAI and Google use the same shape. Vendor documentation nudges you toward the alias, because it means you get improvements without touching code.

Do not do that in production. An alias moving is a change to your output distribution with no diff, no pull request, no eval run and no approval. You cannot bisect it, because your git history says nothing happened. You cannot revert it, because the previous snapshot is no longer reachable through the alias you are calling. Every property you want from a deployment, an alias quietly removes. Use aliases on your laptop and in staging where the surprise is cheap.

Pinning has its own failure, and it is exactly the one that cost us 41 minutes in June: a pin never expires on its own, but the model behind it does. So pinning is only safe when paired with something that watches the calendar for you.

# models.py, tested on Python 3.12.3, anthropic 0.117.0, mlflow 3.14.0
# Dated snapshots in production. Family aliases are for local work only.
# Retirement dates read from platform.claude.com on 20 July 2026. Anthropic
# publishes them as tentative, not sooner than, dates.
import os

PINS = {
    "fast":     "claude-haiku-4-5-20251001",
    "standard": "claude-sonnet-4-5-20250929",
    "deep":     "claude-opus-4-5-20251101",
}

RETIRES = {
    "claude-haiku-4-5-20251001":  "2026-10-15",
    "claude-sonnet-4-5-20250929": "2026-09-29",
    "claude-opus-4-5-20251101":   "2026-11-24",
}

def model_for(tier: str) -> str:
    """Pin, unless an operator has overridden this tier at runtime."""
    return os.environ.get(f"MODEL_{tier.upper()}") or PINS[tier]

if __name__ == "__main__":
    for tier in PINS:
        m = model_for(tier)
        print(tier, m, "retires", RETIRES.get(m, "unknown"))

fast claude-haiku-4-5-20251001 retires 2026-10-15
standard claude-sonnet-4-5-20250929 retires 2026-09-29
deep claude-opus-4-5-20251101 retires 2026-11-24

That environment variable is not decoration. It is the rollback path for a model pin, and it is the difference between 40 seconds and a 9 minute image build when a new snapshot turns out to be worse than the one it replaced.

Now make the calendar somebody else problem. I could not find a public API on any provider that returns retirement dates as data, so we keep the dates beside the pins and let CI shout at us. A 90 day threshold is deliberate: it fires before the 60 day notice email arrives, which means the migration is scheduled work rather than a surprise.

# check_model_expiry.py, runs in CI on every build
import sys
from datetime import date
from models import PINS, RETIRES

WARN_DAYS = 90   # longer than the 60 day notice we are promised

today = date.today()
failed = False
for tier, model in PINS.items():
    days = (date.fromisoformat(RETIRES[model]) - today).days
    flag = "OK"
    if days < WARN_DAYS:
        flag, failed = "FAIL", True
    print(f"{flag:4s} {tier:9s} {model:27s} {days:4d} days left")

sys.exit(1 if failed else 0)

FAIL fast      claude-haiku-4-5-20251001     87 days left
FAIL standard  claude-sonnet-4-5-20250929    71 days left
OK   deep      claude-opus-4-5-20251101     127 days left

Without that check, here is what you get instead, on a morning of the calendar choosing. This is the traceback from our summariser, trimmed to the line that matters:

Traceback (most recent call last):
  File "/app/jobs/summarise_tickets.py", line 61, in run
    resp = client.messages.create(model=MODEL, max_tokens=1024, messages=msgs)
anthropic.NotFoundError: Error code: 404 - {'type': 'error', 'error':
  {'type': 'not_found_error', 'message': 'model: claude-sonnet-4-20250514'}}

Worth noticing why this hurt so much: a 404 is not retryable, so our retry and backoff layer correctly gave up immediately, and a batch job that fails fast at 02:00 looks identical to a batch job that had nothing to do. Add a startup assertion that every pin in PINS resolves to a live model, and you turn a silent overnight failure into a container that refuses to start.

Prompts as versioned artifacts

Two homes are on offer for prompt text, and teams argue about them as if it were either or. Git gives you review, blame and a single source of truth. A prompt registry gives you a version pointer you can move without shipping code. My verdict after doing both badly: author in git, sync to a registry from CI, and promote by alias. Git owns what the prompt says; the registry owns which version is live. Storing prompts only in a registry loses code review, which is where most of our bad prompt changes actually got caught.

MLflow and Langfuse both do the registry half well and neither locks you in. MLflow suits teams already tracking models there, per the registry patterns from the Data Science Series. Versions are immutable in both, which is the property that makes rollback meaningful.

# prompt_deploy.py, mlflow 3.14.0. Called from CI after the eval gate passes.
# MLFLOW_TRACKING_URI comes from the environment, never hardcoded.
import mlflow

TEMPLATE = """You answer staff questions using only the context below.
If the context does not contain the answer, say so and stop.

Context:
{{ context }}

Question: {{ question }}
"""

p = mlflow.genai.register_prompt(
    name="support-assistant-answer",
    template=TEMPLATE,
    commit_message="Tighten refusal wording when context is empty",
    tags={"author": "pranay", "eval_run": "2026-07-20-a"},
)
print(f"registered version {p.version}")

# Promote. Reverting is the same call with the previous number. [VERIFY]
# alias argument names against the mlflow 3.14 API reference before use.
mlflow.genai.set_prompt_alias(
    name="support-assistant-answer", alias="production", version=p.version
)

live = mlflow.genai.load_prompt(
    "prompts:/support-assistant-answer@production", cache_ttl_seconds=0
)
print(f"production now serves version {live.version}")

registered version 7
production now serves version 7

Shadow, canary and full rollout

Shadow mode runs the new version alongside the old, serves the old, and compares offline. Canary serves the new version to a slice of real traffic. Full rollout is what happens when the canary survives.

My rule splits by what kind of change it is. For a prompt edit, skip shadow and go straight to a 5 percent canary: reversal is 8 seconds, and shadowing doubles your token bill on every shadowed request for information your eval set already gave you. For a model swap, shadow first for a day, because the failure mode is a quiet quality drop that error rates and latency dashboards will not show you at all.

Whatever the split, hash the tenant rather than rolling a die per request. A stable hash gives every tenant a consistent experience and lets you widen the canary without shuffling who is in it.

# rollout.py, Python 3.12.3, standard library only
import hashlib
from collections import Counter

def variant_for(tenant_id: str, canary_pct: int) -> str:
    h = hashlib.sha256(tenant_id.encode()).digest()
    bucket = int.from_bytes(h[:4], "big") % 100
    return "canary" if bucket < canary_pct else "stable"

ids = [f"tenant-{i}" for i in range(10000)]
print(Counter(variant_for(t, 5) for t in ids))
print(Counter(variant_for(t, 25) for t in ids))

Counter({'stable': 9526, 'canary': 474})
Counter({'stable': 7492, 'canary': 2508})

A 5 percent setting produced 474 tenants, not 500, and a 25 percent setting produced 2,508. Hash splits are lumpy at small percentages and that is fine; what you buy is that tenant-42 lands in the same bucket on every request, and that widening from 5 to 25 keeps everyone already in the canary in the canary. Swap the hash for random.random() and you get exact proportions plus a support manager watching the assistant change personality between two questions in one conversation. I have shipped that version. The bug report was hard to read and entirely fair.

Saying a canary must hold for 24 hours is useless unless you have written down what holding means. We learned this by running a canary for a day, agreeing it looked fine, promoting it, and discovering four days later that refusal rate had doubled inside the slice while every dashboard we were watching stayed flat. Four gates now decide, and any one of them tripping reverts the alias automatically.

Notice that three of those four are cheap counters, not judgements. Automated quality scoring belongs in the eval gate before the canary, where it can take its time and cost what it likes. Inside a live canary you want signals that compute in a query and cannot themselves be wrong, which is a lesson I took the long way round after wiring an LLM judge into a rollout alarm and getting paged by it at 03:00 over a formatting difference.

Rollback speed and what it costs you

We timed four mechanisms with a stopwatch, measuring from decision to last replica serving the old behaviour. Note the axis: it is logarithmic, because the spread is two orders of magnitude.

None of this is free. Every artifact you lift out of the container image makes that image a less complete description of what ran. A tag no longer tells you which prompt version answered a question last Tuesday. Pay that back by recording the prompt version and the resolved model ID as attributes on every trace, exactly as in Part 25, and by keeping the whole promotion path in CI so it leaves an audit trail, the same discipline as CI/CD for ML pipelines.

Coupling between prompt version and model version

A prompt is tuned against a model whether you meant it or not. When we moved the answering tier from Sonnet 4.5 to Sonnet 5 without touching a word of the template, our eval score fell from 0.87 to 0.79, and the cause turned out to be an over-specified refusal instruction that the newer model followed too literally. Two hours of prompt work recovered it. Had we shipped that model swap in the same release as a prompt edit, I would have spent a day arguing about which one broke it.

So the real deployable unit is the pair, and two rules follow. Run your eval set against the pair, never the prompt in isolation. And never ship a prompt change and a model change inside the same 24 hours, however tempting the batching feels on a Friday.

MLflow lets you attach a model_config block to a prompt version, which looks like the answer to this. Read the small print: model config is explicitly mutable and can be changed after the version is created, unlike the template. That makes it a helpful hint for whoever picks the prompt up next, and a poor record of what actually ran. Your trace is the record. This is the same lesson as evaluating GenAI output: measure the system you are shipping, not a component of it.

Deployment setup I would run for a two person team

Six pieces, and none of them takes more than a morning. One module owning every model string, pinned to dated snapshots, with a per tier environment variable override. A CI check that fails the build at 90 days of remaining runway. Prompts authored in git, registered from CI once the eval gate passes, promoted by alias. One config service for retrieval settings, routing thresholds and the canary percentage. A stable tenant hash for the split, 5 percent, held for 24 hours. And a monthly rollback drill, timed.

What I would not build at this size: a bespoke prompt management service, per request A/B infrastructure, or automated promotion on eval score alone. Each of those solves a problem you get at a hundred prompts, not at six, and every one of them adds a component that can fail between your user and your model.

Part 29 takes the next step and asks what happens when the provider itself becomes the thing you want to roll back, which is the same problem with a much larger blast radius.

Your turn. On Monday, grep your repository for model name strings and count the call sites. If that number is greater than one, you already know what your first task of the week is. Tell me what the count was; mine was three, and I thought it was one.

References

• Anthropic model deprecations, lifecycle states, snapshot naming, retirement table and the 60 day notice commitment, read 20 July 2026.
• MLflow Prompt Registry, register_prompt, aliases, immutable versions, mutable model config and cache TTL defaults, mlflow 3.14.0.
• Langfuse prompt caching, 60 second default TTL, stale while revalidate behaviour and measured fetch latency, langfuse 4.14.1.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts