A prompt is code with no compiler behind it. You can delete a sentence from a system prompt, watch three answers come back looking fine, merge it, and only find out five weeks later that a whole class of question started failing on the day of that commit.

Cost of an untested prompt edit

Last part we calibrated the judge until it agreed with two human labellers at Cohen kappa 0.767, which is the point at which its verdicts are worth acting on. This part takes that judge, plus the 200 golden cases from Part 20, and turns them into a gate: a thing that runs on every pull request touching a prompt file, and on every model change, and tells you a number before your users do.

Here is the edit that made me build it. Our documentation assistant was verbose, staff complained, and I added seven words to the system prompt: be concise and answer in under three sentences. Reasonable. I checked it against four questions by hand and shipped it on a Tuesday. Suite pass rate went from 172 of 200 to 158 of 200, a drop from 0.860 to 0.790, and every one of the 14 newly failing cases was a multi step configuration question where the correct answer genuinely needs five sentences. Nobody filed a ticket, because staff who get a truncated answer just stop using the tool. I found it eleven days later while doing something else.

Four separate things can move your output quality, and only one of them appears in your git history:

• Your prompt. Visible, reviewable, and the only one most teams test.
• Your corpus. Someone republishes a docs page, chunk boundaries shift, retrieval returns different context for the same question.
• Your model version. A pinned snapshot gets deprecated, an alias silently points somewhere new, or you deliberately upgrade.
• Provider side infrastructure. Serving stack changes, quantisation changes, routing changes. You get no commit, no changelog and no notice.

That last category is why a regression suite for an LLM feature is not optional in the way a regression suite for a pure function is optional. Your dependency changes without telling you. A scheduled nightly run against a frozen eval set is the only instrument that catches it.

Pinning a baseline before you change anything

Reach for snapshot testing and you will find good tooling waiting: syrupy, pytest-snapshot, inline-snapshot. All of them store the exact output of a call and fail when it differs. For an LLM feature this is a trap, and it is the single most common wrong turn I see. Model output is not stable enough to diff, and it does not need to be. Two answers can be textually unrelated and both correct.

What you pin is the verdict, not the prose. Run the suite, record whether each case passed, and store that alongside the git SHA and the exact model identifier. Store one sample answer per case too, so a human can eyeball a diff during triage, but never assert on it.

# baseline.py -- Python 3.10.12, openai 2.46.0
# Freezes a run record. Assert on results, keep text only for triage.
import json, os, subprocess, datetime
from assistant import answer          # our app entry point
from evalset import load_cases        # the 200 cases from Part 20
from grade import grade_case          # tiered grader, next section

MODEL = os.environ["ASSISTANT_MODEL"]   # never hardcode, never an alias

def git_sha():
    return subprocess.check_output(
        ["git", "rev-parse", "--short", "HEAD"]
    ).decode().strip()

def run_baseline(path="baseline.json"):
    cases = load_cases()
    results = {}
    for case in cases:
        out = answer(case["question"], model=MODEL)
        verdict = grade_case(case, out)
        results[case["id"]] = {
            "passed": verdict.passed,
            "tier": verdict.tier,
            "sample": out[:400],     # triage only, never asserted on
        }
    passed = sum(1 for r in results.values() if r["passed"])
    record = {
        "git_sha": git_sha(),
        "model": MODEL,
        "recorded_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
        "n_cases": len(cases),
        "pass_rate": round(passed / len(cases), 4),
        "results": results,
    }
    with open(path, "w") as f:
        json.dump(record, f, indent=2)
    print(f"{passed}/{len(cases)} passed, rate {record['pass_rate']}")
    return record

if __name__ == "__main__":
    run_baseline()

$ ASSISTANT_MODEL=gpt-5-mini-2025-08-07 python baseline.py
172/200 passed, rate 0.86

Now run it twice with nothing changed. I did, on the same commit, same model snapshot, temperature pinned at zero, and got 172 then 169. Nine individual cases flipped verdict in one direction or the other; the net was three. Temperature zero selects the highest probability token, it does not make the arithmetic underneath deterministic, and floating point non-determinism in batched GPU inference is enough to change a token, which changes everything after it. The GenAI Series piece on hallucination and temperature covers why that dial does less than its name promises.

Assertion tiers that keep the suite affordable

Grading runs in three tiers, cheapest first, and a case that fails a cheap tier never reaches an expensive one. Deterministic string and structure checks are free and instant. Schema validation costs a millisecond. Only the residue goes to the judge, which costs money and 900 milliseconds a call. On our suite that split is 118 cases settled deterministically, 82 sent to the judge, which is what keeps a full run at 62 cents instead of about 1.50.

# tests/test_regression.py -- pytest 9.0.3
import json, os, pytest
from assistant import answer
from evalset import load_cases
from grade import grade_case

BASELINE = json.load(open("baseline.json"))
TOLERANCE = 0.02      # absolute pass rate drop we will accept
CRITICAL = set(BASELINE.get("critical_ids", []))   # 40 cases, zero tolerance
CASES = load_cases()
_observed = {}

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_case(case):
    out = answer(case["question"], model=os.environ["ASSISTANT_MODEL"])
    verdict = grade_case(case, out)
    _observed[case["id"]] = verdict.passed
    if case["id"] in CRITICAL:
        assert verdict.passed, f"critical case regressed: {verdict.reason}"
    # non-critical cases do not fail individually; the gate below decides

def test_aggregate_gate():
    assert len(_observed) == len(CASES), "gate ran before the cases"
    current = sum(_observed.values()) / len(_observed)
    base = BASELINE["pass_rate"]
    drop = base - current
    print(f"baseline {base:.3f} current {current:.3f} drop {drop:+.3f}")
    assert drop <= TOLERANCE, (
        f"pass rate fell {drop:.3f} against baseline {base:.3f} "
        f"(model {BASELINE['model']}, sha {BASELINE['git_sha']})"
    )

Two details in there earn their keep. Ordering matters, so the aggregate test must be collected last; pytest runs tests in file order, and putting the gate at the bottom of the file is the plainest way to get that without a plugin. And critical cases assert individually, because a 2 point tolerance on the whole suite would happily absorb the four questions about billing that legal wants answered correctly every single time.

$ pytest tests/test_regression.py -q -n 8
................F.....F..........F....F..........  [ 25%]
..........F..............F.......................  [ 50%]
.......F......F.........F........................  [ 75%]
..............F.........F.......F.......F........  [100%]F

--------- Captured stdout call ---------
baseline 0.860 current 0.790 drop +0.070

-------------- short test summary info --------------
FAILED tests/test_regression.py::test_aggregate_gate - AssertionError:
  pass rate fell 0.070 against baseline 0.860
  (model gpt-5-mini-2025-08-07, sha 4a1c9e2)
42 failed, 159 passed in 259.77s (0:04:19)

That is the be-concise commit, caught in four minutes instead of eleven days. Note what the failure message carries: the model identifier and the git SHA of the baseline. Six months from now, when someone reopens this run, those two strings are the difference between a diagnosis and a shrug.

Wiring the gate into a pipeline

My first version of this gate failed the build if any case regressed. It felt principled and it lasted thirteen days. Given a noise floor of nine flipping cases, that rule blocked 6 of the 9 pull requests we opened in a fortnight, every one of them a false alarm. Two engineers started re-running the job until it went green, which is the same as having no gate but slower. Then a genuine 7 point regression went in behind a red X that everyone had learned to scroll past. Rewriting it to an aggregate tolerance with a locked critical subset took an afternoon and the gate has been believed ever since. A test people route around is worse than no test, because it also costs you their trust in the next one you build.

Rebaselining deserves its own warning. If a branch can write baseline.json, every 1 point drop becomes the new normal and six merges later you have lost 6 points with a green pipeline the whole way. Baseline updates happen on main only, in a commit whose message says why, reviewed by a human.

If you would rather not own the harness, promptfoo does the same job declaratively, exits non-zero when assertions fail, and mixes deterministic assertion types with model graded ones in the same file. It is a reasonable default for a team that does not already live in pytest:

# promptfooconfig.yaml
prompts:
  - file://prompts/assistant_system.txt
providers:
  - openai:gpt-5-mini-2025-08-07
tests:
  - vars:
      question: What is the rate limit on the export endpoint
    assert:
      - type: contains
        value: 100 requests per minute
      - type: llm-rubric
        value: answers the question asked and cites a docs section
  - vars:
      question: How do I rotate an API key
    assert:
      - type: not-contains
        value: contact support
      - type: llm-rubric
        value: gives concrete steps, invents no configuration keys

Run it with an API key from the environment and a pass rate threshold in the pipeline step. The trade off against the pytest version is control: you get the assertion library for free, and you give up the ability to express a gate like ours, where 40 named cases behave differently from the other 160.

Model upgrades break on parameters first

Everyone plans for a model upgrade to change answer quality. In practice the first thing that breaks is the request itself. Anthropic now lists temperature, top_p and top_k as deprecated parameters that return a 400 error when set to a non default value on Claude Opus 4.7 and later, including Opus 4.8, and on Claude Sonnet 5. Every eval harness I have written pins temperature to zero as a matter of hygiene, which means every one of them fails on first contact with those models:

$ ASSISTANT_MODEL=claude-opus-4-8 pytest tests/test_regression.py -q
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE  [100%]

assistant.py:41: in answer
    resp = client.messages.create(model=model, temperature=0, ...)
anthropic.BadRequestError: Error code: 400 - temperature is not supported
  by this model

200 errors in 6.11s

Two hundred identical errors in six seconds is actually the good outcome, because it is loud. Build the sampling parameters as a dict that a per-model capability map populates, rather than passing temperature unconditionally, and this class of failure disappears for good. It is worth internalising the wider point: the parameter surface is versioned too, not just the weights, and a provider can remove a knob you built a habit around.

Migration runbook for a deprecated model

Deprecation notices are routine now rather than exceptional. Anthropic retired Claude Sonnet 4 and Opus 4 on 15 June 2026 and has Opus 4.1 retiring on 5 August 2026; OpenAI notified developers on 11 June 2026 that older GPT-5 and o3 snapshots leave the API on 11 December 2026. Assume you will do this two or three times a year, and make it boring. This is the sequence I follow, and it is the artifact from this part worth saving:

1. Confirm the blast radius. Export API usage by key and model from the provider console and find every service on the retiring snapshot, not just the one you remembered.
2. Freeze a fresh baseline on the old model while it still works. After the retirement date you cannot reproduce it, and a migration without a before number is a guess.
3. Run the suite on two or three candidates. Include a cheaper tier, because upgrades are the only moment anyone will let you re-examine cost.
4. Fix parameter errors before reading any quality number. A 400 is not a quality signal and will contaminate the comparison.
5. Compare per tier, not just headline. A candidate can gain 3 points overall while losing your 40 critical cases; the aggregate hides that and the critical assertions do not.
6. Shadow run for 5 to 7 days. Send live traffic to both, log both answers, serve only the incumbent. Real questions surface failure modes your golden set does not contain.
7. Cut over behind a flag, rebaseline on main, keep rollback for two weeks. Rollback means the old model string still resolves, which stops being true on the retirement date, so do not cut over in the final fortnight.

Step five is where our last migration nearly went wrong. Here is what the four candidates scored on the same 200 cases, run three times each and averaged:

Newer tier won on aggregate by 2.5 points and I very nearly shipped it on that basis. It failed 3 of the 40 critical cases, all three refund policy questions, because it had become markedly more hedging about anything resembling a commitment. Aggregate up, the questions legal cares about down. We shipped it anyway, with a retrieval fix and two extra sentences in the system prompt aimed squarely at those three cases, and rebaselined at 0.890. Without the critical subset asserting individually, that would have gone out unnoticed and been discovered by a customer.

Regression symptoms and their usual causes

When the gate goes red, the shape of the failure tells you where to look before you read a single answer. Keep this near the runbook:

Row five catches people out more than any other. Your judge is a model call, so it is subject to every failure mode in this article. When judged cases fall and deterministic ones hold, suspect the instrument before the thing being measured.

Gate on aggregate pass rate, not on individual cases

If you take one rule from this part: gate on aggregate pass rate with a tolerance wider than your measured noise floor, and carve out a small set of cases that may never regress. Avoid exact output snapshot testing entirely, however well the tooling markets itself, because you will spend your credibility on diffs that mean nothing. Avoid a per case zero tolerance rule for the same reason, and I say that as someone who shipped one and watched it get routed around inside two weeks.

On Monday, do the smallest version of this: run your eval set once, write the pass rate to a file with today’s git SHA and your exact model string next to it, and commit it. That single file converts every future argument about whether the assistant got worse from an opinion into a subtraction. Everything else here is refinement on top of having a number from before.

Part 23 turns to guardrails: input filtering, output validation and PII handling. A regression gate tells you the assistant got worse on average; a guardrail stops one specific answer reaching a user. Different jobs, different failure modes, and you need both before this thing faces customers.

References

• Anthropic, Model deprecations, for retirement dates, the 60 day notice commitment and the temperature, top_p and top_k parameter deprecation.
• OpenAI, Deprecations, for the 6 month notice period on generally available models and the current shutdown schedule.
• promptfoo, Assertions and metrics, for the deterministic and model graded assertion types used in the YAML config above.
• pytest, How to parametrize fixtures and test functions, for the parametrize and ids arguments used in the test module.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts