Twelve cases. That was the whole eval set for the first version of the documentation assistant, and it passed all twelve, and I shipped it feeling reasonably good about my life. Support reopened 41 percent of the answers it gave in week one. Twelve cases had told me nothing, because I had written all twelve after the feature existed, by asking myself what the assistant seemed good at. An eval set assembled that way is a mirror. It reflects the thing you already built.

Eval sets before features, and why that order

Last part gave the assistant approval gates, so a human signs off before it does anything that changes state. Every capability from Part 8 onward is now in place: it ingests docs, retrieves, calls tools, escalates. Not one line of that tells me whether its answers are any good, and every prompt tweak from here is a coin flip I cannot score. That is what this part fixes.

Ordering matters more than most people expect. When you write cases after the feature works, you unconsciously write cases it passes, because your sense of what the assistant is for has already been shaped by watching it succeed. Anthropic’s guidance puts this plainly: design evals that mirror your real world task distribution, and factor in edge cases. Your intuition about the task distribution is worthless once you have spent a fortnight staring at your own happy path.

Anthropic also makes a recommendation that reads as wrong until you have lived it: prioritise volume over quality, because many questions with slightly noisy automated grading beat a handful of hand graded ones. I resisted that for months. Forty crudely graded cases catch a regression at 4am; eight beautifully graded ones catch nothing, because eight cases have no statistical resolution at all. A single case flipping moves your score by 12.5 percent, so every run looks like a catastrophe or a triumph and you learn to ignore it.

Anatomy of a golden case

A case is not a question and a model answer. Storing a full reference answer sounds rigorous and is a trap, because there are twenty acceptable phrasings of any correct reply and string comparison against one of them fails all nineteen others. What you want instead are assertions: things that must appear, things that must never appear, and the source that should have been cited. Those survive rewording.

Five cases in JSONL, one object per line. Three positives drawn from tickets and two negatives, because an assistant that answers everything confidently is worse than one that answers less.

# evalset.jsonl -- one JSON object per line, no trailing blank line
{"id": "doc-001", "question": "How do I rotate an API key without downtime?", "intent": "howto", "source": "ticket-48213", "must_include": ["create the new key first", "revoke"], "must_not_include": ["contact support"], "expect_citation": "docs/security/api-keys.md", "answerable": true}
{"id": "doc-002", "question": "What is the rate limit on the bulk export endpoint?", "intent": "lookup", "source": "ticket-48901", "must_include": ["120"], "must_not_include": [], "expect_citation": "docs/api/limits.md", "answerable": true}
{"id": "doc-003", "question": "Does the Enterprise plan include SAML?", "intent": "lookup", "source": "changelog-2026-03", "must_include": ["SAML"], "must_not_include": [], "expect_citation": "docs/plans.md", "answerable": true}
{"id": "neg-001", "question": "What will pricing be in 2028?", "intent": "unanswerable", "source": "handwritten", "must_include": ["do not have"], "must_not_include": ["$"], "expect_citation": null, "answerable": false}
{"id": "neg-002", "question": "Can you refund my invoice right now?", "intent": "out_of_scope", "source": "ticket-49110", "must_include": ["support"], "must_not_include": ["refunded"], "expect_citation": null, "answerable": false}

Loose JSONL rots quickly, so validate it on load rather than discovering a typo three weeks later in a CI run nobody reads. A pydantic model plus a duplicate id check is about fifteen lines and has caught more of my mistakes than any grader.

# evalcase.py -- tested with Python 3.10.12, pydantic 2.13.4
# No API key is used or needed anywhere in this file.
from typing import Literal
from pydantic import BaseModel, Field
import pathlib, collections

Intent = Literal['howto', 'lookup', 'troubleshoot', 'unanswerable', 'out_of_scope']

class EvalCase(BaseModel):
    id: str
    question: str
    intent: Intent
    source: str
    must_include: list[str] = Field(default_factory=list)
    must_not_include: list[str] = Field(default_factory=list)
    expect_citation: str | None = None
    answerable: bool

def load(path: str) -> list[EvalCase]:
    cases, seen = [], set()
    for n, line in enumerate(pathlib.Path(path).read_text().splitlines(), 1):
        if not line.strip():
            continue
        c = EvalCase.model_validate_json(line)
        if c.id in seen:
            raise ValueError(f'duplicate case id {c.id} on line {n}')
        seen.add(c.id)
        cases.append(c)
    return cases

if __name__ == '__main__':
    cases = load('evalset.jsonl')
    print(f'loaded {len(cases)} cases')
    print('by intent:', dict(collections.Counter(c.intent for c in cases)))
    print('negatives:', sum(1 for c in cases if not c.answerable), 'of', len(cases))

$ python3 evalcase.py
loaded 5 cases
by intent: {'howto': 1, 'lookup': 2, 'unanswerable': 1, 'out_of_scope': 1}
negatives: 2 of 5

Add a case by hand and forget a field, which you will, and pydantic stops you at the door rather than letting the grader silently treat a missing flag as false:

Traceback (most recent call last):
  File "evalcase.py", line 22, in load
    c = EvalCase.model_validate_json(line)
pydantic_core._pydantic_core.ValidationError: 1 validation error for EvalCase
answerable
  Field required [type=missing, input_value={'id': 'doc-004', 'questi...ation': 'docs/audit.md'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.13/v/missing

Seeding a set from real traffic

Nobody wants to hear that the first forty cases are manual work, and they are. Sit with a week of support tickets and the assistant’s own request log, and take questions verbatim. Do not paraphrase, because paraphrasing quietly repairs the ambiguity that made the question hard.

Four sources gave me a usable seed set in an afternoon. Reopened tickets, which are questions the docs failed to answer once already. Search queries with no click, which are gaps. Changelog entries from the last two quarters, which catch answers that were correct last year. And a deliberate block of negatives, written by hand, since nobody files a ticket to report that a bot confidently invented a price.

You can ask a model to expand a seed set, and both Anthropic and OpenAI suggest it. Use it for phrasing variants of cases you already have, never for inventing new questions. Generated questions cluster around what the generating model finds natural to ask, which correlates with what your assistant finds easy to answer, and you are back to writing a mirror.

Coverage gaps you cannot see without measuring

Here is the twenty line script that embarrassed me. It compares the intent mix of the eval set against the intent mix of a week of live traffic, and prints the gap.

# coverage.py
import collections
from evalcase import load

# Intent counts from one week of real assistant traffic, from the request log.
# Replace with your own query; the shape is what matters.
TRAFFIC = {'lookup': 4120, 'howto': 2890, 'troubleshoot': 1755,
           'out_of_scope': 690, 'unanswerable': 402}

def report(cases):
    total_t = sum(TRAFFIC.values())
    have = collections.Counter(c.intent for c in cases)
    total_c = len(cases)
    print('{:<14}{:>9}{:>7}{:>8}'.format('intent', 'traffic', 'eval', 'gap'))
    for intent, n in sorted(TRAFFIC.items(), key=lambda kv: -kv[1]):
        share_t = n / total_t
        share_c = have.get(intent, 0) / total_c if total_c else 0
        print('{:<14}{:>8.1%}{:>7}{:>+8.1%}'.format(
            intent, share_t, have.get(intent, 0), share_c - share_t))

if __name__ == '__main__':
    report(load('evalset.jsonl'))

$ python3 coverage.py
intent          traffic   eval     gap
lookup           41.8%      2   -1.8%
howto            29.3%      1   -9.3%
troubleshoot     17.8%      0  -17.8%
out_of_scope      7.0%      1  +13.0%
unanswerable      4.1%      1  +15.9%

Troubleshoot at 17.8 percent of traffic and zero cases is the finding. My assistant could have degraded to uselessness on nearly a fifth of what people asked it, and every dashboard would have stayed green. Run this before you write another prompt. It takes ten minutes and it will tell you something you did not want to know.

Running an eval set with pytest

Resist building an eval framework. You already have one, it is called pytest, and parametrize turns a JSONL file into named test cases with almost no code. Graders come first, and at this stage they are deterministic string checks, which Anthropic rates as the fastest and most reliable method available. LLM as judge belongs in Part 21, once you have something to check the judge against.

# graders.py
from dataclasses import dataclass
from evalcase import EvalCase

@dataclass
class Answer:
    text: str
    citations: list[str]

@dataclass
class Grade:
    passed: bool
    reasons: list[str]

def grade(case: EvalCase, ans: Answer) -> Grade:
    reasons = []
    low = ans.text.lower()
    for phrase in case.must_include:
        if phrase.lower() not in low:
            reasons.append(f'missing required phrase: {phrase!r}')
    for phrase in case.must_not_include:
        if phrase.lower() in low:
            reasons.append(f'contains banned phrase: {phrase!r}')
    if case.expect_citation and case.expect_citation not in ans.citations:
        reasons.append(f'missing citation: {case.expect_citation}')
    if not case.answerable and ans.citations:
        reasons.append('cited a source for an unanswerable question')
    return Grade(passed=not reasons, reasons=reasons)

Note the last rule. Citing a source for a question you should have refused is its own failure, and it is the one that generates angry tickets, because a fabricated answer wearing a real filename is far more convincing than a fabricated answer alone.

# test_evalset.py -- tested with pytest 9.1.1
import pytest
from evalcase import load
from graders import Answer, grade

# Swap this for your real pipeline. Recorded replies keep the suite runnable
# offline and free; the nightly CI job points it at the live assistant,
# reading OPENAI_API_KEY or ANTHROPIC_API_KEY from the environment.
RECORDED = {
    'doc-001': Answer('Create the new key first, deploy it, then revoke the old one.',
                      ['docs/security/api-keys.md']),
    'doc-002': Answer('The bulk export endpoint allows 120 requests per minute.',
                      ['docs/api/limits.md']),
    'doc-003': Answer('Yes, SAML single sign on is included on Enterprise.',
                      ['docs/plans.md']),
    'neg-001': Answer('I do not have pricing information for 2028.', []),
    'neg-002': Answer('I cannot issue refunds. Please raise this with billing.', []),
}

CASES = load('evalset.jsonl')

@pytest.mark.parametrize('case', CASES, ids=[c.id for c in CASES])
def test_case(case):
    g = grade(case, RECORDED[case.id])
    assert g.passed, '; '.join(g.reasons)

$ python3 -m pytest -q test_evalset.py
....F                                                                    [100%]
=================================== FAILURES ===================================
______________________________ test_case[neg-002] ______________________________

case = EvalCase(id='neg-002', question='Can you refund my invoice right now?', intent='out_of_scope', ...)

    @pytest.mark.parametrize('case', CASES, ids=[c.id for c in CASES])
    def test_case(case):
        g = grade(case, RECORDED[case.id])
>       assert g.passed, '; '.join(g.reasons)
E       AssertionError: missing required phrase: 'support'
E       assert False
E        +  where False = Grade(passed=False, reasons=["missing required phrase: 'support'"]).passed

test_evalset.py:23: AssertionError
=========================== short test summary info ============================
FAILED test_evalset.py::test_case[neg-002] - AssertionError: missing required...
1 failed, 4 passed in 0.08s

Read that failure carefully, because it is not what it looks like. Case neg-002 requires the word support. Assistant said "please raise this with billing", which is a better answer than the one I demanded, since billing is exactly who handles refunds. My case was wrong. Expect this. In my first real run, 6 of the 11 failures were defects in the eval set rather than the assistant, and fixing them is the process working, not the process failing.

Wiring the set into CI without blocking every merge

Sooner or later someone asks whether a failing eval should block a merge, and answering yes on day one is how a young eval set gets deleted. Forty cases against a live model took about 90 seconds and roughly 25 cents per run in my setup, which is cheap, but a set that is still 20 percent wrong about its own assertions will red-light honest pull requests until a frustrated colleague adds a skip marker and nobody ever removes it.

Two tiers solved it. On every pull request that touches a prompt, a retrieval parameter or the chunker, pytest runs against recorded responses, which is free and finishes in under a second, so it catches grader and schema breakage without spending a token. A nightly job runs the same cases against the live model and writes the pass rate to a file committed in the repository.

Gate on the delta, not the absolute. A pull request fails if it drops the pass rate below last night’s committed baseline, and it passes at 71 percent if 71 percent is where you already were. Absolute thresholds punish you for adding hard cases, which is exactly the behaviour you want to encourage, and within a month of setting a 90 percent floor I noticed I had stopped writing cases I suspected would fail. That is the failure mode nobody warns you about: a threshold that quietly turns your eval set back into a mirror.

Storage, ownership and vendor eval platforms

Every provider will offer to host your eval set, and this is where I part company with the obvious advice. Keep ground truth in your repository, versioned next to the prompts it grades, in a format you could parse with the standard library. Hosted platforms are fine as a runner. They are a poor home.

OpenAI has just made that argument for me. Its hosted Evals platform is deprecated, going read only for existing users on 31 October 2026 and shutting down on 30 November 2026, with the documentation now pointing new users at Datasets instead. Anyone whose eval definitions, graders and run history lived only in that product now has a migration on their roadmap that produces zero value for their users. A JSONL file and a pytest suite would have carried across untouched.

Portability is the other half of it. Part 29 covers switching providers without a rewrite, and an eval set is what makes that switch a measurement rather than an act of faith. If your set only runs inside one vendor’s console, you cannot benchmark that vendor against its competitor, which is a conflict of interest hiding in your tooling choice.

Once the set exists, it becomes a loop rather than a document, and the loop is what keeps it honest as traffic shifts under you.

Forty cases from last week’s tickets

Here is what I would do on Monday. Open last week’s support queue, pull forty questions verbatim, and give each one a must_include list, a must_not_include list and an expected citation. Ten of the forty should be negatives where refusal is the correct answer. Save it as JSONL in the repository, wire it to pytest, and run the coverage script against your traffic mix before you touch a single prompt.

Forty is the number I would defend. Twenty gives you a score that swings 5 percent on one flaky case; two hundred is a week of work you will abandon halfway through and never finish. Forty took me about four hours, caught a retrieval regression within a fortnight, and grew to 180 cases over the following quarter by the only healthy route, which is one case added every time something went wrong in production.

Deterministic graders will carry you a surprisingly long way, and then they will stop, because no string check tells you whether an answer was actually helpful. Part 21 hands that judgement to a model and looks hard at where the model lies to you. Structuring this as an installable package rather than four loose files is covered in the Data Science Series part on going from notebook to package, and it is worth doing early, because an eval suite nobody else can run is an eval suite nobody else runs.

Build the set this week, before the next prompt change, so that when someone asks whether the new version is better you have an answer with a number in it.

References

• Define success criteria and build evaluations, Anthropic documentation, for task specific eval design, the recommendation to prioritise volume over quality, and the ranking of code based, human and LLM based grading.
• Working with evals, OpenAI documentation, for the deprecation notice: read only for existing users on 31 October 2026, platform shutdown on 30 November 2026, and the pointer to Datasets.
• Evaluation best practices, OpenAI documentation, for designing evals against production behaviour and iterating on them as the application changes.
• Models, pydantic 2 documentation, for model_validate_json and the ValidationError shown above.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts