4.6 out of 5. That was the mean score my judge handed the documentation assistant across a rolling window of 200 answers, it sat on a dashboard next to latency and cost, and I looked at it most mornings with something close to pride. Support had reopened 31 of those same 200 answers. A badly built judge is worse than no judge, because a number on a dashboard stops you from looking at the answers yourself.

Where automated grading actually fits

Last part we built the eval set: 200 golden cases drawn from real tickets, each carrying must_include and must_not_include assertions. This part runs a model over the cases that substrings cannot settle, and then, which matters far more, checks whether that model agrees with a person.

String assertions closed 118 of our 200 cases cleanly. Those are the questions with a fact in them: a rate limit, a header name, a CLI flag. Either the string is there or it is not. What remains is everything a substring cannot reach. Did this answer respond to the question that was asked, or to a nearby one. Did it invent a configuration key that appears nowhere in the retrieved chunks. Did it hedge when the documentation is unambiguous, which annoys staff far more than a wrong answer does. Judgement of that kind is what a model grader buys you, and it is worth being precise about what you are buying.

OpenAI splits its graders into much the same taxonomy, string_check, text_similarity, score_model and a Python grader, which is a useful sanity check that this carving is not idiosyncratic. My rule from that table is blunt: reach for a model grader only for the criteria that a deterministic check genuinely cannot express, because every model grader you add is another instrument you now have to calibrate and keep calibrated.

Anatomy of a judge prompt that survives review

Four things separate a judge prompt that measures something from one that produces agreeable noise. Give it the retrieved context and not only the answer, or it will grade prose style, because style is the only signal it has. Ask independent yes or no questions rather than one holistic score. Let it reason before it decides, then throw the reasoning away, which Anthropic recommends and which moved my agreement by about four points. And pin temperature to zero, because a judge that disagrees with itself between runs cannot detect a regression smaller than its own variance.

# judge.py -- openai 2.46.0, pydantic 2.13.4, Python 3.10.12
import os
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])   # from env, never hardcoded
JUDGE_MODEL = os.environ["JUDGE_MODEL"]                  # a small instruct tier

class Verdict(BaseModel):
    reasoning: str        # written first, discarded after parsing
    grounded: bool        # every claim traceable to the CONTEXT
    responsive: bool      # answers the question that was asked
    no_invention: bool    # no flag, endpoint or field absent from CONTEXT
    scoped: bool          # says it does not know when CONTEXT is silent

SYSTEM = (
    "You grade a support assistant. You receive a QUESTION, the CONTEXT chunks "
    "it retrieved, and its ANSWER. Write brief reasoning, then answer four "
    "independent yes or no questions. Judge only against CONTEXT. Length, tone "
    "and confidence are irrelevant to every criterion. Return JSON only."
)

def judge(question, context, answer):
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"QUESTION:n{question}nn"
                                          f"CONTEXT:n{context}nn"
                                          f"ANSWER:n{answer}"},
        ],
    )
    return Verdict.model_validate_json(resp.choices[0].message.content)

v = judge(q, ctx, ans)
print(v.model_dump(exclude={"reasoning"}))
print("pass" if all(v.model_dump(exclude={"reasoning"}).values()) else "fail")

Output on a case where the assistant answered a key rotation question but invented a flag that appears nowhere in the docs:

{'grounded': True, 'responsive': True, 'no_invention': False, 'scoped': True}
fail

Run that over a few hundred real answers and you will meet this, probably within the first fifty:

Traceback (most recent call last):
  File "judge.py", line 38, in judge
    return Verdict.model_validate_json(resp.choices[0].message.content)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Verdict
grounded
  Input should be a valid boolean, unable to interpret input
  [type=bool_parsing, input_value='partially', input_type=str]

Obvious response: add a validator that coerces partially to False and move on. Wrong response. A bool_parsing error on a judge is nearly always a rubric defect rather than a parsing defect. Read the reasoning field and you find the model was right to hedge, because grounded was doing two jobs at once: are the claims supported, and are the citations correct. An answer can be supported by a chunk it failed to cite. Splitting that criterion into grounded and cited removed the hedging entirely, and cost me two of the four points I later gained on agreement. Every time a judge refuses to give you a boolean, it is telling you the question was compound.

Measuring your judge against human labels

Here is the step almost everyone skips, and it is the whole point of this part. Two support engineers labelled 200 real answers pass or fail, independently, with a third breaking the 14 ties. That took them a little over five hours between them. Then I compared my judge to those labels.

# calibrate.py -- scikit-learn 1.7.2
import json
from sklearn.metrics import cohen_kappa_score

rows  = [json.loads(line) for line in open("labelled.jsonl")]
human = [r["human_pass"] for r in rows]
model = [r["judge_pass"] for r in rows]

agree = sum(h == m for h, m in zip(human, model)) / len(rows)
print(f"n={len(rows)} agreement={agree:.3f} kappa={cohen_kappa_score(human, model):.3f}")
print(f"human pass rate={sum(human)/len(human):.3f} "
      f"judge pass rate={sum(model)/len(model):.3f}")

Version one of the judge, a single 1 to 5 score with 4 and above counted as a pass, printed this:

n=200 agreement=0.710 kappa=0.322
human pass rate=0.600 judge pass rate=0.860

Seventy one percent agreement reads like a passing grade until you sit with the second line. Humans passed 60 percent of answers; my judge passed 86 percent. A judge hardcoded to return pass, with no model behind it at all, would have scored 60 percent agreement on this set. Cohen kappa is the statistic that removes that free credit, and 0.322 says the judge was adding about a third of the available signal above guessing. Landis and Koch call that fair, which is a generous word for an instrument I had been reading off a dashboard for two weeks.

Version two replaced the single score with the four binary criteria shown earlier, and printed n=200 agreement=0.890 kappa=0.767 against the same labels, with a judge pass rate of 0.640 against the human 0.600. My gate is kappa 0.6. Below that I do not let a judge write to a dashboard, because a number nobody can trust is worse than a blank space where people know to go and read answers instead.

Failure modes of LLM as judge

Zheng and colleagues catalogued these in the MT-Bench paper back in 2023, and the list has aged well: position bias, verbosity bias, self enhancement bias. Later work quantifying judge biases adds style bias, which several groups now find larger than position bias. My own encounter was with position, and it was embarrassing. I had switched to pairwise comparison for choosing between two prompt versions, prompt B won 61 percent of 150 comparisons, and I shipped it. A colleague asked what happened if I swapped the order. Twenty minutes of work:

# swap_test.py -- pairwise(...) returns the string 'first' or 'second'
sticky = 0
for case in pairwise_cases:                      # 150 cases, two candidates each
    a = pairwise(case.q, case.ctx, case.x, case.y)
    b = pairwise(case.q, case.ctx, case.y, case.x)   # same pair, order reversed
    if a == b:                                   # same SLOT won twice, not same answer
        sticky += 1

n = len(pairwise_cases)
print(f"order decided the verdict: {sticky}/{n} = {sticky/n:.1%}")

order decided the verdict: 33/150 = 22.0%

Twenty two percent of my verdicts were about slots rather than answers, and prompt B had been in the second slot every single time. Once I ran both orders and scored a disagreement as a tie, B won 52 percent, which is inside the noise on 150 cases. I had shipped a prompt change on the strength of an ordering artefact. That cost a week, and the fix is four lines. Table below is the one I keep pinned, and it is the artefact worth taking from this part.

Judge ensembles and majority voting

Everyone asks this next, so let me answer it with the measurement rather than an opinion. If one judge is noisy, why not run three and take the majority. I tried exactly that: three judges, two model families, majority vote per criterion, against the same 200 human labelled answers. Kappa went from 0.767 to 0.791. Cost went up threefold and p50 latency on the 200 case run went from 70 seconds to 194 seconds. That is a 3 percent gain on the metric for a 200 percent increase in spend, and on a set of 200 cases a 0.024 kappa difference is well inside the sampling noise anyway.

One trap is worth naming, because I fell into it first. My initial ensemble was the same model called three times, which at temperature 0 returns the same verdict three times, so the vote is unanimous by construction and tells you nothing. I spent an afternoon admiring a 100 percent consensus rate before working out that I had simply paid for the same answer three times. A meaningful ensemble needs either different model families or nonzero temperature, and nonzero temperature reintroduces exactly the run to run variance you were trying to remove. Determinism and diversity pull against each other here; sampling behaviour and why temperature does this is covered in the GenAI Series piece on hallucination and temperature.

Where an ensemble does earn its cost is building the calibration set itself. Run two judges from different families over a few hundred candidate answers, then send only the cases where they disagree to a human. On our corpus that was 47 of 300, which cut the labelling effort by roughly six sevenths and gave the humans the interesting cases rather than the obvious ones. Ensemble to spend human attention well, not to squeeze a third of a point out of a CI gate.

Cost and latency of running a judge

One judgement on our corpus costs 1,025 input tokens, which breaks down as roughly 25 for the question, 600 for the retrieved chunks, 220 for the answer and 180 for the rubric, plus about 140 output tokens once you allow room for reasoning. Multiply out and the numbers stop being scary. Rates assumed below are $0.25 and $2.00 per million input and output tokens for a small tier, and $3.00 and $15.00 for a frontier tier; provider list prices move often, so check the current pricing page before you quote these [VERIFY]. Latency figures are measured p50 at eight way concurrency against a 200 case set.

Calibration loop and when to retire a judge

Calibration is not a launch task, it is a standing one. Your documentation changes, your traffic mix shifts, and one day a provider silently retires the model your judge runs on. Any of those can move kappa without moving a line of your code. My loop runs quarterly and on every model change, and it costs about ninety minutes of two people’s time.

Wire the gate into the same pytest suite that runs your eval set, so an uncalibrated judge cannot quietly start grading:

# test_judge_calibration.py
MIN_KAPPA = 0.60
MAX_LABEL_AGE_DAYS = 100

def test_judge_is_calibrated(calibration_set):
    assert calibration_set.age_days <= MAX_LABEL_AGE_DAYS, "labels are stale, resample"
    k = cohen_kappa_score(calibration_set.human, calibration_set.judge)
    assert k >= MIN_KAPPA, f"judge kappa {k:.3f} below gate {MIN_KAPPA}"

Start with binary rubrics and a kappa gate of 0.6

My pick is a per criterion binary rubric on a small model tier, calibrated against 150 to 200 human labelled cases, gated at kappa 0.6. It costs 54 cents per thousand judgements, it runs a 200 case set in seventy seconds, and it survived a rubric rewrite and a model change without needing a redesign. What I would avoid is the single 1 to 5 holistic score, which is the shape both OpenAI’s score_model grader example and Anthropic’s grading tips nudge you toward. On open ended support answers that scale collapsed onto two buckets and gave me an instrument with a kappa of 0.322, and I had no way of knowing that until I paid two engineers for five hours of labelling. Pairwise sits in the middle: excellent for choosing between two prompt versions, unusable until you run both orders, and wrong for CI.

One thing to do on Monday: take fifty outputs your current judge has already scored, label them yourself pass or fail without looking at its verdicts, and compute Cohen kappa. It is twenty lines of code and an hour of reading. If the number comes back under 0.4, you have been steering by an instrument that is barely better than a coin, and you now know it before your users tell you. Part 22 takes a calibrated judge and turns it into a regression gate, so a prompt edit or a model upgrade cannot silently make the assistant worse. Bring your kappa number with you.

References

• Graders, OpenAI documentation, for the grader taxonomy of string_check, text_similarity, score_model and Python graders, the score_model schema with its range and pass_threshold fields, and the note that non numeric grader output defaults to 0.
• Define success criteria and build evaluations, Anthropic documentation, for the ranking of code based, human and LLM grading, the detailed rubric guidance, and the recommendation to have the judge reason before scoring and then discard the reasoning.
• Judging LLM as a Judge with MT-Bench and Chatbot Arena, Zheng et al., NeurIPS 2023, for the original catalogue of position, verbosity and self enhancement bias and the finding that strong judges reach over 80 percent agreement with human preference.
• Justice or Prejudice, Quantifying Biases in LLM as a Judge, for the systematic measurement of judge biases beyond the original three.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts