My tuned Granite model scored 0.02 lower on MMLU than the base checkpoint it grew out of, and I shipped it anyway. Honest scoring means knowing which number is allowed to fall and which one is not, and MMLU was never the number that decided this model.

Where the assistant stands before evaluation

Last part multi phase training handed back one winner, checkpoint samples_1945, scored 6.81 on MT-Bench, and left a pile of lower scoring siblings. This part asks the harder question: 6.81 against what, and better at what. A number with no baseline is decoration, so evaluation here means scoring the tuned checkpoint and the base granite-7b-starter side by side, on the same benchmarks, and deciding honestly whether nine hours of tuning bought the support assistant anything real.

Two terms before any command. A benchmark is a fixed set of questions with known answers or a known way to grade them, run against a model to produce a comparable number. A judge model is a separate, stronger model that reads a candidate answer and scores it, used when the task has no single correct string, such as a multi turn support reply. Knowing which benchmark leans on a judge, and which grades itself against a fixed key, is half of reading these reports correctly.

Four benchmarks and what each one measures

InstructLab ships four core benchmarks, plus DK-Bench for custom questions. Two of them, mmlu and mt_bench, score a model against a public standard that has nothing to do with your data. MMLU is 57 fact based multiple choice tasks graded against a fixed key, so it needs no judge and returns a score from 0 to 1. MT-Bench asks 80 multi turn questions and has the judge model rate each answer out of 10. Neither can see your taxonomy, which is exactly why the branch variants exist. Keep the table below beside your first run; it is the fastest way to pick the right benchmark for a question.

Two of these, mmlu_branch and mt_bench_branch, are the ones that matter for a tuned assistant, because they measure the delta your taxonomy created rather than absolute standing against every other model on the internet. mmlu_branch has a teacher model generate multiple choice questions from the knowledge document in your taxonomy branch, then scores both models on those, so a rise means the tuned model absorbed facts the base did not have. mt_bench_branch uses your seed questions and seed answers as the reference and asks the judge to compare, which is the closest automated proxy for whether a support reply improved.

Running the branch benchmarks against the base

Both branch benchmarks take the tuned checkpoint, the base model, and the SDG output from Part 9. Point mmlu_branch at the node_datasets directory whose timestamp matches the JSONL files you trained on; a mismatched timestamp is the most common first error, covered below. Run this from the Git branch of your taxonomy where the qna.yaml lives.

flowchart LR
  A[Tuned checkpoint samples_1945] --> C[ilab model evaluate]
  B[Base granite-7b-starter] --> C
  C --> D[mmlu and mmlu_branch, knowledge]
  C --> E[mt_bench and mt_bench_branch, skills]
  D --> F[Compare tuned against base]
  E --> F
  F --> G[Ship decision on branch scores]

# RHEL AI 1.5, bundled InstructLab, judge prometheus-8x7b-v2-0
# knowledge delta: did the model learn the Acme Relay docs
$ ilab model evaluate --benchmark mmlu_branch 
    --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945 
    --tasks-dir ~/.local/share/instructlab/datasets/2026-07-28_205421/node_datasets_2026-07-28T20_54_21 
    --base-model ~/.cache/instructlab/models/granite-7b-starter

# KNOWLEDGE EVALUATION REPORT

## BASE MODEL (SCORE)
.../models/granite-7b-starter (0.71/1.0)

## MODEL (SCORE)
.../hf_format/samples_1945 (0.82/1.0)

### IMPROVEMENTS (0.0 to 1.0):
1. acme_relay_admin_guide: 0.71 -> 0.82 (+0.11)

That plus 0.11 is the whole point of Part 10 made numeric: on questions generated from the product documentation, the tuned model answers 82 percent correctly against the base at 71 percent. Now the skills side. mt_bench_branch needs the judge model and the taxonomy branch names, and it reports per leaf, so you see exactly which contributions helped and which regressed.

# skills delta: did the model learn the support reply shape
$ ilab model evaluate --benchmark mt_bench_branch 
    --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945 
    --judge-model ~/.cache/instructlab/models/prometheus-8x7b-v2-0 
    --branch acme-support --base-branch main

# SKILL EVALUATION REPORT

## BASE MODEL (SCORE)
.../models/granite-7b-starter (5.78/10.0)

## MODEL (SCORE)
.../hf_format/samples_1945 (6.00/10.0)

### IMPROVEMENTS (0.0 to 10.0):
1. compositional_skills/writing/grounded/support_reply/qna.yaml: 5.33 -> 6.67 (+1.34)

### REGRESSIONS (0.0 to 10.0):
1. foundational_skills/reasoning/temporal_reasoning/qna.yaml: 5.67 -> 5.00 (-0.67)

Read the regression, do not skip it. Teaching the support reply skill nudged temporal reasoning down two thirds of a point, a small tax that is normal when you tune hard on one leaf. For a support assistant that trade is fine, because reply quality is the job and temporal puzzles are not, but you only get to make that call if you actually read the regressions block instead of celebrating the improvement line. Standardized mmlu and mt_bench round out the picture, and mt_bench prints a per turn split worth a glance.

$ ilab model evaluate --benchmark mt_bench 
    --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945

# SKILL EVALUATION REPORT

## MODEL (SCORE)
.../hf_format/samples_1945 (6.81/10.0)

### TURN ONE (0.0 to 10.0):
7.02

### TURN TWO (0.0 to 10.0):
6.60

Turn two sits below turn one, 6.60 against 7.02, which is the usual pattern: models hold the first answer well and drift on the follow up. For a support assistant that handles multi turn tickets, that gap is the number to watch over time, because a session assistant that fades on the second question frustrates users even at a healthy average.

Reading scores without fooling yourself

Line the numbers up and a clear story appears. Standard MMLU barely moved, 0.74 base to 0.73 tuned, while mmlu_branch jumped from 0.71 to 0.82 and mt_bench rose from 6.42 to 6.81. That shape is exactly right for a knowledge tune: the model did not get broadly smarter, it got specifically smarter about your product, and the only benchmark that can detect the latter is the branch one. Chart it and the flat MMLU bar stops looking like a failure and starts looking like proof the tune was targeted.

Here is where honest scoring earns its name, with a mistake I have made. On an earlier project I shipped the checkpoint with the higher standard MMLU, 0.79 against 0.77, because it looked like the smarter model. A week later a support lead flagged that reply quality had slipped, and mt_bench_branch on our own skills had actually dropped 0.4 points on the build I shipped. I had optimised the one number that could not see the product and ignored the two that could. We rolled back to the lower MMLU build, and branch scores have gated every ship since. Picking the right yardstick is the same discipline the AI Engineering series frames under building an eval set before you build the feature, and the same reason the GenAI series treats evaluating GenAI output as its own hard problem.

Where evaluation breaks and how to tell

My first mt_bench run died after 40 minutes of answer generation, which is the worst place to fail because the model work was done and only the judging remained. I had not pulled the judge model onto this node. mt_bench and mt_bench_branch both stand up a vLLM server for prometheus-8x7b-v2-0, and without it the run collapses at the judging step, not at the start.

$ ilab model evaluate --benchmark mt_bench 
    --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945

Generating answers... 100%|##########| 80/80
Starting judge model server...
FileNotFoundError: Model path not found:
  ~/.cache/instructlab/models/prometheus-8x7b-v2-0

# pull the judge model, then re-run
$ ilab model download --repository docker://registry.redhat.io/rhelai1/prometheus-8x7b-v2-0 [VERIFY]
$ ilab model evaluate --benchmark mt_bench --model .../samples_1945
# SKILL EVALUATION REPORT ... (6.81/10.0)

One more trap that is not an error at all: variance. Run mt_bench three times on the same checkpoint and you will see scores like 6.81, 6.74 and 6.88, because the judge samples its own outputs. A 0.07 wobble is noise, not progress, so a build that scores 0.05 higher than the last one has not necessarily improved. Run the benchmark at least three times when a decision hangs on it, average, and treat anything under roughly 0.2 points as a tie. Keep this lookup beside the run and a red traceback becomes a two minute fix.

DK-Bench for questions your users actually ask

Branch benchmarks measure the delta from your taxonomy, but they do not ask the questions your staff type at 2pm on a Tuesday. DK-Bench closes that gap. You write a JSONL file of real questions and reference answers, and the judge grades each response from 1 to 5 against the reference, so you score the model on your ground truth rather than a generated proxy. This is the benchmark I trust most for a ship decision, because it is the one built from the tickets the assistant will actually see.

# one line per question, drawn from the real support ticket archive
$ cat ~/acme/support_questions.jsonl
{"user_input":"What is the default session timeout in Acme Relay 4.2?","reference":"It is 15 minutes, reduced from 30 in earlier releases."}
{"user_input":"How do I rotate an Acme Relay API key?","reference":"Rotate it under Settings, Security, API keys, then revoke the old key."}

$ ilab model evaluate --benchmark dk_bench 
    --input-questions ~/acme/support_questions.jsonl 
    --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945

# DK-BENCH REPORT

## MODEL: samples_1945
Question #1:  5/5
Question #2:  4/5
...
----------------------------
Average Score:  4.20/5
Total Score:    42/50

Score the base model on the same file and the comparison lands hard: base granite-7b-starter averaged 3.30 out of 5 on these questions, the tuned checkpoint 4.20. Nearly a full point per answer on the exact questions users ask is a result you can take to the business, and it is far more legible to a support lead than an MT-Bench decimal. Build this file once from your ticket archive, keep it in version control next to the taxonomy, and re-run it on every checkpoint; it becomes the held out set the whole series leans on. A term you have already met carries through here, since these graded answers are a small evaluation set in the same sense Part 9 built one for training.

Scoring the support assistant honestly

My verdict for this assistant is a three number gate. mmlu_branch tells me the product facts landed, mt_bench_branch tells me the reply shape improved without wrecking general skills, and a DK-Bench file built from real tickets tells me it answers what people actually ask. Standard MMLU stays on the dashboard purely as a tripwire; if it craters, something in training went wrong, but a flat or slightly lower MMLU on a good branch result is a pass, not a concern. I would avoid shipping on mt_bench alone, because a single run with 0.2 point variance can flip the wrong way and it never sees your domain.

Scored honestly, samples_1945 earned its place: plus 0.11 on product knowledge, plus 0.39 on general skills, plus 0.90 on real support questions, against a one hundredth dip on trivia it was never meant to learn. That is a model worth serving. Making it fast and cheap to serve, first the base serving story then serving at scale on OpenShift AI, is where Part 12 and Part 13 go next.

References

• Red Hat Enterprise Linux AI 1.5, evaluating the model
• InstructLab eval library, benchmark definitions
• Zheng et al, Judging LLM as a Judge with MT-Bench

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts