A Granite prompt scored 87 on a public summarization benchmark, so the team shipped it. Two weeks later support tickets showed the same prompt inventing account numbers on our own documents. The benchmark was real. It just measured a task that was not ours, on data that was not ours, against references we never wrote. That gap, between a leaderboard number and the number that matters for your use case, is what evaluation on watsonx exists to close.

Why leaderboard scores mislead you

A benchmark is a fixed test set with a fixed scoring rule, run once so different models get one comparable number. That is useful for the people who build models. It is close to useless for deciding whether your prompt works, and the reason is simple. A leaderboard measures a generic task on generic data. Your application answers questions about your policies, your product names, your contract clauses. A model that tops a summarization board can still miss the one field your support agents need, because that field never appeared in the board’s data.

There is a second trap. Most public boards score against a single reference answer with a text-overlap metric. Real answers can be correct and phrased nothing like the reference, or fluent and confidently wrong. Overlap does not see either case. So the score moves for reasons that have little to do with whether a user got helped. On watsonx the fix is not a better leaderboard. It is measuring your task, on your data, with metrics chosen for that task, and doing it repeatably.

watsonx.governance is IBM’s layer for that, a service that evaluates and monitors models from any vendor, watsonx.ai as well as models on Amazon Bedrock, Amazon SageMaker, Google Vertex AI, and Microsoft Azure. This part uses it for the offline job: score and compare candidates before you ship. The live monitoring job, watching quality drift in production, is the governance work we reach in a later part; here the goal is to pick the right prompt and model in the first place.

What watsonx measures, metric by metric

watsonx.governance groups metrics by the task type you declare for the prompt. Declare the task wrong and it offers the wrong metrics, so this choice comes first. The supported task types are classification, summarization, generation, question answering, entity extraction, and retrieval augmented generation. Each pulls a different metric set.

Two families run underneath. Content quality metrics compare output text to a reference you supply: ROUGE, which counts overlapping word sequences against a reference summary and reports rouge1, rouge2, and rougeLSum; BLEU and METEOR and SARI, related overlap scores; exact match; and sentence similarity by Jaccard or cosine distance. These need a labeled reference output for every test row, which is the expensive part. Gen AI quality metrics judge the output on its own or against retrieved context: faithfulness, answer relevance, context relevance, answer similarity, and the unsuccessful answer rate. Alongside both sit safety and readability checks, PII detection, HAP for hate, abuse, and profanity, and a readability grade, all computed automatically with no reference needed.

For anything RAG-shaped, three metrics carry most of the weight, and they split the blame cleanly. Context relevance asks whether retrieval returned passages that actually bear on the question, so it grades your retriever, not your model. Faithfulness asks whether the answer is grounded in those passages, and it returns attributions, the specific source sentences that support the claim, so a low score points you at the exact hallucinated line. Answer relevance asks whether the answer addresses the question at all. Read together they tell you which stage failed. Bad context relevance and good faithfulness means the retriever missed and the model honestly worked with junk. Good context relevance and bad faithfulness means the passages were there and the model made something up anyway.

Evaluation Studio, comparing prompts and models side by side

Evaluation Studio is the browser tool in watsonx.governance for exactly one question: of my candidates, which is best. A candidate is a prompt template, meaning a saved prompt with named variables and a chosen model behind it. You select up to five, upload one test dataset, pick the metrics, and run a single experiment that scores all of them on the same rows. That last detail is the point. Same data, same metrics, one run, so the comparison is fair rather than five separate tests you eyeball later.

The results view is built for choosing. You set one template as the reference, and every other template’s cells highlight better or worse against it. Scores that cross a threshold you set are flagged, so a template that quietly leaks PII does not hide behind a strong faithfulness number. You can build a custom ranking, weighting the metrics that matter for your use case and letting a formula order the candidates, which beats arguing about which single metric wins. Every run is captured in an AI factsheet, a tracked record of what was evaluated, on what data, with what result, so the choice is auditable later.

The guardrails are worth knowing before you plan a run. You must evaluate at least two templates and at most five. They must share a task type, and the same variable names mapped to the same test columns. Detached prompt templates, the ones that point at a model hosted outside watsonx, cannot go through Studio; those you score with the SDK below. Reference-based metrics like ROUGE and BLEU only compute if your test data carries a reference output column. And you need the Editor or Admin role on the project. None of this is heavy, but a run that fails at step six because two templates named a variable differently wastes twenty minutes.

flowchart LR
  G[Golden test set] --> S{Where}
  S -->|browser, compare 2 to 5| E[Evaluation Studio]
  S -->|code, gate a build| K[ibm-watsonx-gov SDK]
  E --> T[Thresholds and ranking]
  K --> T
  T --> F[AI factsheet]
  T -->|pass| P[Ship candidate]
  T -->|fail| R[Fix prompt or retriever]
  R --> G

Which metrics for which task

Picking metrics is where most evaluations go soft. The task type narrows the list, but you still choose, and the wrong choice produces a confident number about the wrong thing. A few rules I hold to. For a RAG assistant, lead with context relevance and faithfulness and treat answer relevance as a floor, because a grounded, on-topic answer is the whole job and hallucination is the failure that hurts. For classification or entity extraction, use exact match and the F1, precision, and recall trio, because there is a right answer and overlap metrics only muddy it. For summarization where a house style exists, ROUGE against a couple of gold summaries is fair; where it does not, drop ROUGE and score faithfulness and readability instead.

Two metrics belong on nearly every run regardless of task. PII, because a model that repeats a customer’s identifiers is a compliance problem no quality score offsets, and it is the kind of thing Granite Guardian from Part 10 also watches at runtime. HAP, for the same reason on toxic output. These cost you nothing to add and catch the failures that end up in an incident review rather than a quality report.

Scoring a RAG prompt with the SDK

The browser is fine for a one-time bake-off. For anything that ships more than once you want evaluation in code, so it runs in the same pipeline that builds the app and blocks a merge when quality drops. The ibm-watsonx-gov Python SDK computes the same gen AI quality metrics on a dataframe. Here is the shape of a RAG scoring run over a small set of rows, each with a question, the retrieved context, and the generated answer.

# pip install ibm-watsonx-gov
import pandas as pd
from ibm_watsonx_gov.evaluators import MetricsEvaluator
from ibm_watsonx_gov.metrics import (
    FaithfulnessMetric, AnswerRelevanceMetric, ContextRelevanceMetric,
)

# one row per test question; context is the retrieved passages
df = pd.DataFrame({
    "question": ["What is the refund window on a damaged item?"],
    "context":  ["Damaged goods may be returned within 30 days of delivery."],
    "generated_text": ["You have 30 days from delivery to return a damaged item."],
})

evaluator = MetricsEvaluator()
result = evaluator.evaluate(
    data=df,
    metrics=[FaithfulnessMetric(), AnswerRelevanceMetric(), ContextRelevanceMetric()],
)
print(result.to_df())

# gate the build: fail CI if any row is ungrounded
scores = result.to_df()
assert scores["faithfulness"].min() >= 0.7, "ungrounded answer in test set"

Expected output: a dataframe with one row per question and a column per metric, each scored 0 to 1, for example faithfulness 0.98, answer_relevance 0.95, context_relevance 0.99. The final assert passes, so CI stays green.

Failure mode: if the context column is empty or the passages do not mention the answer, faithfulness drops toward 0 and the assert throws, failing the build. That is the point. A MissingValueError or a 401 instead means credentials or the column names are wrong, not the model, so check those before you blame the prompt. The exact class and method names track the installed SDK version, so pin it.

Reading the numbers without fooling yourself

A single averaged score hides the answer you need. The worked example above averages fine, but the decision lives in the shape. Plot the two prompts on the three RAG metrics and the trade is obvious in a way a table of means never is.

Three habits keep the numbers honest. Look at the worst rows, not the mean, because one row scoring 0.2 on faithfulness is a hallucination a user will meet, and a 0.86 average buries it. Hold the retriever fixed when you compare prompts, and hold the prompt fixed when you compare retrievers, or you cannot tell which change moved the score. And be skeptical of tiny gaps. A two-point difference on fifty rows is noise; I want a clear separation or a bigger set before I let a number decide.

Benchmarking across models, and what it costs

Comparing models is the same machinery pointed at a different variable. Hold the prompt and the test set fixed, swap the model behind each template, and you get a fair read on whether Granite 8B is enough or a larger model earns its cost on your task. This is where an evaluation on your own data pays off twice, because the honest answer is often that the smaller, cheaper model clears your threshold and the expensive one buys a rounding error.

Evaluation itself is not free, and the cost scales in a way that surprises people. watsonx.governance bills evaluation by the experiment, and the resource an experiment consumes climbs with three things multiplied together: the number of templates, the number of test rows, and the number of metrics. Reference-free gen AI metrics that call a judge model cost the most per row. Five templates times two hundred rows times a full metric panel is not five times one template; it is a much larger job. The scaling is worth picturing before you queue a run.

The practical move is to shortlist before you spend. Run a cheap first pass, two candidates, a small metric panel, the golden set, then only take the survivors into a bigger run with the full panel and more rows. IBM has also been extending this into agentic evaluation, scoring whole tool-using agents rather than single prompts, which matters once your app is the agent you built in Part 15 rather than one call. If you want to see how another platform frames the same offline-then-online split, the Azure Foundry evaluation part covers it from the Microsoft side.

Build the golden set before you pick a model

If you do one thing from this part, write the golden set first, before you compare a single model. Fifty to two hundred questions from your own domain, each with the answer you would accept, split across the ordinary cases and the ones that scare you. Then the tooling falls into place. Evaluation Studio picks between prompts in an afternoon, the SDK turns that judgment into a CI gate that fails a bad merge automatically, and the factsheet gives you the paper trail when someone asks why this model and not that one. The leaderboard was never going to answer that. Your fifty rows will.

Start this week by writing twenty questions and their accepted answers, and run your current prompt against them in Evaluation Studio. The number you get back is the first true measurement of your app, and everything after is improving it. The next parts move from picking a model to watching it in production, where these same metrics become live monitors under watsonx.governance.

References

• Comparing AI assets with Evaluation Studio, IBM Documentation
• Evaluate metrics with the ibm-watsonx-gov SDK, IBM watsonx docs
• Generative AI quality evaluations, IBM watsonx docs
• watsonx.governance model governance, IBM

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts