Shipping a prompt change because the new answer read better on two hand-picked examples is how a regression reaches production. The two examples you looked at improved. The forty you did not look at got worse, and the first you hear of it is a support ticket. A spot check is not an evaluation. It is a guess with a nice feeling attached, and it does not survive contact with a model swap, a temperature tweak, or a retrieval change. You need a number that moves when quality moves, computed the same way every time.
What the service actually measures
The Gen AI evaluation service is a managed part of Vertex AI that takes a set of prompts, the responses your model or agent produced, and a list of metrics, then returns a score per row and an aggregate across the set. You give it a table. Each row has the input, the generated response, and optionally a reference answer you consider correct. It hands back the same table with a score column per metric, plus a summary. That is the whole shape. Everything else is which metric you pick and whether the score comes from a formula or from another model reading the answer.
Two things make it worth using over a script you write yourself. It ships built-in metrics with tested definitions, so groundedness or instruction following means the same thing across your team and across runs. And it records every run in Vertex AI Experiments, the tracking store, so you can line up today’s score against last week’s and see the direction. A number you cannot compare to a prior number is not much use. The service exists to make the comparison honest and repeatable.
Three kinds of metric
Pick the metric family before you write a line of code, because it decides what data you need. Computation metrics compare your response to a reference answer with a formula. Exact match, ROUGE for overlap of text spans, BLEU for translation-style overlap. They are cheap, deterministic, and need a ground-truth reference for every row. They work when there is one right answer, a classification label, a known summary, a fixed extraction, and they fall apart the moment the task is open-ended, because two good answers can share almost no words.
Model-based metrics fix that by having another model read the answer and judge it. That judge is called an autorater. A pointwise metric asks the autorater to score one response against written criteria, on a scale of 0 to 5, so you learn how good this answer is on fluency, coherence, groundedness, or safety. A pairwise metric shows the autorater two responses and asks which is better, which is the right tool when you are choosing between two models or two prompts and absolute scores do not matter, only the winner. Pointwise tells you the level. Pairwise tells you the direction of a change.
| Metric family | How the score is made | Needs a reference? | Reach for it when |
|---|---|---|---|
| Computation | A formula compares text to a reference (exact match, ROUGE, BLEU) | Yes, always | One right answer exists: labels, extraction, fixed summaries |
| Pointwise model-based | An autorater scores one response 0 to 5 on written criteria | Optional | You want a level for quality, safety, or groundedness |
| Pairwise model-based | An autorater compares two responses and picks the better one | Optional | You are choosing between two models or two prompts |
Pointwise scoring with an autorater
A pointwise run is the workhorse. You define a metric as a prompt template that tells the autorater what to reward and what to penalize, the service sends each response through the judge, and you get a score from 0 to 5 with a short rationale for each row. The built-in templates cover the common axes: fluency reads whether the language is clean, coherence whether it hangs together, groundedness whether every claim is supported by the provided context, instruction following whether it did what was asked, and safety whether it stayed inside policy. You can also write your own template when your quality bar is specific, say a legal-tone metric or a metric that penalizes any answer that invents a policy number.
The reason to score several axes rather than one overall number is that they move independently, and the average hides it. Here is a run I would not have caught with a single score: a prompt change lifted fluency and coherence but dropped groundedness, because the smoother phrasing came from the model filling gaps with plausible invention. The overall average barely moved. The per-axis view showed groundedness falling almost a full point, which is the one axis a support bot cannot afford to lose. Score the axes separately and read them separately.
When should you use pairwise instead?
Reach for pairwise the moment the question is which one wins rather than how good is this. Absolute scores drift. An autorater that scores a batch at 4.1 today might score the same batch at 4.0 next week after a judge model update, and that wobble can swamp the small real change you are trying to detect. Pairwise sidesteps it. You show the judge both responses for the same prompt and ask it to pick, so a systematic bias in the judge cancels out because it applies to both sides. For A versus B decisions, the swap from an old model to a new one, prompt v3 against prompt v4, pairwise gives you a cleaner signal and a win rate you can read at a glance.
The catch is that pairwise does not give you a level. A 70 percent win rate says the new prompt beats the old one most of the time; it says nothing about whether either is good enough to ship. So the pattern I use is both. Pairwise to decide the change is an improvement, pointwise on the axes that matter to confirm the absolute quality clears the bar. Use pairwise to choose, pointwise to gate. For readers who run more than one cloud, this is the same judge-model idea Amazon calls LLM-as-a-judge on Bedrock, which I covered in the AWS evaluation part; the mechanics differ, the trap is identical.
Rubric evaluation and Gecko for media
A flat pointwise score has a weakness: one number for a whole answer buries which part was wrong. Rubric-based evaluation breaks the judgment into pieces. The service first generates a set of specific, checkable rubrics for each individual data point, small yes-or-no criteria drawn from that prompt and response, then the autorater checks the answer against each rubric and reports which passed. Instead of coherence is 3, you learn that the answer named the right policy, missed the effective date, and invented a phone number. That is a debuggable result. You can see the failing criterion and fix the prompt or the retrieval that caused it.
For image and video generation the autorater is a purpose-built one called Gecko, a rubric-based and interpretable autorater for generative media. Gecko uses a Gemini model to break the input prompt into semantic pieces, the entities, their attributes, and the relationships between them, then checks the generated image or video for each piece. So a prompt for a red bicycle beside a blue door gets scored on whether the bicycle is present, whether it is red, whether the door is present and blue, and whether the two are positioned as asked. That is far more useful than a single aesthetic score, because it tells you what the model dropped, which is usually one attribute or one spatial relationship rather than the whole scene.
Evaluate the judge before you trust it
Here is the step almost everyone skips, and it is the one that makes the rest worth anything. The autorater is a model, and models are wrong sometimes. If your judge disagrees with a human on which answer is better, every downstream number it produces is noise dressed as signal. So before you gate a release on an autorater, you evaluate the autorater itself. Build a small benchmark of examples that humans have already rated, run the judge over the same examples, and measure how often the judge agrees with the human verdict. If agreement is high, you can trust the judge to scale to thousands of rows no human will ever read. If it is low, fix the metric template or pick a stronger judge model before you rely on it.
This is not busywork. It is the difference between an evaluation you can defend and a plausible-looking dashboard that quietly steers you wrong. A hundred human-labeled examples is enough to start. Measure agreement, and re-measure it whenever you change the judge model or the metric definition, because a judge update can shift agreement without any warning in the score itself. Trust the autorater exactly as far as you have measured it against people, and no further.
Run an evaluation in the SDK
The code is short because the service does the heavy lifting. You put your prompts and responses in a table, name the metrics, and call EvalTask. This is a bring-your-own-response run, meaning you already generated the responses and just want them scored, which is the mode I use most because it separates generation from grading. The example scores a small set on two built-in pointwise metrics.
import pandas as pd
from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples
# Bring-your-own-response: prompts and the answers you already generated
eval_df = pd.DataFrame({
'prompt': [
'Summarize our refund window in one sentence.',
'What is the parental leave policy?',
],
'response': [
'Refunds are accepted within 30 days of delivery.',
'Staff get 16 weeks of paid parental leave after 12 months.',
],
})
task = EvalTask(
dataset=eval_df,
metrics=[
MetricPromptTemplateExamples.Pointwise.INSTRUCTION_FOLLOWING,
MetricPromptTemplateExamples.Pointwise.GROUNDEDNESS,
],
experiment='refund-policy-eval', # tracks the run in Vertex AI Experiments
)
result = task.evaluate() # autorater defaults to a Gemini judge model
print(result.summary_metrics) # aggregate scores across the set
print(result.metrics_table) # per-row scores and rationaleExpected output: summary_metrics prints a mean score per metric, for example an instruction_following mean near 4.5 and a groundedness mean the judge computes from the text, and metrics_table gives a per-row score with the autorater’s reason. Failure mode: pass a groundedness metric with no context column and the judge grades against nothing, returning scores that look fine and mean nothing. Give groundedness the retrieved context each answer used, or do not run it. Import path and metric enum names are current for the Vertex evaluation SDK.
Agents need trajectory metrics, not just answers
An agent can reach the right answer the wrong way, and a final-answer score will not catch it. If your support agent calls the refund tool, then the order-lookup tool, then answers correctly, the answer looks fine while the agent did a needless and possibly costly step. The evaluation service scores the trajectory, the ordered sequence of tool calls the agent made, against a reference trajectory you consider correct. It works across frameworks, agents built with the Agent Development Kit, LangGraph, or CrewAI, hosted locally or on Agent Engine, so the multi-agent graph from Part 21 evaluates the same way a single agent does.
The trajectory metrics come in a few shapes, and the one you pick encodes how strict you want to be about the path. Exact match wants the same tool calls in the same order. In-order match allows extra calls as long as the required ones happen in order. Any-order match drops the ordering requirement. Precision asks what fraction of the calls the agent made were actually relevant. Single tool use just checks that a specific tool was called at all. Score the trajectory and the final answer together, because a right answer down a wrong path is a bug waiting for a harder question.
| Trajectory metric | Passes when | How strict |
|---|---|---|
| trajectory_exact_match | Same tool calls, same order, nothing extra | Strictest |
| trajectory_in_order_match | Required calls in order, extra calls allowed | Strict |
| trajectory_any_order_match | Required calls present, order ignored | Loose |
| trajectory_precision | Fraction of made calls that were relevant | Graded 0 to 1 |
| trajectory_single_tool_use | A named tool was called at least once | Narrow check |
flowchart LR
D[Eval dataset: prompts and references] --> R[Generate or bring responses]
R --> M{Metric type}
M -->|Computation| F[Formula vs reference]
M -->|Pointwise| J[Autorater scores 0 to 5]
M -->|Pairwise| P[Autorater picks winner]
F --> X[Vertex AI Experiments]
J --> X
P --> X
X --> G{Score clears bar?}
G -->|Yes| S[Ship]
G -->|No| B[Block and fix]
What an evaluation run costs
The bill has two parts, and people worry about the wrong one. Model-based metrics call a judge, and the judge charges for tokens like any Gemini call. But the judge prompt is small: the input, the response, the criteria, and a short rationale back. On a fast judge model that is a fraction of a cent per row. The part that actually costs is regenerating candidate responses across the whole set on every run, especially with a large or reasoning-heavy model, plus the human hours to label the benchmark that validates the judge. Optimize the thing that is expensive, which is not the judge.
Worked example
Take a pointwise run on a fast Gemini judge. Each judged row sends about 1,500 input tokens and gets about 250 output tokens back. At roughly $0.30 per million input tokens and $2.50 per million output tokens, that is about $0.0011 a row. So a 200-row smoke test costs about $0.22, a 1,000-row nightly run about $1.10, and a 5,000-row release gate about $5.50 in judge tokens. Pairwise puts two responses in the prompt, so budget closer to $0.0016 a row, still cents.
Now the real number. Regenerating 5,000 candidate responses on a large model at, say, 800 output tokens each is the line item that dwarfs the judge, and a human labeling 100 benchmark examples to validate the autorater costs more in salary than a year of nightly judge runs. Keep the nightly set small and representative, run the 5,000-row gate only before a release, and spend the saved money on better human labels.
Wire one eval gate before you swap the model
Do not try to evaluate everything at once. Start with one gate. Pick the single quality axis your product cannot lose, groundedness for a support bot, instruction following for a form filler, and build a small dataset of 100 to 200 real prompts with the answers you would accept. Score them pointwise on that one axis, and before you trust the number, spend an afternoon checking the autorater against your own human verdicts on those rows. Once the judge agrees with you, put the run in your pipeline so a prompt change or a model swap that drops the axis fails the build instead of reaching users. One measured gate beats ten metrics nobody validated.
From there, grow it deliberately. Add pairwise when you are choosing between two models so the judge’s drift cancels out. Add trajectory metrics when an agent starts calling tools. Keep the nightly set small and the release-gate set broad. The next part moves from a single answer to everything the model can produce beyond text, the multimodal side with Veo and Imagen, where Gecko-style rubric scoring is how you keep image and video quality honest. Before you touch the next model version, wire the one gate you can defend, and let it tell you the truth your spot check was hiding.
References
Run an evaluation, Gen AI evaluation service (Google Cloud docs)
How to evaluate your gen AI at every stage (Google Cloud Blog)
Evaluate your AI agents with Vertex Gen AI evaluation service (Google Cloud Blog)
Evaluate your gen media models on Vertex AI, Gecko autorater (Google Cloud Blog)
About The Author
DrJha