An agent that passes every demo can still be wrong on real traffic. I have watched one clear a curated test set and then slip badly within a week of going live, because the questions users actually ask never look like the ones you wrote for the demo. Evaluation is how you catch that slip. Observability is how you find out why it happened, before it turns into a support queue.

What evaluation and observability actually mean here

Two words get used loosely, so let me pin them down. Evaluation means scoring the output of your application against a metric. You give it a question, the answer your app produced, and often the context it retrieved, and an evaluator returns a number. Observability means being able to see what happened inside a run, which documents the retriever pulled, how many tokens the model used, which tools the agent called and in what order. Evaluation tells you the answer was weak. Observability tells you the retriever returned the wrong section.

An evaluator is just a scoring function. Some are rule based, like checking whether a tool call matched the expected arguments. Most of the interesting ones are AI assisted, meaning a second model reads the answer and grades it against a rubric. A trace is the record of one end to end run, made of spans, where each span is one step, a retrieval, a model call, a tool invocation. Foundry collects both, and the design point that matters is that the same evaluators run everywhere, so a groundedness score from your laptop means the same thing as one from production.

This used to need three different tools. You would test locally with one script, gate in CI with another, and monitor production with a dashboard that measured something unrelated. When the definitions differ, you cannot tell whether a regression is real or just a change in how you measured. Foundry closing that gap is the real improvement, more than any single metric.

Quality, safety, and agent evaluators

Built in evaluators fall into three families, and knowing which family you need keeps you from measuring the wrong thing. Quality evaluators judge the answer itself. Groundedness checks whether the response is supported by the retrieved context, which is the single most useful metric for any RAG app because it catches the model inventing facts. Relevance checks whether the answer addresses the question. Coherence and fluency judge how the text reads. Retrieval scores how good the pulled chunks were, separate from what the model did with them.

Safety evaluators are a different mechanism. Hate and unfairness, violence, sexual content, and self harm detectors are backed by the Foundry safety service rather than a rubric you write, and they return severity levels. Indirect attack detection, sometimes called cross prompt injection, checks whether text hidden in a retrieved document tried to hijack the model. Protected material detection flags copyrighted text. These tie back to the content filters from Part 14, but here you run them as measurements across a whole test set rather than as a live gate on one request.

Agent evaluators are the newest family and the reason this part sits after the agent parts. Intent resolution measures whether the agent understood what the user wanted. Tool call accuracy checks whether it called the right tool with the right arguments. Task adherence measures whether it followed its instructions and finished the job. If you built anything with the Agent Service in Part 13 or Copilot Studio in Part 22, these are how you grade it. Microsoft reported meaningful accuracy and speed gains on the intent resolution and tool call evaluators in recent SDK releases.

The chart is one prompt and retriever fix on a RAG agent, scored before and after on the same set. That is the usual shape. Fluency barely moves because the model was always articulate, groundedness jumps because the answer is now anchored to the right text.

How tracing ties a bad answer to its cause

A score without a trace is a dead end. You know groundedness dropped, but not whether the retriever failed, the context window truncated, or the model ignored good context. Tracing answers that. Foundry tracing is built on OpenTelemetry, the open standard for distributed traces, and writes to Azure Monitor Application Insights, so your AI traces sit next to the rest of your application telemetry instead of in a separate silo.

A single run becomes a tree of spans. The top span is the request. Under it you see the retrieval span with the query and the chunks it returned, the model span with token counts and latency, and a span for each tool the agent called. When an answer is wrong, you open its trace and read down the tree until you find the step that went sideways. Nine times out of ten it is visible in seconds, the retrieval span shows the wrong document, or the tool span shows a bad argument.

Foundry emits these traces for the common frameworks without much wiring, including LangChain, LangGraph, the OpenAI Agents SDK, and the Microsoft Agent Framework from Part 21. The Agent Monitoring Dashboard then aggregates the same data into token usage, latency, and success rates per agent, so you get both the single run view for debugging and the fleet view for spotting a trend.

Running your first evaluation in the SDK

The portal has a click through evaluation flow, and it is fine for a one off. But evaluation earns its keep when it runs in code, on a schedule or in a pipeline, so here is the shape in the Azure AI Evaluation SDK. You install the package, point it at a dataset of test cases, and pass a dictionary of evaluators to the evaluate function. Field names shift across SDK versions, so treat this as the shape and check the current reference before you ship.

# pip install azure-ai-evaluation
from azure.ai.evaluation import (
    evaluate,
    GroundednessEvaluator,
    RelevanceEvaluator,
    ToolCallAccuracyEvaluator,
)

judge = {
    'azure_endpoint': 'https://YOUR-RESOURCE.openai.azure.com',
    'azure_deployment': 'gpt-4o-mini',
    'api_version': '2024-10-21',
}

result = evaluate(
    data='eval_set.jsonl',
    evaluators={
        'groundedness': GroundednessEvaluator(model_config=judge),
        'relevance': RelevanceEvaluator(model_config=judge),
        'tool_accuracy': ToolCallAccuracyEvaluator(model_config=judge),
    },
    azure_ai_project='https://YOUR-PROJECT.services.ai.azure.com/api/projects/eval',
    output_path='results.json',
)

print(result['metrics'])

Expected result: a dictionary of aggregate scores printed to your terminal, something like groundedness near 4.2, relevance near 4.4, and tool call accuracy near 0.83, with the per row scores uploaded to the Foundry project so you can read the failures, not just the averages.

Failure mode: leave the context field out of your JSONL and GroundednessEvaluator has nothing to check the answer against, so it errors or returns a blank score. Each evaluator needs specific columns, query and response for relevance, plus context for groundedness and recorded tool calls for the tool evaluator. A missing column fails that one evaluator, not the whole run, which is easy to miss if you only read the average.

Run this locally against a golden set of a few dozen hand checked cases before every release. Passing azure_ai_project uploads the scored rows to Foundry, which is the difference between knowing the average fell and knowing which five questions caused it.

From a CI gate to continuous evaluation on live traffic

The same evaluate call has two homes beyond your laptop. The first is your CI pipeline. You add a step that runs the golden set on every pull request and fails the build if groundedness drops below a threshold you set, say 4.0 on the five point scale. That turns quality into a gate, the same way you already gate on unit tests. A prompt change that quietly wrecks groundedness never reaches production, because the build goes red first.

The second home is production. Continuous evaluation samples live traffic, runs the evaluators on that sample, and pushes the scores to Azure Monitor. You do not evaluate every request, that would double your model bill for no reason. You sample, catch drift, and alert. Set an Azure Monitor alert on the groundedness metric and you learn about a regression from a page, not from a customer complaint two weeks later.

That dip is the pattern that makes continuous evaluation worth the wiring. The app still returned fluent answers the whole time, they were just wrong more often, so no error rate would have caught it. Only a quality metric on live traffic sees a regression that looks fine from the outside.

What continuous evaluation costs, and when it earns it

AI assisted evaluators call a model, so continuous evaluation has a running cost, and the lever is your sampling rate. Here is the math on a mid size workload.

The honest reading of that table is that sampling everything is a waste. A five to ten percent sample costs a few tens of dollars and catches drift within a day. Evaluating all of it costs ten times more and tells you almost nothing extra.

Red teaming before you ship

Evaluation on your own test set tells you how the app does on questions you thought of. Red teaming tells you how it does against an adversary. Foundry includes an AI Red Teaming Agent that runs automated attacks using Microsoft PyRIT, the open framework for probing model safety, and reports which attack categories got through. You point it at your deployment, it generates adversarial prompts across harm categories, and you get a scorecard of what leaked.

Run it before a launch and on a schedule after, because a prompt change or a new tool can reopen a hole you already closed. This is the safety counterpart to the quality gate. The quality evaluators keep the app useful, the red teaming keeps it from being embarrassing or dangerous. Neither replaces the content filters you run live from Part 14, they test that those filters actually hold under pressure.

Where I would start: three evaluators and one trace pipeline

If you are standing up evaluation from nothing, do not try to turn on all twelve evaluators and continuous monitoring in week one. Start with three, groundedness and relevance for quality, plus the one agent evaluator that matches what your app does, tool call accuracy for an agent, retrieval for a RAG app. Build a golden set of thirty to fifty hand checked cases, because a small honest dataset beats a large sloppy one. Wire those three into your CI pipeline as a gate with a threshold you can defend.

Turn on tracing to Application Insights the same day, with a redaction step, so the first time a score drops you already have the traces to explain it. Add continuous evaluation only after you have a stable baseline, and start at a five to ten percent sample. Save red teaming for the pre launch checklist and a monthly schedule. Next in the series I move from measuring text to measuring everything else, multimodal evaluation, where the inputs are images, audio, and documents rather than clean strings. Before you read it, open your pipeline config and check whether a single failing evaluation actually blocks a deploy today. If it does not, that is the first thing to fix.

References

• Microsoft Learn: Observability in generative AI
• Microsoft Foundry blog: Evaluations, monitoring, and tracing GA
• Azure AI Evaluation client library for Python
• Microsoft Learn: Agent Monitoring Dashboard

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts