Azure Monitor already knows how many tokens your deployment burned last night. It does not know which user prompt caused the latency spike at 2am, and by default it never will. That gap is the whole subject of this part. Platform metrics are free and always on, but they are aggregate and anonymous, so the moment you need to answer which call did that you are into a second and a third layer of telemetry that you have to turn on, pay for, and lock down on purpose. Get the three layers straight and you can see everything. Turn them all on blindly and you get a compliance liability and a surprising Log Analytics bill.

Three layers of telemetry, and what each one can and cannot see

Every Azure OpenAI resource emits telemetry at three levels, and confusing them is the most common reason teams either miss a problem or overspend chasing one. The first level is platform metrics: numeric counters that Azure Monitor collects automatically the instant the resource exists. Nobody configures them, they cost nothing, and they answer how much and how fast in one minute buckets. The second level is resource logs, sometimes called diagnostic logs. These are per request records, one row per API call, and they do not exist until you create a diagnostic setting that routes them somewhere. The third level is application traces, the spans your own code emits through OpenTelemetry into Application Insights, which is the only layer that can tie a bad answer to the exact retrieval and prompt that produced it.

The reason this matters is that each layer sees a different slice, and none of them sees everything. Metrics know a 429 happened but not who got throttled. Resource logs know the call failed, the model, the status code, and the duration, but by default they do not carry the prompt or the completion. Only the trace layer, and only when you deliberately turn on message content capture, holds the actual text. So the question is never should I have observability, it is which layer answers the question I am actually asking, because the cost and the risk climb sharply as you move down the stack.

Table 1. The three layers side by side. Read it top to bottom as increasing detail, increasing cost, and increasing privacy exposure.

What Azure Monitor gives you for free

Open any Azure OpenAI resource, go to Monitoring and then Metrics, and there is a chart builder already wired to live data. You pick a metric, a split, and an aggregation, and you get a graph going back thirty days. This costs nothing extra and needs no setup, which makes it the first place I look when someone says the app feels slow or the bill jumped. Azure also ships a prebuilt workbook, the Azure OpenAI insights view, that groups the same metrics into four panels: HTTP requests, token based usage, PTU utilization, and fine tuning. For a first look, open the workbook rather than building charts by hand.

The catch is dimensionality. Platform metrics can split by a handful of built in dimensions, the model deployment name, the operation, the status code, but there is no dimension for the calling application, the API key, or the end user. So a metric can tell you that generated tokens doubled at noon, and it cannot tell you that the growth marketing team’s batch job was the cause. That blind spot is exactly why the log and trace layers exist, and it is the single most common thing people expect metrics to do that they cannot.

Which metrics actually matter for a GenAI deployment

There are dozens of metrics on the resource, but five carry most of the operational weight, and I watch these on every deployment I run. Azure OpenAI Requests is the raw call count, and split by status it is your throttling early warning, because a rising share of 429 responses means you are hitting a rate limit before users complain. Processed Prompt Tokens and Generated Completion Tokens are input and output token counters, and together they are your cost meter, since billing tracks tokens. Provisioned managed Utilization V2 is the one that matters if you bought PTUs, the reserved throughput units from Part 6, because it reports how full that reservation is as a percentage. The older Provisioned managed Utilization metric is deprecated, so alert on the V2 version. Latency metrics round out the set for anyone holding a response time target.

Here is the practical mapping I keep in my head. If the question is about money, watch the two token metrics. If it is about reliability, watch requests split by status code and the throttling that shows up there, which ties back to the quota model from Part 8. If it is about capacity on a PTU deployment, watch utilization V2 and nothing else. Three questions, three metrics, and you have covered most of what a platform team is on the hook for.

Table 2. Five metrics that carry most of the operational load. Names verified against the Azure OpenAI monitoring data reference, July 2026.

Turning on diagnostic logs, and what they hold

When metrics run out of answers you move to resource logs, and those do not exist until you create a diagnostic setting on the resource. A diagnostic setting is a routing rule: it says take these log categories and these metrics, and send them to a Log Analytics workspace, a storage account, or an event hub. Azure OpenAI offers three log categories. Audit records control plane actions, the who deployed and who changed what. RequestResponse records one row per inference call with the model, the operation, the status code, and the duration. Trace carries lower level diagnostic detail. For request level monitoring you enable RequestResponse, usually alongside Audit for security, and add the AllMetrics bucket so your metric history also lands in the workspace.

Read this carefully, because it trips up almost everyone the first time. RequestResponse logs capture metadata about each call, not the prompt and not the completion. That is a privacy choice by Azure, and it means you cannot open Log Analytics and read what a user asked. If you need the text of the conversation, that is the trace layer with content capture turned on, which I get to below, and it carries a data protection burden that request logs do not. So request logs are perfect for a security audit trail and a per call cost breakdown, and useless for debugging a wrong answer. Match the layer to the question.

One Azure CLI command wires RequestResponse and Audit logs, plus all metrics, into a workspace. Replace the three resource ids with your own.

az monitor diagnostic-settings create 
  --name aoai-to-law 
  --resource /subscriptions/SUB/resourceGroups/RG/providers/Microsoft.CognitiveServices/accounts/MY-AOAI 
  --workspace /subscriptions/SUB/resourceGroups/RG/providers/Microsoft.OperationalInsights/workspaces/MY-LAW 
  --logs    '[{"category":"RequestResponse","enabled":true},{"category":"Audit","enabled":true}]' 
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Expected output: a JSON object describing the new setting, with an id ending in /diagnosticSettings/aoai-to-law and your two log categories shown as enabled. Give it a few minutes, then rows appear in the AzureDiagnostics table. Failure mode: a ResourceNotFound almost always means a wrong workspace id, check the OperationalInsights path. A category validation error means a misspelled name, the value is RequestResponse, not RequestResponseLog, and the CLI will list the valid categories in the error.

Reading request logs with KQL

Once logs are flowing, you query them in Log Analytics with Kusto Query Language. In the default Azure diagnostics mode the rows land in the AzureDiagnostics table, and you filter by category. KQL reads like a pipeline, each line takes the rows from the line above and transforms them, so you build a query by stacking filters and a summarize at the end. The query below finds throttling: it keeps only the RequestResponse rows, keeps the ones whose status starts with 429, and counts them in five minute buckets so you can see when the rate limit bit.

AzureDiagnostics
| where Category == "RequestResponse"
| where ResultSignature startswith "429"
| summarize throttled = count() by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

Expected output: a two column table, a timestamp bucket and a count, that you can chart as a bar to see throttling spikes line up with your traffic peaks. Failure mode: an empty result usually means the diagnostic setting has not started writing yet, or your column names differ. Field names in AzureDiagnostics vary by resource and by whether you use resource specific tables, so confirm ResultSignature and Category against your own rows before you wire this into an alert.

The reason to keep logs in KQL rather than only staring at metric charts is that you can group by things metrics cannot see. Join RequestResponse rows against an API Management log and you get per caller token counts. Filter by a correlation id from your app and you follow one user session across calls. That is the payoff of the log layer, and it is why I turn RequestResponse on for anything that a finance team or a security team will ask questions about later.

Application Insights and OpenTelemetry traces

The third layer is where you finally see inside a single run. Application Insights is the application monitoring arm of Azure Monitor, and Azure GenAI telemetry reaches it through OpenTelemetry, the vendor neutral standard for traces and metrics. Your app, or a framework it uses, emits spans following the OpenTelemetry generative AI semantic conventions, a shared vocabulary of span attributes for model calls, token counts, tool calls, and retrievals. Because the vocabulary is standard, the Application Insights Agents view can render a multi agent run as a readable tree, and the same spans would be legible in any other OpenTelemetry backend. This is the same trace stream that powers the quality tracing in Part 23, seen from the platform side rather than the evaluation side.

Turning it on is mostly wiring the exporter and setting a few environment variables. You point the Azure Monitor OpenTelemetry distro at your Application Insights connection string, and then a set of opt in flags decides how much detail you keep. To record the actual prompt and completion text on each span you set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to SPAN_AND_EVENT, opt into the experimental GenAI semantic conventions, and enable GenAI tracing on the Azure client. Those flags are the switch that turns a privacy safe trace into one that carries user text, so treat them as a deliberate decision, not a default.

What observability costs when you log everything

Metrics are free, but the two lower layers bill you by the gigabyte of data ingested, and the jump from metadata logging to full content capture is where the number gets uncomfortable. Log Analytics charges for ingestion, and a per request record is small, but multiply a couple of kilobytes by millions of calls and it adds up. Full prompt and completion capture in traces multiplies that again, because now every span carries the entire conversation instead of a status code. The trap is enabling everything at once on a busy resource and meeting the bill at month end.

Alerts that page you before users notice

Dashboards are for looking. Alerts are for not having to look. Azure Monitor metric alerts fire on a metric crossing a threshold, and for a GenAI deployment two of them earn their keep on day one. First, a metric alert on Provisioned managed Utilization V2 above ninety percent for a sustained window, because that is when a PTU deployment starts queuing and latency climbs, exactly the crossing in the utilization chart above. Second, an alert on the 429 share of Azure OpenAI Requests, because a rising throttle rate means you are outgrowing your quota and users are getting errors. Both are metric alerts, so they cost almost nothing and need no logs.

Beyond those two, log alerts run a KQL query on a schedule and fire on the result, which is how you alert on things metrics cannot express, a spike in a specific error signature or a caller exceeding a token budget. Log alerts cost more than metric alerts because they run queries against ingested data, so I reserve them for conditions that genuinely need the log layer. Wire an action group to each alert so it reaches a human or an incident tool, an unrouted alert is just a red dot nobody sees.

What to turn on, and in what order

Here is the order I set up on a new Azure GenAI workload, and why the sequence matters. Day one, do nothing but read the platform metrics that are already there, and add the two metric alerts, PTU utilization V2 and the 429 rate. That is free, it takes an hour, and it covers the reliability and capacity risks that actually page you. Next, create a diagnostic setting for RequestResponse and Audit logs into Log Analytics, because the audit trail and the per call cost breakdown are worth the small ingestion charge and every security review will ask for them. Set a daily cap on the workspace the same day. Only then, and only where you own the application code, wire Application Insights traces over OpenTelemetry, and keep content capture in development and a sampled slice of production, never on all traffic.

What I would not do is start at the bottom. Turning on full content capture before you have a metric alert is the classic mistake, a huge bill and a privacy exposure in exchange for detail you rarely open, while the throttling that would actually take your app down goes unwatched. Build up the stack, not down it. If you run the same workload on AWS, the parallel is Bedrock and CloudWatch, which I walk through in the AWS series observability part, and the layering instinct carries over even though the service names do not.

One thing to do next: open your Azure OpenAI resource, check whether a diagnostic setting exists at all, and if it does not, you have been flying on metrics alone. That single check tells you which of the three layers you are actually running today.

References

• Monitoring data reference for Azure OpenAI, Microsoft Learn
• Monitor Azure OpenAI in Microsoft Foundry Models, Microsoft Learn
• Enable diagnostic logging for Azure AI services, Microsoft Learn
• Application Insights OpenTelemetry overview, Microsoft Learn

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts