A p99 alert fired on a Friday. The log line it pointed me at had a timestamp, a 200 status, and a duration of 8.2 seconds, and nothing else. It did not say which model call was slow, whether retrieval or the model ate the time, or whether the user even got a useful answer. A green status code on an eight second request is the shape of a problem you cannot fix, because the number that tripped the alert is the only number you have.

Observability on Vertex AI is how you turn that one lonely number into a story: which call, how long each step took, what it cost, and whether the answer was any good. Google Cloud gives you three tools for it, they are already wired to your model calls, and most of what you need is on by default. This part is the map of where each signal lives and which failure it catches.

Three signals, one platform

Observability is the practice of understanding what a system did from the data it emits, and it rests on three kinds of signal. A metric is a number measured over time, requests per second, latency, error count. A trace follows one request through every step it touched and records how long each step took. A log is a timestamped record of a discrete event, one line per request or per error. You need all three because each answers a different question. Metrics tell you that something changed. Traces tell you where in the request the time or the failure went. Logs tell you exactly what happened on the one request you are staring at.

On Google Cloud these map to three products you do not have to install. Cloud Monitoring holds the metrics and draws the dashboards. Cloud Trace stores the traces and shows you the waterfall of a single request. Cloud Logging keeps the logs and lets you search them. Vertex AI already emits into all three. The managed models publish metrics automatically, Agent Engine emits traces once you flip tracing on, and everything writes structured logs. Your job is not to build a telemetry pipeline. It is to know which of the three to open when something breaks, and to keep the expensive one from surprising you on the bill.

What the built-in dashboard already gives you

Here is the part people miss because it needs no code. If you call managed models from Vertex AI, Google added a Model Observability view to the Vertex AI dashboard in March 2025. Open the Vertex AI home page, find the model observability metrics, and click Show all metrics to land on a prebuilt Cloud Monitoring dashboard. It shows query rate, character and token throughput, latency, and error rates for your Gemini and other managed model calls, with no dashboard building on your side. For most teams this is the first and often the only monitoring they need for a while.

The single most useful thing on that dashboard is the error breakdown, and one code matters more than the rest. A 429 means the serving region ran out of aggregate capacity across all customers, not that you did anything wrong. When you see a run of 429s, the fixes are the ones from Part 6: buy provisioned throughput so you have reserved capacity, move the workload to a less busy processing location, or push non urgent work to batch. Google ships a recommended alert that pages you when more than one percent of requests return 429 over a window, and turning it on is a two minute job from the Vertex AI integration in Cloud Monitoring. Do that before you build anything custom.

Which metric catches which failure

A metric is only useful if you know which failure it moves for. Latency is the one to read as percentiles, not as an average, because the average hides the tail. A percentile is the value below which a given share of requests fall, so p50 is the median and p99 is the slow one in a hundred. The mean can sit at a comfortable one second while p99 is at eight, and it is the p99 users complain about. For streaming responses, watch time to first token separately, because a stream that takes two seconds to say its first word feels broken even if the full answer arrives quickly.

Token throughput is the metric that predicts your bill and your quota headroom, since cost on Vertex AI is priced per token as covered in Part 6. Watch it climb before it becomes a cost review. The table below maps the built-in metrics to the failures they catch. The metric type strings are the Cloud Monitoring identifiers for managed publisher models; confirm the exact suffix in the Cloud Monitoring metrics reference for your model, since Google adjusts the list as new model families land.

Why a dashboard cannot tell you what broke

Go back to that 8.2 second request. The metric told me it was slow. It could not tell me whether the model was slow, the retrieval step was slow, a tool call hung, or a safety check stalled, because a metric is an aggregate and the request is a sequence. To see the sequence you need a trace, and a trace is built from spans. A span is a single timed unit of work with a name, a start, and a duration, and a trace is the tree of spans for one request. The parent span is the whole request; child spans are the retrieval, the model call, the guardrail, the response. Lay them on a timeline and the long bar is your answer.

This is where the model dashboard and the trace divide the labor. The dashboard tells you the fleet of requests got slower this afternoon. The trace tells you that on this specific slow request the model call took 7.5 of the 8.2 seconds while retrieval took 0.4, so the fix is the model tier or streaming, not the vector index. Without the trace you would have spent the afternoon tuning retrieval that was never the problem. Metrics find the fire. Traces tell you which room it is in.

flowchart LR
  A[Agent or app request] --> B[OpenTelemetry SDK]
  B --> C[Cloud Trace spans]
  B --> D[Cloud Monitoring metrics]
  A --> E[Cloud Logging request logs]
  C --> F[Trace Explorer]
  C --> G[Observability dataset]
  G --> H[BigQuery analysis]
  D --> I[Dashboards and alerts]

Turn on tracing for an agent

Agent Engine, the managed runtime for agents from Part 13, has tracing built around OpenTelemetry, the vendor neutral standard for emitting traces. When you deploy an agent you enable tracing, and the runtime emits spans for the model calls and tool calls to Cloud Trace, where the Trace Explorer draws the waterfall. For agents built with the Agent Development Kit or LangGraph, the spans are created for you, so you get the tree without hand instrumenting every step. You can also view agent relationships, a topology graph of which agent called which, when you run a multi-agent system.

If you run your own service rather than a deployed agent, you instrument it with the OpenTelemetry SDK and point the exporter at Cloud Trace. The snippet below wraps a Gemini call in a span and exports it. Method and package names on the OpenTelemetry Google Cloud exporter move between releases, so check the current package before a scheduled run.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from google import genai

# Send spans to Cloud Trace in this project
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(CloudTraceSpanExporter(project_id='my-project'))
)
tracer = trace.get_tracer('genai-app')
client = genai.Client(vertexai=True, project='my-project', location='us-central1')

with tracer.start_as_current_span('handle_request') as root:
    with tracer.start_as_current_span('model_call') as span:
        resp = client.models.generate_content(
            model='gemini-2.5-flash',
            contents='Summarize the refund policy in one sentence.',
        )
        # Safe to record counts, not the prompt text
        span.set_attribute('output_tokens', resp.usage_metadata.candidates_token_count)
    root.set_attribute('status', 'ok')

print(resp.text)

Expected output: the answer prints locally, and within a minute a trace named handle_request appears in Trace Explorer with a child span model_call carrying the token count attribute. Failure mode: forget the BatchSpanProcessor and spans never leave the process, so Trace Explorer stays empty and the call still succeeds, which is the quiet failure that makes people think tracing is broken; attach the full prompt as an attribute and you have just written user data into Cloud Trace.

Reading a trace when p99 jumps

Back to the Friday alert, now with a trace in hand. I opened Trace Explorer, filtered to requests over five seconds, and picked one. The waterfall told the whole story in a glance. The request span was 8.2 seconds. Under it, auth and setup took 120 milliseconds, the Vector Search retrieval took 380, the Gemini call took 7,500, the Model Armor safety check took 180, and serializing the response took 20. One bar dwarfed the rest. The model call was the entire problem, and it was the reasoning tier doing long generations under load, not anything in my code.

That reading changed the fix. I did not touch retrieval. I moved the interactive path from the reasoning tier to Gemini Flash for the common questions, kept the heavier model for the rare hard ones, and turned on streaming so the user saw a first token fast even when the full answer was long. The p99 came down because the slow span shrank. Every step of that decision came from one trace, and none of it was visible on the model dashboard, which had shown a model that was slow but not why or where.

What to log, and what to never log

Logs are where observability gets expensive and where it gets you in trouble. The temptation is to log the full prompt and full response on every request, because that is the record you want when a user complains. Two problems. The prompts and responses are large, so at real volume the log bill climbs fast and is priced per gibibyte ingested. And those payloads carry whatever the user typed, which routinely includes personal data, so a full-payload log is a data governance liability sitting in Cloud Logging where more people can read it than you think. The governance rules from Part 10 apply to your logs, not just your model calls.

My default is a tiered logging policy. Log the metadata on every request, the timestamp, latency, token counts, model, status, and a request ID, because that is small, cheap, and free of user text. Log the full prompt and response only on a small sample, or only on errors, and redact obvious identifiers before they land. When you do need payloads for debugging, put them behind tighter access than the metadata logs and set a short retention. Cardinality, the number of distinct label values on a metric or log, is the other trap: tag a metric with the user ID and you create a separate time series per user, which explodes storage and cost. Keep high cardinality fields in logs and traces, never as metric labels.

Where I would start, and the trace worth wiring first

Here is the order I actually follow. Turn on the built-in Model Observability dashboard, which costs nothing and takes minutes, and switch on the recommended 429 alert the same afternoon. That alone answers the first wave of production questions about capacity and latency. Do not build custom dashboards until the built-in one leaves a question unanswered, because it usually does not for a while. When latency gets slow and the dashboard cannot say where, that is the day you wire tracing, and Agent Engine gives it to you almost for free through OpenTelemetry. Keep logging tiered from day one: metadata on everything, payloads only sampled and access controlled. Sample your traces so they stay cheap, and never put a user ID on a metric label.

The one thing I would wire before you think you need it is a single end to end trace on your critical path, one request instrumented from entry to response, so the first time a metric lies you already have the waterfall. That trace is the difference between a Friday afternoon of guessing and a five minute fix. Once you can see latency, errors, and cost per request, the next question is what all this is costing you across the account and how to hold it down, which is the next part on cost governance and FinOps. Before you move on, open your Vertex AI dashboard and turn on the 429 alert. It is the cheapest reliability win on this list.

Compare with the same layer on another cloud: Azure Monitor observability for GenAI, from metrics to traces.

References

• Built-in performance monitoring for Vertex AI Model Garden (Google Cloud Blog)
• Agent observability overview, Cloud Trace and topology (Google Cloud docs)
• Cloud Monitoring metrics for Vertex AI (Google Cloud docs)
• Set up tracing on Agent Runtime (Google Cloud docs)

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts