gen_ai.usage.input_tokens: 41208

That attribute came off a trace of an ordinary question, asked by a support agent on an ordinary Tuesday. Forty one thousand input tokens, for an answer that ran to two sentences. Our logs said nothing was wrong. Latency was 6.1 seconds, which is slow but not alarming; the answer was correct; the agent gave it a thumbs up. Only the trace showed that a retrieval filter had stopped applying after a config change, and we were assembling thirty chunks into a prompt sized for six. We had been shipping that request shape for nine days, at roughly six times the intended cost, and nothing in our monitoring had a name for it.

Last part put a permission layer around the assistant tools and an injection corpus alongside the eval suite. Everything so far has been about deciding whether the system is correct. This part is about being able to answer a different question, the one you get at 4pm on a Friday: why did this request do that. That question needs a record of one request, in order, with the intermediate values preserved, and that record is a trace.

Logs, metrics and traces in an LLM system

Ordinary web services get away with logs and metrics because their failures are loud. A 500 appears in a counter; a slow query appears in a histogram; a stack trace names a line. An LLM feature fails quietly and correctly typed. Retrieval returns nothing, the model writes a fluent paragraph anyway, the response is a valid 200, and every metric you own stays green. Aggregate numbers describe the population and tell you nothing about the individual, which is exactly the wrong shape of information when a specific user is unhappy about a specific answer.

Tracing solves a narrower problem than monitoring, and it is the problem you actually have. A trace is one request, decomposed into nested spans, each carrying its own timing and attributes. Reconstructing that request from log lines is possible in theory and miserable in practice, because the pieces are interleaved with other requests, the retrieved chunks are too large to log comfortably, and by the time you have grepped them back together the interesting one has rotated out. Monitoring in the drift and decay sense, which I covered for classical models in the Data Science Series, remains useful and remains separate. Population level drift tells you something is changing. A trace tells you what happened.

One more reason applies only to this class of system. A non deterministic component means you cannot re run a request and expect the same failure. If you did not capture the inputs at the moment it went wrong, that specific failure is gone. Traces are the only artifact that survives.

Span layout for one assistant request

Before writing any instrumentation, decide what a span is. Too few and the trace tells you nothing you did not already know; too many and every request becomes a wall of noise you stop reading. My rule is one span per boundary where the request can go wrong in a distinguishable way. For our documentation assistant that means nine spans on a typical request, and eleven when a tool fires.

Nesting matters more than people expect. If the second model turn sits as a sibling of the first, a reader has to infer that a tool caused it. If it nests under the tool call, causation is visible at a glance, and an agent that loops three times looks like a loop instead of like six unrelated generations. Getting that hierarchy right is most of what makes a trace readable six months later, when the person reading it is not you.

Instrumenting with OpenTelemetry GenAI conventions

OpenTelemetry now publishes semantic conventions for generative AI, which is worth caring about for one unglamorous reason: attribute names are the schema your dashboards, alerts and queries are written against. Pick your own names and you will rewrite everything when you change vendor. Use the standard ones and any conformant backend understands your spans. Span names follow the pattern of operation plus model, so a chat call becomes chat gpt-4o-mini. Here is the smallest honest instrumentation of a single call, printed to console so you can see the span itself.

# trace_call.py, Python 3.12.3
# opentelemetry-sdk 1.44.0, openai 2.46.0, verified 20 July 2026
import os
from openai import OpenAI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)
tracer = trace.get_tracer("docs-assistant")
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])   # read from env, never hardcode

MODEL = "gpt-4o-mini"

def answer(question: str) -> str:
    with tracer.start_as_current_span(f"chat {MODEL}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "openai")
        span.set_attribute("gen_ai.request.model", MODEL)
        span.set_attribute("gen_ai.request.temperature", 0.2)
        r = client.chat.completions.create(
            model=MODEL,
            temperature=0.2,
            messages=[{"role": "user", "content": question}],
        )
        span.set_attribute("gen_ai.response.model", r.model)
        span.set_attribute("gen_ai.response.id", r.id)
        span.set_attribute("gen_ai.response.finish_reasons",
                           [r.choices[0].finish_reason])
        span.set_attribute("gen_ai.usage.input_tokens", r.usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", r.usage.completion_tokens)
        return r.choices[0].message.content

print(answer("What does error SSO-402 mean?")[:48])

SSO-402 indicates the identity provider returned a
{
    "name": "chat gpt-4o-mini",
    "context": {
        "trace_id": "0x4f1c3d0a9b7e2a55c1d84f60b3e9a712",
        "span_id": "0x9c2b71ea44d0f318"
    },
    "start_time": "2026-07-20T21:04:11.482913Z",
    "end_time": "2026-07-20T21:04:13.117640Z",
    "attributes": {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "openai",
        "gen_ai.request.model": "gpt-4o-mini",
        "gen_ai.request.temperature": 0.2,
        "gen_ai.response.model": "gpt-4o-mini-2024-07-18",
        "gen_ai.response.id": "chatcmpl-C7xR2mQhVd0ka9",
        "gen_ai.response.finish_reasons": ["stop"],
        "gen_ai.usage.input_tokens": 19,
        "gen_ai.usage.output_tokens": 84
    },
    "status": {"status_code": "UNSET"}
}

Adding message content is where a lot of people lose an afternoon. Conventions define gen_ai.input.messages and gen_ai.output.messages, so passing your message list straight in looks obviously correct. It is not, and it does not raise.

>>> span.set_attribute("gen_ai.input.messages", messages)
WARNING:opentelemetry.attributes:Invalid type dict in attribute
'gen_ai.input.messages' value sequence. Expected one of
['bool', 'str', 'bytes', 'int', 'float'] or None
>>> dict(span.attributes)
{'gen_ai.usage.input_tokens': 1200}

# Fix: span attributes take scalars or homogeneous sequences of scalars.
# Serialise first.
import json
span.set_attribute("gen_ai.input.messages", json.dumps(messages))

Miss that warning in a noisy log and you will believe you have prompt capture for weeks. I did, on a staging environment, and only found it when a colleague asked why every trace in the UI had an empty input panel. Two minutes to fix, eleven days to notice.

Attributes that make a trace debuggable

Model attributes come free with any instrumentation library. Attributes that actually resolve incidents are ones only you can add, because they describe your pipeline rather than the provider. We ran for about two months with textbook GenAI attributes and no pipeline attributes, and it cost us badly. A team reported that the assistant kept citing a deprecated migration guide. I could see the question and the answer, I could see that retrieval had run and how long it took, and I could not see which documents came back. Three weeks of intermittent complaints, several failed attempts to reproduce it by hand, and a growing suspicion that the model was confabulating the citation. Then I added one attribute holding retrieved chunk ids and reran the next complaint through the pipeline. Twenty minutes later the cause was obvious: a stale index shard was still serving the pre migration document set to about one request in seven. Nothing was wrong with the model at all.

Language specific SDKs make the pipeline layer easier than raw OpenTelemetry, because they give you a decorator and sensible defaults for LLM shaped work. I use Langfuse for this, and it is built on OpenTelemetry underneath, so the standard attributes still flow through and you are not locked in. Below is the retrieval and answer path with observation types set, session and user propagated to every child span, and a score attached from user feedback.

# pipeline.py, langfuse 4.14.1, verified 20 July 2026
# env: LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_BASE_URL
from langfuse import get_client, observe, propagate_attributes

langfuse = get_client()

@observe(as_type="retriever")
def retrieve(question: str, k: int = 6):
    hits = vector_store.search(question, k=k)          # your store from Part 11
    langfuse.update_current_span(metadata={
        "doc_ids": [h.id for h in hits],
        "scores": [round(h.score, 3) for h in hits],
        "index_version": vector_store.index_version,
    })
    return hits

@observe(as_type="generation")
def generate(question: str, hits):
    r = call_model(question, hits)                     # openai client from above
    langfuse.update_current_generation(
        model=r.model,
        usage_details={"input": r.usage.prompt_tokens,
                       "output": r.usage.completion_tokens},
        metadata={"prompt_version": PROMPT_VERSION},
    )
    return r.choices[0].message.content

@observe(name="answer-request")
def answer(question: str, user_id: str, session_id: str) -> str:
    with propagate_attributes(user_id=user_id, session_id=session_id,
                              tags=["docs-assistant"]):
        return generate(question, retrieve(question))

text = answer("What does error SSO-402 mean?", "u_4417", "s_90ab")
langfuse.create_score(trace_id=langfuse.get_current_trace_id(),
                      name="user-feedback", value=1, data_type="NUMERIC")
langfuse.flush()   # required in short lived scripts, see the gotcha below

Context loss across threads and async workers

Any assistant worth shipping fans out. We query three index shards in parallel, and we run guardrail checks concurrently with the first token arriving. Context propagation in OpenTelemetry rides on context variables, and a thread pool does not carry those across the boundary. Nothing errors. Your children just quietly become root spans of their own traces, and your beautifully nested pipeline arrives at the backend as four unrelated fragments.

from concurrent.futures import ThreadPoolExecutor
from opentelemetry import trace, context as otel_context

def child(i):
    with tracer.start_as_current_span(f"retrieve shard {i}") as s:
        return format(s.get_span_context().trace_id, "032x")

# Broken: the pool threads start with an empty context
with tracer.start_as_current_span("answer") as root:
    root_id = format(root.get_span_context().trace_id, "032x")
    with ThreadPoolExecutor(max_workers=3) as ex:
        ids = list(ex.map(child, range(3)))

root   763d27fec13898aa111da37051c677b9
child  df83aaf6fed18adb191539c6e4479227  ORPHAN
child  57c6fc1b4dc2ea44508d226fe3696400  ORPHAN
child  1364d384c16fc4c76bf7a014a2276cd7  ORPHAN

# Fix: capture the context on the calling thread, attach it inside the worker
def with_ctx(ctx, fn, arg):
    token = otel_context.attach(ctx)
    try:
        return fn(arg)
    finally:
        otel_context.detach(token)

with tracer.start_as_current_span("answer") as root:
    ctx = otel_context.get_current()
    with ThreadPoolExecutor(max_workers=3) as ex:
        ids = list(ex.map(lambda i: with_ctx(ctx, child, i), range(3)))

root   d549b8220c27f69d4a3c8f25ca27c8e1
child  d549b8220c27f69d4a3c8f25ca27c8e1  same
child  d549b8220c27f69d4a3c8f25ca27c8e1  same
child  d549b8220c27f69d4a3c8f25ca27c8e1  same

Worth checking this deliberately rather than assuming it works, because orphaned spans do not look like a bug. They look like a system that simply makes more requests than you thought, which is a story people accept for months.

Reading a trace when the answer is wrong

Instrumentation is the easy half. Reading a trace under pressure is a skill, and it is mostly a matter of having a small set of shapes memorised. Below is the lookup I keep pinned; it covers roughly four out of five investigations we open, and it is the single most useful thing in this post. Work down the retrieval column first, always, because that is where 61 percent of our quality complaints resolved.

Latency deserves its own paragraph because intuition here is consistently wrong. Ask any team where their time goes and they will say the model. We said the model. Then I pulled the span durations from 2,014 traced requests and plotted them, and our own reranker, a cross encoder we had left running on CPU because it was fine at prototype volume, turned out to own 35 percent of p95.

Sampling, content capture and trace storage cost

Here is where I disagree with almost every vendor quickstart you will read. Observability platforms for LLM applications capture full prompt and completion text by default, and present that as the point of the product. OpenTelemetry does the opposite: content capture is opt in, because prompts and completions are user data and because they are enormous. Vendor defaults optimise for a good demo. OpenTelemetry optimises for running this for a year.

Our numbers, at 1.2 million requests a month and nine spans per trace: with full content, a trace averaged 14 KB and we were storing about 16.8 GB a month. Metadata only, keeping token counts, durations, doc ids, versions and scores but dropping message bodies, a trace averaged 1.1 KB, roughly 1.3 GB a month. Cost went from about 340 dollars a month to 61. More importantly, the PII surface shrank to almost nothing, which matters given the handling rules from Part 23.

# sampling.py, our production policy
# Metadata on 100 percent of traces. Content on the ones worth reading.
import random

def capture_content(latency_ms: float, error: bool,
                    feedback: int | None, doc_count: int) -> bool:
    if error or latency_ms > 6000:      # anything anomalous
        return True
    if feedback is not None and feedback < 1:
        return True                      # every thumbs down
    if doc_count == 0:                   # empty retrieval, always interesting
        return True
    return random.random() < 0.10         # 10 percent baseline

# Applied at span close:
if capture_content(dur_ms, is_error, score, len(hits)):
    span.set_attribute("gen_ai.input.messages", json.dumps(messages))
    span.set_attribute("gen_ai.output.messages", json.dumps(output))

Retention deserves a decision rather than a default. We keep metadata for 90 days because that is the window in which someone asks about a cost or latency regression, and content for 14 days because nobody has ever asked me to read a prompt from five weeks ago. Traces attached to a failed eval case get pinned indefinitely, since those become regression fixtures in the sense described in Part 21.

One connection worth making explicitly, because teams keep building these as separate systems. Traces and evaluation are the same data viewed twice. A trace with a low judge score is an eval failure with full context attached; an eval case is a trace someone decided to keep. Scores on traces, as in the code above, give you a production feedback loop that feeds the eval set from Part 20 without anyone hand writing cases. If you have used an experiment tracker such as MLflow, described in the Data Science Series, this is the same instinct applied to inference rather than training, and the reasoning behind evaluating generative output carries over directly.

Instrument the retrieval boundary first

If you do one thing on Monday, do not install an observability platform. Add three attributes to whatever spans you already have: the ids of the documents retrieved, the version of the prompt used, and a session id that ties consecutive requests together. Those three cost an afternoon, need no vendor decision, and would have resolved every incident described in this post. Retrieval first, because that is where the causes live even when the symptom appears in the answer.

After that, pick a backend and use OpenTelemetry attribute names inside it, so migrating later is a configuration change rather than a rewrite. My pick is a Langfuse style LLM native platform over a general purpose tracing tool, because the observation types and score model save real work; the one I would avoid is a bespoke logging table in your own database, which every team builds once and regrets around month four. Turn content capture down to a conditional sample on day one rather than after the first invoice.

Next part uses this instrumentation for something other than debugging. Once you can see where a request spends its time and its tokens, caching, batching and latency engineering stop being guesswork, and the reranker story above becomes a repeatable method rather than a lucky afternoon. Instrument first, then optimise. If you have a trace shape that fooled you, add it to the lookup table and tell me what it was.

References

• OpenTelemetry GenAI attribute registry, for every attribute name used above, including the usage, retrieval and tool call groups
• Inside the LLM Call: GenAI Observability with OpenTelemetry, for the opt in position on capturing message content
• Langfuse Python SDK overview, for the client, observation types and the OpenTelemetry mapping used in the pipeline example
• OpenTelemetry Python documentation, for tracer providers, span processors and context propagation
• OpenAI chat completions reference, for the usage and finish reason fields read in the first code block

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts