Agent observability: knowing what an AI agent did, and whether it worked

What 'working' means for an agent, and the traces, logs and evaluations that let you tell.

The problem

A request/response dashboard tells you an LLM call returned 200 in 900 ms. It does not tell you which model the router picked, which tools the agent called and in what order, whether the answer was right, or what it cost. An agent can return a fluent, well-formatted, wrong answer at normal latency, and every HTTP metric stays green.

Agents make this worse because failure is spread across steps. A retrieval step returns the wrong document, the model reasons correctly from it, a tool call succeeds with the wrong argument, and the final answer looks plausible. The error happened at step 1 and the evidence shows up at step 4. Per-request monitoring sees one call and a status code.

So "working" has to be defined per task: the classification matched the label, the extracted fields were correct, the customer's question was answered, the action the agent took was the one a person would have approved. Then you need three kinds of evidence: what the system did (traces and structured logs), whether it was right (evaluation), and what it cost (token usage per tenant and per task).

At an EU FinTech start-up running a second-hand marketplace, the lack of observability was one of the reasons the Claude prototype could not go to production, together with the lack of a safe way to release prompt or routing changes. When I later cut Claude inference cost there by 38%, the number only counted because quality held, verified through evaluation pipelines. Cost and quality are two readings of the same system. This topic is about taking both.

The levers that actually work

Trace every LLM and agent step end to end

Give every step its own OpenTelemetry span: the agent invocation, each model call, each tool execution, each retrieval. Parent them under the request's trace, so one trace ID shows the whole decision path, including retries and fallbacks. A retry that silently moved a request from a small model to a large one is invisible in a latency chart and obvious in a trace.

OpenTelemetry has semantic conventions for generative AI. They now live in their own repository, semantic-conventions-genai, and as of September 2026 they are still marked Development, OpenTelemetry's label for not yet stable. Attribute names can change, so pin the version you instrument against. The conventions define span names such as invoke_agent {gen_ai.agent.name}, execute_tool {gen_ai.tool.name} and, for a model call, {gen_ai.operation.name} {gen_ai.request.model}. The attributes that matter most:

Prompt and completion content (gen_ai.input.messages, gen_ai.output.messages) is opt-in, and the specification warns it is likely to contain personal data. Keep it out of the general trace pipeline unless it passes the same masking as your logs. Then add your own attributes for what the conventions do not cover: the routing class, the reason the router chose it, and the tenant.

OpenTelemetry was part of the stack on the marketplace platform. Before that, on an airline group's operations platform spread over four clouds, distributed tracing in one shared observability stack with OpenTelemetry enabled real-time cross-service debugging. The collection side, one OpenTelemetry layer across several clouds, is covered in one OpenTelemetry layer across four clouds.

Structured logs a machine or an agent can read

Traces show the shape of a run. Logs should record the decisions. Write one structured event per decision: model chosen and why, tool called with which arguments, guardrail triggered, fallback taken, answer returned. Every event carries the trace ID, the conversation ID and the tenant, uses one of a fixed set of event types, and keeps its field names stable across releases.

The reason to be strict is that the first reader of these logs is increasingly a program: an alert rule, an evaluation job, or an agent asked to triage an incident. A line written for a person ("sending this one to opus, looks hard") cannot be queried. This can:

{"event": "route.decision", "trace_id": "4bf92f35", "tenant": "t-17", "class": "simple",
 "model": "haiku", "reason": "short (12 tokens), no multi-step markers, no code"}

The open-source llm-router classifier already returns a reason string with every routing decision, and that string belongs in the log.

Kept per conversation, the same records become an audit trail. The marketplace platform had PII masking and a full conversation-level audit trail, and after rollout there were zero PII incidents in production logs. An audit trail for every LLM call goes into how to build one.

Evaluation sets and a judge you have checked

Traces tell you what happened. They cannot tell you whether it was right. For that you need a fixed evaluation set drawn from real traffic, labeled with what a correct answer is, versioned with the prompts and routing config, and scored per class of request. Use exact match where there is a right answer, such as classification; schema and field checks for structured output; and an LLM judge with a rubric for open-ended text.

The judge needs calibrating like any instrument. Research on LLM judges (Zheng et al., 2023) found that strong judges agreed with human preferences over 80% of the time, about as often as humans agree with each other, and documented position, verbosity and self-enhancement biases. So label a sample by hand, measure agreement per class, swap answer order when the judge compares two answers, and check again whenever the judge model or its prompt changes.

Run the evaluation as a release gate. On the marketplace platform, prompts, agents and routing logic shipped through gated releases in the CI/CD pipeline. Is the cheaper model good enough? walks through building that gate for a routing change, with thresholds, a noise floor and a code sketch.

Canary and shadow traffic

An evaluation set is a snapshot, and traffic moves. Three techniques close the gap:

Cost and quality on the same dashboard

A cost dashboard on its own rewards making the product worse. A quality dashboard on its own hides a budget overrun. Put them side by side, per tenant and per task type.

On the marketplace platform, per-tenant token-spend dashboards made cost visible per workspace. On an audio-commerce start-up's voice platform, the dashboards linked audio duration to cost to accuracy. That is the useful shape: the unit of work, what it cost, and whether it came out right, on one screen. The cost levers themselves are covered under LLM cost engineering, and the marketplace case in cutting Claude inference costs by 38%.

How to measure it

Four numbers, per task type and per tenant, cover most of it:

Keep p95 latency and the escalation rate of any fallback chain next to them. A router that often escalates from a small model to a large one pays for two calls and shows up in both.

Common mistakes

Every post on Agent observability & evaluation