admin
Agent pipelines do not fail in one place. A user request can involve a router, model invocation, tool selection, network call, policy guardrail and final response, each served by a different component. The result is an observability problem: aggregate API success rates can look healthy while a tool repeatedly times out for one user segment. OpenTelemetry’s evolving generative AI semantic conventions create a vocabulary for model and agent instrumentation, while cloud-provider telemetry covers system-level behavior. This article sets out a deployment playbook for correlating model decisions with downstream tool execution without turning sensitive prompts into another sprawling log database.
Design one trace around a user-visible outcome
Start the root trace at a genuine user operation, such as requesting a digital avatar animation or retrieving a document. Add child spans for model calls, tool dispatches, retries, guardrail checks and storage writes. Record stable names rather than stuffing full content into span names. A trace should answer which component consumed time and which component decided the outcome. If a UI action creates several asynchronous jobs, propagate trace context into messages so the chain survives queue boundaries. A single request ID can be useful, but distributed context prevents developers from confusing unrelated retries. Treat partial success distinctly from a completed user task; a successful HTTP response from a tool does not mean the fan received a usable answer.
Normalize the attributes that actually support diagnosis
The OpenTelemetry semantic convention registry contains generative AI attributes for operations, providers, token usage and other model activity. Use standardized field names where mature, keep internal extensions in a documented namespace and pin convention versions. For each tool span, include a sanitized tool identifier, outcome class, duration, retry attempt and dependency type. Avoid collecting raw OAuth tokens, sensitive query strings or private document text merely to make dashboards easier to build. Attribute model input and output usage to the appropriate invocation, not to every downstream span, or the accounting totals will be duplicated. Establish a clear owner for schema changes; otherwise different teams will silently interpret the same metric differently.
Separate failure attribution from blame
When a model calls the wrong tool, the tool can execute perfectly and still produce a bad outcome. Conversely, an appropriate call may fail because a third-party service returns a timeout. Define error categories across decision quality, argument validation, dependency failure, permission denial and unsupported user expectation. Preserve a minimal sampled record showing the intended operation and sanitized result. Do not label all tool failures as ‘model hallucination.’ In dashboards, group by both the originating planner version and failing dependency, then investigate representative traces. This distinction matters commercially: changing the prompt is ineffective for network packet loss, while adding retries cannot repair a model’s incorrect selection of a sensitive action.
Make token and latency budgets composable
Model response time, queueing delay, tool execution and rendering time should be visible separately. Instrument elapsed wall-clock time as well as consumed token quantities, because high token count and high perceived delay are related but not interchangeable. A request can spend very few tokens while waiting on a slow external API. Record the budget passed to each child operation, whether it expired and whether cancellation propagated to downstream work. Connect cost estimates to model versions and price schedules with explicit effective dates, rather than assuming provider pricing never changes. For multi-agent systems, publish a policy for shared budget exhaustion and prevent secondary retries from exceeding the user’s initial time or spending limit.
Inspect cardinality and data exposure before scaling
A span attribute containing an entire user prompt creates unbounded variation and may leak personal information into observability backends. Prefer bounded enums for outcome and task type, hashed stable identifiers where strictly needed, and selected exemplars for authorized debugging. Use sampling rules that retain high-severity failures while reducing repetitive successful paths. Monitor exporter failures, storage costs and missing context propagation as independent metrics. A low tracing bill is not evidence of a healthy system if most requests lack spans. Before production rollout, run an adversarial privacy review: search the proposed traces for authentication headers, phone numbers, URLs with signed secrets and accidentally copied conversation content.
Validate the traces against controlled fault injections
Design a test matrix with a wrong-tool selection, expired credential, slow external HTTP request, malformed tool argument, cancellation and final-response rendering failure. Trigger each under known conditions and verify that one trace captures the causal sequence without leaking secrets. AWS Bedrock’s published monitoring guidance illustrates the importance of tracking model and guardrail metrics in addition to application outcomes, but vendor metrics alone cannot explain cross-provider operations. Add a regression test when a new model or tool enters the workflow. Evaluate whether an on-call engineer can identify the failing boundary within a set time. If not, improve trace structure before collecting still more telemetry.
Agent observability should make causal questions cheaper to answer, not merely create attractive charts. The most useful engineering investment is a consistent boundary between planner decisions, tool execution and user-visible results. Once those boundaries appear in traces, teams can allocate reliability work to the component that actually needs it and explain both performance and operating cost with evidence.
