Laptop and smartphone workspace illustrating software telemetry

When an AI agent invokes a tool, a useful production trace should answer what the agent attempted, which policy allowed it, what the tool returned, and how the final answer used that result. A raw transcript alone rarely provides that evidence. At the same time, storing every prompt, customer record and tool payload creates a second sensitive-data system that must be secured and governed. A redaction-first audit log balances operational traceability with data minimization.

1. Define the evidence needed for each tool call

Record a stable trace ID, conversation or job ID, tool name and version, policy decision, start and end time, outcome code, retry count and a reference to the relevant configuration version. Where a tool modifies external state, record a transaction or idempotency key so operators can distinguish a successful action from a retried request. Avoid logging full payloads merely because the logger can capture them.

Different tools need different evidence. A search tool may need a query class, source identifiers and result count; a payment or account-management tool may need an authorization decision and transaction reference. Define fields before implementation and document which are required, optional or prohibited.

2. Redact before long-term storage

Redaction should happen at the logging boundary, not weeks later in an analytics pipeline. Remove secrets, authentication tokens, unnecessary personal identifiers and full message content unless a documented purpose requires a restricted record. Hashing a value does not automatically make it anonymous; stable hashes can still link records. Use pseudonymous identifiers only where correlation is necessary, and control access to the key.

OWASP’s Logging Cheat Sheet emphasizes that logs should be designed around security and operational needs while excluding sensitive data that should not be recorded. Apply that principle to AI agents: retain enough information to reproduce a failure, but not a copy of every user conversation. Provide separate retention classes for routine telemetry, security incidents and legally required records.

3. Connect spans across model and tool boundaries

Use trace and span relationships to connect the model request, retrieval operation, external API call and final response. Each child span should carry a correlation identifier and a clear status. This lets engineers see whether a delay came from model generation, a database query, a third-party service or a retry loop. It also helps identify when a tool returned successfully but the agent failed to interpret the result.

OpenTelemetry’s tracing model provides a common way to represent operations and parent-child relationships. Keep raw user text out of metric labels; put safe identifiers in traces and aggregate performance measures in metrics. This avoids exploding metric storage while preserving useful debugging evidence.

4. Make retries and side effects explicit

Agent workflows often retry after timeouts even when an external tool completed the action. Log the intended side effect, idempotency key, attempt number and final reconciliation result. For irreversible actions, require a clear policy gate and user authorization where appropriate. A timeout is not proof that the action failed.

For each tool, define which errors are retryable, which require human review and which should stop the workflow. A repeated authorization failure should not trigger unlimited retries. A parser error should preserve a small redacted diagnostic sample rather than dumping the entire payload.

5. Build access controls around logs

Separate routine product analytics from security investigations. Apply role-based access, short retention for verbose traces, encryption in transit and at rest, and an audit trail for access to sensitive records. Developers should not automatically receive customer payload access because they can read aggregate metrics. Incident responders may need temporary access under a documented process.

Test redaction with seeded secrets and personal identifiers. Verify that the values do not appear in application logs, error messages, traces, exports or dashboards. A redaction rule that works only on successful responses will fail precisely when an exception includes the original request.

6. Use telemetry to improve reliability, not surveillance

Useful indicators include tool success rate, retry rate, duplicate side effects, latency by dependency and the share of workflows that need human intervention. These metrics help teams prioritize engineering work without scoring individual users or collecting unnecessary conversation content. Review the fields as the agent’s capabilities change.

OWASP provides logging security guidance, and OpenTelemetry describes the trace model used to connect operations. A disciplined schema makes AI workflows easier to debug while reducing the risk that observability becomes an uncontrolled archive of private data.

7. Prepare an incident workflow

When an agent makes an unexpected external call, responders should be able to identify the trace, freeze further retries, determine whether the side effect occurred and hand the case to the correct owner. Define which fields may be inspected during an incident and how that access is recorded. A useful incident packet contains the redacted event sequence, policy decision, tool version, outcome code and reconciliation result, not an unfiltered transcript dump. After the incident, convert the failure into a regression test and review whether the log schema captured enough evidence. This creates a feedback loop between observability and reliability instead of letting logs accumulate without changing the system.

Sources