A voice turn crosses capture, transport, speech recognition, model orchestration, synthesis and playback. A single response-time chart hides which stage slowed down. This runbook proposes observable boundaries and repeatable fault tests, while making a strict distinction between operational telemetry and sensitive conversational content.

Map one voice turn to its actual components

Begin with a logical turn identifier created when the user activates a microphone or submits a recorded voice note. Keep that logical identifier distinct from an HTTP request, because recognition, generation, synthesis and retry requests may be separate network operations. Make capture acknowledgement, upload completion, transcription ready, first generated token, first synthesized packet and playback start visible as individual events. A useful trace graph should explain a delay without requiring the engineer to listen to the user’s audio. For a real-time digital human, include the video synchronization point; the audio and visual streams may advance independently. Record a cancellation as a terminal business state instead of pretending that the missing completion was a slow success. The acceptance test is whether a support engineer can identify a failing subsystem using a synthetic session with no access to transcript text.

Use interoperable trace headers but restrict attributes

The W3C Trace Context Recommendation of November 23, 2021 defines traceparent and tracestate for cross-service correlation. That gives teams a common transport for tracing; it does not authorize transmitting arbitrary user data. Establish a whitelist of fields such as coarse device class, codec, model route identifier, queue bucket, region and error category. Explicitly prohibit raw prompts, speech recognition results, private photographs, access tokens, email addresses and signed media URLs as span attributes. Trace identifiers should be freshly generated and cannot themselves serve as permanent advertising identifiers. Test reverse proxies and asynchronous message handlers because both can drop headers or accidentally propagate sensitive baggage. For one intentionally malformed inbound traceparent, verify that your system safely starts a new trace without weakening authentication.

Primary source for this control: W3C Trace Context, Recommendation, 2021-11-23. The acceptance thresholds in this runbook are proposed engineering tests, not requirements claimed by the source.

Define latency budgets with denominators

Measure perceived delay from microphone stop to first audible reply, and separately measure full speech completion. A hypothetical run might spend 160 milliseconds in transport, 320 in speech recognition, 470 before model output and 230 in speech synthesis startup, producing 1.18 seconds before buffering. These are illustrative inputs, not an observed industry benchmark. Use median and p95 by network class and session temperature; show sample sizes. Cold-start sessions should not be silently merged with warmed-up calls and then excluded when the result looks bad. Track the fraction of valid turns below the stated budget. Put user cancellations, microphone permission denials and abandoned sessions in named parallel cohorts so the aggregate does not become an artificially flattering success rate.

Reproduce retries, disconnections and out-of-order callbacks

The hardest bugs often cross queues. In staging, disconnect the mobile client after upload while transcription still completes, then retry with the same logical turn identifier. Send a duplicated synthesis-ready notification and a model callback after the user has already ended the session. Only one visible reply should be committed for one logical action; all secondary results need a defined discard or compensation rule. Link trace spans to idempotency events without copying private text. Instrument orphaned worker jobs separately because they affect compute cost even if no user sees the result. The test report must show each injected fault, the state transition, the observed user experience and the number of jobs actually billed.

Export only the evidence that operations needs

OpenTelemetry’s tracing specification describes spans, events and attributes; it does not require recording an entire personal conversation. Set short retention for raw diagnostic spans and longer retention only for privacy-safe aggregate metrics when operationally justified. Keep exporter access separate from customer-service access. Use an allowlist and continuous schema linting before a telemetry record reaches an external vendor. Sample a variety of languages and accessibility settings because differing speech lengths change timing patterns. A security review should inspect actual exported JSON and dashboard queries, not merely a configuration screenshot. Give on-call staff a temporary escalation mode with an expiry and audit trail, instead of leaving verbose private diagnostics permanently enabled in production.

Primary source for this control: OpenTelemetry Tracing API, living specification. The acceptance thresholds in this runbook are proposed engineering tests, not requirements claimed by the source.

Publish a rollout gate and reversible release plan

The release checklist should contain synthetic traces for at least a baseline successful turn, a cold start, packet loss, user cancellation, duplicate callback, and failed authentication. Verify parent-child relationships, successful ingestion and that sensitive content scans find no transcript substrings. If trace coverage falls below the team-defined minimum, or private payloads appear in spans, stop rollout and disable the exporter behind a feature flag. Preserve the mapping of test IDs to redacted telemetry for reproducibility. Compare both cost and perceived first-audio time before and after deployment; monitoring overhead is a performance cost. When the next speech vendor or model route changes, re-run these cases rather than assuming that old traces remain comparable.

Sources

Method: original engineering test plan; hypothetical timings are explicitly identified. Image: Unsplash, credited in the media library.