Clock Drift in Real-Time AI Video Calls: An RTP-to-UI Measurement Plan concerns an operational decision that needs a falsifiable test, not a generic performance claim. This analysis proposes a deployable method grounded in public technical specifications, and labels illustrative scenarios as assumptions rather than measured customer data. The referenced material was reviewed on October 30, 2026.

Define which clock the user can perceive

A digital-human call contains several clocks: captured audio, generated speech frames, rendered animation, transport timestamps, client playout and the browser’s rendering loop. Treating each as the same wall-clock time makes the dashboard look tidy while lips visibly lag behind words. Define the observable outcome first: the absolute difference between a recognizable phoneme and its corresponding mouth movement at the screen. Use a local audiovisual test marker and repeated recordings, not a single subjective demonstration. The result must include direction: video leading audio is different from audio leading video. Browser screen capture adds its own delay, so run a hardware reference recording for a subset of tests.

Document every timestamp domain. A Unix millisecond value from an inference service cannot safely be subtracted from an RTP timestamp without conversion and synchronization. Keep a mapping record and clock uncertainty in the event envelope. If the sender’s clock is untrusted, mark the estimate as provisional rather than silently creating a precise-looking metric.

Understand RTP synchronization rather than guessing latency

RTP timestamps encode media sampling progression rather than a synchronized universal wall clock. RTCP sender reports can relate RTP media time to an NTP timestamp, allowing receivers to associate distinct streams under defined assumptions. The RTP specification, RFC 3550 (July 2003) explains the protocol model; engineers should validate how their selected stack exposes this mapping. For generated audio and animated video, use the same logical utterance identifier across both media tracks without assuming the network transports them identically.

Preserve packet sequencing and expected playout information when frame generation stalls. A speech restart might create a fresh stream or discontinuity. The lip-sync monitor must treat that as a new mapping, not as a huge accumulated drift from a previous timestamp epoch. Explicit clock reset events stop a false alert cascade during intentional interruptions.

Use WebRTC statistics as evidence, not as the final answer

The W3C WebRTC Statistics API draft (Candidate Recommendation Draft dated September 25, 2025) defines a structured way to examine transport and media counters. Capture relevant inbound track, jitter buffer, decoded frame and packet-loss observations before and after a visible synchronization event. Some statistics are browser-specific or sampled at different frequencies, so store browser build, device and collection interval alongside results. Do not present packet loss alone as proof of mouth timing failure; clean packets can still carry mismatched synthetic motion.

Build a timeline that shows user speech end, inference first token, audio chunk ready, animation cue ready, media sent and media presented. Correlate with trace IDs but redact conversation text. If the end-user device is CPU-bound, transport statistics may remain normal while rendering falls behind. That distinction changes the remedy from network capacity to animation quality settings or local resource budgets.

Run interruption and reconnect test scenarios

Create deterministic tests for a two-second sentence, a seven-second sentence with punctuation, rapid user barge-in and a Wi-Fi-to-cellular transition. For each test, capture five or more runs per representative device class under a fixed network profile. Intentionally drop video frames and inject 100 milliseconds of network delay to see whether playout re-synchronizes. These are laboratory perturbations, not recommended production target values. Record maximum drift, 95th percentile absolute drift, recovery interval and the fraction of calls that required a track restart.

Interruptions deserve special scrutiny. When the user cuts off an AI utterance, obsolete video frames should not play after the new response begins. Verify that audio and animation share the cancellation identifier. If old frames remain in a client buffer, a faster model cannot fix the embarrassing effect of a face visibly answering the previous question.

Choose a recovery policy users can understand

An application can drop stale motion frames, insert a neutral idle animation or briefly pause speech to recover alignment. None is universally superior: dropping frames may appear jerky, while extending audio latency can make turn-taking feel unresponsive. Test a declared maximum correction interval with users and monitor whether recovery affects meaning. A short voice pause at a sentence boundary is often easier to interpret than a sudden mouth-position jump, but that is a design hypothesis to validate, not a clinical fact or an established standard.

Publish the fallback behavior in the incident runbook. On high packet loss, prioritize intelligible speech and captions if available, then reduce the animation complexity. When the session cannot recover within the chosen budget, offer a transparent reconnect option rather than continuing with obviously mismatched media.

Define release gates and costs in one experiment

Use an illustrative acceptance sheet containing device class, capture rate, measured drift, clock uncertainty, barge-in recovery and bandwidth. Suppose two renderers show equal average drift, but the second produces rare 500-millisecond failures. An average alone conceals the problem, so measure tail behavior and the rate of visibly unacceptable episodes. Compare the quality improvement with extra GPU time, data transferred and battery impact. Costs must use the same observation period and traffic assumptions to support a fair decision.

Ship a dashboard that separately shows transport health, generation timing and end-user synchronized playout. Require that a release improve its target metric without a meaningful regression in another layer. This turns vague reports of an AI avatar feeling out of rhythm into a falsifiable engineering issue with a responsible team and a reproducible test set.

Conclusion. A high-quality AI system measures the outcome users actually see, preserves evidence across failures and states the limits of each measurement. Design the recovery rule and acceptance gate before optimizing the dashboard.

Sources: RTP: A Transport Protocol for Real-Time Applications, RFC 3550 (July 2003); W3C WebRTC Statistics API, CRD (September 25, 2025). Sources reviewed October 30, 2026.