A circuit breaker is straightforward when an API either succeeds or fails before sending a response. Streaming language models are harder: a provider may emit half a sentence, then stall, while the user already sees the partial output. Replaying the request at another provider can duplicate text, generate a contradictory answer or trigger the same external tool action twice. This article defines a gateway contract for partial responses, provider health and explicit fallback behavior. The design assumes a multi-model service but does not require any specific vendor or runtime.

Model three distinct failure moments

First classify failures before the first output byte, during the visible response and after response completion but before persistence acknowledgment. Each needs different handling. Before output begins, a controlled retry at an alternate provider may be safe if the request has no irreversible external side effect. Midstream, the gateway must not silently splice two incompatible completions and present them as one coherent reply. After completion, a failed database write may require idempotent persistence instead of re-generating the answer. Capture a stream event identifier and last acknowledged chunk index. The user interface should expose “reply interrupted” when a completion cannot be trusted, alongside a retry action that makes the consequences clear. Hide internal provider names if they are not relevant, but never pretend the interrupted answer finished normally.

Use provider health separately from request outcome

The AWS Prescriptive Guidance circuit-breaker pattern describes preventing repeated calls to a failing service and probing recovery. Adopt closed, open and half-open states per provider and endpoint type. A provider can be healthy for short text but degraded for long streaming or audio synthesis; merging those statistics can cause poor routing decisions. Define a rolling failure window, minimum sample threshold, observation interval and limited recovery probes. An isolated user cancellation should not mark a provider unhealthy. Nor should an authorization error or malformed client request automatically count as provider failure. Explain the exact error classes that trip the breaker in a runbook, then test the classification with fault injection rather than dashboard intuition.

Bound retries before any irreversible side effect

Gateway retries must consume a shared budget, not initiate another independent retry policy at every layer. Suppose the edge retries twice, the model SDK retries three times and the tool executor retries twice: multiplicative fan-out can vastly exceed the apparent request count. The budget should carry a deadline, attempts remaining and cancellation token across all downstream calls. For read-only generation, retry only when there is adequate time for a useful answer. For payment, publication or email sending, require a server-side idempotency key and an explicit operation receipt. A conversational model’s belief that the previous action failed is not evidence that the external side effect never occurred. Audit records must distinguish “requested,” “committed” and “acknowledged to user.”

Handle HTTP and stream semantics without invention

IETF RFC 9110 (2022) defines HTTP semantics, including status-code classes and request method characteristics. Once a streaming response is underway, the initial HTTP status alone cannot describe all subsequent application failures. Introduce an in-band terminal event such as completed, canceled or failed, and let the client require a terminal event before considering the answer finished. Set application rules for lost connections and resumable event identifiers rather than assuming generic HTTP automatically provides exactly-once delivery. When upstream returns rate limiting, honor documented backoff and avoid piling traffic onto a provider during recovery. If a fallback model has materially different capabilities or data-processing properties, surface that change before using it for sensitive workflows.

Protect conversation consistency across provider changes

Model outputs are not interchangeable database records. A replacement provider can follow a different instruction hierarchy, use different safety filters or tokenize tool declarations differently. Pin tool schemas and policy version to the original request. If a partial answer cannot be resumed deterministically, store it as an interrupted artifact and start a clearly labeled new answer in the same conversation. Do not merge speculative text from another provider into the original artifact. For multimodal replies, track whether generated audio and video correspond to text the user actually received. Memory writes should occur only on confirmed transitions, not every streamed token, or partial hallucinations may become durable facts. This can be tested by forcibly disconnecting the provider after several different chunk positions.

Observe user outcomes and infrastructure cost together

A health dashboard should show trip frequency, time spent open, probe success, fallback adoption, first-token delay, terminal completion rate and cost per useful finished reply. Separate abandoned user sessions from backend failures when attribution is uncertain. Track an interrupted-answer correction metric: how often did users have to restate a request or dispute a tool action after recovery? The breaker may reduce server workload while raising user confusion if it hides partial failures. To estimate impact, hold traffic shape constant and inject controlled faults into a test cohort, never into unrelated production accounts. Show both infrastructure and conversational outcomes with denominators and distribution plots. The goal is stable useful work, not merely a low count of 500 responses.

Write a failure drill that can be repeated monthly

Construct a fake provider that stalls before the first token, stops midstream, sends a malformed chunk and acknowledges a tool execution before cutting the connection. For each scenario, check circuit state transitions, retry count, generated text coherence, idempotent side effects and audit receipts. Include a second simulated client that disconnects voluntarily to ensure cancellation is not misclassified as provider sickness. Require deterministic test artifacts and a documented rollback switch for the new policy. A small number of deep failure drills is more useful than a thousand happy-path prompts. Product and operations teams should agree in advance which outcomes mean degraded-but-usable, which require human intervention and which must stop the workflow entirely.

Sources

AWS, Circuit Breaker Pattern, Prescriptive Guidance; IETF RFC 9110, HTTP Semantics, June 2022.