AI chat services need strict ordering inside one conversation while keeping unrelated conversations parallel. This engineering runbook converts Amazon SQS FIFO semantics into a concrete implementation and failure-test plan for text, audio and video workflows. It also identifies the boundaries where queue guarantees end and application accountability must begin.
A queue does not automatically make every side effect exactly once
The promise of ‘exactly once’ must be evaluated at a specific boundary. Amazon SQS FIFO offers deduplication and ordered message delivery, but downstream database writes, payment calls and media rendering still require their own safeguards. A worker might commit a chat event to MySQL, crash before acknowledging receipt and process the same message after its visibility timeout. Without an application-level unique key, the user can receive duplicate responses or charges. Define three separate properties: queue admission deduplication, worker execution attempts and exactly one durable business effect. Only the last represents the behavior the customer actually experiences. This article assumes a PHP, Node or equivalent backend plus a durable transactional database. It is an implementation proposal, not a measured report on an existing system.
Partition by conversation, not by global tenant
AWS documentation identifies MessageGroupId as the mechanism that serializes events within a FIFO group while allowing different groups to progress independently. Use a stable conversation identifier as the group key when edits, deletions and assistant responses must occur in a defined order. A single group for all customer traffic introduces head-of-line blocking: a slow video task can delay thousands of unrelated text messages. A random group for every event destroys ordering. The right group boundary is the smallest unit with a real state dependency. Multi-party chat may require a common group for shared turns, whereas account-level billing should live in an independently consistent ledger. Document this decision and test it against an actual conversation sequence before adding concurrency. The queue does not know which business entities are logically related.
Make retries safe beyond the broker’s deduplication window
Every event should contain immutable event_id and conversation_id fields, a sequence number, schema version, producer timestamp and action type. The producer sends a deduplication ID derived from the logical event rather than inventing a new token for each retry. AWS documents a five-minute SQS FIFO deduplication interval, so a repeat outside that window must still be handled by application state. Create a unique database constraint for event_id, and write an explicit processing receipt before acknowledging the message. Separate the storage key of a generated media asset from the event that requests it; multiple worker attempts must not surface multiple published clips. For payment-affecting actions, record a distinct idempotency key at the payment boundary. Do not put entire conversation transcripts in queue metadata or monitoring events when narrower identifiers will do.
Treat worker leases and cancellation as first-class cases
A visibility timeout is a temporary delivery lease, not a permanent exclusive lock. Long-running rendering should extend the lease carefully or transition into a state-machine job and send a separate completion event. Before publishing a result, the worker should re-read the authoritative conversation state: the user might have edited the prompt, cancelled the action or revoked media sharing. Failed messages need bounded retries and a dead-letter queue. The dead-letter record should include an error category, event ID and version, not unnecessary private content. When a consumer restarts, a durable processing ledger must tell it whether the result was committed, sent to the UI or still awaiting finalization. Streaming provisional text can be displayed optimistically, but it cannot replace an auditable committed event.
A repeatable fault-injection test with explicit pass criteria
Prepare three message groups: a busy thread with fifty alternating edits and sends, a normal thread with ten independent text turns, and a third thread with a deliberately slow media task. Inject a duplicate event, terminate a worker immediately after the database commit, then retry the same logical event after the five-minute broker window. The pass conditions are zero duplicate committed effects, ordered sequence numbers within each group and independent progress for other groups. Measure p95 queue waiting time, event retry counts, poison-message rate, cancellation acknowledgment time and the interval between a provisional UI update and its authoritative commit. These are proposed acceptance metrics, not externally observed benchmarks. Repeat the exercise with two producers racing to send updates to the same conversation, since a superficially successful single-producer demo is insufficient.
Rollout controls that product engineers can verify
Start with a shadow run that records prospective group keys and event IDs without altering the live chat flow. Check IAM access, FIFO queue configuration, producer timestamps, service-region latency and the long-job visibility policy. Alarm on per-group backlog age rather than only total queue length. Replay dead-letter events in a staging environment before any production replay, since replay can violate ordering expectations if older actions are re-applied without state guards. Keep a versioned event contract listing all state mutations and whether they are safe to retry. During a limited rollout, compare ordering, duplicate rates, user cancellation success and media visibility with the legacy path. Only expand traffic after observed correctness holds under failures. A single consistent reply to a single action is a product requirement, not just an infrastructure optimization. Document the rollback owner, the on-call communication path and the evidence required to declare recovery complete.
Sources
- AWS documentation: Message group IDs for SQS FIFO queues (current documentation reviewed 10 Nov 2026)
- AWS documentation: Exactly-once processing in SQS FIFO queues (current documentation reviewed 10 Nov 2026)
Illustration: Ionela Mat / Unsplash.