AI media workflows often start when a user uploads a photo, audio sample or video source. The convenient design sends an S3 event straight to a processing worker. The reliable design first establishes which object version the worker is allowed to transform, then treats notifications as hints rather than the final source of truth.

The delivery contract is weaker than the product promise

AWS documents that S3 event notifications are delivered at least once and may arrive out of order. This means a second upload can finish first while a delayed event for an earlier object arrives later. A consumer who assumes a single ordered message stream might overwrite a new avatar preview with an old one. The event notification is an observation that something happened, not permission to publish a result. Introduce a durable job record keyed by account, asset, upload generation and requested transformation. A new generation must supersede earlier generations explicitly. Keeping the ordering rule in application data makes it testable across Lambda, queues and alternate cloud processors. Avoid interpreting the server’s clock alone as a causal guarantee; different components can observe the same object at different times. The right invariant is that an accepted newer generation cannot be replaced by an older result.

Define stable identities before processing starts

Separate four identifiers that are often incorrectly collapsed into one: a user-visible asset ID, an object key, a storage version identifier when versioning is enabled, and a transformation job ID. A retry should repeat the same logical operation with the same idempotency key. A deliberate user replacement should create a new generation even if the destination object key is unchanged. Store the original checksum, media type, declared consent scope and expected derivative types with the job. Workers should compare stored version and checksum against the object they read before they compute anything expensive. The S3 event payload contains structured fields that help identify an occurrence, but event payload details can evolve; parse supported versions defensively and preserve the raw event for diagnostics. Never use a filename supplied by an upload client as the sole database identity for media authorization.

Treat duplicate events as normal workflow inputs

The simplest duplicate filter is an atomic transition, not an in-memory set. Insert or claim a ledger row with a uniqueness constraint on the exact operation identity. If the row already belongs to a completed job, acknowledge the repeat and return the existing result reference. If it is running, use a lease with a bounded expiration and an owner token so two workers cannot both publish. If a worker crashes after creating a derivative but before recording completion, the next attempt must find that derivative deterministically and validate it before doing more work. This is the difference between idempotent side effects and merely ignoring a duplicate notification. AWS explicitly cautions that rare duplicate notifications can occur. The application should make that condition routine enough that no on-call engineer needs to guess what happened.

Keep output publication behind a version fence

Media generation produces intermediate artifacts: resized images, extracted transcripts, embeddings, animated clips or rendered previews. Write them under immutable generation paths instead of replacing a public asset immediately. The worker may update an output manifest only if its generation remains current and its quality and permission checks pass. A conditional database update can compare current_generation with expected_generation; zero affected rows means the result is stale. This is particularly important when the user revokes access or deletes the underlying media while processing continues. Before exposing a URL, re-evaluate the asset’s publication status and authorization state. Never assume that successful processing grants a right to publish. The two systems have different sources of truth: storage indicates bytes exist; the product database indicates whether those bytes may be shown.

Inject failures into the pipeline deliberately

Build a small deterministic test matrix before increasing traffic. Deliver the same event twice, deliver generation three before generation two, kill a worker after upload but before commit, revoke consent during rendering, and deliver a queue event after a job has been canceled. For each case record whether there is exactly one active public version and whether orphaned artifacts can be cleaned later. Monitor queue age, duplicate suppression count, stale-generation rejection, object-not-found retries and user-visible time to readiness. An unexpected rise in duplicate suppression may be benign, but a rise in stale publications is a correctness defect. Include a replay harness that reads saved event examples without generating new public assets. The team should be able to prove its invariants from logs, not just assert that a message queue is reliable.

Roll out by separating correctness from throughput

A safe release begins with a low-volume shadow pipeline that consumes notifications and records what it would have published. Compare its chosen object generations with the existing system. Next enable publication for a small cohort while retaining rollback to the previous stable version. Set alerts on invariant violations rather than only on throughput and invocation errors. AWS documents that directly targeting an SQS FIFO queue from S3 event notifications is not supported; EventBridge can be part of a design when FIFO routing is necessary, but using it does not by itself settle all application-level ordering questions. The operational takeaway is simple: a resilient AI media service needs both reliable event handling and a version-aware product state machine. Scaling the workers before solving the latter only lets the service publish the wrong result faster.

Sources