AI video and digital-human pipelines routinely create the same media artifact more than once because of retries, fan-out and late callbacks. A conditional S3 write is a small primitive that can prevent a large, hard-to-detect overwrite. This guide combines published AWS behavior with an independently proposed transaction and testing model.
Why AI media outputs race even when jobs have unique IDs
An AI media service often stores an original upload, several generated previews, a voice transcript and a final video. Each artifact can be produced by a different worker. A queue retry, model timeout or callback delivered twice may cause two workers to generate the same object key independently. The problem is not that Amazon S3 is inherently inconsistent: the problem is application intent. A later write may be perfectly valid at the storage API while still overwriting an artifact belonging to the previous attempt. The design therefore needs both a logical job identity and a storage-level precondition. Amazon S3 documents conditional writes using If-None-Match and If-Match. These are not substitutes for an idempotency ledger; they are the last boundary that prevents an unexpected writer from silently winning. Treat the object key as an immutable artifact identifier rather than merely a convenient file path.
Separate creation from replacement with explicit preconditions
For a newly generated thumbnail, use If-None-Match with an asterisk to request a create-only operation. If a current object already occupies the key, S3 rejects the conditional request rather than replacing it. For intentional replacement, read the existing object’s ETag and use If-Match to make the update conditional on that exact observed state. AWS documents both headers, their permissions and failure responses. A failed precondition is a business decision point, not a cue for a blind retry. The worker should query the job ledger, determine which candidate is authoritative, and either discard its attempt or allocate a new versioned key. Multipart completion deserves special attention: the relevant condition is evaluated when completing the upload, not simply when the first part is sent. Verify the operation against the exact S3 API being used, including multipart and copy behavior, rather than assuming that all endpoints interpret headers identically.
Build a monotonic artifact ledger outside object storage
Use a relational record keyed by tenant, job, artifact type and generation version. Store the expected output key, attempt identifier, state, ETag or version ID when available, producer model version, checksum provenance and a human-readable reason for replacement. Move the record from reserved to uploaded to validated to published with controlled transitions. The ledger should reject a stale worker trying to mark an older attempt as current after another attempt has already passed validation. A transaction or conditional update can enforce this without holding a database lock during the expensive video render. In an example schema, a uniqueness constraint on tenant plus job plus artifact plus version prevents duplicate reservations, while a separate pointer identifies the latest approved version. This is an implementation pattern, not an AWS guarantee. Storage conditions protect an object; the ledger protects editorial and product meaning across multiple objects.
Handle 412, 409 and missing objects differently
AWS states that a failed conditional match can return HTTP 412 Precondition Failed. Concurrent deletion and conditional writes may also produce HTTP 409 Conflict or 404 Not Found depending on the operation. These outcomes should not be collapsed into a generic upload failure. On 412, do not overwrite; examine whether a legitimate artifact already exists, compare the expected content and close the duplicate attempt. On a conflict, inspect the relevant API-specific recovery guidance; a multipart completion conflict can require initiating a new multipart upload. On a missing object, reconcile the ledger against actual storage and the latest approved manifest. Use bounded retries only for transient, safely repeatable operations, with a new attempt key when necessary. Log request IDs, object keys and internal job identifiers, while excluding secrets and private customer media URLs from analytics. This turns operational alerts into evidence of race outcomes instead of noisy retry counts.
Test three workers and one deliberate crash
A useful integration test uses a deterministic fixture rather than a production user’s face or voice. Launch three workers against the same intended thumbnail key: one creates the first valid object, another attempts a duplicate create, and a third tries to replace an older ETag after a version has changed. Assert that at most one create-only operation wins and that the stale replace cannot succeed. Then terminate a worker between the successful object upload and the database state transition. The reconciliation process should find the orphan candidate and either resume validation or quarantine it, never automatically surface it. Repeat with multipart uploads, copy operations and a simulated delayed callback. Collect the final S3 object identity, job-ledger transition history and public CDN URL as separate test evidence. A green test requires preservation of the approved media and explicit conflict resolution, not simply a lack of exceptions.
Publish only after provenance and delivery checks
An uploaded artifact is not automatically publishable. Validate its media type, dimensions, duration, audio alignment and permissions before assigning a public delivery path. Where outputs contain AI-generated imagery, preserve a record of the generating model and user-authorized input without exposing personal source files. Separate the private upload bucket from the public delivery origin, and make the final publishing transition atomically reference an approved manifest version. A CDN purge or old signed URL can otherwise expose an older thumbnail after the database claims the new version is live. Measure accidental overwrite rate, rejected duplicate attempts, orphan reconciliation time and end-to-end publication delay in addition to raw S3 success rates. The architecture is strongest when the S3 precondition, job ledger and publishing gate agree on one outcome. For implementation details, consult Amazon S3’s current conditional-write guide and AWS’s October 29, 2025 copy-operation announcement before adapting any request examples.
Sources
Amazon Web Services, How to prevent object overwrites with conditional writes (living technical documentation; checked November 5, 2026). AWS, Amazon S3 adds conditional write functionality to copy operations (published October 29, 2025).