Prevent Duplicate Video Jobs When Clients Retry Requests

When a client times out or loses a response, it may send the same video-creation request again even though the first job is already running. An idempotent request flow lets retries return the original job instead of launching duplicate work. You will need control of the job-creation API, a durable data store, and a way to pass a stable request identifier from the client.

What You’ll Need

• A client that can generate and reuse an idempotency key for one logical video request.
• An API service that validates requests and creates video jobs.
• A database or other durable store with a unique constraint, so concurrent requests cannot claim the same key.
• A job queue or worker system, if video rendering runs asynchronously.
• A defined retention period for idempotency records and a way to observe conflicts and retries.

Step-by-Step

1. Define what counts as the same request. Decide whether a retry is identical only when it has the same idempotency key, or whether the request payload must also match. A key should represent one logical attempt to create a video, not a user, session, or file in general. If a user intentionally starts a second render, the client must use a new key.
2. Have the client create the key once. Generate a sufficiently unpredictable identifier when the user submits the video request. Keep it with the pending operation and send it on every retry, commonly in an Idempotency-Key header. Do not generate a new key each time a network call is retried: that makes every retry look like a new job. Persist the key if the client must recover after a page reload or app restart.
3. Validate and normalize the request. Check required fields, allowed values, and authorization before creating work. Build a stable representation of the meaningful inputs, such as source asset, trim points, output format, and rendering options. Normalize equivalent values consistently before hashing or comparing them; ignore transient fields such as request timestamps. This allows the server to detect a reused key with changed content.
4. Store the key and job association durably. Create an idempotency record containing the key, an account or tenant scope, a request fingerprint, a state, and the resulting job ID once available. Put a unique constraint on the combination that defines the scope, for example tenant plus key. A check-then-insert performed only in application code is not enough: two requests can both pass the check before either writes.
5. Make claiming the key atomic. In a database transaction, attempt to insert the scoped key record. If the insert succeeds, that request owns creation. If the unique constraint reports a conflict, load the existing record and compare its fingerprint. Do not hold a database transaction open while rendering; the transaction should cover only the short state change and reliable job-enqueue operation described next.
6. Connect job creation to queue delivery safely. Avoid committing a job record and then separately publishing to a queue without recovery: a process crash between those actions can leave a job that never runs. A common pattern is a transactional outbox: write the job and an outbox event in the same database transaction, then have a dispatcher publish pending events and mark them delivered. Make the worker safe against receiving the same event more than once, usually by claiming or updating the job with a conditional state transition.
7. Return a consistent response for a retry. When the key already exists with the same fingerprint, return the existing job ID and its current state rather than enqueueing another render. Use a documented response shape and status behavior for both the first request and repeats. If the original request is still being processed, the retry can return that in-progress job; clients can poll its status endpoint.
8. Reject key reuse with different input. If an existing key is submitted with a different fingerprint, return a clear client error rather than silently returning a job for the wrong video or creating another job. Document that the client must use a fresh key for a genuinely new request. Avoid exposing another tenant’s job: look up keys within the authenticated tenant scope and authorize access to the returned job.
9. Choose record retention deliberately. Keep idempotency records for at least as long as clients may retry, including retries after outages. If records expire too soon, a late retry can create a new job. Store enough information to preserve the association without retaining unnecessary sensitive payloads; a fingerprint and job reference may be sufficient. Make expiration behavior explicit and monitor cleanup.
10. Test failure and concurrency cases. Send the same key and payload repeatedly, including simultaneous requests, and confirm only one job is created. Simulate a timeout after the server commits but before the response reaches the client. Also test changed payloads, worker redelivery, database or queue interruptions, and a retry after the record’s retention window. Verify both the API response and the actual number of queued or rendered jobs.

Common Mistakes

Generating a fresh key per HTTP attempt defeats deduplication. So does storing keys only in process memory, which loses protection on restart and fails across multiple API instances. Another frequent mistake is checking for a key and then inserting without a database uniqueness guarantee; concurrent requests can both create work.

Do not treat queue delivery as exactly once. Most queue systems can redeliver messages, so the worker also needs idempotent state changes. Avoid returning an old job when the same key has different parameters, and do not scope keys globally if separate tenants could collide. Finally, do not delete records as soon as rendering finishes if clients can still retry.

Tips

• Document the key’s scope, format, retention period, and behavior when a payload changes.
• Return the same job identifier on successful retries so clients can safely resume polling.
• Track metrics for key conflicts, duplicate attempts, outbox backlog, and jobs stuck in pending states.
• Keep rendering workers independently idempotent; API deduplication alone cannot prevent duplicate effects after queue redelivery.
• Use a stable payload fingerprint and avoid logging raw keys or sensitive video inputs unnecessarily.

Troubleshooting

Retries still create multiple jobs: Confirm the client sends exactly the same key, the database constraint covers the correct tenant scope, and every API instance uses the same durable store. Check whether a new key is being generated after a timeout.

The API returns one job, but it never renders: Inspect the job and outbox records. A committed job with an undelivered outbox event points to a dispatcher or queue issue. Add retry and alerting for pending outbox events; do not ask clients to submit a new key to work around it.

A worker renders twice: Check queue redelivery and worker state transitions. Make the worker claim a job atomically and ensure repeated processing of a completed job has no additional effect. If the rendering provider has its own idempotency mechanism, pass a stable job-derived key where supported.

Clients receive a conflict unexpectedly: Compare the canonical request fields used for the fingerprint. Look for values that change between retries, such as defaults, timestamps, or reordered options. Keep meaningful parameters in the comparison, but normalize representation consistently.

Old retries create new jobs: Review record expiry and the maximum retry period in clients and gateways. Extend retention if necessary, or return a clear expired-key response when you can still identify an expired key; never imply an old key remains protected after its record is deleted.

Frequently Asked Questions

Should an idempotency key be generated by the client or server?
The client should create it before the first attempt so it can reuse the same value after a timeout. The server should validate it and enforce uniqueness.

Can the same key be used for a different video?
No. Treat reuse with different meaningful inputs as a conflict and require a new key for a new logical job.

Does an idempotency key guarantee exactly-once rendering?
No. It prevents duplicate job creation within the key’s scope and retention period. Queue redelivery and worker side effects still need their own safeguards.

What should a retry response contain?
Return the existing job identifier and current status in the documented response format. Clients can then continue polling or fetch the job details.

How long should idempotency records be kept?
Keep them at least as long as supported client, proxy, and operational retry windows. Choose a longer period if delayed retries are plausible, and communicate when protection expires.

Should failed jobs be retried with the same key?
If the retry means checking or resuming the same logical job, keep the key and return that job’s state. If the user is deliberately requesting a new render, create a new key and job.

Related

• Evaluate AI Voice Quality, Latency, and Cost
• Monitor Competitor Websites in India With Screenshots
• Evaluate AI Coding Tools for Accuracy, Privacy and Cost