Background Jobs Failing Quietly During Deploy?

Yes, your background job can silently stop working during a deploy, and it usually does so by abandoning work rather than crashing loudly. This happens because the deploy process interrupts the job runner (like Sidekiq, Celery, or Resque) mid-task, and unless the runner is configured to retry or drain gracefully, the job is simply lost. The fix is a combination of graceful shutdown signals, retry logic, and a deployment strategy that waits for in-flight jobs to finish.

Why Jobs Get Abandoned During Deploy

Background job runners process tasks from a queue. When you deploy new code, you typically restart the application server and the job runner processes. If the job runner receives a termination signal (SIGTERM) while it is in the middle of executing a job, it has a short window to finish or save its state. If it doesn’t, the job is marked as “in progress” in the queue, but the worker is gone. The job never completes, and unless there is a retry mechanism, it stays stuck forever.

The most common scenario is a deploy script that runs systemctl restart sidekiq or kill -TERM without giving the worker time to finish. The default behavior of many runners is to stop accepting new jobs and then wait a few seconds before forcing a kill. If the job takes longer than that timeout, it is interrupted mid-execution.

How Job Runners Handle Shutdown

Modern job runners have a defined shutdown sequence:

• Receive SIGTERM (or a platform-specific signal).
• Stop pulling new jobs from the queue.
• Wait for the current job to finish (up to a configurable timeout).
• If the timeout is exceeded, force-kill the worker.

Sidekiq, for example, has a timeout option in its configuration, defaulting to 25 seconds. Celery has worker_shutdown_timeout (default 60 seconds). Resque uses a simple TERM signal and a polling loop. The key is that the deploy process must send SIGTERM, not SIGKILL, and must wait for the process to exit before moving on. Many deploy tools (Capistrano, Kubernetes, systemd) do this correctly, but a misconfigured script can send SIGKILL immediately, which gives the worker no chance to save state.

What Happens to the Interrupted Job

When a worker is killed mid-job, the job stays in the queue with a status of “in progress” or “running.” Depending on the runner and the queue backend, this can lead to three outcomes:

• Automatic retry: Sidekiq and Celery will re-queue the job after a timeout (default 5 minutes for Sidekiq, 3 minutes for Celery) if the worker dies unexpectedly. This is the most common behavior.
• Stuck forever: If the runner doesn’t have a retry mechanism, or if the job was marked as “done” before the actual side effect (e.g., sending an email) was completed, it may never be re-run.
• Duplicate execution: If the job was partially completed (e.g., a database record was updated) and then re-queued, it will run again, potentially causing double side effects.

The “quietly abandoning” part usually happens when the job is re-queued but the code that processes it has changed during the deploy. The new code may fail to handle the old job’s payload, or the job may be dropped because the queue is purged during deploy (a common mistake with in-memory queues like Redis without persistence).

How to Detect Silent Job Loss

You won’t see a crash log because the process was killed, not crashed. Here are the signs:

• Jobs that are supposed to run (e.g., sending a welcome email) never arrive.
• The queue dashboard shows a growing number of “retry” or “scheduled” jobs.
• Workers disappear from the process list during deploy, but no error is logged.
• Database records that should have been updated by a job remain stale.

To detect this, you need monitoring on the queue itself, not just the worker process. Tools like Sidekiq’s Web UI, Celery Flower, or a custom metric that counts jobs processed vs. jobs enqueued will show a gap. A simpler check: after a deploy, compare the number of jobs in the queue before and after. If it drops but no job completed, that’s a red flag.

Best Practices to Prevent Abandonment

There is no single fix, but a combination of the following will eliminate most silent losses:

1. Send SIGTERM and wait: Your deploy script should send SIGTERM (not SIGKILL) and then wait for the process to exit. In systemd, use TimeoutStopSec= to give the worker enough time. In Kubernetes, use terminationGracePeriodSeconds.
2. Configure a generous shutdown timeout: Set the worker’s timeout to be longer than your longest job. For Sidekiq, that’s timeout: 60 in your config. For Celery, worker_shutdown_timeout: 120.
3. Enable retry with backoff: Ensure your runner is configured to retry jobs that fail due to worker death. Sidekiq does this by default; Celery requires task_acks_late = True and a retry policy.
4. Make jobs idempotent: If a job may run twice, design it so running it twice has no side effects. For example, use a unique key in the database to prevent duplicate emails.
5. Don’t purge the queue on deploy: Some deploy scripts clear the queue to “reset” it. This is almost always a mistake. Instead, let the queue drain naturally.
6. Use a separate queue for deploys: If you have long-running jobs, consider a separate worker pool that is not restarted during deploy. You can deploy code to the main app, then restart the long-running workers after a delay.

Common Misconception: “The Job Was Lost Because of a Bug”

Many developers assume that a missing job is due to a code error, but the deploy itself is often the culprit. A job that runs for 30 seconds and is killed at second 29 will not log an error. The worker process simply disappears. The job is re-queued, but if the queue is not persistent (e.g., Redis without AOF), a restart of Redis during deploy can wipe the queue entirely. That’s why you should always use a persistent queue backend and never rely on in-memory storage for critical jobs.

Comparing Job Runners: Sidekiq vs. Celery vs. Resque

Each runner has different default behaviors around shutdown:

• Sidekiq (Ruby): Handles SIGTERM gracefully by default, waits for the current job, and re-queues it if killed. Requires a timeout setting. Best for Rails apps.
• Celery (Python): More configurable, but defaults to acking tasks early, which means a killed worker loses the task unless task_acks_late=True is set. Best for Django/Flask.
• Resque (Ruby): Simpler, but less graceful. It will mark a job as “failed” if the worker dies, and you need a separate plugin to retry. Best for small apps.

The key difference is whether the runner acknowledges the task before or after execution. If it acks before (Celery default), a crash means the task is lost. If it acks after (Sidekiq default), the task is re-queued. Always check your runner’s ack behavior.

Frequently Asked Questions

How do I know if my background job was abandoned during deploy?

Check your queue dashboard for jobs stuck in “in progress” or “retry” status. Also, compare the number of jobs processed before and after a deploy. If you see a gap, and no error logs, it was likely killed during shutdown.

What is the difference between SIGTERM and SIGKILL?

SIGTERM is a polite request to stop, allowing the process to clean up. SIGKILL immediately terminates the process without any cleanup. Your deploy script must send SIGTERM first and wait for the process to exit before sending SIGKILL as a last resort.

Will a job be re-queued if the worker is killed?

It depends on the runner. Sidekiq re-queues by default. Celery only re-queues if task_acks_late=True is set. Resque does not re-queue by default. Check your runner’s documentation.

Can I prevent a job from running twice after a deploy?

Yes, by making your jobs idempotent. For example, before sending an email, check if a record with a unique key exists. If it does, skip the job. This is the only reliable way to avoid duplicates.

Is it safe to restart Redis during deploy?

Only if Redis has persistence enabled (RDB or AOF). If not, you will lose all queued jobs. For critical jobs, use a database-backed queue (like Sidekiq’s ActiveRecord adapter) or ensure Redis persistence is on.

What is a good shutdown timeout value?

Set it to the maximum time your longest job takes, plus a buffer. If your longest job takes 2 minutes, set the timeout to 180 seconds. This ensures the worker can finish what it started.

Related

• Smart Ring vs Smart Watch: Which Wearable Should You Choose?
• Used Japanese Flip Phone: 10 Checks Before You Buy
• Google Chat vs Slack Emoji Reactions Compared