For most applications, PostgreSQL job queues are the safer default, while Redis queues win when you need very high throughput and can accept some operational complexity. This article compares both approaches on reliability, performance, feature set, and operational cost so you can pick the right one for your project.
Background: why job queues exist
Job queues decouple work from the request that triggers it. Instead of sending an email, resizing an image, or calling a third-party API inside a web request, you push a description of that work into a queue. A separate worker process pulls the job and executes it. This makes your web app respond faster, lets you retry failed work, and lets you scale workers independently.
Two of the most common places to store that queue are PostgreSQL and Redis. PostgreSQL is a relational database that has supported transactional queues for years, either with plain tables or with the SKIP LOCKED clause added in version 9.5. Redis is an in-memory data store that has native list and sorted-set types, which map naturally to queue operations. Both are widely used, but they make different trade-offs.
How PostgreSQL queues work
A PostgreSQL queue is just a table with a status column. You insert a row with status pending, and workers claim it by updating the status to processing. The key trick is SELECT ... FOR UPDATE SKIP LOCKED, which lets multiple workers claim different rows without blocking each other.
Example schema and claim query:
CREATE TABLE jobs ( id BIGSERIAL PRIMARY KEY, payload JSONB NOT NULL, status TEXT NOT NULL DEFAULT 'pending', attempts INT NOT NULL DEFAULT 0, run_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- Worker claims one job atomically UPDATE jobs SET status = 'processing', attempts = attempts + 1 WHERE id = ( SELECT id FROM jobs WHERE status = 'pending' AND run_at <= now() ORDER BY run_at FOR UPDATE SKIP LOCKED LIMIT 1 ) RETURNING *;
This gives you transactional integrity: if the worker crashes after claiming, the job stays in processing and a sweeper can reset it. You can also use the same transaction to update application data and mark the job done, which is impossible with a separate queue system.
Libraries like pg-boss (Node.js), GoodJob (Rails), and django-pgqueue build on this pattern. They add features like scheduled jobs, retries with backoff, and dead-letter queues.
How Redis queues work
Redis queues use the LPUSH and BRPOP commands on a list. A producer pushes a serialized job onto the left side of a list, and workers block on the right side waiting for work. This is the model behind Sidekiq (Ruby), RQ (Python), and Bull (Node.js).
For delayed jobs, Redis uses sorted sets with a score equal to the Unix timestamp when the job should run. A worker periodically moves due jobs from the sorted set to the list. For retries and dead letters, you use additional lists and sorted sets.
Redis is fast because everything lives in memory. A single Redis instance can push and pop tens of thousands of jobs per second. The BRPOP command also gives you blocking semantics, so workers don’t poll the database repeatedly; they just wait until a job arrives.
Key differences: durability and consistency
The most important difference is what happens when the server crashes. PostgreSQL is durable by default: every committed transaction is written to disk and survives a crash. Redis, unless you enable appendfsync always, can lose the most recent writes because it flushes to disk asynchronously. Even with appendfsync always, Redis still has a small window where a crash can lose data, and it is much slower with that setting.
This matters for jobs you cannot afford to lose, such as payment webhooks, order fulfillment, or account changes. If a Redis queue loses a job, that work never happens. PostgreSQL queues do not lose committed jobs.
Another difference is atomicity with your application data. With PostgreSQL, you can update a user’s balance and mark the job as done in the same transaction. With Redis, you have two separate systems, so you need a distributed transaction or an outbox pattern to keep them consistent.
Performance and throughput
Redis is faster for pure queue operations. A benchmark on typical hardware might show Redis handling 50,000–100,000 push/pop operations per second, while PostgreSQL handles 1,000–5,000 per second depending on the query and indexes. Redis also has lower latency because it is in memory.
However, PostgreSQL’s throughput is often sufficient. Most applications do not process more than a few hundred jobs per second. If you need more, you can shard the queue across multiple tables or use a dedicated queue database. PostgreSQL also benefits from connection pooling, but high concurrency on a single table can cause lock contention.
For very high throughput (over 10,000 jobs/second), Redis is the practical choice. But you must accept the durability trade-off or run Redis with careful persistence settings and replica failover.
Operational complexity and cost
If you already run PostgreSQL, adding a queue costs nothing extra. You do not need to deploy, monitor, and back up another service. This is a huge advantage for small teams and single-server applications.
Redis adds a second data store to operate. You need to manage memory, configure persistence (RDB snapshots or AOF), set up replication for high availability, and handle failover. Redis memory usage grows with queue size, so a backlog of millions of jobs can consume gigabytes of RAM. PostgreSQL stores the same jobs on disk, which is cheaper.
There is also a middle path: use Redis for high-throughput transient jobs and PostgreSQL for critical jobs. Some teams run both, routing each job type to the appropriate store.
Feature comparison and ecosystem
Both ecosystems offer mature libraries with similar features:
- Scheduled/delayed jobs: both support them (PostgreSQL via
run_atcolumn, Redis via sorted sets). - Retries with backoff: both have libraries that implement exponential backoff.
- Dead-letter queues: both can move failed jobs to a separate table or list.
- Job priorities: PostgreSQL can order by a priority column; Redis can use multiple lists or a sorted set.
- Rate limiting: Redis has native primitives for this; PostgreSQL requires application logic.
- Unique jobs: PostgreSQL can enforce a unique index; Redis needs a separate set or lock.
- Bulk operations: PostgreSQL can update thousands of jobs in one statement; Redis needs a pipeline.
Redis libraries like Sidekiq and Bull are extremely polished and have large communities. PostgreSQL libraries are slightly less feature-rich but are improving quickly. If you need a feature that is missing from your current library, you can often implement it with a few SQL queries or Lua scripts.
How to choose: a practical decision guide
Start with PostgreSQL queues if any of these apply:
- You already use PostgreSQL and do not want another service.
- You cannot lose jobs (financial, email, critical data).
- Your job rate is under a few thousand per second.
- You need transactional consistency between jobs and application data.
- You are on a single server or a small cluster.
Choose Redis queues if:
- You need very high throughput (over 10,000 jobs/second).
- You need sub-millisecond latency for queue operations.
- You can tolerate losing a few jobs on crash (e.g., analytics, cache warming).
- You already operate Redis and have monitoring and backup in place.
- You need advanced rate limiting or real-time features.
If you are unsure, start with PostgreSQL. You can always migrate to Redis later if you outgrow it, and the job processing logic in your workers will be similar.
Common pitfalls and misconceptions
A common misconception is that PostgreSQL queues are always slow. With proper indexing and SKIP LOCKED, they are fast enough for most workloads. The bottleneck is usually the worker code, not the database.
Another pitfall is forgetting to set a timeout on Redis BRPOP. If the worker blocks forever and Redis restarts, the worker may hang. Use a timeout and reconnect logic.
With PostgreSQL, a common mistake is not adding an index on (status, run_at). Without it, the claim query does a full table scan, which destroys performance. Also, remember to use FOR UPDATE SKIP LOCKED; a plain SELECT will cause workers to claim the same job.
For Redis, be careful with memory. If a producer outpaces consumers, the list grows without bound. Set a maximum queue length or use a monitoring alert. Also, do not use KEYS in production; use SCAN.
Alternatives worth considering
Besides PostgreSQL and Redis, there are dedicated queue systems like RabbitMQ, Apache Kafka, and Amazon SQS. RabbitMQ is a full message broker with advanced routing and delivery guarantees. Kafka is designed for event streaming and replay, not simple job queues. SQS is a managed service that handles scaling and durability for you.
For most applications, these are overkill. If you need exactly-once delivery or complex routing, RabbitMQ is a strong option. If you need to replay events, Kafka is better. If you want zero operational overhead and are on AWS, SQS is convenient.
Frequently Asked Questions
Can PostgreSQL handle a million jobs in a queue?
Yes. A table with a million rows is not a problem for PostgreSQL, especially with an index on (status, run_at). The claim query will be fast because it uses the index. The main concern is disk space and vacuum maintenance, but it is manageable.
Is Redis durable enough for job queues?
It depends on your tolerance for loss. With default settings, Redis can lose the most recent writes on a crash. Enabling appendfsync always reduces the window to near zero but slows writes. For critical jobs, use PostgreSQL or a managed queue like SQS.
Can I use both PostgreSQL and Redis for queues?
Yes. Many teams use Redis for high-throughput, non-critical jobs (like analytics or notifications) and PostgreSQL for critical jobs (like billing or account changes). This gives you the best of both worlds at the cost of operating two systems.
Which is easier to scale horizontally?
Redis is easier to scale for raw throughput because you can add more workers and Redis handles many concurrent connections. PostgreSQL can also scale with read replicas, but writes go to the primary, so you may need to shard the queue table for very high write rates.
Does PostgreSQL have a built-in job queue?
No, but it has the primitives (tables, transactions, SKIP LOCKED) to build one. Several open-source libraries provide a full implementation. Redis also has no built-in queue; it provides data structures that libraries use.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Microsoft Surface Duo: The Foldable Phone Story
- Samsung SM-T813 Hard Reset: Step-by-Step Guide
- Mafia Clash Codes: Real Meanings & Usage
- Remove Programs from Mac: Full Uninstall Guide
- Project Sentinel Morning Digest: Data Stack Failure Alerts
- Best KVM VPS Hosting: Top Picks & Guide