Why PostgreSQL WAL Files Are Not Deleted

PostgreSQL WAL (Write-Ahead Log) files are not deleted when they are no longer needed for crash recovery; they are removed only after they are no longer required by any replication standby, a PITR (Point-In-Time Recovery) backup, or an archive command. If you notice WAL files accumulating, the cause is almost always one of these retention mechanisms, not a disk leak. This article explains the concrete reasons WAL files persist, how to diagnose which one applies to your server, and what to do about it.

What PostgreSQL WAL Files Are For

Write-Ahead Logging is the mechanism PostgreSQL uses to guarantee durability and crash safety. Before any change to a data page is written to disk, a record describing that change is written to the WAL. If the server crashes, PostgreSQL replays the WAL from the last checkpoint to restore the database to a consistent state. The WAL is split into 16 MB segment files (the default wal_segment_size), which are stored in the pg_wal directory (formerly pg_xlog). These files are recycled (renamed and reused) or deleted once their contents are no longer needed for recovery.

However, “no longer needed for recovery� is not the only condition. PostgreSQL also retains WAL files that are still required by:

• Replication slots (both physical and logical)
• Standby servers that are lagging behind the primary
• An archive_command that is failing or too slow
• PITR backups that have not been marked as complete
• The wal_keep_size parameter (or the older wal_keep_segments)

Each of these can cause WAL files to pile up, and the fix differs depending on which one is the culprit.

How PostgreSQL Decides to Delete WAL Files

After a checkpoint, PostgreSQL can discard WAL files that are older than the checkpoint’s redo location, but only if no other component needs them. The server maintains a “minimum recovery point� that is the oldest WAL position that might be required for recovery. This is determined by:

• The last checkpoint’s redo location
• The oldest replication slot’s restart_lsn
• The oldest WAL file still needed by a connected standby (based on its reported flush position)
• The wal_keep_size setting, which forces retention of a certain amount of recent WAL
• The archive module’s status (if archiving is enabled, files must be archived before they can be removed)

PostgreSQL periodically runs a check (during CheckPointWal and other routines) and removes or recycles WAL files that are no longer needed by any of these. If any one of these conditions is not met, the file stays.

Main Reasons WAL Files Accumulate

1. Replication Slots Not Advanced

Replication slots are used to prevent WAL from being removed before a standby or logical decoding consumer has processed it. If a slot’s restart_lsn is far behind the current write position, PostgreSQL will keep all WAL from that point onward. This often happens when:

• A standby is disconnected for a long time.
• A logical replication subscriber is down or slow.
• A slot was created but never used (e.g., for a failed setup).

To check for slots, run:

SELECT slot_name, slot_type, restart_lsn, active FROM pg_replication_slots;

If a slot is inactive or no longer needed, drop it with pg_drop_replication_slot('slot_name') (or the appropriate function for logical slots).

2. Standby Servers Lagging Behind

Even without explicit slots, a connected standby that is behind the primary will cause the primary to keep WAL files until the standby reports that it has flushed them. If the standby is slow (due to network issues, disk I/O, or heavy load), WAL will accumulate. Check the primary’s view:

SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn FROM pg_stat_replication;

If flush_lsn is far behind sent_lsn, the standby is not keeping up. You may need to increase resources on the standby or reduce load on the primary.

3. Archive Command Failing or Too Slow

If archive_mode is on and archive_command is set, PostgreSQL will not remove a WAL file until it has been successfully archived. If the archive command fails (e.g., because the backup destination is full, permissions are wrong, or the command itself has a bug), WAL files will pile up indefinitely. Check the PostgreSQL log for messages like archive command failed with exit code 1. The pg_stat_archiver view shows the last archive attempt and its success:

SELECT * FROM pg_stat_archiver;

If failed_count is increasing, fix the archive command or the destination. You can also manually archive the backlog by running the archive command on the oldest files, or temporarily set archive_mode to off (but that disables PITR).

4. PITR Backups Not Completed

When you take a base backup (e.g., with pg_basebackup), PostgreSQL records that backup’s start location. WAL files generated after that point must be kept until the backup is considered complete (usually when you stop the backup and take a final WAL segment). If the backup process is interrupted or never finalized, the server may retain WAL files indefinitely. In practice, this is less common than the other causes, but it can happen if a backup script is killed.

5. wal_keep_size / wal_keep_segments

These parameters force PostgreSQL to retain a minimum amount of WAL for potential standby connections. If set too high, you will keep more WAL than necessary. The default is 0 (no extra retention), but some administrators set it to a large value for safety. To see the current setting:

SHOW wal_keep_size;  -- or wal_keep_segments on older versions

If you have replication slots and a proper archive, you can usually set this to 0.

How to Diagnose Which Reason Applies

Follow these steps to identify the cause on your server:

1. Check the age of the oldest WAL file in pg_wal (e.g., ls -lt pg_wal | tail). If it’s very old, something is holding it.
2. Run SELECT * FROM pg_replication_slots; – if any slot has restart_lsn far behind, that’s the cause.
3. Run SELECT * FROM pg_stat_replication; – if a standby is lagging, that’s the cause.
4. Check pg_stat_archiver for failures.
5. Check SHOW wal_keep_size; – if it’s non-zero, it might be too high.
6. Look at the PostgreSQL logs for any messages about WAL being “required by� something.

If none of these show an issue, it’s possible that the server simply has a very high write volume and the WAL is being recycled normally, but the directory size is large because the checkpoint frequency is low. In that case, you may need to tune checkpoint_timeout and max_wal_size.

Common Misconceptions and Pitfalls

Misconception: “WAL files are never deleted, only recycled.� It’s true that PostgreSQL recycles WAL segments (renames them for reuse) rather than deleting them, which keeps the directory at a stable size. However, if the retention conditions force keeping more than the usual number, the directory grows. Recycling only happens when the file is no longer needed; if it’s needed, it stays.

Pitfall: Dropping a replication slot that is still in use. If you drop a slot that a standby is using, that standby will fail when it tries to fetch WAL from a position that no longer exists. Always ensure the slot is inactive or that you are intentionally reconfiguring the standby.

Pitfall: Setting wal_keep_size to a huge value “just in case.â€� This can fill your disk with WAL that is never needed. Use replication slots instead, which are more precise.

Costs and Safety of Retaining WAL

The main cost is disk space. Each WAL segment is 16 MB, and with a high write load, you can generate many segments per minute. If you retain them for hours or days, you can easily consume tens of gigabytes. There is also a small performance cost because the server must track and manage more files, but this is usually negligible compared to disk space.

Safety-wise, you should never manually delete WAL files that are still required by a standby or a PITR backup. Doing so will break replication or make it impossible to restore to a point in time. Always fix the root cause rather than deleting files. If you absolutely must free space immediately, you can temporarily disconnect a lagging standby (after ensuring it can be re-synced) or drop an unused replication slot, but do this with caution.

Alternatives and Best Practices

To avoid WAL accumulation in the long term:

• Use replication slots for physical and logical replication, and monitor them for lag.
• Set archive_command to a reliable destination and test it regularly.
• Monitor pg_stat_replication and pg_stat_archiver with a monitoring tool.
• Set max_wal_size appropriately for your workload (default is 1 GB). If you have frequent large transactions, you may need to increase it to avoid too-frequent checkpoints.
• If you do not use PITR or replication, you can disable archiving and replication slots, and set wal_keep_size to 0.
• Consider using pg_receivewal for continuous archiving instead of a custom script, as it is more robust.

If you are already facing a full disk, the immediate fix is to identify the retention cause and resolve it, then let PostgreSQL recycle the WAL normally. In extreme cases, you can move the pg_wal directory to a larger disk (by stopping the server and changing the wal_log_hints or symlink), but that is a last resort.

Frequently Asked Questions

Q: Can I just delete old WAL files to free space?
Only if you are sure they are not needed. Deleting WAL that is still required by a standby or PITR will break recovery and replication. Always diagnose first.

Q: Why is my pg_wal directory so big even though I have no replication slots?
Check if archiving is enabled and failing, or if wal_keep_size is set. Also check if there are any stale replication slots that you might have missed (they can be inactive but still hold WAL).

Q: How do I find out which WAL file is the oldest and why it’s kept?
Use pg_waldump on the oldest file to see its timeline and LSN, then compare with the restart_lsn of slots and the archived status. The PostgreSQL log often says “removing unnecessary WAL segmentsâ€� and lists the oldest required LSN.

Q: Does vacuuming or autovacuum affect WAL retention?
No. WAL retention is independent of vacuum. Vacuum generates WAL, but it does not prevent deletion of old WAL.

Q: What is the difference between wal_keep_size and replication slots?
wal_keep_size simply retains a fixed amount of WAL regardless of whether a standby needs it. Replication slots retain WAL only up to the point a standby has confirmed receipt, which is more efficient.

Q: How often does PostgreSQL check for WAL to remove?
It checks during checkpoints and also when a standby advances its flush position. If you have a long checkpoint interval, WAL may accumulate temporarily, but it will be cleaned at the next checkpoint.

Related

• Sweet Herring Dreamlight Valley: Recipe & How to Cook
• GTA 6 Music System: Stream Playlists Explained
• Dollar Cost Averaging: How It Works