Roll Back a Nomad, Consul, or Vault Upgrade

Upgrading HashiCorp tools usually goes smoothly, but when a new version introduces a regression or breaks compatibility, you need a fast, safe rollback. This guide covers rolling back Nomad, Consul, or Vault to the previous version, including binary replacement, data directory considerations, and service restart steps. You’ll need CLI access to the servers, the previous version’s binary, and a clear understanding of your cluster’s state.

What You’ll Need

SSH or console access to each server running the service (server and client nodes for Nomad, server nodes for Consul/Vault).
The previous version’s binary (download from HashiCorp’s releases page or use your package manager’s cached version).
Backup of the data directory (especially for Vault) – if you didn’t take one before the upgrade, stop and assess before proceeding.
Knowledge of your service manager (systemd, initd, or Windows Service).
For Vault, the unseal keys and root token (if you need to re-unseal after restart).

Step-by-Step

1. Assess the Situation

First, confirm the upgrade is the cause of the problem. Check logs (e.g., journalctl -u nomad or /var/log/consul/consul.log). If the issue is a configuration change, a rollback might not help. Also, note the current version and the previous version you’re rolling back to.

2. Download the Previous Binary

On each server, download the exact previous version. For example, to roll back from Nomad 1.7.3 to 1.7.2:

curl -O https://releases.hashicorp.com/nomad/1.7.2/nomad_1.7.2_linux_amd64.zip
unzip nomad_1.7.2_linux_amd64.zip

Verify the SHA256 checksum against the published checksum file. Replace nomad with consul or vault as needed.

3. Stop the Service

On each server, stop the service cleanly. For systemd:

sudo systemctl stop nomad

For Consul: sudo systemctl stop consul. For Vault: sudo systemctl stop vault. Wait for the process to exit completely (ps aux | grep nomad should show nothing).

4. Replace the Binary

Back up the current binary (optional) and replace it with the previous version:

sudo mv /usr/local/bin/nomad /usr/local/bin/nomad.1.7.3
sudo mv nomad /usr/local/bin/nomad
sudo chmod +x /usr/local/bin/nomad

Verify the version: nomad version should show the old version.

5. Check Data Directory Compatibility

For Nomad and Consul, data directories are generally forward-compatible but not backward-compatible. If the previous version cannot read the data written by the newer version, you may need to restore a backup. For Vault, the data directory is encrypted and version-sensitive; you must restore a backup taken before the upgrade if the new version wrote to it.

6. Restart the Service

Start the service again:

sudo systemctl start nomad

Watch the logs to ensure it starts cleanly. For Vault, you’ll need to unseal it:

vault operator unseal

Enter the unseal keys as prompted.

7. Verify Cluster Health

Check that the node rejoins the cluster and that all services are healthy. For Nomad: nomad node status. For Consul: consul members. For Vault: vault status.

8. Repeat on All Nodes

Roll back each server, one at a time, watching for issues. For Nomad, roll back servers first, then clients. For Consul, roll back servers before agents. For Vault, roll back all nodes, but be careful with HA standby nodes.

Common Mistakes

Not stopping the service before replacing the binary – the running process may keep the old file in memory, leading to confusion.
Forgetting to restore a Vault backup – Vault’s data directory is not backward-compatible; you must restore a pre-upgrade backup.
Rolling back only one node – a mixed-version cluster can cause protocol mismatches and data corruption.
Ignoring configuration changes – if the new version introduced a config parameter that the old version doesn’t understand, the service will fail to start.
Not checking logs after restart – always verify the service is actually healthy, not just running.

Tips

Before any upgrade, snapshot the data directory (especially Vault) and note the exact version numbers.
Use a dedicated rollback plan: document the binary path, service name, and data directory location for each node.
Test the rollback in a staging environment if possible.
Keep the previous version’s binary in a known location (e.g., /opt/hashicorp/rollback/) to avoid re-downloading.
For Vault, consider using a seal that doesn’t require manual unseal (e.g., auto-unseal with cloud KMS) to speed up recovery.

Troubleshooting

Service fails to start after rollback – Check the logs for version-specific configuration errors. Remove any config parameters added in the newer version. If the data directory is incompatible, restore from backup.

Cluster doesn’t form – Ensure all nodes are on the same version. Check network connectivity and RPC ports. For Consul, run consul operator raft list-peers to see if the node is reachable.

Vault won’t unseal – Verify you have the correct unseal keys. If the data directory was written by the new version, the keys may not work; restore the backup.

Data loss or corruption – If you didn’t back up, you may need to restore from snapshots. For Nomad, jobs may need to be re-submitted. For Consul, service registrations may need to be re-registered.

Frequently Asked Questions

Can I roll back without stopping the service?

No, you must stop the service to replace the binary safely. The running process will continue using the old binary even if you replace the file, so a restart is required.

Will I lose data if I roll back?

For Nomad and Consul, data written by the newer version may be incompatible with the older version, potentially causing loss. For Vault, data is encrypted and version-specific, so you must restore a backup.

How do I know which version to roll back to?

Check your upgrade history or package manager logs. The previous version is the one you were running before the upgrade.

Can I roll back a Nomad client without rolling back the server?

It’s not recommended. Mixed versions can cause protocol mismatches. Roll back all servers first, then clients.

What if the previous version can’t read the data directory?

You’ll need to restore a backup of the data directory taken before the upgrade. If no backup exists, you may need to reinitialize the cluster (for Vault) or re-register services (for Consul).

How long does a rollback take?

Typically 5-15 minutes per node, depending on data size and whether you need to restore a backup. Plan for downtime.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →