Consul Production Setup: Step-by-Step Guide

This guide walks you through configuring HashiCorp Consul for a production environment, covering the architecture, security settings, and operational practices needed for a reliable service mesh and service discovery layer. You will learn how to set up a secure, highly available Consul cluster with ACLs, TLS encryption, and proper backup procedures.

What You’ll Need

• At least 3 (ideally 5) Linux servers (physical or virtual) for the Consul servers, with a supported OS like Ubuntu 22.04, RHEL 9, or similar.
• Root or sudo access to each server.
• Network connectivity between all servers on the required ports (8300, 8301, 8302, 8500, 8501, 8600).
• Consul binary (version 1.15 or later) downloaded from the official HashiCorp releases page.
• A DNS name for the Consul servers (e.g., consul.example.com) that resolves to the server IPs.
• Optionally, a load balancer to expose the Consul UI and API to clients.

Step-by-Step

1. Install Consul on each server. Download the binary, verify the checksum, and move it to /usr/local/bin/consul. Create a system user: sudo useradd --system --home /etc/consul.d --shell /bin/false consul.
2. Create configuration directories. Create /etc/consul.d (for config files) and /opt/consul (for data). Set ownership: sudo chown -R consul:consul /etc/consul.d /opt/consul.
3. Generate TLS certificates. Use Consul’s built-in command to create a CA and server certificates: consul tls ca create and consul tls cert create -server. Distribute the CA, server cert, and key to each server (e.g., to /etc/consul.d/tls/). For a production setup, you may use your own internal CA, but the process is similar.
4. Set up gossip encryption. Generate a gossip key: consul keygen. This produces a 32-byte base64 key. Save it securely; you’ll need it on every server.
5. Configure the server agent. On each server, create /etc/consul.d/consul.hcl with the following (adjust the IP addresses and hostname):

   server = true
   datacenter = "dc1"
   data_dir = "/opt/consul"
   log_level = "INFO"
   ui = true
   bind_addr = "0.0.0.0"
   client_addr = "0.0.0.0"
   retry_join = ["<server1_ip>", "<server2_ip>", "<server3_ip>"]
   encrypt = "<gossip_key>"
   tls {
     defaults {
       ca_file = "/etc/consul.d/tls/consul-agent-ca.pem"
       cert_file = "/etc/consul.d/tls/dc1-server-consul-0.pem"
       key_file = "/etc/consul.d/tls/dc1-server-consul-0-key.pem"
       verify_incoming = true
       verify_outgoing = true
     }
   }
   acl {
     enabled = true
     default_policy = "deny"
     enable_token_persistence = true
   }
   

   Note: The retry_join list should contain the IPs of all server nodes. If you have a DNS name, you can use that instead.

6. Create a systemd service. Create /etc/systemd/system/consul.service with the standard unit file (ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/). Enable and start the service: sudo systemctl enable consul && sudo systemctl start consul.
7. Verify cluster formation. On any server, run consul members (you may need to use -token later). You should see all servers listed with the ‘alive’ status.
8. Bootstrap ACLs. Since ACLs are enabled, you need to create the initial bootstrap token. On the first server, run: consul acl bootstrap. This outputs a SecretID (the bootstrap token). Save it immediately; you won’t see it again. Use this token for all subsequent commands.
9. Create an agent token for servers. Using the bootstrap token, create a policy for servers and an agent token: consul acl policy create -name "server-policy" -rules 'agent_prefix "" { policy = "write" } node_prefix "" { policy = "write" } service_prefix "" { policy = "read" }' and then consul acl token create -policy-name "server-policy". Note the SecretID. Add this token to the config file on each server as acl { tokens { agent = "<token>" } } and restart Consul.
10. Configure the UI and API access. By default, the UI is served on port 8500. To access it securely, set up a reverse proxy (e.g., Nginx) with TLS, or use Consul’s built-in HTTPS (port 8501) by enabling ports { https = 8501 } in the config. For production, it’s best to restrict access to the UI via a VPN or a load balancer with authentication.
11. Set up backups. Consul’s state is stored in the data directory. For a production cluster, you should take consistent snapshots. Use the consul snapshot save command with the agent token, and store snapshots offsite (e.g., to S3 or a separate backup server). Schedule a cron job to run this hourly or daily.
12. Monitor the cluster. Enable Consul’s telemetry (Prometheus metrics) by adding telemetry { prometheus_retention_time = "24h" } to the config. Then scrape the metrics endpoint (port 8500/v1/agent/metrics?format=prometheus) with Prometheus, and set up alerts for leader changes, node failures, and high CPU.

Common Mistakes

• Not encrypting gossip traffic. Without the encrypt key, all gossip is plaintext, exposing node metadata and allowing eavesdropping.
• Using the same gossip key across environments. Always generate a unique key per cluster.
• Forgetting to set verify_incoming and verify_outgoing. This leaves TLS half-enabled; clients can still connect without encryption.
• Ignoring ACL default policy. Setting default_policy = "allow" is a common misconfiguration that leaves your cluster open. Always use deny and create explicit policies.
• Not backing up the bootstrap token. If you lose it, you’ll have to reset ACLs, which is disruptive.
• Running a single server. A single server is not a cluster; you need at least 3 for quorum. A 2-node cluster can cause split-brain issues.

Tips

• Use a configuration management tool (Ansible, Terraform) to deploy and manage Consul consistently across nodes.
• Set performance { raft_multiplier = 1 } for lower latency in high-throughput environments (default is 5, which is more conservative).
• Use Consul’s retry_join with a DNS name that resolves to all server IPs, so you don’t have to update the config when adding nodes.
• Enable enable_syslog to send logs to syslog for central collection.
• Regularly test your backup restoration procedure to ensure you can recover from a disaster.
• Use Consul’s verify_server_hostname setting to ensure clients verify the server’s hostname in the TLS certificate, preventing man-in-the-middle attacks.

Troubleshooting

• Cluster doesn’t form: Check that the ports are open (8301 for gossip, 8300 for RPC). Use consul members with the bootstrap token to see the state. Ensure the gossip key matches on all nodes.
• ACL errors (403 Forbidden): You likely haven’t set the agent token correctly. Verify the token in the config file and that the policy includes agent_prefix and node_prefix write permissions.
• TLS handshake failures: Check that the CA file is the same on all nodes, and that the server certificate matches the hostname/IP you’re connecting to. Use consul tls cert create -server with the correct -dc and -node flags.
• Leader election issues: If you see frequent leader changes, check for network latency or clock skew. Use consul operator raft list-peers to see the raft state.
• UI not loading: Ensure ui = true is set and that the client_addr is accessible. If using HTTPS, make sure the port 8501 is open.
• Snapshot restore fails: Ensure you’re using the same Consul version for backup and restore. Restore to a fresh cluster, not an existing one.

Frequently Asked Questions

How many Consul servers do I need for production?

At least 3, but 5 is recommended for larger environments. This ensures quorum (majority) can be maintained even if one node fails. With 3 nodes, you can lose 1; with 5, you can lose 2.

Can I run Consul clients on the same nodes as servers?

Yes, but it’s not recommended for production. Separating servers and clients helps isolate failures and improve performance.

How do I rotate the gossip encryption key?

Use the consul keygen command to generate a new key, then update the encrypt parameter on all nodes and restart them one by one. Consul supports key rotation by specifying multiple keys in the config (using encrypt for the primary and encrypt_verify_incoming for old ones).

What is the difference between a server and a client in Consul?

Servers maintain the cluster state and respond to queries. Clients are lightweight processes that forward requests to servers and participate in health checks. Services register with the local client.

How do I add a new server to an existing cluster?

Install Consul on the new server, configure it with the same gossip key and TLS certs, and set retry_join to include the existing server IPs. Start the agent; it will automatically join the cluster and become a non-voting member initially, then promote itself after the leader approves.

Is it safe to expose the Consul API to the public internet?

No. The API can be used to modify cluster state. Always restrict access via firewalls, VPNs, or a reverse proxy with authentication. Use ACLs to limit even internal access.

Related

• AI Agent Harness vs Framework: What’s the Difference?
• OpenAI Decisions API: Practical Implementation Guide
• AI Cloud Explained: How It Works and Why It Matters