keepcalculator

06 Numbers and Powers

Redundancy Plan for High Availability: How to Choose

Card No. 06·78528 · Updated · Standard formula, shown below

This guide helps you choose a redundancy plan for high availability (HA) by breaking down the key decisions: what to protect, how much downtime you can tolerate, and which redundancy pattern fits your budget and operations. You’ll end with a clear, defensible plan.

What You’ll Need

  • A list of your critical systems (applications, databases, network links, power)
  • Estimated downtime cost per hour for each system (or a rough ranking)
  • Your recovery time objective (RTO) and recovery point objective (RPO) for each system
  • Budget range for infrastructure and operational overhead
  • Access to your current architecture diagram (or a willingness to sketch one)

Step-by-Step

  1. List your critical systems and their dependencies. Write down every component that, if it fails, stops revenue, safety, or core operations. Include power, network, compute, storage, and databases. Note which systems depend on others (e.g., the app needs the database).
  2. Define your RTO and RPO for each system. RTO is how long you can be down before it becomes unacceptable. RPO is how much data you can afford to lose (e.g., 5 minutes, 1 hour, 24 hours). These two numbers drive every redundancy decision. If you don’t know them, start with a simple tier: Tier 1 (RTO < 15 min, RPO < 5 min), Tier 2 (RTO < 1 hour, RPO < 15 min), Tier 3 (RTO < 24 hours, RPO < 1 hour).
  3. Calculate the cost of downtime per hour for each tier. Use a rough estimate: lost revenue + lost productivity + recovery labor + reputational damage. Even a rough number helps justify spending. For example, if a system costs $10,000 per hour to be down, a $50,000 redundancy that reduces downtime by 10 hours pays for itself.
  4. Choose the redundancy pattern that matches your RTO/RPO. The most common patterns, from cheapest to most expensive:
    • Backup and restore – RTO in hours, RPO in hours. Good for non-critical data. You back up to a separate disk or cloud, and restore when needed.
    • Cold standby – RTO in hours (often 4-24), RPO in hours. You have a pre-configured but powered-off server or instance. You turn it on, restore data, and go live.
    • Warm standby – RTO in minutes (often 15-60), RPO in minutes. The standby is running but not serving traffic. You replicate data and switch over with a script or manual step.
    • Active-passive (failover) – RTO in seconds to minutes, RPO in seconds. The standby is always running and receiving replicated data. On failure, you flip a VIP or DNS record to the standby. This is typical for databases and load balancers.
    • Active-active (load balancing) – RTO in seconds, RPO in seconds. Multiple nodes serve traffic simultaneously. If one fails, the others absorb the load. This is the highest availability but requires stateless apps or careful database synchronization.
  5. Match the pattern to your budget and operational skill. Active-active is not just more expensive in hardware; it also requires more complex code and database handling. If your team has never run a cluster, a warm standby with a failover script might be more reliable than an active-active setup you can’t maintain.
  6. Plan for the failure of the redundancy itself. Your backup can fail, your standby can be on the same power grid, or your cloud region can go down. For Tier 1 systems, consider redundancy across two availability zones or two physical sites. For lower tiers, at least store backups off-site.
  7. Document the failover procedure and test it. Write down exactly how to switch from primary to standby: who does it, what commands they run, what they check. Schedule a test at least quarterly. A redundancy plan that has never been tested is a plan that will fail when you need it.
  8. Monitor and alert on both sides. Monitor not just the primary but also the standby’s health, data replication lag, and backup success. Set alerts for when replication falls behind or when the standby is unreachable.
  9. Review and update the plan annually. As systems change, your RTO/RPO may change. Re-run the cost analysis and adjust the pattern if needed.

Common Mistakes

  • Ignoring dependencies. You redundantly protect the database but not the network link to it. The whole chain must be redundant.
  • Setting RTO/RPO too tight without budget. A 5-second RTO for a non-critical internal tool is overkill. Be honest about what you really need.
  • Forgetting about data replication lag. A warm standby that replicates every 15 minutes will lose up to 15 minutes of data. If your RPO is 5 minutes, that’s a fail.
  • Testing failover only in “happy path.” Test what happens when the primary is completely destroyed, not just when it gracefully shuts down.
  • Not automating failover. Manual failover takes time and is error-prone under stress. Automate where possible, but ensure you can also do it manually.
  • Choosing active-active without the right architecture. If your app stores session state locally, active-active will break. You need shared session storage or sticky sessions.

Tips

  • Start with the cheapest pattern that meets your RTO/RPO, then add layers only if the cost of downtime justifies it.
  • Use the cloud’s managed services for redundancy (e.g., managed databases with automatic failover) – they reduce operational burden, but verify their RPO/RTO guarantees.
  • For databases, use synchronous replication for zero data loss (RPO=0) but be aware it adds latency. Asynchronous replication is faster but can lose data.
  • Keep your failover runbook short – one page, with exact commands and expected outputs. Long runbooks are ignored.
  • Consider chaos engineering: intentionally kill a node in a test environment to see if your system survives.
  • Remember that redundancy is not just about hardware; it’s also about people. Have a designated on-call person who knows the runbook.

Troubleshooting

  • Failover didn’t work when you tested it. Check the most common causes: the standby wasn’t actually running, the data replication had stopped, the VIP or DNS record didn’t update, or the application can’t connect to the new database. Fix the specific cause, then re-test.
  • Data loss occurred despite having a standby. Check replication lag. If you use asynchronous replication, you will lose data on failover. Consider switching to synchronous replication or accept the RPO.
  • The standby is slower than the primary. This is common if the standby is smaller. For failover to be seamless, the standby should have at least the same capacity as the primary, or you should be prepared for degraded performance.
  • Failover works, but the system is unstable afterward. Often this is because the application wasn’t designed to handle a new database IP or because sessions were lost. Ensure your app uses a connection string that can be updated, and use shared session storage if needed.
  • You can’t afford a full active-active setup. Reconsider your RTO/RPO. Maybe a warm standby is enough for your business. Or you can use a cheaper active-passive setup with a smaller standby that you scale up after failover.
  • Your team doesn’t know how to operate the redundancy tooling. Invest in training or choose simpler tooling. A complex system that no one understands is a liability.

Frequently Asked Questions

What is the difference between RTO and RPO?

RTO (Recovery Time Objective) is how long it takes to get back online after a failure. RPO (Recovery Point Objective) is how much data you can afford to lose – measured in time, e.g., the last 5 minutes of transactions. RTO is about time to recover, RPO is about data loss.

Is active-active always better than active-passive?

Not necessarily. Active-active gives higher availability and better load distribution, but it’s more complex and can be worse if not implemented correctly. If your application isn’t stateless or your database can’t handle multi-writer, active-passive is more reliable.

Can I have redundancy without a second physical site?

Yes, you can have redundancy within one site (e.g., two servers in the same rack), but that doesn’t protect against site-wide failures like power outages or fires. For true high availability, you need geographic redundancy.

How often should I test my failover?

At least quarterly for critical systems, and after any major change to the system or the redundancy setup. More frequent testing (monthly) is better if the system is complex or the cost of failure is high.

What is the cheapest redundancy plan?

The cheapest is backup and restore – just copy your data to an external disk or cloud. It has the highest RTO and RPO, but for non-critical data it’s often sufficient. The next step up is a cold standby, which is still relatively cheap.

Do I need redundancy for every system?

No. Only for systems where the cost of downtime exceeds the cost of redundancy. For low-impact systems, a simple backup is fine. Focus your budget on the few systems that truly run your business.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →