keepcalculator

06 Numbers and Powers

Blue Green Deployments vs Bad Database Migrations

Card No. 06·37854 · Updated · Standard formula, shown below

Blue-green deployments cannot rescue a bad database migration because they only swap application versions, not the schema or data underneath them. A destructive or incompatible migration (like dropping a column or renaming a table) is already applied to the shared database before the new app version goes live, so rolling back to the “green” environment doesn’t undo the damage. This article explains why that is, what blue-green actually protects you from, and what you can do instead.

How Blue-Green Deployments Work

Blue-green deployment is a release strategy that reduces downtime and risk by running two identical production environments, traditionally called Blue and Green. At any time, only one environment (say Blue) serves live traffic. The new application version is deployed to the idle environment (Green), where it is tested and verified. Once ready, the load balancer or router switches all traffic from Blue to Green in one atomic cutover. If something goes wrong, you can switch back to Blue almost instantly because it still runs the old version.

The key benefit is that the switch is at the network level, not the code level. You are not rolling back code; you are redirecting requests. This makes the rollback fast and clean for the application tier. The database, however, is a shared stateful resource that both environments must use. You cannot have two separate production databases that stay in sync during a release without significant replication complexity, so in practice, Blue and Green point to the same database.

Why the Database Is the Weak Link

Database migrations are changes to the schema (tables, columns, indexes) or the data itself. They are typically run as a separate step before, during, or after the application deployment. Because the database is shared, a migration affects both Blue and Green simultaneously. If the migration is bad—say it deletes a column that the old code still references, or it corrupts data—then both environments are broken. Switching traffic back to Blue does not restore the column or repair the data; the migration has already been executed.

Even if you use a migration tool that supports rollback (like Rails’ revert or Flyway’s undo), the rollback is not automatic and may not be safe. For example, if the migration removed a column, rolling back would re-add it, but any data that was deleted in the process is gone unless you backed it up. If the migration transformed data (e.g., changing a string to an integer), rolling back may not restore the original values. The database is the source of truth for your data, and once you change it, reversing the change is often impossible or lossy.

What Blue-Green Actually Protects Against

Blue-green is excellent for catching application-level issues: a bug in the new code, a misconfigured environment variable, a missing dependency, or a performance regression that shows up under real traffic. It also helps with version compatibility during a rolling upgrade, because you can keep the old version running until the new one is proven stable. The strategy is particularly effective for stateless services that can be scaled and swapped without affecting persistent data.

However, it does nothing to protect against schema drift, data corruption, or irreversible data loss. These are database problems, and they require database solutions. A common misconception is that blue-green is a substitute for proper backup and restore procedures. It is not. If your migration drops a table, you need a backup to recover the data, not a traffic switch.

Common Database Migration Failures

Here are typical ways a migration goes wrong, each of which blue-green cannot fix:

  • Destructive operations: Dropping a column, table, or index that is still needed by the old application version.
  • Data loss: A migration that deletes rows, truncates a table, or overwrites values without a backup.
  • Locking and downtime: A migration that takes a long time, locking tables and causing outages that affect both environments.
  • Incompatible schema changes: Changing a column type in a way that breaks the old code’s queries (e.g., changing a string to an integer).
  • Partial failure: A migration that fails midway, leaving the schema in an inconsistent state.

In each case, the damage is done to the shared database. Blue-green cannot rewind the clock.

How to Actually Handle Bad Database Migrations

Since blue-green can’t save you, you need a different set of practices. The most reliable approach is to make migrations backward-compatible, so that both the old and new application versions can run against the same schema. This is often called expand-and-contract or parallel change.

  1. Expand: Add the new column, table, or index without removing anything old.
  2. Migrate data: Backfill or transform data in a separate step, while the old code still works.
  3. Switch: Deploy the new application version that uses the new schema.
  4. Contract: Once the new version is stable, remove the old schema elements in a later release.

This avoids destructive changes that break the old code. Additionally, always take a full backup before running any migration, and test the migration on a copy of production data. Use tools that support transactional DDL where possible (like PostgreSQL), so a failed migration can be rolled back automatically. Also consider using a migration tool that generates rollback scripts, but verify them in staging.

When Blue-Green Can Help with Migrations

There is a narrow case where blue-green can assist: if the migration is purely additive and reversible, and you have a way to run the migration only on the green environment before switching. For example, you could add a new column to the database, deploy the new code to green, and switch traffic. The old blue environment still runs the old code, but since the new column is additive, the old code ignores it. If the new code has a bug, you switch back to blue, and the database still has the extra column—but that is harmless. This only works if the migration is non-destructive and the old code is compatible with the new schema.

In practice, this requires careful coordination and a migration that is safe to run while the old version is live. Many teams use blue-green for the application and a separate, more cautious migration process for the database, often running migrations before the switch and verifying they are backward-compatible.

Costs and Complexity of Blue-Green

Running two full production environments doubles your infrastructure costs—you pay for twice the compute, storage, and networking. You also need to keep the two environments in sync (except for the deployed version), which adds operational complexity. Database synchronization is the hardest part: you cannot have two independent databases that both accept writes, because they would diverge. So you must use a single database, which defeats the isolation benefit for schema changes.

Some teams mitigate this by using a shared database but with read replicas, but that doesn’t solve the migration problem either. The migration still affects the primary database. The cost and complexity are justified for high-traffic services where zero-downtime deployments are critical, but they are not a silver bullet for database issues.

Frequently Asked Questions

Can blue-green deployments work with zero-downtime migrations?

Yes, but only if the migration is backward-compatible and non-destructive. You can run the migration on the shared database before switching traffic, as long as the old code can still operate with the new schema. For example, adding a column is fine; dropping a column is not.

What is the difference between blue-green and rolling deployment?

Blue-green uses two full environments and switches traffic atomically, while rolling deployment updates instances one by one in the same environment. Rolling is cheaper but has a longer cutover window and can mix versions during the update. Neither handles database migrations.

Should I use a migration tool that supports rollback?

Yes, but rollback is not a substitute for backups. Tools like Flyway, Liquibase, or Alembic can generate rollback scripts, but they may not restore data that was deleted or transformed. Always take a backup before migrating and test the rollback in staging.

How do I test a migration safely?

Run the migration against a copy of your production data in a staging environment. Verify that both the old and new application versions work with the migrated schema. Also test the rollback script. This catches most issues before they hit production.

What is the expand-and-contract pattern?

It is a migration strategy that makes changes backward-compatible by first adding new schema elements, then deploying the new code, and finally removing old elements in a later release. This allows blue-green or rolling deployments to work without breaking the old version.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →