Database Migration: When Are Dual Writes and Shadow Tests Safe?

Dual writes and shadow tests are safe only when you treat them as temporary verification steps, not as a permanent architecture, and when you have a rollback plan that does not depend on reverting the writes themselves. They are safe for validating a migration when the old system remains the source of truth, the new system is treated as a replica, and you can reconcile and discard the shadow data before cutover. This article explains the conditions under which these techniques work, their risks, and how to use them without corrupting your data.

Background: Why Migrations Are Hard

Changing a database schema, moving to a new database engine, or switching from a monolithic datastore to microservices often requires a period where both the old and new systems run side by side. The challenge is that you cannot atomically switch all reads and writes from one system to another without downtime or data loss. Two common strategies to reduce risk are dual writes (writing to both systems during the transition) and shadow testing (sending a copy of production traffic to the new system without using its results). Both are used to build confidence that the new system behaves correctly before you commit to it.

How Dual Writes Work

In a dual-write pattern, every write operation (insert, update, delete) is applied to both the old and new databases. The application typically writes to the old database first, then to the new one, and if the second write fails, the operation is logged or queued for retry. Reads continue to come from the old database until you are ready to switch. This gives you a live, continuously updated copy of your data in the new system, which you can query and compare against the old system.

The key safety condition is that the old system remains authoritative. If the new write fails, you must not fail the user request; you must record the discrepancy and fix it later. This means the application must be instrumented to handle partial failures gracefully. A common implementation is to use a transaction outbox pattern: write the change to the old database and also write an event to a queue, then a separate consumer applies that event to the new database. This decouples the two writes and makes it easier to retry failures.

How Shadow Testing Works

Shadow testing, also called shadow traffic or dark launch, involves sending a copy of real production requests to the new system while still serving the user from the old system. For example, you might intercept every read query and run it against both databases, but only return the result from the old one. The new system’s responses are compared to the old ones to detect differences. Writes are often shadowed too: the new system receives the write but its state is not used for reads yet.

Shadow testing is safe when the new system is isolated and cannot affect production. That means no shared resources, no side effects like sending emails or charging credit cards, and no writes to the old system. You must also ensure that the shadow traffic does not change the behavior of the new system in a way that affects other tests. For example, if the new system has a cache, the shadow traffic may pollute it, so you need to either use a separate cache or clear it before comparing results.

How to Choose Between Them

Choose dual writes when you need a continuously updated copy of the data for a long transition period, such as when you are migrating a large dataset that cannot be copied in one shot. Choose shadow testing when you want to validate the behavior of the new system under real traffic without committing to writing to it. Often you will use both: first shadow test to verify the new system handles the queries correctly, then enable dual writes to keep the new system in sync, then finally switch reads and retire the old system.

• Dual writes are better for data migration where the new system must be up-to-date for a cutover.
• Shadow testing is better for validating query performance and correctness before any writes are made.
• Both require a comparison mechanism to detect discrepancies automatically.
• Both require the ability to roll back without losing data.

Common Problems and How to Avoid Them

The most common problem with dual writes is the dual-write consistency issue: if the write to the new database fails after the old one succeeds, the two systems diverge. This is often called the dual-write problem. To avoid it, use a transactional outbox or a change data capture (CDC) pipeline that reads the old database’s transaction log and applies changes to the new system. This ensures that every committed change is eventually applied to the new system, even if the application crashes.

Another problem is that the new system may have different constraints or data types. For example, the old system may allow NULL in a column that the new system requires NOT NULL. When you shadow test, you may see errors that did not occur in production. You need to handle these by either transforming the data or fixing the schema. Also, beware of non-deterministic functions like NOW() or RAND() – they will produce different results in the two systems, so you need to exclude them from comparisons.

Shadow testing can also cause performance issues because you are doubling the load on your infrastructure. You need to ensure that the new system can handle the extra traffic without degrading. Use a separate environment or throttle the shadow traffic if necessary.

Costs and Risks

The main cost is operational complexity. You are running two systems, which means double the storage, compute, and maintenance. You also need to build and maintain the comparison tooling. The risk is that you might not notice a subtle data divergence until it is too late. For example, if a write to the new system fails silently and you do not have a reconciliation job, the new system will be missing data at cutover.

Another risk is that the new system may have different behavior under load. Shadow testing can reveal performance issues, but it cannot reveal issues that only appear when the new system is the primary. For example, if the new system has a bug that only triggers when it receives writes from multiple application instances, you may not see it until you switch over. To mitigate this, do a full rehearsal of the cutover, including a rollback, before the final switch.

Safety Checklist for Using These Techniques

1. Ensure the old system remains the source of truth until the final cutover.
2. Implement a reconciliation job that periodically compares the old and new data and reports differences.
3. Use a transactional outbox or CDC to avoid dual-write consistency issues.
4. Isolate the new system during shadow testing – no side effects, no shared caches.
5. Have a rollback plan that does not rely on reverting the writes (e.g., keep a full backup of the old system).
6. Test the cutover and rollback procedures multiple times in a staging environment.
7. Monitor both systems for errors and performance during the entire transition.

Alternatives to Dual Writes and Shadow Tests

If dual writes and shadow tests seem too risky, there are alternatives. One is the expand-contract pattern (also called parallel change): first expand the schema to support both old and new formats, then migrate data in the background, then contract by removing the old code. This is safer because it avoids dual writes altogether – you only write to one system but in a way that is compatible with both. Another alternative is to use a database migration tool that supports online schema changes, such as pt-online-schema-change or gh-ost for MySQL, which copy the table in the background while applying changes to the original.

For large-scale migrations, you might also consider using a data replication tool that handles the consistency for you, such as AWS Database Migration Service or Debezium for CDC. These tools reduce the risk of manual dual writes but still require careful validation.

Frequently Asked Questions

Can I use dual writes permanently?

Technically yes, but it is not recommended because it doubles the cost and complexity, and the two systems will inevitably diverge if not perfectly synchronized. Permanent dual writes are usually a sign that you have not fully migrated – you are still depending on the old system for some reads or writes. The goal should be to eventually retire the old system.

What is the difference between shadow testing and canary testing?

Shadow testing sends a copy of traffic to the new system but does not use the results; canary testing sends a small percentage of real traffic to the new system and serves the results to those users. Canary testing is riskier because real users are affected, but it provides more realistic feedback. Shadow testing is safer because no user sees the new system’s output.

How do I compare results between old and new systems?

You can compare at the database level by running checksums or row counts, or at the application level by logging the responses from both systems and diffing them. For queries, you can capture the query and its result from the old system, run the same query on the new system, and compare. For writes, you can compare the state of the two databases after each write. Automated tools like data-diff or custom scripts are common.

What if the new system is faster but has slightly different results?

If the results differ, you need to determine whether the difference is acceptable. For example, if you are migrating from a SQL database to a NoSQL database, the query semantics may differ slightly. You need to define what “correct� means for your application and test against that. If the differences are due to floating-point precision or time zones, you may need to standardize the data format.

Is it safe to shadow test with production data?

Yes, as long as the new system is isolated and does not leak data. You must ensure that the new system has the same security controls as the old one, and that the shadow traffic is not logged in a way that exposes sensitive data. Also, be aware of data residency requirements – if the new system is in a different region, you may violate compliance.

How long should the dual-write phase last?

It depends on how quickly you can validate the new system. A common practice is to run dual writes for a few weeks to cover different business cycles (e.g., month-end processing). You should have a clear exit criterion, such as zero reconciliation errors for a week, before you switch reads.

Related

• Music Gifts for Commuters & Vinyl Fans
• Iron Soul Dungeon Black Thursday Schedule
• Synth Riders DLC Without Full Game: Can You?