Green Retry: Hooks, Fixtures & Flaky Budgets

Green retry is a testing strategy where a failed test is automatically re-run to see if it passes on a second attempt, and the question of when to use it comes down to whether you are retrying the right thing. You should use green retry only for flaky tests—tests that fail intermittently due to environment or timing issues—and you must pair it with a strict budget for how many flakes you tolerate, otherwise you hide real bugs and slow down your suite. This article explains what green retry actually does, how hooks and fixtures interact with it, and how to set a practical flake budget that keeps your CI honest.

Formula

The calculator estimates the time cost of retrying flaky tests. The formula for time lost is:

Time lost = flaky failures per run × retry attempts × (total test time / 100)

Variables:

• t – total test time in minutes (the full suite duration without retries).
• f – number of flaky failures per run (tests that fail but would pass on retry).
• r – retry attempts (how many times you re-run a failed test).

The flake rate is simply flaky failures / total tests × 100, but since we don’t have total test count, we use the ratio of flaky failures to 100 as an approximation for the percentage. In practice, you’d replace 100 with your actual test count.

Worked Example

Using the defaults: total test time = 10 minutes, flaky failures = 2, retry attempts = 1.

Time lost = 2 × 1 × (10 / 100) = 2 × 1 × 0.1 = 0.2 minutes (12 seconds).

Flake rate = (2 / 100) × 100 = 2%.

So with a 10-minute suite, two flaky tests, and one retry each, you add about 12 seconds to every CI run. That might be acceptable, but if you have 20 flaky tests, you’re adding 2 minutes—and that’s just the retry time, not the debugging time.

What Green Retry Actually Does

Green retry is a feature in test runners like Jest, Cypress, or Playwright that automatically re-runs a failed test. The idea is that if a test fails due to a transient issue—like a network timeout, a slow animation, or a race condition—it might pass on the second run. The test then shows as “passed� in the final report, even though it failed once. This is different from a “retry on failure� in a CI pipeline, which re-runs the entire job; green retry re-runs only the specific test that failed.

Retries are useful for flaky tests, but they can also mask real bugs. If a test fails because the code is actually broken, retrying it will just make it fail again (unless the bug is intermittent). That’s why you need a flake budget—a maximum number of retries or flaky failures you’re willing to accept per run. Without a budget, you might end up with tests that always fail once and then pass, and you’ll never notice that they’re flaky.

Hooks and Fixtures: The Hidden Interaction

Hooks (like beforeEach, afterEach) and fixtures (test data or setup) can cause flakiness that retries don’t fix. If a hook fails because a database connection is slow, retrying the test will also retry the hook—so it might still fail. Worse, if a fixture is not cleaned up properly, a retry might run the test with leftover data, causing a different failure.

When you enable green retry, consider what runs before and after the test. If the hook is the flaky part, you might need to fix the hook rather than retry the test. Similarly, if a fixture is shared across tests, retrying one test could affect others. A better approach is to make hooks idempotent (safe to run multiple times) and fixtures isolated per test.

How to Choose a Retry Strategy

Not all tests should be retried. Here are factors to consider:

• Test type: Unit tests are usually deterministic; retrying them is rarely needed. Integration and end-to-end tests are more prone to flakiness.
• Cost of retry: If a test takes 5 minutes to run, retrying it twice adds 10 minutes. That might be too expensive.
• Flake history: Check if a test has failed in the past. If it’s consistently flaky, fix it instead of retrying.
• Retry limit: Most runners allow you to set a maximum retry count (e.g., 2). Set it low—1 or 2—to avoid masking real issues.

For example, in Cypress, you can set retries: 2 in the configuration. That means each test can fail up to twice before being marked as failed. But if you have a test that always fails on the first run and passes on the second, you should investigate why, not just rely on the retry.

Setting a Practical Flake Budget

A flake budget is a threshold for how many flaky failures you tolerate in a given time period (per run, per day, or per week). Here’s a practical way to set one:

1. Measure your current flake rate – Run your suite for a week and count how many tests fail on the first attempt but pass on retry.
2. Set a target – A common target is less than 1% of tests flaky per run. If you have 1000 tests, that’s fewer than 10 flakes.
3. Track flaky tests – Use a CI tool or a simple script to flag tests that use retries. If a test uses a retry more than a few times, it’s a candidate for fixing.
4. Enforce the budget – If the flake rate exceeds your target, fail the build or require a fix before merging.

For example, if your suite has 500 tests and you see 10 flakes per run, that’s 2% flake rate—too high. You’d want to fix the flaky tests or reduce the retry count.

Common Problems and Misconceptions

One misconception is that green retry “fixes� flaky tests. It doesn’t; it just hides them. Another is that retrying all tests is safe. It’s not—it can double your CI time and mask regressions. A common problem is that retries can cause side effects: if a test creates data, a retry might create duplicate data, causing a different failure. Also, retries don’t help with flaky hooks or fixtures, as mentioned earlier.

Another issue is that retries can make it harder to debug. If a test fails twice and then passes, you might not investigate the failure at all. That’s why you should always log retry attempts and make them visible in the report.

Alternatives to Green Retry

If you’re dealing with flaky tests, retries are a band-aid. Better alternatives include:

• Fix the root cause – Use waits instead of fixed sleeps, clean up fixtures, and make tests independent.
• Quarantine flaky tests – Move them to a separate suite that doesn’t block CI, and fix them later.
• Use test sharding – Split tests across multiple machines to reduce load and flakiness.
• Improve test isolation – Ensure each test runs in a clean state, with its own database or container.

For example, if a test fails because a dropdown animation is slow, instead of retrying, use a wait for the element to be visible. That fixes the flake for everyone, not just on retry.

Frequently Asked Questions

Does green retry work for all test types?

No. It’s most useful for end-to-end and integration tests that interact with browsers, networks, or databases. Unit tests are usually deterministic and shouldn’t need retries. If a unit test is flaky, that’s a code smell—fix it.

How many retries should I allow?

Start with 1 or 2. More than that masks real bugs and adds significant time. If a test needs more than 2 retries to pass, it’s not flaky—it’s broken.

What’s the difference between retry and rerun?

A retry re-runs the failed test within the same test run, usually immediately. A rerun (or re-run) means running the entire test suite again, often in CI after a failure. Retries are more targeted and faster.

Can retries cause false positives?

Yes. If a test fails due to a real bug but passes on retry because of a race condition, you get a false positive. That’s why you should track retry usage and investigate any test that uses a retry more than a few times.

How do I know if my flake budget is too high?

If you’re spending more than a few minutes per day investigating flaky tests, or if your suite has a flake rate above 1%, your budget is too high. Reduce the retry count and fix the flakiest tests.

Related

• Crunchyroll Watch Party: How It Works & How to Host One
• Fix Slow Internet Lag on Galaxy S6: Step-by-Step
• Photography Studio Websites: Best Portfolio Displays

in t | Total test time (min) | min | 10 | 0
in f | Flaky failures per run | - | 2 | 0
in r | Retry attempts | - | 1 | 0 | 5
out time | Time lost to retries (min) | min | f * r * (t / 100) | 2
out pct | Flake rate (%) | % | (f / 100) * 100 | 2