Monitor Flaky Tests & Alerts: A Practical Guide

This guide explains how to set up a system for monitoring automated test results, specifically to identify flaky tests (tests that pass and fail without code changes) and to configure alerts so you know when something needs attention. You’ll learn how to aggregate results, track flakiness over time, and set up notifications that help your team act on real problems without being overwhelmed by noise.

What You’ll Need

• A CI/CD system (e.g., Jenkins, GitHub Actions, GitLab CI, CircleCI) where your tests run.
• Access to the CI system’s logs and artifacts, or a way to export test results (JUnit XML, JSON, etc.).
• A place to store and query test history (e.g., a database, a time-series database, or a dedicated test analytics tool like TestRail, Allure, or Grafana).
• A notification channel (email, Slack, Microsoft Teams, or a paging service like PagerDuty) for alerts.
• Basic scripting skills (Python, Bash, or similar) to parse results and call APIs.

Step-by-Step

1. Define what “flaky” means for your team. A common definition: a test that fails on one run and passes on the next without any code change, or fails intermittently across runs. Write this down so everyone agrees on what you’re tracking.
2. Export test results from your CI pipeline. Ensure your test runner outputs a structured format like JUnit XML or JSON. Most CI systems can save these as artifacts. For example, in GitHub Actions, use actions/upload-artifact; in Jenkins, use the JUnit plugin.
3. Store test results in a central location. Set up a database or use a test analytics tool. If you’re building your own, create a table with columns: test name, suite, branch, commit, timestamp, status (pass/fail/error), and duration. For a quick start, use a simple CSV file, but a database is better for querying.
4. Parse and ingest results. Write a script (e.g., Python) that reads the JUnit XML files and inserts records into your storage. Run this script as a post-build step in your CI pipeline, or as a scheduled job that fetches artifacts.
5. Calculate flakiness metrics. For each test, track the pass rate over a rolling window (e.g., last 50 runs). A test is considered flaky if its pass rate is below 100% but above some threshold (e.g., 80%). Also track the number of consecutive failures followed by a pass—this is a strong signal.
6. Create a flaky test report. Build a dashboard or a simple HTML page that lists tests by flakiness score. Include the last failure time, failure rate, and a link to the latest failure log. Update this report after each CI run.
7. Set up alerting rules. Define conditions that trigger alerts. Examples:
   • A test that was passing for the last 20 runs now fails.
   • A test’s flakiness score crosses a threshold (e.g., fails more than 5% of runs in a day).
   • A test fails on more than 3 consecutive runs (likely a real bug, not flaky).
8. Integrate with your notification channel. Use webhooks or APIs. For Slack, use an incoming webhook; for email, use SMTP; for PagerDuty, use the Events API. Include the test name, failure rate, and a link to the log in the alert message.
9. Add a quarantine mechanism. When a test is flagged as flaky, automatically move it to a “quarantine” list so it doesn’t block CI. This prevents flaky tests from masking real failures. The test still runs, but its result doesn’t fail the build.
10. Review and act on alerts. Set a weekly review where the team triages new flaky tests. Decide whether to fix the test, fix the code, or delete the test. Update the flakiness threshold if needed.
11. Continuously refine. Monitor the number of alerts per day. If you’re getting too many, adjust thresholds or quarantine more aggressively. If you’re missing real issues, tighten thresholds.

Common Mistakes

• Ignoring flaky tests until they block a release. Flaky tests erode trust in the test suite. Deal with them early.
• Setting alert thresholds too low. If you alert on every single failure, you’ll get alert fatigue. Start with a threshold that catches real problems (e.g., 3 consecutive failures).
• Not quarantining flaky tests. Allowing a flaky test to fail the build wastes time and causes developers to skip running tests.
• Only looking at the latest run. Flakiness is a pattern over time. You need historical data to identify it.
• Using a single pass/fail metric. A test that fails 1 in 100 runs is different from one that fails 50 in 100. Use pass rate and frequency.

Tips

• Use a dedicated test analytics tool if you don’t want to build your own. Tools like Allure TestOps, TestRail, or even Grafana with a test results data source can save time.
• Tag tests with metadata (e.g., @flaky, @slow) to make filtering easier.
• Include environment info in your results (e.g., browser version, OS) because flakiness often depends on the environment.
• Set up a weekly automated report that lists the top 10 most flaky tests, so the team can review them without waiting for alerts.
• Use a separate branch for flaky test fixes so you don’t mix them with feature work.

Troubleshooting

• No alerts are being sent. Check that your webhook URL or SMTP settings are correct. Test the notification channel with a manual message. Also check that the script that evaluates conditions is actually running (look at logs).
• Alerts are too noisy. Adjust thresholds upward. For example, require 5 consecutive failures before alerting. Or add a cooldown period—don’t alert on the same test more than once per hour.
• Flaky tests are not being detected. Your detection logic might be too strict. For example, if you only look at the last 10 runs, a test that fails once in 20 runs won’t be flagged. Increase the window size.
• Test results are not being ingested. Check that the artifact path is correct in your CI config. Verify that the parser can read the JUnit XML format. Add logging to your ingestion script to see what it’s reading.
• Quarantined tests are not being excluded from the build. Make sure your CI pipeline reads the quarantine list. If you’re using a script, ensure it’s called before the build step that fails on test failures.

Frequently Asked Questions

What is the difference between a flaky test and a broken test?

A flaky test gives inconsistent results (pass/fail/pass) without code changes. A broken test consistently fails because of a real defect in the code or the test itself.

How many runs do I need to detect flakiness?

At least 10-20 runs to see a pattern. The more runs you have, the more confident you can be. A test that fails 1 in 20 runs is flaky, but you might need 50 runs to see that failure.

Should I automatically quarantine a flaky test?

Yes, but with a review process. Auto-quarantine prevents build failures, but you should review the test within a week to decide if it’s worth fixing or deleting.

What if a flaky test is actually a real bug?

That’s possible. If a test fails consistently for a while, it’s likely a real bug. Use the alert system to escalate after N consecutive failures. Then investigate the code.

Can I use the same system for UI tests and unit tests?

Yes, as long as you can export results in a common format. Many teams use the same pipeline for all test types, but you may want separate thresholds because UI tests are often more flaky.

How do I prevent alert fatigue?

Use thresholds, cooldown periods, and quarantine. Only alert on tests that are new failures or have a high flakiness score. Also, route alerts to different channels based on severity (e.g., Slack for warnings, PagerDuty for critical).

Related

• Mortal Shell 2: Getting & Using the Makeshift Projectile
• Async Work Norms & When to Meet
• Insta Account Hack: What It Really Takes