Save Baseline Screenshots as CI Artifacts

This guide shows you how to save baseline screenshots as CI artifacts so your visual regression tests have a stable reference to compare against. You’ll set up a pipeline step that generates and stores baseline images, then configure the artifact to persist across builds—this prevents tests from failing when the baseline isn’t available locally. By the end, you’ll have a repeatable process for capturing and reusing baselines in any CI system.

What You’ll Need

• A CI system (GitHub Actions, GitLab CI, Jenkins, CircleCI, etc.) with YAML configuration support.
• A visual testing tool that can generate screenshots (e.g., Playwright, Puppeteer, Cypress, or a dedicated tool like Percy, but we’ll focus on self-hosted options).
• A project with a test suite that includes visual regression tests.
• Basic familiarity with your CI platform’s artifact storage features.

Step-by-Step

1. Configure your visual test tool to output screenshots to a specific directory. In your test setup, ensure that baseline screenshots are saved to a dedicated folder, e.g., __screenshots__/baseline/. For Playwright, you might set outputDir in playwright.config.ts. For custom scripts, use a consistent path like ./baseline-screenshots.
2. Create a CI job that generates the baseline screenshots. Add a job in your CI configuration (e.g., .github/workflows/visual-tests.yml for GitHub Actions) that checks out the code, installs dependencies, and runs a command to capture screenshots. For example, a script npm run test:visual:baseline that runs your visual tests in “update” mode to create new baseline images.
3. Upload the baseline directory as a CI artifact. After the baseline generation step, add an artifact upload action. In GitHub Actions, use actions/upload-artifact@v4 with path: baseline-screenshots. In GitLab CI, use the artifacts keyword with paths: [baseline-screenshots/]. In Jenkins, use the archiveArtifacts step.
4. Set the artifact to be retained (if needed). By default, artifacts are retained for a limited time (e.g., 90 days on GitHub). For long-term baselines, you may want to increase retention or store them in a separate repository or cloud storage. Adjust the retention policy in your CI settings or via the artifact configuration.
5. Download the baseline artifact in subsequent test runs. In your regular visual test job (the one that runs on every commit), add a step to download the baseline artifact before running tests. For GitHub Actions, use actions/download-artifact@v4 with the same artifact name. For GitLab, use artifacts: download in the job definition.
6. Run visual tests against the downloaded baseline. Ensure your test command uses the baseline directory as the reference. For Playwright, set toHaveScreenshot to compare against the downloaded baseline. For custom tools, point your comparison logic to the artifact’s directory.
7. Optionally, automate baseline updates on a schedule or on demand. Create a separate workflow that regenerates baselines (e.g., on a weekly schedule or via a manual trigger) and uploads them as a new artifact version. This keeps baselines fresh without manual intervention.

Common Mistakes

• Not using a consistent artifact name. If the artifact name changes (e.g., includes a timestamp), the download step will fail. Use a static name like visual-baselines.
• Uploading baselines from the test job instead of a dedicated job. If you upload baselines from the same job that runs tests, you might accidentally upload screenshots of failures. Always separate baseline generation from test running.
• Forgetting to clean up old baselines. Over time, the artifact storage fills up. Set a retention policy or manually delete old artifacts.
• Ignoring file permissions. On Linux CI runners, ensure the screenshot directory is writable and readable by the user running the tests.
• Not handling the case where no baseline exists yet. The first run should generate baselines, not fail. Make sure your test script creates the baseline directory if it doesn’t exist.

Tips

• Use a separate branch or repository for baselines. Some teams store baselines in a dedicated Git repository so they can be versioned and reviewed. This also avoids CI artifact expiration issues.
• Compress the baseline directory before upload. If you have many screenshots, zip them to reduce upload/download time. Use actions/upload-artifact with compression-level or a tar command.
• Name your artifact with the project and branch. If you have multiple projects or branches, include a prefix like myproject-main-baselines to avoid collisions.
• Log the artifact name and path in CI output. This helps debugging when something goes wrong.
• Consider using a dedicated visual testing service. If managing baselines becomes a burden, services like Percy or Applitools handle baseline storage and comparison automatically, but they come with a cost.

Troubleshooting

Artifact download fails with “not found”. Check that the artifact name matches exactly and that the artifact was uploaded in a previous job. If you’re using GitHub Actions, ensure the actions/download-artifact step runs after the upload step in the same workflow, or use needs to order jobs.

Baselines are empty or missing files. Verify that the screenshot generation script actually writes files to the expected directory. Run the script locally to confirm. Also check that the directory path is relative to the workspace root, not absolute.

Tests fail because baselines are stale. If your UI has changed, you need to regenerate baselines. Run the baseline generation job manually or on a schedule, then download the new artifact. Make sure the artifact version is updated (e.g., by using a unique name or overwriting).

CI runner runs out of disk space. Screenshots can be large. Clean up temporary files after tests, and consider compressing the baseline directory. Also, increase the runner’s disk size if possible.

Artifact upload times out. If the baseline directory is huge, split it into multiple artifacts or use a different storage solution like an S3 bucket. You can also exclude unnecessary files (like logs) from the artifact.

Frequently Asked Questions

Q: Can I use the same artifact for multiple branches? Yes, but be careful: baselines may differ per branch. Use branch-specific artifact names or store baselines in a versioned folder within the artifact.

Q: How long should I keep baseline artifacts? It depends on your project’s release cycle. For most projects, 30-90 days is enough. If you need longer, consider moving baselines to a Git repository or cloud storage.

Q: What if my visual tests need to run on multiple operating systems? Screenshots can differ across OS due to font rendering. Generate separate baselines per OS, and name artifacts accordingly (e.g., baselines-linux, baselines-windows).

Q: Can I generate baselines on a local machine and upload them? Yes, but it’s not recommended because local environments vary. It’s better to generate baselines in a controlled CI environment to ensure consistency.

Q: How do I update baselines when the UI changes intentionally? Run the baseline generation job after merging the UI changes. The new baselines will be uploaded and used for future tests. Review the changes in a pull request if possible.

Q: What if my CI provider doesn’t support artifacts? Most do, but if not, you can use a third-party storage like AWS S3 or a simple HTTP server. The principle is the same: upload the baseline directory and download it before tests.

Related

• Honda vs Toyota: 5 SUVs Dominating Sales Charts?
• How to Generate an OpenSSL Certificate Signing Request
• 4K Monitors High Refresh Rates: What to Know