Recommended Workflow
The purpose of TestRadar is to prevent flaky tests from blocking code merge. With that in mind, we recommend a 3 step workflow: detect, prioritize, and combat.
Detecting Flaky Tests
Detecting flaky tests is a breeze with TestRadar. Simply setup TestRadar's plugins, then every time your CI runs your test results will be immediately available on TestRadar's dashboard. The dashboard only shows data for your default branch, it's expected that all tests should pass on this branch, so detecting flaky tests on it as opposed to a working branch reduces false positives. We recommend running your tests as part of your main pipeline, i.e. when changes are pushed to your default branch. If for whatever reason you don't run your tests as part of your main pipeline, then you can create a scheduled pipeline that runs your tests several times a day. You may want to do this even if you run your tests as part of your main pipeline because the more data TestRadar has the better it can detect and prioritize your flakiest tests.
Prioritizing Flaky Tests
It's important to prioritize what flaky tests to address first because you want to reduce disruptions as much and as quickly as possible. TestRadar's dashboard assigns every test a severity (i.e. high, medium, and low severity) based on how many times the test ran and how many times it failed due to flake. It also displays a summary so that you know how many tests of each severity you have at a glance. This makes it easy to address your flakiest tests first.
Combating Flaky Tests
There are several actions you can take to combat flaky tests: test retries, the ol' re-run pipeline button, dropping your current work to fix the unrelated tests, and several others. Choosing one of those feels like choosing the least bad option though. We recommend auto quarantining. TestRadar's auto quarantining opens pull requests when flaky tests are detected, moving them from your pre-merge pipeline to a separate quarantine pipeline. This prevents flaky tests from blocking code merge. 🙌
Auto quaranting is not yet available.