Bazel pass flaky test if 1 out of 3 retries pass

Question

I have a Java based test suite running Bazel 4.2.2, and my goal is to collect code coverage regardless of test flakiness. I tried to add these options:

bazel coverage ... --runs_per_test=3 --cache_test_results=no ...

but it looks like if 1/3 of those fail, then the test is failed and coverage data is not collected for failing tests.

Does Bazel have any flags to take the first passing result, and retry only on failures?

The full command I've tried is

bazel coverage --jobs=6 --runs_per_test=3 --cache_test_results=no --combined_report=lcov --coverage_report_generator="@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main" -- //$TARGET/... 2>&1

Thanks!

Answer 1

Answer to my question (can't accept it yet): there's an option in the documentation I found https://docs.bazel.build/versions/0.25.0/command-line-reference.html

--flaky_test_attempts=<a positive integer, the string "default", or test_regex@attempts. This flag may be passed more than once> multiple uses are accumulated

Each test will be retried up to the specified number of times in case of any test failure. Tests that required more than one attempt to pass would be marked as 'FLAKY' in the test summary. If this option is set, it should specify an int N or the string 'default'. If it's an int, then all tests will be run up to N times. If it is not specified or its value is ' default', then only a single test attempt will be made for regular tests and three for tests marked explicitly as flaky by their rule (flaky=1 attribute).

Answer 2

Another option is using the flaky attribute on your test rules for the problematic tests. That will run them up to 3 times even with normal bazel test, without needing any flags.