Skip to content
GitLab

Flaky tests

What's a flaky test?

It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.

Quarantined tests

When a test frequently fails in main, create a ~"failure::flaky-test" issue.

If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be quarantined by assigning the :quarantine metadata with the issue URL, and add the ~"quarantined test" label to the issue.

it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do
  expect(response).to have_gitlab_http_status(:ok)
end

This means it is skipped unless run with --tag quarantine:

bin/rspec --tag quarantine

Once a test is in quarantine, there are 3 choices:

  • Fix the test (that is, get rid of its flakiness).
  • Move the test to a lower level of testing.
  • Remove the test entirely (for example, because there's already a lower-level test, or it's duplicating another same-level test, or it's testing too much etc.).

Automatic retries and flaky tests detection

On our CI, we use RSpec::Retry to automatically retry a failing example a few times (see spec/spec_helper.rb for the precise retries count).

We also use a home-made RspecFlaky::Listener listener which records flaky examples in a JSON report file on main (retrieve-tests-metadata and update-tests-metadata jobs).

This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021.

If you want to enable retries locally, you can use the RETRIES environment variable. For instance RETRIES=1 bin/rspec ... would retry the failing examples once.

Problems we had in the past at GitLab

Order-dependent flaky tests

These flaky tests can fail depending on the order they run with other tests. For example:

To identify the tests that lead to such failure, we can use scripts/rspec_bisect_flaky, which would give us the minimal test combination to reproduce the failure:

  1. First obtain the list of specs that ran before the flaky test. You can search for the list under Knapsack node specs: in the CI job output log.

  2. Save the list of specs as a file, and run:

    cat knapsack_specs.txt | xargs scripts/rspec_bisect_flaky

If there is an order-dependency issue, the script above will print the minimal reproduction.

Time-sensitive flaky tests

Array order expectation

Feature tests

Capybara viewport size related issues

Capybara JS driver related issues

Capybara expectation times out

Hanging specs

If a spec hangs, it might be caused by a bug in Rails:

Resources

Return to Testing documentation