flaky_tests.md 6 KB
Newer Older
1 2 3 4 5 6 7
# Flaky tests

## What's a flaky test?

It's a test that sometimes fails, but if you retry it enough times, it passes,
eventually.

Rémy Coutable's avatar
Rémy Coutable committed
8 9
## Quarantined tests

10 11 12 13
When a test frequently fails in `master`,
[a ~"broken master" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master)
should be created.
If the test cannot be fixed in a timely fashion, there is an impact on the
14
productivity of all the developers, so it should be placed in quarantine by
15
assigning the `:quarantine` metadata.
Rémy Coutable's avatar
Rémy Coutable committed
16

17 18 19
This means it will be skipped unless run with `--tag quarantine`:

```shell
Rémy Coutable's avatar
Rémy Coutable committed
20 21 22
bin/rspec --tag quarantine
```

23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
**Before putting a test in quarantine, you should make sure that a
~"broken master" issue exists for it so it won't stay in quarantine forever.**

Once a test is in quarantine, there are 3 choices:

- Should the test be fixed (i.e. get rid of its flakiness)?
- Should the test be moved to a lower level of testing?
- Should the test be removed entirely (e.g. because there's already a
  lower-level test, or it's duplicating another same-level test, or it's testing
  too much etc.)?

### Quarantine tests on the CI

Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:

- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE)
- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only)

41 42 43 44 45 46 47 48 49 50
## Automatic retries and flaky tests detection

On our CI, we use [rspec-retry] to automatically retry a failing example a few
times (see [`spec/spec_helper.rb`] for the precise retries count).

We also use a home-made `RspecFlaky::Listener` listener which records flaky
examples in a JSON report file on `master` (`retrieve-tests-metadata` and `update-tests-metadata` jobs), and warns when a new flaky example
is detected in any other branch (`flaky-examples-check` job). In the future, the
`flaky-examples-check` job will not be allowed to fail.

51
This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13021>.
52 53 54 55 56 57

[rspec-retry]: https://github.com/NoRedInk/rspec-retry
[`spec/spec_helper.rb`]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/spec_helper.rb

## Problems we had in the past at GitLab

58 59 60 61
- [`rspec-retry` is bitting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-ce/issues/29242): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9825>
- [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-ce/issues/28307#note_24958837): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9846>
  - Follow-up: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10688>
  - [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-ce/issues/33779): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12224>
62
- FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!):
63 64 65 66
  - [Make `spec/mailers/notify_spec.rb` more robust](https://gitlab.com/gitlab-org/gitlab-ce/issues/20121): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10015>
  - [Transient failure in spec/requests/api/commits_spec.rb](https://gitlab.com/gitlab-org/gitlab-ce/issues/27988#note_25342521): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9944>
  - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-ce/issues/29643): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10184>
  - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-ce/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10404>
67 68 69

### Time-sensitive flaky tests

70 71
- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10046>
- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10306>
72 73 74

### Array order expectation

75
- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10148>
76 77 78

### Feature tests

79 80 81 82
- [Be sure to create all the data the test need before starting exercize](https://gitlab.com/gitlab-org/gitlab-ce/issues/32622#note_31128195): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12059>
- [Bis](https://gitlab.com/gitlab-org/gitlab-ce/issues/34609#note_34048715): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12604>
- [Bis](https://gitlab.com/gitlab-org/gitlab-ce/issues/34698#note_34276286): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12664>
- [Assert against the underlying database state instead of against a page's content](https://gitlab.com/gitlab-org/gitlab-ce/issues/31437): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10934>
83 84 85

#### Capybara viewport size related issues

86
- [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-ce/issues/29241#note_26743936): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10411>
87 88 89

#### Capybara JS driver related issues

90 91
- [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-ce/issues/30461): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10454>
- [Bis](https://gitlab.com/gitlab-org/gitlab-ce/issues/34647): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12626>
92 93 94

#### PhantomJS / WebKit related issues

95
- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12003>
96 97 98 99 100 101 102 103 104 105 106 107

## Resources

- [Flaky Tests: Are You Sure You Want to Rerun Them?](http://semaphoreci.com/blog/2017/04/20/flaky-tests.html)
- [How to Deal With and Eliminate Flaky Tests](https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests)
- [Tips on Treating Flakiness in your Rails Test Suite](http://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html)
- ['Flaky' tests: a short story](https://www.ombulabs.com/blog/rspec/continuous-integration/how-to-track-down-a-flaky-test.html)
- [Using Insights to Discover Flaky, Slow, and Failed Tests](https://circleci.com/blog/using-insights-to-discover-flaky-slow-and-failed-tests/)

---

[Return to Testing documentation](index.md)