Review flaky QA failure issues to identify potential transient bugs
It's possible that some transient bugs have been mistaken for flaky QA tests.
For example, if the UI sometimes takes longer than 10 seconds to update, we might make it wait longer and then the test passes. But if the UI should update in fewer than 10 seconds the passing test is a false positive - we should figure out why it sometimes takes longer than it should.
The purpose of this issue is to review flaky QA failure issues to identify potential transient bugs that were misidentified as flaky tests.
Part of gitlab-com/www-gitlab-com#9504 (closed)
Of 126 closed ~QA ~"failure::flaky-test" issues:
- 30 are ~"devops::create" (which might concern MR features)
- 9 are ~"devops::verify" (which might concern CI features)
Of the 30 devopscreate issues (plus a few more I stumbled across), the following are MR-related:
-
gitlab-org/gitlab#254680 (closed) -
gitlab-org/gitlab#215876 (closed) -
gitlab-org/gitlab#37924 (closed) -
gitlab-org/gitlab#35797 (closed)
The following are not MR-specific but did involve changes to waits, so are worth reviewing:
-
gitlab-org/gitlab#267508 (closed) -
gitlab-org/gitlab#257877 (closed) -
gitlab-org/gitlab#196713 (closed) -
gitlab-org/gitlab#196521 (closed) -
gitlab-org/gitlab#196034 (closed)
These all look like the same issue, that the files sometimes take a while to load in the project overview:
-
Project overview loading lag
This one is an open failure issue that looks a lot like a bug:
This one is more of a performance issue, but might be worth opening an issue for, even if it's not a transient bug:
This looks like a transient problem with Staging, but was fixed by retrying, so we should look at subsequent passing jobs to see how often if passes after a retry. The issue might need to be relabeled as failuretest-environment
Some were inconclusive so could be worth reviewing if we don't find many bugs elsewhere:
-
gitlab-org/gitlab#239320 (closed) -
gitlab-org/gitlab#36414 (closed) (possibly a transient Staging issue)
Of the 9 devopsverify issues, these suggest pipeline loading lag that could be worth looking into:
-
gitlab-org/gitlab#268210 (closed) -
gitlab-org/gitlab#267981 (closed) -
gitlab-org/gitlab#255164 (closed)
This involved changing a wait:
This one was inconclusive: