Remove transient tests and associated documentation
Background
Transient tests were introduced about four years ago to address specific issues related to intermittent failures in our test suite (see gitlab-com/www-gitlab-com#9506 (closed) and #759 (closed)). But they no longer seem to provide value:
- Only two transient tests remain in our test suite after four years.
- Jobs running tests for transient bugs were removed in gitlab-org/quality/pipeline-common!276 (merged) as part of gitlab-org/quality/engineering-productivity/team#198 (closed)
- The tests still run in the "airgapped" job, but probably only because it doesn't exclude tests with the
:transientmetadata
Proposal
Given the limited number of remaining transient tests and the changes in our pipelines, we should:
- Remove the remaining transient tests from our test suite: gitlab-org/gitlab!176058 (merged)
- Update documentation to reflect the removal of transient tests
- in
gitlab-org/gitlab: gitlab-org/gitlab!176058 (merged) - in gitlab-qa: gitlab-org/gitlab-qa!1406 (merged)
- in
Rationale: Since only two transient tests remain after four years, and we tried to remove them already, they clearly don't provide the value we hoped. Removing what remains of them should help avoid confusion and unnecessary efforts to deal with failures (such as https://gitlab.com/gitlab-org/gitlab/-/issues/509492 - the failure was not related to the feature under test).
Note: We will still have guidance on preventing transient bugs, and identifying transient bugs as part of issue triage