Remove rspec-retry gem usage in e2e tests
rspec-retry gem that e2e tests heavily depend on for retry mechanism has been unmaintained for a while so it's already a problem just waiting to manifest itself sooner or later.
Unit/feature specs already use native rspec --only-failures
flag, we should switch e2e tests to the same approach.
Cons
- Reliance on a more complicated CI setup as that approach requires running rspec command again. This is more complicated especially with custom
bin/qa
script and tests being executed bygitlab-qa
orchestrator in a separate docker container.
Pros
- Removing unmaintained gem
- Retries will be correctly displayed in allure reports
Retry
tab making it easier to understand which tests are flaky in particular run -
before(:context)
hooks should work correctly given we will retry the whole process, they are not retried in case ofrspec-retry
gem which is capable of only retrying separate specs
Implementation
This should be done iteratively. We can start with pipelines that doesn't use gitlab-qa
orchestrator to run the tests and then move to gitlab-qa
.
-
test-on-gdk
-
test-on-cng
-
package-and-test
-
live env tests
-
cdot
-
gem and respective code removal from framework
Edited by Andrejs Cunskis