PoC: Use confiner to auto-quarantine failing tests
requested to merge 361672-use-the-confiner-tool-to-automatically-quarantine-tests-that-failed-repeatedly into master
What does this MR do and why?
Depend on gitlab-org/ruby/gems/confiner!18 (closed).
TODO:
-
Replace the value of FLAKY_TESTS_PROJECT_TOKEN
with an actual bot token (currently a personal one from me). -
Remove the last commit which has been ported to !115533 (merged)
Screenshots or screen recordings
https://gitlab.com/gitlab-org/gitlab/-/jobs/4023503009#L793
(2023-03-29 12:43:34) QUARANTINE Quarantining Login with two-factor authentication with valid username/password does not show a "You are already signed in." error message (1 >= 1)
(2023-03-29 12:43:34) ISSUE Found issue https://gitlab.com/gitlab-org/quality/engineering-productivity/flaky-tests-playground/-/issues/1380 for `Login with two-factor authentication with valid username/password does not show a "You are already signed in." error message`
(2023-03-29 12:43:35) RSPEC Found match on line spec/features/users/login_spec.rb:304
(2023-03-29 12:43:35) DRY_RUN Skipping creating branch, committing and filing merge request
(2023-03-29 12:43:35) QUARANTINE Done Quarantining Login with two-factor authentication with valid username/password does not show a "You are already signed in." error message
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #361672 (closed)
Edited by Rémy Coutable