Skip to content

PoC: Use confiner to auto-quarantine failing tests

Rémy Coutable requested to merge 361672-use-the-confiner-tool-to-automatically-quarantine-tests-that-failed-repeatedly into master

What does this MR do and why?

Depend on gitlab-org/ruby/gems/confiner!18 (closed).

TODO:

  • Replace the value of FLAKY_TESTS_PROJECT_TOKEN with an actual bot token (currently a personal one from me).
  • Remove the last commit which has been ported to !115533 (merged)

Screenshots or screen recordings

https://gitlab.com/gitlab-org/gitlab/-/jobs/4023503009#L793

(2023-03-29 12:43:34)	QUARANTINE  			Quarantining Login with two-factor authentication with valid username/password does not show a "You are already signed in." error message (1 >= 1)
(2023-03-29 12:43:34)	ISSUE       				Found issue https://gitlab.com/gitlab-org/quality/engineering-productivity/flaky-tests-playground/-/issues/1380 for `Login with two-factor authentication with valid username/password does not show a "You are already signed in." error message`
(2023-03-29 12:43:35)	RSPEC       				Found match on line spec/features/users/login_spec.rb:304
(2023-03-29 12:43:35)	DRY_RUN     				Skipping creating branch, committing and filing merge request
(2023-03-29 12:43:35)	QUARANTINE  			Done Quarantining Login with two-factor authentication with valid username/password does not show a "You are already signed in." error message

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #361672 (closed)

Edited by Rémy Coutable

Merge request reports