Skip to content

Quarantine both flakiness::1 and flakiness::2 specs

Jennifer Li requested to merge jennli-auto-quarantine-flakiness-2 into master

What does this MR do and why?

Closes #463486 (closed)

dry run looks like the following::

bundle exec gitlab-housekeeper -d -k Keeps::QuarantineFlakyTests
Running keep Keeps::QuarantineFlakyTests
.query_api: https://gitlab.com/api/v4/projects/278964/issues/?order_by=updated_at&state=opened&labels[]=test&labels[]=failure::flaky-test&labels[]=flakiness::1&not[labels][]=QA&not[labels][]=quarantine&per_page=20
.query_api: https://gitlab.com/api/v4/projects/278964/issues/?order_by=updated_at&state=opened&labels[]=test&labels[]=failure::flaky-test&labels[]=flakiness::1&not[labels][]=QA&not[labels][]=quarantine&per_page=20&page=2
.query_api: https://gitlab.com/api/v4/projects/278964/issues/?order_by=updated_at&state=opened&labels[]=test&labels[]=failure::flaky-test&labels[]=flakiness::2&not[labels][]=QA&not[labels][]=quarantine&per_page=20
Merge request URL: (known after create), on branch quarantine-flaky-tests-spec-models-organizations-organization_spec-rb-307.
=> QuarantineFlakyTests: spec/models/organizations/organization_spec.rb: 307
=> Title:
Quarantine a flaky test

=> Description:
The `Organizations::Organization#add_owner adds user as an owner`
test has either ~"flakiness::1" or ~"flakiness::2" label set, which means the number of reported failures
is at or above 95 percentile, indicating unusually high failure count.

This MR quarantines the test. This is a discussion starting point to let the
responsible group know about the flakiness so that they can take action:

- accept the merge request and schedule the associated issue to improve the test
- close the merge request in favor of another merge request to delete the test

Please follow the
[Flaky tests management process](https://handbook.gitlab.com/handbook/engineering/infrastructure/engineering-productivity/flaky-tests-management-and-processes/#flaky-tests-management-process)
to help us increase `master` stability.

Please let us know your feedback
[in the dedicated issue](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/447).

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/463107.


=> Attributes:
Labels: maintenance::refactor, test, failure::flaky-test, pipeline:expedite, quarantine, quarantine::flaky, group::tenant scale, automation:gitlab-housekeeper-authored
Assignees:
Reviewers: OmarQunsulGitlab

=> Diff:
diff --git a/spec/models/organizations/organization_spec.rb b/spec/models/organizations/organization_spec.rb
index 3fb2d928f5a3..868fd2cc5835 100644
--- a/spec/models/organizations/organization_spec.rb
+++ b/spec/models/organizations/organization_spec.rb
@@ -304,7 +304,7 @@
       organization.add_owner(user)
     end

-    it 'adds user as an owner' do
+    it 'adds user as an owner', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/463107' do
       expect(organization.owner?(user)).to eq(true)
     end
   end


Dry run complete. Housekeeper would have created 1 MR on an actual run.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Jennifer Li

Merge request reports