[Feature flag] Rollout of `rebalance_issues`

What

Remove the :rebalance_issues feature flag ...

Owners

• Team: Create / Source Code
• Most appropriate slack channel to reach out to: #g_create_source_code
• Best individual to reach out to: @alexkalderimis

Expectations

What are we expecting to happen?

Problematic issue sets (ones that show large numbers of timeouts and NoSpaceLeft exceptions) should begin rebalancing, and the incidence of these errors should decline.

What might happen if this goes wrong?

This may consume a large number of resources in the worker, since this can be a long running worker, which may itself timeout. We should watch for spikes in queue processing time.

What can we monitor to detect problems with this?

We should watch sentry for NoSpaceLeft and any error in the IssueRebalancingWorker. We should watch kibana to observe queue sizes:

• NoSpaceLeft
• IssueRebalancingWorker
• https://log.gprd.gitlab.net/app/kibana#/visualize/edit/AW2moVRdUOguGaJ_mRPG
• https://log.gprd.gitlab.net/goto/f4c17f162d25f00eedd8e3f1f2216117
• https://dashboards.gitlab.net/d/redis-sidekiq-main/redis-sidekiq-overview?orgId=1

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

• Beta groups/affected user: ["Project:10152778", "Project:10986423", "Project:11283490", "Project:11638590", "Project:14963009", "Project:18668865", "Project:4326131", "Project:4482940"]

Gitlab has too many issues to use this worker at present - future updates may allow for this.

Roll Out Steps

• Enable on staging
• Test on staging
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour
• Coordinate a time to enable the flag with #production and #g_delivery on slack.
• Announce on the issue an estimated time this will be enabled on GitLab.com
• Enable on GitLab.com by running chatops command in #production
• Cross post chatops slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channelb