Rebalance issues FF rollout

What

Enable the :rebalance_issues feature flag.

Owners

• Team: Create / Source Code
• Most appropriate slack channel to reach out to: #g_create_source_code
• Best individual to reach out to: @acroitor

Expectations

What are we expecting to happen?

Problematic issue sets (ones that show large numbers of timeouts and NoSpaceLeft exceptions) should begin rebalancing, and the incidence of these errors should decline.

What might happen if this goes wrong?

This may consume a large number of resources in the worker, since this can be a long running worker, which may itself timeout. We should watch for spikes in queue processing time.

What can we monitor to detect problems with this?

We should watch sentry for NoSpaceLeft and any error in the IssueRebalancingWorker. We should watch kibana to observe queue sizes:

• NoSpaceLeft
• Issues::RelativePositionRebalancingService

Roll Out Steps

• Enable on staging
• Test on staging
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour
• Coordinate a time to enable the flag with #production and #g_delivery on slack.
• Announce on the issue an estimated time this will be enabled on GitLab.com
• Enable on GitLab.com by running chatops command in #production
• Cross post chatops slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channelb