[Feature flag] Rollout of `load_balancing_for_build_hooks_worker`

What

Remove the :load_balancing_for_build_hooks_worker feature flag ...

Owners

• Team: groupmemory
• Most appropriate slack channel to reach out to: #g_memory
• Best individual to reach out to: @nmilojevic1

Expectations

What are we expecting to happen?

BuildHooksWorker should now utilize new Sidekiq read-only database replicas capabilities, and hit the replica most of the time, since we set data consistency for this worker to always

What might happen if this goes wrong?

It can affect the sidekiq connection pool saturation.

What can we monitor to detect problems with this?

• Metric: Sidekiq queue detail dashboard
• Metric: Postgres Async (Sidekiq) replica Connection Pool Utilization per Node
• Metric: Queue lenghts
• Metric: Postgres RSS Memory usage
• Metric: Sidekiq connection pool saturation details
• Log: https://log.gprd.gitlab.net/goto/6b54e091862b1fb63eb5e89fc42fe2f0
• Metric: database_chosen metric

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

• gitlab-org/gitlab project
• gitlab-org/gitlab-com groups
• ...

Roll Out Steps

• Enable on staging (/chatops run feature set load_balancing_for_build_hooks_worker true --staging)
• Test on staging
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab load_balancing_for_build_hooks_worker true)
• If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: 10%, 50%, 100%. Proposed minimum time between increments is 15 minutes.
• Coordinate a time to enable the flag with the SRE oncall and release managers
  • In #production mention @sre-oncall and @release-managers. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
• Announce on the issue an estimated time this will be enabled on GitLab.com. Note: Once a feature rollout has started, it is not necessary to inform @sre-oncall/@release-managers at each stage of the gradual rollout.
• Check if the feature flag change needs to be accompagnied with a change management issue: Change management issue: gitlab-com/gl-infra/production#4377 (closed)
• Enable on GitLab.com by running chatops command in #production (/chatops run feature set load_balancing_for_build_hooks_worker true)
• Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry. Ensure that the feature flag definition YAML file has been removed in the same MR that is removing the feature flag from the code
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set --project=gitlab-org/gitlab load_balancing_for_build_hooks_worker false