Corrective action: Set up Gitaly Weight Assigner in GSTG

Summary

The file-01-stor-gstg.c.gitlab-staging-1.internal VM in staging became saturated from normal use. There are multiple gitaly VMs in staging, but no automation was in place to start to curtail new projects on the saturating VM and direct them to other gitaly VMs.

Related Incident(s)

Originating issue(s): production#8134 (closed)

Desired Outcome/Acceptance Criteria

Set up the Gitaly Weight Assigner to run in Staging so that as vms begin to fill up, new projects are directed to the least full VMs.

• Created a scheduled job for staging 👉 https://ops.gitlab.net/gitlab-com/gl-infra/gitaly-shard-weights-assigner/-/pipeline_schedules/319/edit
• Create a bot admin user for using its PAT for adjust weights of gitaly servers 👉 https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/20125
• Update scheduled job to use PAT of bot admin user from above and enable the job

Note: There is an incoming legacy webhook with name Gitaly Weights Assigner which by default posts to #production but it can be overrided to post to #staging too. It is already configured in the draft scheduled job for staging. Reference: https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/19974#note_1235410473

Associated Services

Corrective Action Issue Checklist

• Link the incident(s) this corrective action arose out of
• Give context for what problem this corrective action is trying to prevent from re-occurring
• Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
• Assign a priority (this will default to 'Reliability::P4')