## Summary The `file-01-stor-gstg.c.gitlab-staging-1.internal` VM in staging became saturated from normal use. There are multiple gitaly VMs in staging, but no automation was in place to start to curtail new projects on the saturating VM and direct them to other gitaly VMs. ## Related Incident(s) <!-- Note the originating incident(s) and link known related incidents/other issues. The relation will happen automatically if you are creating this issue from an incident, if this isn't done already please uncomment the following line: --> Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8134 ## Desired Outcome/Acceptance Criteria Set up the `Gitaly Weight Assigner` to run in Staging so that as vms begin to fill up, new projects are directed to the least full VMs. - [x] Created a scheduled job for staging :point_right: https://ops.gitlab.net/gitlab-com/gl-infra/gitaly-shard-weights-assigner/-/pipeline_schedules/319/edit - [ ] Create a bot admin user for using its PAT for adjust weights of gitaly servers :point_right: https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/20125 - [ ] Update scheduled job to use PAT of bot admin user from above and enable the job Note: There is an incoming legacy webhook with name `Gitaly Weights Assigner` which by default posts to #production but it can be overrided to post to #staging too. It is already configured in the draft scheduled job for staging. Reference: https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/19974#note_1235410473 ## Associated Services <!-- Apply the appropriate services associated with this corrective action if applicable. ~Service::SERVICE_NAME --> ## Corrective Action Issue Checklist * [x] Link the incident(s) this corrective action arose out of * [x] Give context for what problem this corrective action is trying to prevent from re-occurring * [x] Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') * [x] Assign a priority (this will default to 'Reliability::P4')