Add note in gitaly-weights-assigner that it has assigned 0% to too many nodes
Summary
On 2022-03-18, we got " Firing 1 - Number of active Gitaly shards is low There are less than 10 Gitaly shards that actively receive new project, this could cause them to fill up quickly and put more stress on them. Consider adding a new Gitaly shard https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation"
The gitaly weights assigner had just run and assigned one more node the 0% weight putting the threshold below the alert.
We should add messaging in https://gitlab.com/gitlab-com/gl-infra/gitaly-shard-weights-assigner/-/blob/master/assigner.rb that notes that it has taken the count below 10 (or whatever we want).
This would keep the SREs watching #production or on call from wondering what happened as the link between the alert and ci-job were not totally clear.
Related Incident(s)
Originating issue(s): gitlab-com/gl-infra/production#ISSUE_ID
Desired Outcome/Acceptance Criteria
Associated Services
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose out of -
Give context for what problem this corrective action is trying to prevent from re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'priority::4')