Corrective action: The Horizontal Pod Autoscaler Desired Replicas resource of the sidekiq service (main stage) has a saturation exceeding SLO and is close to its capacity limit.
## Summary The catch-all shard of sidekiq is hitting it's HPA limits. While this might be aggravated by an ongoing incident with GCP HPA scaling, we should try to grow this deployment to handle the volume of work. 1. Verify that `gitlab-sidekiq-catchall-v2` shard is still low on replicaset headroom. 2. Calculate the increase of additional pods and if we need more host nodes added. 3. Increase the replica maximum. ## Related Incident(s) <!-- Note the originating incident(s) and link known related incidents/other issues. The relation will happen automatically if you are creating this issue from an incident, if this isn't done already please uncomment the following line: --> Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6615 ## Desired Outcome/Acceptance Criteria During peak load, we should not hit 100% HPA limits. ## Associated Services <!-- Apply the appropriate services associated with this corrective action if applicable. ~"Service::Sidekiq" --> ## Corrective Action Issue Checklist * [x] Link the incident(s) this corrective action arose out of * [x] Give context for what problem this corrective action is trying to prevent from re-occurring * [x] Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') * [x] Assign a priority (this will default to 'priority::4')
issue