## Summary We frequently have problems with sidekiq where it doesn't scale horizontally because the HPA is only specified for CPU usage. This doesn't work for [memory bound workers](https://gitlab.com/gitlab-com/gl-infra/capacity-planning/-/issues/36#note_937983898). We used [custom metrics](https://cloud.google.com/kubernetes-engine/docs/concepts/custom-and-external-metrics) in the past to scale on quue length for [pubsubbeat](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/665). This helps scale the number of pods depending on the amount of work that needs to be done. ## Proposal We should try and do the same thing for Sidekiq so that we horizontally scale-out not just on CPU but also on queue length. Few things to consider: 1. We need to upstream this change to the [GitLab helmchart ](https://gitlab.com/gitlab-org/charts/gitlab) since that is where the HPA definition lives. 2. Figure out if we can get Prometheus metrics available for the HPA in GKE. --- *Props to @mwasilewski-gitlab for suggesting this* ## Originating incidents - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8173