Add more hosts to the pipeline fleet
The sidekiq pipeline fleet now exclusively runs GitLab CI pipeline jobs, previously besteffort
nodes were picking up a large amount of pipeline jobs. See https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7725 for more details.
Reviewing the metrics, it appears that this fleet is not always able to keep up with demand, leading to queues forming.
This is probably best captured in this chart of queue lengths for pipeline jobs, over the past 30 days:
For this reason, I reckon we should immediately add to more sidekiq pipeline hosts now and reassess in a few days time.
cc @devin for