Web pods are being throttled
Currently web pods are getting throttled quite frequently, this shows that the 99th quantile of pods get about 100ms of throttling per second (10% of the time).
This is currently not affecting SLIs. Reducing throttling could improve the perceived performance for our users, at the possible cost of having to run more pods.
This came to attention when we were looking in to https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2028#note_2539046133, we've discussed this before in https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2122
The web pods currently run 7 puma-processes, requests 5vCPU and allows bursting up to 6vCPU. As @nduff mentioned in https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2122#note_2502805197, we should probably increase the limit to allow all processes to utilize 1 full vCPU, however, we need to take into account overall node utilization before doing that.