Add api queue limit options to workhorse

Summary

Workhorse will queue with no limit by default, for production configuration we set the following options:

-apiLimit 5 -apiQueueDuration 60s -apiQueueLimit 200 -apiCiLongPollingDuration 50s

Issue where this was initially enabled in production https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/608

This is especially important when pods are brought online and start receiving requests immediately, in this case it is possible for so many requests to queue that it can cause additional latency. (Summarize the bug encountered, concisely as possible)

Also consider that apiCiLongPollingDuration MUST be shorter than the value of the terminationGracePeriodSeconds of the deployment. Otherwise clients will recieve an HTTP502 as the Pods are cycled and the long poll response is never sent back to the client. The behavior should be re-evaluated pending the outcome of: gitlab-org/gitlab#325114 (closed)

changed the description

added production request label

added For Scheduling devopssystems groupdistribution labels

mentioned in issue gitlab-org/distribution/team-tasks#745 (closed)

added sectioncore platform label

mentioned in issue gitlab-com/gl-infra/delivery#1072 (closed)

mentioned in merge request !1633 (merged)

This has been identified as a problem for the API service migration. Currently the use of extraArgs is configured for all webservice deployments. However, we use differing values for differing deployments. Ideally we would need this added per webservice deployment.

Adding kubernetes-migration-blocker for evaluation prior to our next service fleet migration.

After some testing, I was incorrect about this assumption. After testing we are indeed able to take workhorse configurations separately just fine. This can be seen in the following MR: gitlab-com/gl-infra/k8s-workloads/gitlab-com!869 (merged)

Removing the blocker label.

mentioned in issue gitlab-com/gl-infra/delivery#1761 (closed)

added kubernetes-migration-blocker label

marked this issue as related to gitlab-com/gl-infra/delivery#1761 (closed)

removed kubernetes-migration-blocker label

changed the description

Due to incident: gitlab-com/gl-infra/production#4666 (closed) I've modified the issue description to take account for the longpoll utilized by workhorse.

mentioned in issue gitlab-com/gl-infra/delivery#1766

added group::distributiondeploy label

marked this issue as related to gitlab-org/gitlab#323105 (closed)