## Completed work 1. Infrastructure should be responsible for provisioning sufficient capacity given jobs high-level criteria. https://gitlab.com/gitlab-com/gl-infra/scalability/issues/162 1. Corrective action: Correct the threadcount for multithreaded environments https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/152 1. Add runbook to remove background jobs based on queue metadata https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/89 1. Better introspection of background jobs https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/173 1. Improve problematic background job queues https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/175 1. Application developers should be responsible for routing their jobs to the correct queue based on the job's latency requirements, priority, and resource needs. https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/97 - this has been moved to the Advocacy Epic &209 1. Define queuing time and job latency thresholds https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/25 1. Route background jobs by their characteristics https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/174 ## Status 2020-06-11 It is now possible for application developers to define the requirements of their workers based on the job's latency requirements, priority, and resource needs. Based on the characteristics defined, these background jobs are routed appropriately to fleets that are provisioned to best serve these needs. Visibility into these queues has been enhanced, and we used that increased visibility to improve problematic background job queues.