## Description Builds queuing recently is causing some problem in our Infrastructure. We recently started a Rapid Action to accelerate work on problematic SQL queries originating from the Verify area of codebase. See https://gitlab.com/groups/gitlab-org/-/epics/5434. There are multiple issues that are problem, but this epic describes a work that needs to be done to resolve queuing problem in the short-term, before we re-architect this solution. Related issues: * https://gitlab.com/gitlab-org/gitlab/-/issues/292047 * https://gitlab.com/gitlab-org/gitlab/-/issues/118495 ## Proposal 1. Understand better 409s: - check disparity between query used to build a queue and query used to check if a build from the queue can be picked by a runner what is a a possible area of contention - find longest queues in the Kibana and check the reason of contention there 1. Scrutinize existing Prometheus metrics - `job_register_attempts_failed_total`, "Counts the times a runner tries to register a job" - `job_register_attempts_total`, "Counts the times a runner tries to register a job" - `job_queue_duration_seconds`, 'Request handling execution time' 1. Ship more metrics: - a build can not be picked event - histogram for a single queue size - counter for the number of unique queues 1. Prove or reject hypotheses: - Reducing the individual queue size can help to reduce 409s - Putting a limit on the queue and shuffling builds can help with reducing conflicts 1. Implement the most promising solution behind a features flag: - FF needs to be toggled for a runner and incrementally rolled out /cc @andrewn @ayufan @darbyfrey