2026-10-08: CI job finalization & webhook delays — Sidekiq concurrency-limiter deferral latch

CI job finalization & webhook delays — Sidekiq concurrency-limiter deferral latch (Severity 2)

Problem: CI jobs finished executing but GitLab took minutes to accept their final state, because Ci::BuildTraceChunkFlushWorker was stuck in a Sidekiq concurrency-limiter backlog. A separate backlog of Ci::ExecuteBuildHooksWorker delayed webhook and build-hook delivery. The limiter deferred every new job while its backlog was non-empty, even with free capacity. The main-stage web deployment-health gate briefly blocked auto-deploys, but web kept serving traffic.

Impact: Delayed CI job finalization from about 14:12 to 16:00 UTC (main trace-flush episode). Sampled customer-managed runners received repeated final-state HTTP 202 responses for about five minutes starting around 14:29. Some jobs stayed in "running" while runners held VMs, the queue grew (no_free_executor), and some jobs timed out or failed. Webhook/build-hook delivery was delayed by a separate backlog that began at 12:19 and was still about 1.26M at 17:16; the backlog alert cleared by about 19:00 UTC. Secondary: the saas-linux-small-amd64 runner apdex SLI breached its threshold from about 15:16 to 15:23 UTC.

Causes: The initial trigger is not established. The 14:12 trace-flush slowdown was in GCS artifact-upload time, with almost no DB time. The ComplianceManagement fan-out at about 12:00 UTC (about 3,900 DB-heavy jobs) is a likely contributing trigger of DB pressure, but not the sole cause. It does not explain the trace-flush stall. The amplifier is confirmed: a pre-existing concurrency-limiter behaviour defers all new jobs while the deferred list is non-empty. This is the third occurrence, after INC-13984 and INC-14022.

Response strategy: No targeted mitigation for the limiter backlog is recorded, and no feature flag changed for either worker on 2026-10-08. The trace-flush backlog cleared at about 15:59–16:00, but the recovery mechanism is not established. The build-hook backlog was still about 1.26M at 17:16, and its drain time is not recorded. The deployment-health gate self-cleared at 17:26.