2026-03-10: Sidekiq queueing apdex SLO violation on low-urgency-cpu-bound shard

Sidekiq queueing apdex SLO violation on low-urgency-cpu-bound shard (Severity 2)

Problem: Queueing SLOs were violated for Sidekiq jobs on the low-urgency-cpu-bound and catchall shards, with apdex dropping well below targets due to database connection saturation.

Impact: Multiple teams experienced moderate queueing delays on the low-urgency-cpu-bound shard and severe delays on the catchall shard. Deployments and feature flags were blocked, and GitLab CI/CD pipelines saw job creation delays. Three customer tickets referenced delays, but service is now stabilizing.

Causes: PgBouncer saturation on GitLab production Sidekiq caused by Sidekiq workers such as MergeRequestMergeabilityCheckWorker and WebHooks::LogExecutionWorker holding long-lived database transactions. This led to pool exhaustion and backlog.

Response strategy: We manually deleted affected Sidekiq pods to restore service and disabled the auto_disabling_web_hooks feature flag to reduce queue pressure. No additional worker disabling was needed as queue lengths and PgBouncer saturation have improved. We continue to monitor Sidekiq and CI/CD health.

This ticket was created to track INC-8246, by incident.io 🔥