2026-09-10: Sidekiq jobs slow to process and being deferred
Sidekiq jobs slow to process and being deferred (Severity 2)
Problem: Security policy evaluations were delayed, blocking some merge request approvals.
Impact: Some merge request approvals for SaaS customers were blocked or delayed. We also confirmed multiple customer reports, and at least one affected customer saw policy evaluations time out.
Causes: Security policy evaluation workers were hitting a fixed concurrency limit and queuing behind it. We also confirmed there was sustained main database connection wait time and saturation during the incident.
Response strategy: We confirmed the affected queue drained to zero at 9:13pm UTC, and we re-enabled the concurrency deferral limit for the worker to reduce the risk of a surge of security policy evaluation work crowding out other Sidekiq jobs.