2025-10-09: Sidekiq queueing SLO violation on catchall shard (apdex 67.38%)
Sidekiq queueing SLO violation on catchall shard (apdex 67.38%) (Severity 3 (Medium))
Problem: Sidekiq jobs on the catchall shard experienced severe queueing delays, with queue lengths staying high and performance dropping below target levels.
Impact: Jobs on the catchall Sidekiq shard experienced roughly 20 minute delays, with only 2.7% of jobs meeting queueing targets. This caused significant slowdowns for affected services over several days.
Causes: A recent reduction in the pgbouncer pool size on September 24 has correlated with repeated Sidekiq queueing issues and more frequent pgbouncer saturation events. No database issues or specific problematic jobs, projects, or users were found. The durability team will investigate next steps.
Response strategy: We reviewed metrics and dashboards, escalated the incident, involved release managers and the database team, and identified the correlation between pgbouncer pool size reduction and queue saturation. The incident resolved without direct intervention, but a follow-up will be opened to review pgbouncer configuration.
This ticket was created to track INC-4634, by incident.io