# [INC-8246: Sidekiq queueing apdex SLO violation on low-urgency-cpu-bound shard](https://app.incident.io/gitlab/incidents/8246) Generated by Scott Hampton on 11 Mar 2026 17:42. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | CI jobs delayed for all users of GitLab.com | | Requests Affected | | | Incident Severity | Severity 2 | | Impact Start Time | Tue, 10 Mar 2026 12:00:00 UTC| | Impact End Time | | | Total Duration | 8 hours, 24 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21498 | # Summary **Problem**: Queueing SLOs were violated for Sidekiq jobs on the low-urgency-cpu-bound and catchall shards, with apdex dropping well below targets due to database connection saturation. **Impact**: Multiple teams experienced moderate queueing delays on the low-urgency-cpu-bound shard and severe delays on the catchall shard. Deployments and feature flags were blocked, and GitLab CI/CD pipelines saw job creation delays. Three customer tickets referenced delays, but service is now stabilizing. **Causes**: PgBouncer saturation on GitLab production Sidekiq caused by Sidekiq workers such as MergeRequestMergeabilityCheckWorker and WebHooks::LogExecutionWorker holding long-lived database transactions. This led to pool exhaustion and backlog. **Response strategy**: We manually deleted affected Sidekiq pods to restore service and disabled the auto_disabling_web_hooks feature flag to reduce queue pressure. No additional worker disabling was needed as queue lengths and PgBouncer saturation have improved. We continue to monitor Sidekiq and CI/CD health. **To address this and other recent issues related to Sidekiq we created an FCL https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/86** # What went well? The team came together and we started [an FCL](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/86) to signal boost the recent instability we have been having around the non-urgent pool. # What was difficult? There was some uncertainty about the jobs that contribute to delayed pipelines and why some of these jobs are assigned to the non-urgent shard. To help with this in the future we opened: 1. https://gitlab.com/gitlab-org/gitlab/-/work_items/593064+ 2. https://gitlab.com/gitlab-org/gitlab/-/work_items/592819+ # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2026-03-10** **12:00:00** **Impact started at** Custom timestamp "Impact started at" occurred **12:21:26** **Incident reported in triage by Prometheus Alertmanager alert** Prometheus Alertmanager alert reported the incident Severity: None Status: Triage **12:48:08** **Image posted by Pravar Gauba** Pravar Gauba posted an image to the channel src: https://log.gprd.gitlab.net/app/r/s/zheEq Increase in errors with: `ReactiveCaching::LowUrgencyWorker`  **12:58:31** **Image posted by Pravar Gauba** Pravar Gauba posted an image to the channel okay.. we have quite a number of jobs queued by concurrency limiter of this worker.. src: https://dashboards.gitlab.net/d/sidekiq-concurrency/sidekiq3a-worker-concurrency-detail[…]activeCaching::LowUrgencyWorker&viewPanel=panel-2282485207 (https://dashboards.gitlab.net/d/sidekiq-concurrency/sidekiq3a-worker-concurrency-detail?orgId=1&from=2026-03-09T12:57:37.991Z&to=2026-03-10T12:57:37.991Z&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-worker=ReactiveCaching::LowUrgencyWorker&viewPanel=panel-2282485207)  **13:35:12** **Incident accepted** Pravar Gauba shared an update Severity: ~~None~~ → Severity 4 Status: ~~Triage~~ → Investigating 2h later **15:48:48** **Severity upgraded from Severity 4 → Severity 2** Stephanie Jackson shared an update Severity: ~~Severity 4~~ → Severity 2 We are seeing moderate queueing SLI impact on the low-urgency-cpu-bound shard, and severe impact on the catchall shard. The catchall shard's queue lengths are not visible, but based on reports from other teams, there are likely pipeline delays affecting GitLab CI/CD. Sidekiq pods on the gprd regional cluster have experienced slow bootups and crashloops, with repeated database connection errors observed in logs. Manual pod deletions have been performed to try to restore service. This incident now appears to affect both low-urgency and catchall Sidekiq shards, with potential for broader CI/CD impact. Further updates will follow as more information on user-facing effects and remediation progress becomes available. Relevant logs and dashboards: • [Pod log example](https://log.gprd.gitlab.net/app/r/s/YAGly "Pod log example") • [Queueing SLI dashboard](https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?from=2026-03-10T12:41:39.478Z&to=2026-03-10T15:41:39.478Z&var-environment=gprd&var-stage=main&var-shard=low-urgency-cpu-bound&var-shard=catchall&var-shard=catchall-b&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&viewPanel=panel-3643825112 "Queueing SLI dashboard") **15:48:48** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **15:48:48** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **16:38:53** **Update shared** Scott Hampton (via @incident) shared an update We're continuing to investigate Sidekiq queueing SLO violations on the low-urgency-cpu-bound and catchall shards. PgBouncer saturation on GitLab production Sidekiq is confirmed, with demand exceeding available database connection capacity. The team is analyzing which Sidekiq job classes are dominating Postgres connections, aiming to identify candidates for deferral or prioritization to reduce impact on critical workflows like CI/CD. There is active discussion about tuning isolation boundaries and queue prioritization to focus resources on the most customer-visible jobs. Multiple teams remain affected, and deployments and feature flags are still blocked. No immediate fix is available. Mitigation efforts are focused on identifying and deferring less critical job classes to free up capacity for high-impact work. Investigation is ongoing using Prometheus and Kibana metrics to attribute connection usage by job class. Relevant resources: • [Queueing SLI dashboard](https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?from=2026-03-10T12:41:39.478Z&to=2026-03-10T15:41:39.478Z&var-environment=gprd&var-stage=main&var-shard=low-urgency-cpu-bound&var-shard=catchall&var-shard=catchall-b&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&viewPanel=panel-3643825112) • [Pod log example](https://log.gprd.gitlab.net/app/r/s/YAGly) • [User report: Zendesk ticket #700334](https://gitlab.zendesk.com/agent/tickets/700334) **17:14:49** **Update shared** Scott Hampton (via @incident) shared an update We've observed substantial recovery in Sidekiq queueing and database saturation metrics. Recent analysis identified several Sidekiq workers, including `MergeRequestMergeabilityCheckWorker` and `WebHooks::LogExecutionWorker`, as major contributors to prolonged database transactions and pool exhaustion. We've reviewed transaction states over the last hour, confirming these endpoints frequently remain in 'idle in transaction' or 'active' states, contributing to the backlog. No additional worker disabling was required as queue lengths and PgBouncer saturation have improved. Only three customer tickets referencing delays were identified, and internal metrics suggest service is stabilizing, though some CI pipelines may still be delayed in creating new jobs. Relevant resources: • [Prometheus query results screenshot](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22c5x%22:%7B%22datasource%22:%22mimir-gitlab-gprd%22,%22queries%22:%5B%7B%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gprd%22%7D,%22expr%22:%22sum_over_time%28sum%28pg_stat_activity_marginalia_sampler_active_count%7Benv%3D%27gprd%27,%20type%3D%27patroni%27,%20state%21%3D%5C%22idle%5C%22,%20database%3D%5C%22gitlabhq_production_sidekiq%5C%22%7D%20and%20on%20%28instance%29%20pg_replication_is_replica%20%3D%3D%200%29%20by%20%28endpoint,state%29%5B1h:%5D%29%22,%22format%22:%22time_series%22,%22interval%22:%2230s%22,%22intervalFactor%22:1,%22legendFormat%22:%22%7B%7B%20endpoint%20%7D%7D%22,%22refId%22:%22A%22,%22editorMode%22:%22code%22,%22range%22:false,%22instant%22:true,%22exemplar%22:false%7D%5D,%22range%22:%7B%22from%22:%22now-3h%22,%22to%22:%22now%22%7D,%22compact%22:false%7D%7D&orgId=1) • [Queueing SLI dashboard](https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?from=2026-03-10T12:41:39.478Z&to=2026-03-10T15:41:39.478Z&var-environment=gprd&var-stage=main&var-shard=low-urgency-cpu-bound&var-shard=catchall&var-shard=catchall-b&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&viewPanel=panel-3643825112) • [User report: Zendesk ticket #700334](https://gitlab.zendesk.com/agent/tickets/700334) • [GitLab production work item](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21498) We will continue monitoring Sidekiq and CI/CD health, focusing on persistent job classes if further delays are observed. **17:57:51** **Status changed from Investigating → Monitoring** Scott Hampton shared an update Status: ~~Investigating~~ → Monitoring Current metrics show Sidekiq queueing and database saturation have recovered, and CI wait times are back to normal for the last hour. The team agrees that the incident can be transitioned to monitoring, as recent data indicates the backlog and connection pool issues have subsided. We will continue to observe Sidekiq and CI/CD health for any recurring delays, but no further urgent investigation is required at this time. **17:57:51** **Monitoring at** Custom timestamp "Monitoring at" occurred **17:57:51** **Identified at** Custom timestamp "Identified at" occurred **18:33:06** **Update shared** Scott Hampton (via @incident) shared an update All Sidekiq queueing and database saturation metrics have recovered, and CI/CD processing times are back to normal. We identified PgBouncer saturation as the primary cause, triggered by long-lived transactions from specific Sidekiq workers. Remediation included manual Sidekiq pod deletions and disabling the auto_disabling_web_hooks feature flag, which reduced queue pressure. Only three customer tickets were reported, and no new impact has been observed. We are monitoring for recurrence and further customer reports. For more details, see the [GitLab production work item](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21498#note_3147543145) and [Queueing SLI dashboard](https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?from=2026-03-10T12:41:39.478Z&to=2026-03-10T15:41:39.478Z&var-environment=gprd&var-stage=main&var-shard=low-urgency-cpu-bound&var-shard=catchall&var-shard=catchall-b&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&viewPanel=panel-3643825112). **18:45:53** **Incident paused** Scott Hampton shared an update Status: ~~Monitoring~~ → Paused All Sidekiq queueing and database saturation metrics have recovered. Because we believe this incident will occur at the same time tomorrow, we are going to pause this incident and unpause tomorrow so that all context remains. **19:15:54** **Incident resumed** incident.io shared an update Status: ~~Paused~~ → Monitoring **19:21:35** **Incident paused** Scott Hampton shared an update Status: ~~Monitoring~~ → Paused **19:22:47** **Update shared** Scott Hampton shared an update **20:43:07** **Message from Hordur Yngvason** Hordur Yngvason pinned their own message I forgot to link it earlier, but here's a very related epic with follow ups from a past incident https://gitlab.com/groups/gitlab-org/-/work_items/20903 **2026-03-11** **07:57:36** **Message from Matt Smiley** Matt Smiley pinned their own message In summary: Setting FF `disable_git_http_fetch_writes` to `true` should be roughly equivalent to freeing 6 out of 45 backend connections (13%) in sidekiq's non-urgent connection pool to the primary node of the `main` db -- one of the two connection pools that saturated during the incident. 5h later **13:50:04** **Incident resumed** Stephanie Jackson shared an update Status: ~~Paused~~ → Investigating **15:13:07** **Message from Hordur Yngvason** Hordur Yngvason pinned their own message Just stumbled upon this very related issue https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/work_items/289. I'm not sure how much of it was already discussed, but @Matt Smiley already found CPU saturation to be a major contributor to pgbouncer saturation (https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/work_items/289#note_2814450208). So this could have been the cause of at least some of our problems over the last two days. #fcl-incident-7872 was already working on that for the _urgent_ sidekiq shards **15:44:14** **Incident paused** Scott Hampton shared an update Status: ~~Investigating~~ → Paused **17:25:23** **Incident resumed** Scott Hampton shared an update Status: ~~Paused~~ → Documenting Queueing SLI metrics for Sidekiq are stable, with recent dips in problematic shards like catchall-b having recovered. The main contributing factor to improvement appears to be enabling the `disable_git_http_fetch_writes` feature flag, which helped reduce database saturation. Monitoring data and smoke tests show no significant delays in pipeline processing at this time. Internal guidance and dashboard links have been collected for future troubleshooting. The team has decided to close the incident, as Sidekiq health has stabilized and all customer-impacting symptoms have subsided. # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Follow-ups **Follow-up** 1. Evaluate jobs in catchall to see if they can use PG replicas instead of the primary https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/work_items/376 2. increase connection pool size for gitlabhq_production_sidekiq pool https://gitlab.com/gitlab-org/database-team/team-tasks/-/work_items/598 3. pipeline duration must be an SLI https://gitlab.com/gitlab-org/gitlab/-/work_items/592819 4. Workers related to pipeline execution should be set to urgency high https://app.incident.io/gitlab/incidents/8246?tab=post-incident # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [ ] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks