Incident Review: INC-6458: Sidekiq queueing SLO violation on urgent-cpu-bound shard

INC-6458: Sidekiq queueing SLO violation on urgent-cpu-bound shard (apdex 73.88%)

Generated by Rehab Hassanein on 6 Jan 2026 16:38. All timestamps are local to Etc/UTC

Key Information

Metric	Value
Customers Affected	Customer emergency tickets, 682732, 682726, 682730, 682731, 682728 in gitlab.com
Requests Affected	Website, API, and CI/CD components affected by degraded performance. Sidekiq job backlog peaked at 600,000–700,000 jobs
Incident Severity	Severity 2
Impact Start Time	Tue, 06 Jan 2026 06:45:00 UTC
Impact End Time	Tue, 06 Jan 2026 09:21:27 UTC
Total Duration	2 hours, 36 minutes
Link to Incident Issue	2026-01-06: Sidekiq queueing SLO violation on u... (#21046 - closed)

Summary

RCA Summary

Between Tue, 06 Jan 2026 09:21:27 UTC and Tue, 06 Jan 2026 09:21:27 UTC a required index on the ssh_signatures table was removed during a recent post-deployment migration, causing a surge of slow queries. The old index index_ssh_signatures_on_commit_sha (commit_sha) was removed and queries looking up ssh_signatures via the commit_sha column only started slowing down (or timing out), this overwhelmed the primary database and saturated the connection pool, leading to widespread delays.

We manually recreated the missing database index, which restored performance. Related incidents for API and web service Apdex SLO violations were merged into this incident, and a follow-up was created to add PDM annotations to Grafana for improved monitoring.

Incident Details

Investigation RCA can be found here: 2026-12-06 - Sidekiq queueing SLO violation on urgent-cpu-bound shard

Problem: Sidekiq queueing performance on the urgent-cpu-bound shard dropped sharply, with the apdex value falling to 73.88%.

Impact: Customers experienced slow merge requests and delayed pipeline triggers. Multiple customer emergencies were reported, and QA tests confirmed recurring timeouts during repository interactions and merge request preparation. The Sidekiq job backlog, which peaked at 600,000–700,000 jobs, is now nearly cleared. All key alerts for web and API services have resolved, and there is no remaining customer impact.

Causes: A required index on the ssh_signatures table was removed during a recent post-deployment migration, causing a surge of slow queries. This overwhelmed the primary database and saturated the connection pool, leading to widespread delays.

Response strategy: We manually recreated the missing database index, which restored performance. Deployments and feature flag changes remain blocked until final checks are completed. Related incidents for API and web service Apdex SLO violations were merged into this incident, and a follow-up was created to add PDM annotations to Grafana for improved monitoring.

What went well?

Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared.

Example:

We quickly discovered a recently changed feature flag through the event log which enabled fast mitigation of the impact, as well as pulling in the engineer involved to further diagnose.
We escalated through dev escalations, which brought in Person X. They knew that Person Y had expertise with the component in question, which enabled faster diagnosis.

What was difficult?

Use this section to highlight opportunities for improvement discovered during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. If the improvement seems like a simplest change, consider adding it as a corrective action above instead. Think about how to improve response next time, and consider any patterns pointing to broader issues, like “key person risk.”

Example:

The runbooks/playbooks for this service are out of date and did not contain the information necessary to troubleshoot the incident.
The incident happened at a time when nobody with expertise on the service was available.

Investigation Details

Timeline

Incident Timeline

2026-01-06

06:45:00 Impact started at

Custom timestamp "Impact started at" occurred

07:18:25 Incident reported in triage by Prometheus Alertmanager alert

Prometheus Alertmanager alert reported the incidentSeverity: NoneStatus: Triage

07:29:24 Image posted by Rehab Hassanein

Rehab Hassanein posted an image to the channel

Might be early to say, but this might correlate with the most recent CNY deploy 18.8.202601060351-4a0309fa428.b355c04a60c

07:30:27 Incident accepted

Rehab Hassanein shared an update

Severity: ~~None~~ → Severity 3

Status: ~~Triage~~ → Investigating

07:41:38 Message from Rehab Hassanein

Rehab Hassanein's message was pinned by Siddharth Kannan

The diff doesn't seem that significant 🤔

https://gitlab.slack.com/archives/C8PKBH3M5/p1767679650313149?thread_ts=1767679643.902179&cid=C8PKBH3M5

07:44:50 Message from Rehab Hassanein

Rehab Hassanein's message was pinned by Siddharth Kannan

@siddharthkannan wdyt about trying to disable canary to see if that mitigates the issue?

07:45:13 Message from Siddharth Kannan

Siddharth Kannan pinned their own message

Sure, we ~~can~~ may be able to try it, but Sidekiq is shared between main and canary?

07:45:38 Message from Rehab Hassanein

Rehab Hassanein's message was pinned by Siddharth KannanYes - I was just about to verify that... it is shared.

07:52:34 Severity upgraded from Severity 3 → Severity 2

Rehab Hassanein shared an update

Severity: ~~Severity 3~~ → Severity 2

We've confirmed that the Sidekiq queueing degradation on the urgent-cpu-bound shard is causing user impact. Support has reported customer emergencies related to slow merge requests and pipeline triggers.QA tests and job logs show recurring timeouts in repository interactions and merge request preparation, which aligns with reported customer issues. There is no evidence so far that recent deployments or configuration changes are the root cause.Deploys have been blocked to prevent further impact, and release managers have been notified.

07:52:35 Escalated to GitLab.com Production

Your workflow manually escalated the incident to the escalation path GitLab.com Production

07:52:35 Escalated to Incident Manager Oncall (IMOC)

Your workflow manually escalated the incident to the escalation path Incident Manager Oncall (IMOC)

07:57:49 Update shared

Rehab Hassanein shared an updateWe've escalated the incident in response to multiple customer emergencies reporting slow merge requests and delayed pipelines. The impact is confirmed by support tickets and recurring timeouts in QA tests for repository interactions and merge request preparation.The team has observed a sharp increase in Sidekiq inflight jobs on the urgent-cpu-bound shard, with the spike coinciding with the recent 'cny' deployment. As a mitigation, we've decided to drain the 'cny' deployment to check if this reduces errors and queue durations.Deploys remain blocked, and release managers have been notified. Consideration is being given to updating the public status page in light of the user impact.Relevant resources:

• Incident issue

• QA job failures

• Sidekiq inflight jobs dashboard

• Customer emergency tickets, 682732, 682726, 682730, 682731, 682728

07:58:14 Message from Siddharth Kannan

Siddharth Kannan's message was pinned by Rehab Hassanein

Done: https://gitlab.slack.com/archives/C0139MAV672/p1767686287501049

08:01:16 Message from Siddharth Kannan

Siddharth Kannan pinned their own message

Draining canary is complete: https://gitlab.slack.com/archives/C0139MAV672/p1767686452763429

08:05:30 Message from Patrick Bajao

Patrick Bajao's message was pinned by Siddharth KannanCould this also be related to post deploy migrations that recently run on production?

08:06:20 Message from Siddharth Kannan

Siddharth Kannan pinned their own message

List of PDMs that ran: gitlab-org/release/tasks#22844 (comment 2985400021)

08:11:37 Escalated to tier2 - Database Operations On-Call

Rehab Hassanein manually escalated the incident to the escalation path tier2 - Database Operations On-Call

08:23:00 Image posted by Stan Hu

Stan Hu posted an image to the channel

08:29:21 Image posted by Stan Hu

Stan Hu posted an image to the channel

https://log.gprd.gitlab.net/app/r/s/DwLbO

08:29:21 Update shared

Rehab Hassanein shared an updateAfter draining the canary ('cny') deployment, there has been no improvement in Sidekiq queueing or overall system performance. Investigation has shifted to potential database-related causes, specifically the recent post-deployment migrations that ran following the end of the change freeze. This larger batch of migrations may have increased pressure on the database.

Monitoring dashboards confirm that PG Bouncer is fully saturated, especially for Sidekiq connections, resulting in prolonged wait times. A 100% saturation event was observed on the 'pgbouncer_async_primary_pool' component starting at 07:12 UTC. Additionally, a sustained spike in CPU usage was noted on the primary database instance (patroni-main-v16-102-db-gprd).

The team has reviewed currently running background migrations, with one paused and others still active. There is ongoing discussion about pausing additional background migrations to help relieve database load. No CI-related jobs or integrations are standing out as a unique source of the problem.Key references:

• Sidekiq queue inflight jobs dashboard

• PG Bouncer async pool saturation

• Primary DB CPU load

• List of post-deployment migrations

Deploys and feature flag changes remain blocked. Customer impact continues as previously described, with slow merge requests and delayed pipelines.

08:30:12 Message from Dat Tang

Dat Tang pinned their own message

https://log.gprd.gitlab.net/app/discover#/?\\\\\\\_g=h@ccf8cb2&\\\\\\\_a=h@fd9d82a

08:31:07 Message from Dat Tang

Dat Tang's message was pinned by Siddharth Kannan

This PDM? https://gitlab.com/gitlab-org/security/gitlab/-/blob/e7dea8d5b7593779b84ccb957ffaa\\\\\\\[…\\\\\\\]te/20251211182111_reintroduce_composite_index_ssh_signatures.rb (https://gitlab.com/gitlab-org/security/gitlab/-/blob/e7dea8d5b7593779b84ccb957ffaa88fe3f91720/db/post_migrate/20251211182111_reintroduce_composite_index_ssh_signatures.rb)

08:51:36 Update shared

Rehab Hassanein shared an update

We've identified the root cause of the Sidekiq queueing and database saturation as a missing index on the ssh_signatures table, which was removed during a recent post-deployment migration. This led to a surge of slow queries overwhelming the primary database and causing widespread delays.

The team manually recreated the dropped index, and initial monitoring shows a significant drop in query latency, pool saturation, and error rates. Metrics and logs confirm that performance is recovering, but there is still a backlog of jobs to process, and degraded performance may persist until queues are cleared.

Deploys and feature flag changes remain blocked while we continue to monitor system health. Updates have been provided on the status page and stakeholders are being kept informed.

Key references:

• Incident issue• Post-deployment migrations• Sidekiq inflight jobs dashboard• PG Bouncer async pool saturation• Primary DB CPU load

We'll continue to monitor until we're confident the backlog is resolved and normal performance is restored.

09:05:17 Image posted by Alexander Sosna

Alexander Sosna posted an image to the channel

Pool Saturation (https://dashboards.gitlab.net/goto/af9c5c7d7in0ge?orgId=1) decreased substantially.

09:05:49 Status changed from Investigating → Fixing

Rehab Hassanein shared an update

Status: ~~Investigating~~ → Fixing

System performance continues to improve following the manual recreation of the missing database index on the ssh_signatures table. Key metrics show positive trends: database connection pool saturation has dropped below 80% (previously >99%), and the Sidekiq job backlog is trending down from a peak of 600,000–700,000 jobs.Deployments and feature flag changes remain blocked. The team has agreed not to run further post-deployment migrations until the index recreation is included in the next release, to avoid rolling back the fix. Two follow-ups have been created: one to re-enable the 'cny' deployment when mitigation is complete, and another to ensure the recreated index is formally included in the next release.We'll continue monitoring until the job backlog fully clears and normal performance is restored.

09:05:49 Identified at

Custom timestamp "Identified at" occurred

09:06:58 Message from Dat Tang

Dat Tang pinned their own message

Canary enabled https://gitlab.slack.com/archives/C0139MAV672/p1767690333223159

09:21:27 Fixed at

Custom timestamp "Fixed at" occurred

09:21:27 Monitoring at

Custom timestamp "Monitoring at" occurred

09:21:27 Status changed from Fixing → Monitoring

Rehab Hassanein shared an update

Status: ~~Fixing~~ → Monitoring

Database connection pool saturation on the urgent-cpu-bound shard dropped after the index recreation, but showed brief spikes back to 100% as Sidekiq continued processing the remaining job backlog. Monitoring confirmed these fluctuations aligned with periods of heavy queue processing, and recent metrics show both queue length and saturation now trending down to near zero.The Sidekiq job backlog, which peaked at 600,000–700,000, is now nearly cleared. Key alerts for the web service have resolved, and the team has confirmed there is no remaining customer impact. The incident is moving to monitoring, with deployments and feature flag changes to be unblocked after final checks.Related incidents for API and web service Apdex SLO violations were merged into this incident, and a follow-up was created to add PDM annotations to Grafana for improved monitoring. The status page will be updated to reflect resolution.

09:21:36 Message from Adam Hegyi

Adam Hegyi's message was pinned by Rehab Hassanein

I notified the team, if I don't hear from them I'll open an MR to restore the index: https://gitlab.slack.com/archives/CNU5W2F5M/p1767690880760569

09:31:10 Incident resolved and entered the post-incident flow

Rehab Hassanein shared an update

Status: ~~Monitoring~~ → Documenting

The incident has been fully resolved. Database performance and Sidekiq queueing have recovered after manually recreating the missing index, and all user-facing metrics—including merge request creation and pipeline triggers—are back to normal.All alerts related to the incident have cleared, and no further customer impact is observed. The status page has been set to green. Deployments and feature flag changes will be unblocked after follow-up checks, and tasks remain open to ensure the recreated database index is formally included in the next release and to add annotations for post-deployment migrations in Grafana for future visibility.

Investigation Notes

Any details you may want to add about the investigation can go here.

Key findings

Group: Source Code

Post-deployment migrations need better safeguards to prevent dropping critical indexes without verification. A Danger bot check was added to detect DROP INDEX operations in Rails migrations and warn reviewers about potential risks.
Documentation was updated with guidance on composite index column order when dropping indexes, ensuring developers understand that queries using only the trailing column(s) of a composite index will not be covered.
Duo MR review instructions were updated to include database guidance about DROP INDEX operations, helping AI-assisted reviews catch risky index removals.

The FCL action items here (gitlab-org#20517 (closed)) are targeted specifically at addressing these 3 contributing factors.

Group: Production Engineering

A follow-up was created to add PDM (post-deployment migration) annotations to Grafana for improved visibility into when migrations run and their potential impact.

Group: Release and Deploy

The team agreed to lock PDM execution until the index recreation was formally included in the next release, preventing accidental rollbacks.

Follow-ups

Follow-up	Owner
ChatOps command to lock PDM	- (workflow-infraCancelled)
Add annotations to Grafana about PDMs	Rehab Hassanein
The recreated database index must be included in the next release to prevent recurrence of the issue	Gavin Hinfey
Follow up: Re-enable `cny` after the incident is mitigated	Dat Tang

Review Guidelines

This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.

For the person opening the Incident Review

Set the title to Incident Review: (Incident issue name)
Assign a Service::* label (most likely matching the one on the incident issue)
Set a Severity::* label which matches the incident
In the Key Information section, make sure to include a link to the incident issue
Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.

For the assigned DRI

Edited Jan 20, 2026 by Javiera Tapia