2026-06-30: CI/CD pipelines not being triggered on merge requests.

CI/CD pipelines not being triggered on merge requests. (Severity 1)

Problem: CI/CD pipelines on merge requests were not being triggered or were experiencing major delays for all customers on GitLab.com due to Sidekiq CPU saturation and a backlog on the urgent-ci-pipeline shard.

Impact: All customers on GitLab.com experienced significant delays or failures triggering CI/CD pipelines on merge requests. Pipeline status updates were outdated, jobs were deferred, and many users could not run CI/CD workflows. At peak, the backlog in the urgent-ci-pipeline queue reached over 800,000 jobs. Pipelines are now being created successfully, but some jobs are still processing with a delay as the system works through the remaining queued work. Monitoring shows throughput and processing rates have returned to normal, though some minor delays persist.

Causes: A single user created a large number of merge requests which saturated the urgent-ci-pipeline Sidekiq shard and created a backlog. Even after blocking this user and deleting their queued jobs, the backlog persisted due to high overall demand and concurrency throttling for MergeRequests::CreatePipelineWorker. No individual user is now dominating the queue.

Response strategy: We blocked the user responsible for the initial backlog at 09:45 UTC and deleted their queued jobs. We expanded job deletions to other queues and then increased the Sidekiq urgent-ci-pipeline worker capacity from 1200 to 1500 replicas at 10:56 UTC. This scaling cleared the backlog, restored normal pipeline creation, and reduced job deferment and worker saturation. Some jobs are still processing with a delay as the system catches up, but overall rates are back to normal levels as of 11:22 UTC (Etc/UTC).

This ticket was created to track INC-11596, by incident.io 🔥