_This ticket was created from_ [_INC-12753_](https://app.incident.io/gitlab/incidents/12753) _using_ [_incident.io_](https://app.incident.io) 🔥 <!-- BEGIN inc-automate enrichment --> ## Context Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22645+ ## Contributing cause this Corrective Action addresses The `urgent-cpu-bound` Sidekiq shard's apdex degraded to 93.94% over a 6-hour window. Responders identified `UpdateMergeRequestsWorker` as the dominant consumer of CPU seconds on the shard, and traced the elevated usage to a single project disproportionately triggering those jobs. The mitigating action taken during the incident — moving `UpdateMergeRequestsWorker` to the `urgent-ci-pipeline` shard via [MR !5694](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5694) — does not explain *why* that project's jobs were slow. The root cause (unusually large revision deltas, a high count of open merge requests targeting `main`, mirroring/import activity, or excessive retries) remains unconfirmed. ## Proposed Action Define and document the specific performance bottleneck causing `UpdateMergeRequestsWorker` jobs triggered by the identified project to exceed their execution duration target: query Sidekiq logs and Kibana for that project's job execution times, commit-range sizes, open merge request count against `main`, push source type, and retry counts (suggested — confirm with assignee); confirm which hypothesis — large revision delta, MR fan-out, mirroring load, or retry storms — is the primary driver; and capture findings in [gitlab-com/gl-infra/production-engineering#29490](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29490) with a recommended code-level or configuration change to prevent recurrence on similarly active projects. --- ## AI Slop FWIW The signature — *CPU-bound (cpu/wall ≈ 0.93), always on main* — narrows it down a lot. It's not slow Gitaly or Postgres; it's Ruby loops chewing on large in-memory collections. Here's what almost certainly drives it. ## The root lever: `@commits` is unbounded `find_new_commits` (refresh_service.rb:299) calls: ``` @commits = @project.repository.commits_between(@push.oldrev, @push.newrev) ``` ... and commits_between (repository.rb:177) is called *with no limit*. So @commits is every commit between oldrev and newrev. For a normal push that's a handful. But when oldrev is far behind newrev on main, this can be tens or hundreds of thousands of commits. Everything downstream scales with |@commits|. When does main move by a huge delta repeatedly? The usual culprits: - *Pull mirroring / repository import / a sync bot* repointing main in big jumps. - A *force-push or history rewrite* on main where oldrev..newrev spans a large range. - Automation that keeps pushing large merges to main. Since all the long runs target main on one project, I'd bet this project has one of the above feeding large deltas. ## Where the CPU actually burns (both scale with |@commits|) *1. post_merge_manually_merged — an O(commits × MRs) linear scan (refresh_service.rb:94-107)* — my prime suspect. ``` commit_ids = @commits.map(&:id) # huge Array ... .select do |merge_request| commit_ids.include?(merge_request.diff_head_sha) && ... # Array#include? = O(n) end ``` `commit_ids` is a plain *Array*, and `Array#include?` is a linear scan run *once per open MR targeting main*. So cost ≈ |commits| × |open MRs targeting main| string comparisons. 100k commits × a few thousand open MRs = billions of comparisons — pure CPU, exactly matching your 0.93 ratio. (A Set would make this O(1); it isn't one.) *2. BranchPushMergeCommitAnalyzer (line 113 / analyzer file)* — allocates a CommitDecorator per commit into a hash and walks the parent graph. Linear in |@commits|, but with huge commit counts the allocation + GC pressure alone burns significant CPU. It also runs @commits.reverse first (another big allocation). Both of these only even reach the expensive part when some MR heads are in the push, but the O(commits × MRs) include? scan in gitlab-com/gl-infra/production-engineering#1 runs regardless of matches. ## Amplifiers - *Retries*: sidekiq_options retry: 3 and the worker is explicitly *non-idempotent*. If a run OOMs or is killed, it retries and re-does the entire expensive computation from scratch — so one bad push can cost 3–4× the work. - *Force push* (line 194): if any of these are force pushes to main, every MR whose source or target is main gets reload_diff — that adds heavy work on top (though that's more I/O than CPU). ## How to confirm (quick, targeted) 1. *Measure the delta for the actual revs* — this is the smoking gun: ``` git rev-list --count e41e1dbe1417ee2efea2b3eef7d3e2af0cc7c212..d0b9a464dcfb8e91930082007f5f1c460e7a06b2 ```     If that's in the tens/hundreds of thousands, @commits is your problem. 2. *Count open MRs targeting main* for project 72783132 — the other multiplier: ``` Project.find(72783132).merge_requests.opened.where(target_branch: 'main').count ``` 3. *Check what's moving main* on that project: is it a *pull mirror*, an active *import*, or a bot/automation? (Project#mirror?, import state, recent events/audit for who pushed those revs.) 4. *Confirm these are distinct jobs vs. retries* of the same one (check the jid / whether retry_count > 0 in the logs). If (1) comes back huge, the fix direction is capping/short-circuiting the commit set (and/or converting that commit_ids Array to a Set) rather than anything about main specifically — but let's confirm the numbers first. --- _Last updated by [`inc-automate`](https://gitlab.com/gitlab-com/gl-infra/inc-automate) on 2026-08-05 06:58 UTC · synthesis model: `claude-sonnet-4-6` · request id: `1ec0f20eab81bf62`._ <!-- END inc-automate enrichment -->