Investigate why a certain project that triggered UrgentMergeRequestsWorker jobs was running slow

This ticket was created from INC-12753 using incident.io 🔥

Context

Incident: 2026-08-05: sidekiq_execution apdex 93.94% on u... (gitlab-com/gl-infra/production#22645 - closed)

Contributing cause this Corrective Action addresses

The urgent-cpu-bound Sidekiq shard's apdex degraded to 93.94% over a 6-hour window. Responders identified UpdateMergeRequestsWorker as the dominant consumer of CPU seconds on the shard, and traced the elevated usage to a single project disproportionately triggering those jobs. The mitigating action taken during the incident — moving UpdateMergeRequestsWorker to the urgent-ci-pipeline shard via MR !5694 — does not explain why that project's jobs were slow. The root cause (unusually large revision deltas, a high count of open merge requests targeting main, mirroring/import activity, or excessive retries) remains unconfirmed.

Proposed Action

Define and document the specific performance bottleneck causing UpdateMergeRequestsWorker jobs triggered by the identified project to exceed their execution duration target: query Sidekiq logs and Kibana for that project's job execution times, commit-range sizes, open merge request count against main, push source type, and retry counts (suggested — confirm with assignee); confirm which hypothesis — large revision delta, MR fan-out, mirroring load, or retry storms — is the primary driver; and capture findings in gitlab-com/gl-infra/production-engineering#29490 (moved) with a recommended code-level or configuration change to prevent recurrence on similarly active projects.

AI Slop FWIW

The signature — CPU-bound (cpu/wall ≈ 0.93), always on main — narrows it down a lot. It's not slow Gitaly or Postgres; it's Ruby loops chewing on large in-memory collections. Here's what almost certainly drives it.

The root lever: @commits is unbounded

find_new_commits (refresh_service.rb:299) calls:

@commits = @project.repository.commits_between(@push.oldrev, @push.newrev)

... and commits_between (repository.rb:177) is called with no limit. So @commits is every commit between oldrev and newrev. For a normal push that's a handful. But when oldrev is far behind newrev on main, this can be tens or hundreds of thousands of commits. Everything downstream scales with |@commits|.

When does main move by a huge delta repeatedly? The usual culprits:

  • Pull mirroring / repository import / a sync bot repointing main in big jumps.
  • A force-push or history rewrite on main where oldrev..newrev spans a large range.
  • Automation that keeps pushing large merges to main.

Since all the long runs target main on one project, I'd bet this project has one of the above feeding large deltas.

Where the CPU actually burns (both scale with |@commits|)

1. post_merge_manually_merged — an O(commits × MRs) linear scan (refresh_service.rb:94-107) — my prime suspect.

commit_ids = @commits.map(&:id)          # huge Array
...
.select do |merge_request|
  commit_ids.include?(merge_request.diff_head_sha) && ...   # Array#include? = O(n)
end

commit_ids is a plain Array, and Array#include? is a linear scan run once per open MR targeting main. So cost ≈ |commits| × |open MRs targeting main| string comparisons. 100k commits × a few thousand open MRs = billions of comparisons — pure CPU, exactly matching your 0.93 ratio. (A Set would make this O(1); it isn't one.)

2. BranchPushMergeCommitAnalyzer (line 113 / analyzer file) — allocates a CommitDecorator per commit into a hash and walks the parent graph. Linear in |@commits|, but with huge commit counts the allocation + GC pressure alone burns significant CPU. It also runs @commits.reverse first (another big allocation).

Both of these only even reach the expensive part when some MR heads are in the push, but the O(commits × MRs) include? scan in gitlab-com/gl-infra/production-engineering#1 (closed) runs regardless of matches.

Amplifiers

  • Retries: sidekiq_options retry: 3 and the worker is explicitly non-idempotent. If a run OOMs or is killed, it retries and re-does the entire expensive computation from scratch — so one bad push can cost 3–4× the work.
  • Force push (line 194): if any of these are force pushes to main, every MR whose source or target is main gets reload_diff — that adds heavy work on top (though that's more I/O than CPU).

How to confirm (quick, targeted)

  1. Measure the delta for the actual revs — this is the smoking gun:

    git rev-list --count e41e1dbe1417ee2efea2b3eef7d3e2af0cc7c212..d0b9a464dcfb8e91930082007f5f1c460e7a06b2

    If that's in the tens/hundreds of thousands, @commits is your problem.

  1. Count open MRs targeting main for project 72783132 — the other multiplier:

    Project.find(72783132).merge_requests.opened.where(target_branch: 'main').count

  2. Check what's moving main on that project: is it a pull mirror, an active import, or a bot/automation? (Project#mirror?, import state, recent events/audit for who pushed those revs.)

  3. Confirm these are distinct jobs vs. retries of the same one (check the jid / whether retry_count > 0 in the logs).

If (1) comes back huge, the fix direction is capping/short-circuiting the commit set (and/or converting that commit_ids Array to a Set) rather than anything about main specifically — but let's confirm the numbers first.

Last updated by inc-automate on 2026-08-05 06:58 UTC · synthesis model: claude-sonnet-4-6 · request id: 1ec0f20eab81bf62.

Edited by 🤖 GitLab Bot 🤖