Draft: Move AutoMergeProcessWorker to high urgency

Marc Shawrequested to merge
631913-auto-merge-process-worker-high-urgency into master

Marks AutoMergeProcessWorker as urgency :high and updates app/workers/all_queues.yml to match. The worker currently defaults to low urgency, so it shares the low-urgency database pool and can be delayed when that pool is saturated. Merge-when-checks-pass MRs then wait longer than they should. This is a draft until the execution times on GitLab.com are measured against the high-urgency SLO.

Related to #631913 and #631893.

Detailed context for AI agents

Why

#631893 follows a Sidekiq incident where merge trains were repeatedly delayed. The likely cause was a saturated PgBouncer main pool shared by low-urgency Sidekiq workers. High-volume workers such as WebHooks::LogExecutionWorker contribute to this (see !259143 (merged)). Proposal item 3 in #631893 asks whether AutoMergeProcessWorker should follow the merge train workers off that pool. Discussion in that issue agreed it probably should, and #631913 tracks the work.

What the worker does and when it runs

  • app/workers/auto_merge_process_worker.rb has no urgency, so it defaults to :low. That routes it to the low-urgency shard and the shared low-urgency database pool.
  • Other attributes: data_consistency :sticky, worker_resource_boundary :cpu, queue_namespace :auto_merge, feature_category :continuous_delivery, sidekiq_options retry: 3.
  • It declares deduplicate :until_executed, if_deduplicated: :reschedule_once, but the worker is not marked idempotent, so deduplication is effectively inert. This MR does not change that.
  • It is enqueued on pipeline completion (parent and child pipelines), on pushes to a target branch (up to 20 MRs), and on MR state changes such as approvals, draft toggles and mergeability rechecks.
  • It is user-facing. An MR set to merge when checks pass merges only when this worker runs, so queue delay is visible to users.

The SLO and why it gates un-drafting

From doc/development/sidekiq/worker_attributes.md, high-urgency workers have a 10 s queue scheduling target and a 10 s execution latency target. Median execution should be under 1 s and p99 under 10 s. A worker that cannot meet this should be redesigned or split into a fast high-urgency part and a slow low-urgency part. CPU-bound high-urgency workers are allowed. Memory-bound ones are not. High urgency with :sticky data consistency is allowed.

Moving a slow worker to the urgent shards would hurt other urgent workers. So this MR should not leave draft until the numbers support it.

Prior investigation in #631600 found:

  • A successful run is Gitaly-bound: about 77% of wall time and 24 RPCs, mostly in the synchronous mergeability check.
  • There are many redundant no-op runs per MR, around 11 per MWCP MR.
  • Production had roughly 61,796 jobs over 10 s per week.

The last figure suggests the worker may not meet the p99 target today. It must be re-checked before this ships.

Still to measure

Tracked in #631913:

  1. Current p50, p95 and p99 execution time and job volume for AutoMergeProcessWorker on GitLab.com (Kibana and Sidekiq dashboards). Compare with the high-urgency targets.
  2. The shard it would land on. It is CPU-bound, so likely the urgent CPU-bound shard. Estimate the added RPS and execution time there, following the "Changing a queue's urgency" section of the Sidekiq docs.
  3. Whether a burst of auto-merge jobs could starve other urgent workers.

If the measurements show the SLO is met, un-draft and ship. If not, split the worker before changing urgency.

Alternatives considered

  • Concurrency cap (concurrency_limit or max_concurrency_limit_percentage): could be added alongside this change to stop a burst of auto-merge jobs from starving other urgent workers. Not included here. It depends on the load measurement above.
  • Split the worker: keep a fast high-urgency job and move the slow Gitaly-bound mergeability work to a low-urgency job. This is the fallback if the SLO is not met. It is a larger change and was not attempted before measuring.
  • Dedicated routing rule by worker name: route this one queue to a different shard without changing its urgency. Rejected as the first option. Urgency is the supported way to express this, and a name-based rule would hide the intent in routing config.

Rollout and revert

Urgency is a static class attribute, so there is no feature flag. Rollout is a normal deploy. The revert is a single MR that removes the urgency :high line and regenerates app/workers/all_queues.yml. This should ship together with or after the merge train refresh worker change from #631893.

Verification

  • bundle exec rspec spec/workers/every_sidekiq_worker_spec.rb passes locally.
  • bundle exec rspec spec/workers/auto_merge_process_worker_spec.rb passes locally.
  • app/workers/all_queues.yml was regenerated, and the diff is limited to the AutoMergeProcessWorker entry changing from :urgency: :low to :urgency: :high.

MR checklist

  • Changelog trailer on the commit: Changelog: performance.
  • Documentation: no change needed.
  • Feature flag: none. Urgency is a static class attribute.
  • Tests: existing worker specs cover the change, no new spec needed.
  • Un-draft after the SLO measurement in #631913 confirms the worker fits.

Merge request reports