Skip enqueuing a mergeability check while one is running

Reads that land while a mergeability check is running enqueue another MergeRequestMergeabilityCheckWorker job, which then does nothing. At peak about 28% of the worker's jobs are no-ops like this. This skips the enqueue when the merge status is checking or cannot_be_merged_rechecking and the check's exclusive lease is held. It's behind mergeability_check_skip_enqueue_while_running.

Behind the same flag, a check marks the merge request checking only once it holds the lease, and a check that discards its result as stale queues the next one. Together these make sure a skipped read never leaves the merge request without a check.

For #637923. Part of &24375.

Detailed context for AI agents

Why

  • Kibana, 2026-11-02 15:00-16:00 UTC: 225k done jobs (62.5/s) for 117k distinct MRs, 28% of them no-ops (no transaction opened). Off-peak (03:00-04:00) 14.7 jobs/s, 11% no-ops.
  • The no-ops come from reads while a check is in flight. MergeRequest#check_mergeability returns early only unless recheck_merge_status?, and check_state? includes checking / cannot_be_merged_rechecking, so every read during a check enqueues again. The worker is idempotent! with the default until_executing dedup, so a job enqueued while one executes is not dropped.
  • Biggest callers by share of jobs (share that were no-ops): ConcurrencyLimit::ResumeWorker 36% (16%), single MR API GET 30% (22%), MergeRequestsController#show 12% (51%), NewMergeRequestWorker 9% (14%), cached_widget 6% (84%).
  • No-op jobs open no transaction, so this barely moves the worker's own idle-in-transaction time. The win is fewer jobs on catchall: less concurrency-limit deferral and less GVL contention, which shortens in-transaction round trips for every worker on the shard. That matters for the PgBouncer saturation the epic tracks.

How

  • MergeabilityCheckService#async_execute returns without enqueuing when check_running? is true: flag on (actor: project), status is checking or cannot_be_merged_rechecking, and Gitlab::ExclusiveLease.get_uuid("mergeability_check:<id>") is present.
  • For that to mean "a running check will write the status", checking has to belong to the lease holder. Before this, two paths marked checking before taking the lease: MergeRequestMergeabilityCheckWorker (at the start of perform) and MergeabilityCheckBatchWorker (the whole batch up front, from the group MR list API). A job that then lost the lease left a checking that check_running? attributed to the running check (Patrick's !267363 (comment 4157570417)).
  • Behind the flag:
    • execute(mark_as_checking: true) marks the merge request checking right after taking the lease, and only when the loaded record still awaits a check, so a job with nothing to write never marks it. Both workers pass it and skip their own early mark (MergeRequest#check_mergeability passes it through for the batch worker). A job that loses the lease now leaves the merge request unchecked, so reads keep enqueuing, the same as today.
    • Any uncheck after the holder marked checking resets it, so checking plus the lease means nothing moved since the holder started, except in the window between a diff reload and the refresh's uncheck (below).
    • When the status write is discarded as stale (STALE_MERGE_STATUS_DISCARDED), execute queues MergeRequestMergeabilityCheckWorker after releasing the lease, so the new job can take it. Reads during the check may have been skipped, so the check queues their replacement itself. Discards need the source diff or target branch to change mid-check, so this is rare.
    • It queues only when a fresh check could write: the row still awaits a check (a final status there was already written for its current inputs, so nothing more is needed), and its latest_merge_request_diff_id matches the diff a check reads. A row where that doesn't match (for example an import that failed after saving the diff) discards every check, so requeuing would loop.
  • Checking without the lease still enqueues: the job died (retries exhausted, killed), and today the next read's enqueue is what heals it.
  • The read is a plain Redis GET, only for MRs in a checking state. ExclusiveLease#try_obtain's inside-transaction check does not apply to reads.
  • Covers every check_mergeability(async: true) caller: API entity and list endpoint, MergeRequestsController#show, cached_widget, AfterCreateService / UpdateService (these set unchecked first, so they never skip), conflicts ResolveService, and the EE dependency security update scheduler. Sync callers (execute) don't mark checking, so reads during them still enqueue.
  • Cost with the flag on: the batch worker marks per merge request it actually checks (2 small UPDATEs each, next to the ~65 queries of the check) instead of 2 UPDATEs for the whole batch.

Edge cases

Probed with a spec that runs a real check holding the lease and interleaves a push, another job and a read at merge_to_ref. "Stuck" means the merge request ends with no final status and no job queued:

Case 9e432fb2 (flag on) This version
Push mid-check, the worker loses the lease and marks checking, read, check discards stuck unchecked, job queued
Same, with the batch worker losing the lease stuck unchecked, job queued
Same, with a sync execute(recheck: true) holding the lease stuck unchecked, job queued
Read between the refresh's diff reload and its uncheck, check discards stuck job queued
Sync holder raises after another job marked checking stuck unchecked, job queued

With the flag off, every case has the read's job queued, as before.

  • Window between taking the lease and marking checking: the status is still unchecked, so reads enqueue.
  • A check that runs longer than the 1 minute lease TTL: the lease expires and reads enqueue again, the same as today.
  • A worker check that raises: Sidekiq retries it, the same as today.

Verification

Run in an rspec slot on 0041661b:

  • mergeability_check_service_spec.rb, merge_request_mergeability_check_worker_spec.rb, mergeability_check_batch_worker_spec.rb, mergeability_check_batch_service_spec.rb: 96 examples, 0 failures. merge_request_spec.rb #check_mergeability: 9 examples, 0 failures.
  • The 10 new examples fail with the app code from 9e432fb2: the three "when the source moves while a check runs" cases, mark_as_checking, the requeue on discard, and the worker and batch worker lease cases.
  • The flag-off variants keep the old worker behaviour, including the batch worker's "2 queries regardless of MR count".
  • A break matrix (72 cases, flag on and off, flag off compared with master's service and workers) passes. It found the requeue loop above, fixed in 2cf24396 with a spec that fails without it.
  • The 2 examples added in f897e9fb (a job with nothing to write doesn't mark checking, no requeue for a row that already has a final status) fail with the app code before it. With the break matrix and the race probes, the service, worker and batch worker specs run 182 examples, 0 failures.
  • Flag-off gating: a probe recorded every SQL statement, enqueued job and final status for six entry points (worker full check, worker losing the lease, async read on unchecked, async read on checking with the lease held, batch worker, sync check discarding after a retarget), on this branch and on master's app/. Both runs are identical apart from merge-commit SHAs. The only flag-off additions are Feature.enabled? lookups: one per worker job, one per project per batch, one per async read, and one per discarded check. The flag actor is built with Project.actor_from_id, so no lookup loads the project.

Rollout

  • !267328 (closed) (guarded merge status write, #637920) and #637921 (closed) (skip the merge-head diff rebuild) are in the same epic. Independent of this MR.
  • Other options from the issue, not done here: until_executed dedup, per-MR rate limit on the API path.
Edited by Marc Shaw

Merge request reports

Loading
Loading