Skip enqueuing a mergeability check while one is running
Reads that land while a mergeability check is running enqueue another MergeRequestMergeabilityCheckWorker job, which then does nothing. At peak about 28% of the worker's jobs are no-ops like this. This skips the enqueue when the merge status is checking or cannot_be_merged_rechecking and the check's exclusive lease is held. It's behind mergeability_check_skip_enqueue_while_running.
Behind the same flag, a check marks the merge request checking only once it holds the lease, and a check that discards its result as stale queues the next one. Together these make sure a skipped read never leaves the merge request without a check.
Detailed context for AI agents
Why
- Kibana, 2026-11-02 15:00-16:00 UTC: 225k
donejobs (62.5/s) for 117k distinct MRs, 28% of them no-ops (no transaction opened). Off-peak (03:00-04:00) 14.7 jobs/s, 11% no-ops. - The no-ops come from reads while a check is in flight.
MergeRequest#check_mergeabilityreturns early only unlessrecheck_merge_status?, andcheck_state?includeschecking/cannot_be_merged_rechecking, so every read during a check enqueues again. The worker isidempotent!with the defaultuntil_executingdedup, so a job enqueued while one executes is not dropped. - Biggest callers by share of jobs (share that were no-ops):
ConcurrencyLimit::ResumeWorker36% (16%), single MR API GET 30% (22%),MergeRequestsController#show12% (51%),NewMergeRequestWorker9% (14%),cached_widget6% (84%). - No-op jobs open no transaction, so this barely moves the worker's own idle-in-transaction time. The win is fewer jobs on catchall: less concurrency-limit deferral and less GVL contention, which shortens in-transaction round trips for every worker on the shard. That matters for the PgBouncer saturation the epic tracks.
How
MergeabilityCheckService#async_executereturns without enqueuing whencheck_running?is true: flag on (actor: project), status ischeckingorcannot_be_merged_rechecking, andGitlab::ExclusiveLease.get_uuid("mergeability_check:<id>")is present.- For that to mean "a running check will write the status",
checkinghas to belong to the lease holder. Before this, two paths markedcheckingbefore taking the lease:MergeRequestMergeabilityCheckWorker(at the start ofperform) andMergeabilityCheckBatchWorker(the whole batch up front, from the group MR list API). A job that then lost the lease left acheckingthatcheck_running?attributed to the running check (Patrick's !267363 (comment 4157570417)). - Behind the flag:
execute(mark_as_checking: true)marks the merge requestcheckingright after taking the lease, and only when the loaded record still awaits a check, so a job with nothing to write never marks it. Both workers pass it and skip their own early mark (MergeRequest#check_mergeabilitypasses it through for the batch worker). A job that loses the lease now leaves the merge requestunchecked, so reads keep enqueuing, the same as today.- Any uncheck after the holder marked
checkingresets it, socheckingplus the lease means nothing moved since the holder started, except in the window between a diff reload and the refresh's uncheck (below). - When the status write is discarded as stale (
STALE_MERGE_STATUS_DISCARDED),executequeuesMergeRequestMergeabilityCheckWorkerafter releasing the lease, so the new job can take it. Reads during the check may have been skipped, so the check queues their replacement itself. Discards need the source diff or target branch to change mid-check, so this is rare. - It queues only when a fresh check could write: the row still awaits a check (a final status there was already written for its current inputs, so nothing more is needed), and its
latest_merge_request_diff_idmatches the diff a check reads. A row where that doesn't match (for example an import that failed after saving the diff) discards every check, so requeuing would loop.
- Checking without the lease still enqueues: the job died (retries exhausted, killed), and today the next read's enqueue is what heals it.
- The read is a plain Redis
GET, only for MRs in a checking state.ExclusiveLease#try_obtain's inside-transaction check does not apply to reads. - Covers every
check_mergeability(async: true)caller: API entity and list endpoint,MergeRequestsController#show,cached_widget,AfterCreateService/UpdateService(these setuncheckedfirst, so they never skip), conflictsResolveService, and the EE dependency security update scheduler. Sync callers (execute) don't markchecking, so reads during them still enqueue. - Cost with the flag on: the batch worker marks per merge request it actually checks (2 small UPDATEs each, next to the ~65 queries of the check) instead of 2 UPDATEs for the whole batch.
Edge cases
Probed with a spec that runs a real check holding the lease and interleaves a push, another job and a read at merge_to_ref. "Stuck" means the merge request ends with no final status and no job queued:
| Case | 9e432fb2 (flag on) | This version |
|---|---|---|
Push mid-check, the worker loses the lease and marks checking, read, check discards |
stuck | unchecked, job queued |
| Same, with the batch worker losing the lease | stuck | unchecked, job queued |
Same, with a sync execute(recheck: true) holding the lease |
stuck | unchecked, job queued |
| Read between the refresh's diff reload and its uncheck, check discards | stuck | job queued |
Sync holder raises after another job marked checking |
stuck | unchecked, job queued |
With the flag off, every case has the read's job queued, as before.
- Window between taking the lease and marking
checking: the status is stillunchecked, so reads enqueue. - A check that runs longer than the 1 minute lease TTL: the lease expires and reads enqueue again, the same as today.
- A worker check that raises: Sidekiq retries it, the same as today.
Verification
Run in an rspec slot on 0041661b:
mergeability_check_service_spec.rb,merge_request_mergeability_check_worker_spec.rb,mergeability_check_batch_worker_spec.rb,mergeability_check_batch_service_spec.rb: 96 examples, 0 failures.merge_request_spec.rb#check_mergeability: 9 examples, 0 failures.- The 10 new examples fail with the app code from 9e432fb2: the three "when the source moves while a check runs" cases,
mark_as_checking, the requeue on discard, and the worker and batch worker lease cases. - The flag-off variants keep the old worker behaviour, including the batch worker's "2 queries regardless of MR count".
- A break matrix (72 cases, flag on and off, flag off compared with master's service and workers) passes. It found the requeue loop above, fixed in 2cf24396 with a spec that fails without it.
- The 2 examples added in f897e9fb (a job with nothing to write doesn't mark
checking, no requeue for a row that already has a final status) fail with the app code before it. With the break matrix and the race probes, the service, worker and batch worker specs run 182 examples, 0 failures. - Flag-off gating: a probe recorded every SQL statement, enqueued job and final status for six entry points (worker full check, worker losing the lease, async read on
unchecked, async read oncheckingwith the lease held, batch worker, sync check discarding after a retarget), on this branch and on master'sapp/. Both runs are identical apart from merge-commit SHAs. The only flag-off additions areFeature.enabled?lookups: one per worker job, one per project per batch, one per async read, and one per discarded check. The flag actor is built withProject.actor_from_id, so no lookup loads the project.
Rollout
gitlab_com_derisk, default off, no changelog. Rollout issue: #637935, with the Kibana dashboard https://log.gprd.gitlab.net/app/dashboards#/view/8e797f40-49d1-45f3-86d4-66525fb045f3.- What to watch:
MergeRequestMergeabilityCheckWorkerjob rate and no-op share (the epic dashboard),ConcurrencyLimit::ResumeWorkervolume for this worker, and that MRs don't stay inchecking(merge status age, widget reports).
Related
- !267328 (closed) (guarded merge status write, #637920) and #637921 (closed) (skip the merge-head diff rebuild) are in the same epic. Independent of this MR.
- Other options from the issue, not done here:
until_executeddedup, per-MR rate limit on the API path.