Skip AutoMergeProcessWorker for pipelines without auto-merge MRs

Marc Shawrequested to merge
631600-pipeline-auto-merge-process-guard into master
  • Every completed pipeline enqueued AutoMergeProcessWorker, even when no merge request has auto-merge enabled. On GitLab.com that is roughly two thirds of the worker's jobs, and most of them likely find nothing to do.
  • Behind the auto_merge_process_pipeline_guard feature flag, the transition now checks for an open merge request with auto-merge enabled first, with one indexed query on the primary, and enqueues only if one exists.
  • Flag off: behaviour is unchanged.
  • The new query runs once per completed pipeline (about 40/s on the main primary). Plans from the gitlab-production-main clone:

Branch pipeline, branch with an open merge request

SELECT 1 AS one FROM merge_requests
WHERE source_project_id = 278964
  AND source_branch = '631600-pipeline-auto-merge-process-guard'
  AND state_id = 1
  AND merge_when_pipeline_succeeds = TRUE
LIMIT 1
Limit  (cost=0.56..3.58 rows=1 width=4) (actual time=3.011..3.012 rows=0 loops=1)
  Buffers: shared hit=3 read=4
  ->  Index Scan using idx_merge_requests_on_source_project_and_branch_state_opened on public.merge_requests  (cost=0.56..3.58 rows=1 width=4) (actual time=3.010..3.011 rows=0 loops=1)
        Index Cond: ((merge_requests.source_project_id = 278964) AND ((merge_requests.source_branch)::text = '631600-pipeline-auto-merge-process-guard'::text))
        Filter: merge_requests.merge_when_pipeline_succeeds
        Buffers: shared hit=3 read=4

Branch pipeline, branch with many open merge requests (master)

SELECT 1 AS one FROM merge_requests
WHERE source_project_id = 278964
  AND source_branch = 'master'
  AND state_id = 1
  AND merge_when_pipeline_succeeds = TRUE
LIMIT 1
Limit  (cost=0.56..12.81 rows=1 width=4) (actual time=0.757..0.758 rows=0 loops=1)
  Buffers: shared hit=6 read=1
  ->  Index Scan using idx_merge_requests_on_source_project_and_branch_state_opened on public.merge_requests  (cost=0.56..135.29 rows=11 width=4) (actual time=0.756..0.756 rows=0 loops=1)
        Index Cond: ((merge_requests.source_project_id = 278964) AND ((merge_requests.source_branch)::text = 'master'::text))
        Filter: merge_requests.merge_when_pipeline_succeeds
        Buffers: shared hit=6 read=1

Merge request pipeline

SELECT 1 AS one FROM merge_requests
WHERE id = 545188689
  AND state_id = 1
  AND merge_when_pipeline_succeeds = TRUE
LIMIT 1
Limit  (cost=0.56..3.58 rows=1 width=4) (actual time=2.605..2.605 rows=0 loops=1)
  Buffers: shared hit=3 read=4
  ->  Index Scan using idx_merge_requests_on_unmerged_state_id on public.merge_requests  (cost=0.56..3.58 rows=1 width=4) (actual time=2.604..2.604 rows=0 loops=1)
        Index Cond: (merge_requests.id = 545188689)
        Filter: (merge_requests.merge_when_pipeline_succeeds AND (merge_requests.state_id = 1))
        Buffers: shared hit=3 read=4
Detailed context for AI agents

What does this MR do and why?

Change in app/models/ci/pipeline.rb:

  • The after_transition any => completed_statuses callback (around app/models/ci/pipeline.rb:385) enqueues AutoMergeProcessWorker with { 'pipeline_id' => id } for every completed pipeline. It is the only one of the worker's 7 enqueue sites that does not first check that a merge request has auto-merge enabled. The worker then loads the pipeline, resolves pipeline.all_merge_requests.with_auto_merge_enabled, and for most pipelines finds nothing.
  • New behaviour, behind the gitlab_com_derisk feature flag auto_merge_process_pipeline_guard (actor: project, default off): enqueue only if an open merge request with auto-merge enabled exists for the pipeline.
    • Merge request pipelines: MergeRequest.where(id: merge_request_id).
    • Branch pipelines: MergeRequest.where(source_project_id: project_id, source_branch: ref).
    • Both are scoped with with_auto_merge_enabled and checked with exists?. One query.
  • The check deliberately skips the sha filter that all_merge_requests applies (by_commit_sha, an EXISTS join over merge_request_diffs and merge_request_diff_commits). It is a superset: it may still enqueue for a pipeline of an older commit on a branch that has an auto-merge MR, and the worker narrows it down as before. Skipping the sha keeps it an index lookup.
  • The check reads the main primary via ::Gitlab::Database::LoadBalancing::SessionMap.current(MergeRequest.load_balancer).use_primary.
    • Load balancing stickiness is per database. The pipeline transition writes ci_pipelines (ci database), so only the ci session is sticky to the primary; the main session (merge_requests) is not. Verified on GDK: inside the after_commit of a ci_pipelines write, main session use_primary? = false, ci session use_primary? = true.
    • PipelineProcessWorker is data_consistency :always, overrides: { main: :sticky }, so it reads main from replicas.
    • If auto-merge is enabled within replica lag before the pipeline finishes, a replica read could miss it and skip the enqueue. The run that enabling auto-merge triggers (app/services/auto_merge/base_service.rb:16) already saw CI still running, so the MR would be stuck. Today the job runs later, after queueing, which gives replicas time to catch up.
    • On single-database installs this block is a no-op (main is already primary there).
  • Precedent for checking before enqueueing: the after_transition any => :can_be_merged callback in app/models/merge_request.rb (around line 303-309) does MergeRequest.where(id: merge_request.id, auto_merge_enabled: true).exists? before enqueueing AutoMergeProcessWorker.
  • No coverage lost: enabling auto-merge always enqueues its own run (app/services/auto_merge/base_service.rb:16), so skipping a pipeline whose MRs have no auto-merge yet misses nothing. Flag off: behaviour unchanged, the only extra work is the Feature.disabled? check.
  • History: !167095 (merged) (17.6, infradev) moved all_merge_requests out of this transition into the worker, because that query (with the sha EXISTS join) from PipelineProcessWorker was 5.71% of main primary CPU, and accepted more enqueued jobs in exchange. This MR does not bring that query back: the new check has no sha join and uses the partial index idx_merge_requests_on_source_project_and_branch_state_opened (source_project_id, source_branch WHERE state_id = 1) for branch pipelines, or the primary key for merge request pipelines.

Why it matters (production numbers):

  • GitLab.com created about 4.16M pipelines in 24 hours (global pipeline id went from 2922184019 at 2026-10-07 11:58 UTC to 2926347681 at 2026-10-08 11:59 UTC). AutoMergeProcessWorker runs about 6M jobs/day (40-77 jobs/s, Mimir). So pipeline completion is roughly two thirds of the worker's jobs.
  • The share of those that find no auto-merge MR is an estimate, not measured. It can be confirmed in Kibana by filtering AutoMergeProcessWorker on json.meta.caller_id: PipelineProcessWorker and json.db_count <= 6.
  • Measured on GDK through the real Sidekiq client and server middleware chains: a pipeline-completion job that finds no auto-merge MR costs 5 SQL (3 for a merge request pipeline) and 7 Redis calls server-side, plus 3 Redis calls client-side and the push itself - about 16 round trips for a job that does nothing.
  • On GitLab.com 30-70% of this worker's job time in the sampled pods is GVL wait on the low-urgency-cpu-bound shard, which makes every round trip slow (discussion: !259376 (comment 3971487486)).
  • For a pipeline without an auto-merge MR, the enqueuing side swaps 3 client-side Redis calls plus the push for 1 indexed SQL query.

Query

Plans and costs are in the summary above. merge_when_pipeline_succeeds is in neither index, so the query does one heap fetch per open merge request on the branch. auto_merge_enabled is an alias for that column. The execution time on the clone is nearly all cold reads; on the primary these are recent, hot pages.

Verification

  • bundle exec rspec spec/models/ci/pipeline_spec.rb:3106 (the 'auto merge' block): 12 examples, 0 failures. Covers:
    • branch pipeline with an auto-merge MR enqueues on succeed!, drop!, cancel!, skip!
    • the check uses the primary
    • a pipeline for another commit of the branch still enqueues (superset)
    • closed MR does not enqueue
    • merge request pipeline with auto-merge enqueues, without does not
    • branch pipeline whose MR has no auto-merge does not enqueue, but does with the flag disabled
    • pipeline with no MR does not enqueue
  • GDK end to end (real pipeline.succeed! transition, jobs captured rather than pushed):
Case Flag off Flag on
Branch pipeline, no MR enqueued not enqueued
Branch pipeline, MR without auto-merge enqueued not enqueued
Branch pipeline, MR with auto-merge enqueued enqueued
Merge request pipeline, MR with auto-merge enqueued enqueued
Merge request pipeline, MR without auto-merge enqueued not enqueued

With the flag on, each case ran exactly one guard query.

Out of scope

  • Skipping child pipelines entirely (they can never be a merge request's head pipeline).
  • The inert deduplicate declaration on AutoMergeProcessWorker, which costs Redis calls on every job without deduplicating anything (the worker is not idempotent).
  • The same transition still runs all_merge_requests with the sha join in other callbacks (for example app/models/ci/pipeline.rb around line 504 and Ci::ExpirePipelineCacheService); not touched here.
  • The concurrent check-order change is !260427 (merged) (fewer round trips per run that does reach an MR); this MR removes whole runs.

References

Screenshots or screen recordings

Not applicable, backend only.

Edited by Marc Shaw

Merge request reports