[FF] merge_request_refresh_batched_commit_lookup -- batch the push commit overlap lookup in MergeRequests::RefreshService

Summary

merge_request_refresh_batched_commit_lookup batches the commit-membership check that MergeRequests::RefreshService runs against open merge requests on a pushed branch.

Today, for every push, the service asks each open merge request individually whether its diff already contains any of the pushed commits (MergeRequestDiff#includes_any_commits?), re-serialising the full commit list into bind parameters each time. Cost scales with merge request count x push size. With mr_diff_commits_read_new_table off (production's current setting), each 1,000-commit chunk costs two queries per merge request - a metadata lookup plus a sha-column fallback.

With the flag on, the whole set is resolved in one batched query per 1,000-commit chunk via MergeRequestDiff.ids_including_any_commits, independent of merge request count. Same merge requests get selected, same diffs get reloaded, no commits are dropped.

This is corrective action for INC-12753 (gitlab-com/gl-infra/production#22645 (closed)), where large mirror pushes ran UpdateMergeRequestsWorker for 5-10 hours at cpu/wall ~0.93 and degraded the shared urgent-cpu-bound Sidekiq shard.

Measured on GDK with synthetic history (30,000-commit push, 200 open merge requests on the branch): CPU 83.1s -> 0.84s, queries 12,019 -> 79, allocations 128M -> 1.7M. Query count is flat in merge request count with the flag on, linear without it.

  • Actor type: project. Default: disabled. Type: gitlab_com_derisk. Milestone: 19.3.
  • Introduced by !248701 (merged) (merged 2026-08-19, deployed to production 2026-08-20 07:51 UTC - code is live, flag is off).
  • Feature issue: #608151 (closed).
  • DRI: @marc_shaw. Team Slack channel: #g_code_review.

What could go wrong?

The batched lookup decides exactly one thing: whether a merge request that the push only reaches through its target branch gets its diff reloaded. Everything else is untouched - a push to a merge request's own source branch reloads unconditionally, and a force push reloads every diff and skips the lookup entirely. Failure modes:

  • A merge request that should have had its diff reloaded does not. Result is a stale diff until the next push to that merge request's own source branch, which rebuilds it.
  • An extra merge request has its diff reloaded unnecessarily. Harmless, just wasted work.
  • Query errors or timeouts in the new batched statements, surfacing as UpdateMergeRequestsWorker failures.

Confounder: mr_diff_commits_read_new_table changes which branch of the batched lookup runs - the sha-column fallback union disappears when it is on. If its rollout percentage moves during this rollout, observed query counts shift for reasons unrelated to this flag. Check its current state before comparing before/after numbers.

Monitoring

  • Kibana baseline query (saved by the DRI): https://log.gprd.gitlab.net/app/r/s/5Yhng
  • Compare json.class: UpdateMergeRequestsWorker per project on json.db_count, json.cpu_s, and json.duration_s before and after each step.
  • urgent-cpu-bound Sidekiq shard apdex and CPU saturation: https://dashboards.gitlab.net
  • UpdateMergeRequestsWorker exception rate.

Rollout

  1. Non-production. Leave for 24 hours.

    /chatops run feature set merge_request_refresh_batched_commit_lookup true --dev --pre --staging --staging-ref

  2. Production, single low-risk actor first (marc_shaw/test), to verify correctness by hand rather than scale: the stacked case (a merge request targeting the pushed branch whose diff already contains pushed commits must still get its diff reloaded), a push to a merge request's own source branch, and a force push.

    /chatops run feature set --project=marc_shaw/test merge_request_refresh_batched_commit_lookup true

  3. Production, gitlab-org/gitlab. Every push to master iterates all open merge requests targeting master (roughly 2,800 today), so each merge train push currently costs several thousand queries and should drop to a handful. Continuous push traffic gives a clean before/after comparison with no synthetic load.

    /chatops run feature set --project=gitlab-org/gitlab merge_request_refresh_batched_commit_lookup true

  4. Production, the project identified in INC-12753. Verify UpdateMergeRequestsWorker durations for that project drop.

    /chatops run feature set --project=<project-from-incident> merge_request_refresh_batched_commit_lookup true

  5. Percentage of actors, then global. Wait at least 15 minutes between steps.

    /chatops run feature set merge_request_refresh_batched_commit_lookup 25 --actors
    /chatops run feature set merge_request_refresh_batched_commit_lookup 100 --actors
    /chatops run feature set merge_request_refresh_batched_commit_lookup true

Rollback: set the flag to false for the affected actor, or globally. No data migration and nothing to repair - the next push takes the old code path.

/chatops run feature set merge_request_refresh_batched_commit_lookup false

Cleanup

Once stable at 100%, remove the flag checks and the YAML definition, then delete the flag:

/chatops run feature delete merge_request_refresh_batched_commit_lookup --dev --pre --staging --staging-ref --production
Edited by Marc Shaw