Merged-manually detection misses are silent and permanent

Summary

MergeRequests::RefreshService#post_merge_manually_merged (running in UpdateMergeRequestsWorker) processes all candidate MRs of a push in one unprotected loop. If processing one MR raises, the whole worker fails and every remaining MR in the push is skipped. Because detection only ever inspects the commits of the push being processed, MRs missed this way are never detected again: their heads are no longer "new commits" in any later push. The MR stays open forever while its commits are on the target branch.

Confirmed production failure class from Kibana (gprd, 2026-08-17 09:18:22 UTC, jid 3b5c02955dc8d786aabe3039, correlation a2c7944f2a4a56cd-IAD):

ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
  ee/lib/analytics/merge_request_metrics_calculator.rb:31  `first_comment_at`
  ee/app/services/ee/merge_request_metrics_service.rb:14   `merge`
  app/services/merge_requests/post_merge_service.rb:117    `block in create_event`
  app/services/merge_requests/post_merge_service.rb:31     `execute`
  app/services/merge_requests/refresh_service.rb:148       `block in post_merge_manually_merged`

The EE productivity-metrics read (first_comment_at: a notes UNION query with commit_id IN (up to 100 shas)) hit the ~15s statement timeout on a project with a very large notes table, inside PostMergeService#create_event's Event.transaction, inside the detection loop.

Related

Production evidence

The affected workflow batch-merges open MRs' exact heads externally (git merge --no-ff of refs/merge-requests/N/head) and pushes once; detection normally marks the whole batch merged within a minute. Three pushes on the same large project, same workflow, three outcomes:

Push MRs in push Detected Notes
2026-08-22 11 11 healthy control: sequential merged_at ~1s apart starting 41s after the push, each merge_commit_sha = the pre-existing in-push integration merge
2026-08-17 11 10 (on retry) attempt 1 died on the first MR (Kibana log above) after mark_as_merged persisted but before event/metrics/note; the retry skipped it and salvaged the other 10; that MR is left half-merged
2026-07-10 10 0 all attempts failed; nothing was marked; 8 MRs were later merged by hand via UI/API and 2 remain open today with their heads reachable from the target branch

All detection preconditions (open state, matching target branch, exact head SHA inside the pushed range, non-empty diff, existing source project) were verified as satisfied for the stranded MRs. Full forensic evidence, including publicly verifiable push contents and the affected merge requests, is documented in the internal Request for Help: https://gitlab.com/gitlab-com/request-for-help/-/work_items/5263#note_3751075727

Additional data-integrity defect

mark_as_merged persists independently of the event/metrics transaction in PostMergeService, so a failure between them produces a merge request that is merged with merged_at: null, merged_by: null, and zero resource state events (no "merged" entry in the activity feed at all). A live specimen is referenced in the RFH above.

Proposed fixes

  1. CE: per-MR error isolation in the post_merge_manually_merged loop (refresh_service.rb:126-148): rescue per iteration, track_exception, continue. This alone would have saved 9 of the 10 MRs lost on 2026-07-10.
  2. EE: rescue ActiveRecord::QueryCanceled around productivity_data in EE::MergeRequestMetricsService#merge and proceed with base metrics. An analytics read must not roll back merge state. This alone would have prevented both incidents.
Edited by 🤖 GitLab Bot 🤖