## Summary When a Sidekiq pod shuts down while `MergeTrains::RefreshWorker` is mid-merge, `Sidekiq::Shutdown` is raised in the thread. Because `RefreshMergeRequestService#execute` rescues `StandardError` (which catches `Sidekiq::Shutdown`), it calls `abort` — destroying the merge train car — and then swallows the exception. The job is re-queued by Sidekiq, but the re-queued run finds no merge train car and does nothing. The git merge commit already exists in the repository but the MR is never marked as merged. Observed in the wild: gitlab-runner MR gitlab-runner!6682, merge commit `16f5b764`. ## Log evidence Application log (Kibana): https://log.gprd.gitlab.net/app/r/s/rrRg4 The key sequence from `application_json` logs, all with correlation ID `09f9edea7c844f9286fa48b448582efb`: ``` json.time,json.message,json.merge_request_info 2026-05-01T16:41:12,gitlab-org/gitlab-runner!6682 - Git merge started on JID 2026-05-01T16:41:19,gitlab-org/gitlab-runner!6682 - Git merge finished on JID commit 16f5b76429b84915e9a710d2b3a76ca6d7bc6a2d <-- NOTHING after this for !6682 --> 2026-05-01T16:42:37,gitlab-org/gitlab-runner!6693 - Git merge started on JID 2026-05-01T16:42:41,gitlab-org/gitlab-runner!6693 - Git merge finished on JID commit e1b300e8... 2026-05-01T16:42:46,gitlab-org/gitlab-runner!6693 - Merge request marked in progress 2026-05-01T16:42:46,gitlab-org/gitlab-runner!6693 - Post merge started on JID with state locked 2026-05-01T16:43:38,gitlab-org/gitlab-runner!6693 - Post merge finished on JID with state merged 2026-05-01T16:43:58,gitlab-org/gitlab-runner!6659 - Git merge started on JID 2026-05-01T16:44:00,gitlab-org/gitlab-runner!6659 - Git merge finished on JID commit a8c9c631... 2026-05-01T16:44:01,gitlab-org/gitlab-runner!6659 - Merge request marked in progress 2026-05-01T16:44:01,gitlab-org/gitlab-runner!6659 - Post merge started on JID with state locked 2026-05-01T16:44:11,gitlab-org/gitlab-runner!6659 - Post merge finished on JID with state merged ``` For gitlab-runner!6682, "Git merge finished" is logged but nothing follows — not even "Merge request marked in progress". For gitlab-runner!6693 and gitlab-runner!6659, the normal post-merge sequence completes. ### Sidekiq shutdown log From https://log.gprd.gitlab.net/app/r/s/7kpyq, the Sidekiq pod running this job was shutting down: - `2026-05-01T16:40:56` — `Shutting down` - `2026-05-01T16:41:21` — `Terminating 1 busy threads` - `2026-05-01T16:41:21` — `Pushed job d002c3a448089afa9cf0f26a back to queue queue:low_urgency_cpu_bound` (MergeTrains::RefreshWorker) - `2026-05-01T16:41:22` — `Requesting keep-around reference` (MergeTrains::RefreshWorker) The Sidekiq shutdown grace period (25s from `16:40:56` to `16:41:21`) expired while the thread was inside the git merge for gitlab-runner!6682. Sidekiq raised `Sidekiq::Shutdown` in the thread, pushed the job back to the queue, and then the re-queued job didn't re-merge. ## Root cause ### `Sidekiq::Shutdown` swallowed by `rescue StandardError` `Sidekiq::Shutdown` inherits from `RuntimeError < StandardError`. `RefreshMergeRequestService#execute` has: ```ruby rescue StandardError => e Gitlab::ErrorTracking.track_exception(e, ...) abort(ProcessError.new("unexpected error occurred - correlation id: #{...}")) end ``` When `Sidekiq::Shutdown` is raised mid-merge: 1. It propagates through `MergeService#execute` (not caught by `rescue MergeError`) 2. `in_locked_state`'s `ensure` runs: `unlock_mr if locked?` → MR transitions `locked → opened` 3. `after_transition any => :opened` fires → `UpdateHeadPipelineForMergeRequestWorker` enqueued at 16:41:22 ✓ 4. `rescue StandardError` in `RefreshMergeRequestService#execute` catches `Sidekiq::Shutdown` 5. `abort(ProcessError.new(...))` is called → **merge train car is destroyed** 6. `Sidekiq::Shutdown` is **swallowed** — the job "completes" normally from Ruby's perspective 7. But Sidekiq had already re-queued the job (step from the log above) 8. The re-queued job runs, finds no merge train car (destroyed in step 5), and does nothing The git merge commit (`16f5b764`) already exists in the repository — the ff-merge to the target branch succeeded at 16:41:19. But `after_merge` was never called, so `PostMergeService` / `mark_as_merged` never ran. ## Proposed fix ### Re-raise `Sidekiq::Shutdown` before the `rescue StandardError` handler `ee/app/services/merge_trains/refresh_merge_request_service.rb`: ```ruby rescue ProcessError => e abort(e) rescue Sidekiq::Shutdown raise rescue StandardError => e Gitlab::ErrorTracking.track_exception( e, merge_request_id: merge_request.id, merge_request_iid: merge_request.iid, project_id: merge_request.target_project_id ) abort( ProcessError.new( "unexpected error occurred - correlation id: #{Labkit::Correlation::CorrelationId.current_or_new_id}" ) ) end ``` With this fix, when `Sidekiq::Shutdown` is raised: - The merge train car is **not** destroyed - `Sidekiq::Shutdown` propagates to `Processor#process`, which pushes the job back to the queue - The re-queued job runs, finds the merge train car intact, and can re-attempt the post-merge steps The re-queued job would also need to handle the case where the git merge already succeeded (the ff-merge is idempotent for `FromTrainRef` since the train ref commit is already on the target branch, but the strategy's `outdated_source_sha?` check may need to account for this). ### Additional consideration: extend Sidekiq shutdown timeout for merge train workers The merge operation for this MR took ~7 seconds (16:41:12 to 16:41:19). The Sidekiq grace period was already set to 25s. If the job had started the merge later in the grace period window, there would be no time for post-merge steps. Consider increasing the timeout or marking merge train workers as not interruptible during merge. ## Impact - MR appears permanently open despite having a corresponding git merge commit in the repository - Merge train car appears in the "merged" tab but the MR state is inconsistent - Repository history has the merge commit but GitLab's MR tracking does not reflect it - The error is only visible in Sentry (not in Kibana application logs), making it hard to diagnose ## Raw log data (application_json, correlation ID 09f9edea7c844f9286fa48b448582efb) Kibana link: https://log.gprd.gitlab.net/app/r/s/rrRg4 ```csv "json.time","json.correlation_id","json.message","json.meta.caller_id","json.meta.root_caller_id","json.subcomponent","json.jid","json.merge_request_info" "2026-05-01T16:44:11.031",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6659 - Post merge finished on JID with state merged","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6659" "2026-05-01T16:44:01.754",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6659 - Post merge started on JID with state locked","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6659" "2026-05-01T16:44:01.749",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6659 - Merge request marked in progress","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6659" "2026-05-01T16:44:00.611",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6659 - Git merge finished on JID commit a8c9c6310bc05a86cc57000d9520ee86a0e1316e","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6659" "2026-05-01T16:43:58.093",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6659 - Git merge started on JID ","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6659" "2026-05-01T16:43:38.432",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6693 - Post merge finished on JID with state merged","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6693" "2026-05-01T16:42:46.686",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6693 - Merge request marked in progress","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6693" "2026-05-01T16:42:46.686",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6693 - Post merge started on JID with state locked","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6693" "2026-05-01T16:42:41.773",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6693 - Git merge finished on JID commit e1b300e8d8451700ecd366085ec555a735a8ef3a","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6693" "2026-05-01T16:42:37.478",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6693 - Git merge started on JID ","MergeTrains::RefreshWorker","POST /api/:version/internal/post_receive","application_json","-","gitlab-org/gitlab-runner!6693" "2026-05-01T16:41:19.270",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6682 - Git merge finished on JID commit 16f5b76429b84915e9a710d2b3a76ca6d7bc6a2d","MergeTrains::RefreshWorker","PUT /api/:version/jobs/:id","application_json","-","gitlab-org/gitlab-runner!6682" "2026-05-01T16:41:12.718",09f9edea7c844f9286fa48b448582efb,"gitlab-org/gitlab-runner!6682 - Git merge started on JID ","MergeTrains::RefreshWorker","PUT /api/:version/jobs/:id","application_json","-","gitlab-org/gitlab-runner!6682" ``` ## Relevant code - `ee/app/services/merge_trains/refresh_merge_request_service.rb:26-38` — the `rescue StandardError` that swallows `Sidekiq::Shutdown` - `app/services/merge_requests/merge_service.rb` — `commit` method and `in_locked_state` - `app/models/merge_request.rb:2134-2148` — `in_locked_state` and `unlock_mr` ensure