- Deep dive into `AutoMergeProcessWorker` (transaction length and general speed), done on 2026-09-30 as a follow-up to #628868. Traces, scripts and reports are attached (see Pick-up notes). - Transactions are short: none holds a Gitaly call, an HTTP call or a per-approver loop. The only plausible >10 min path is a wait on the `merge_requests` row lock behind someone else's long transaction. - A successful run is Gitaly-bound (24 RPCs, 77% of wall on GDK). Three of its four Gitaly-heavy steps are redundant or deferrable. - Most runs are no-ops: an estimated 11 runs for a typical `merge_when_checks_pass` MR, and 1 can merge. The dedup is inert and the worker re-enqueues itself. - First fixes (small, low risk, measurable): B1 skip `can_git_merge?`, B3 reuse the merge ref commit id, C1 drop the self re-enqueue, A2 move side effects out of transactions, B5 and B7 cut needless check passes (IDs: see "What to fix or improve"). - Absolute timings are GDK scale. The shares and counts are what transfer to production. <details><summary>Context and production signal</summary> ### Why this was done - Work item #628868 ("MR workers executing long running transactions"); the assignee of this issue is its DRI. `MergeWorker` (!255421, !255430, !255631) and `UpdateMergeRequestsWorker` (!251113, !253951) already got this treatment. - The ask for `AutoMergeProcessWorker`: find long transactions and general speed improvements, with a full example, tracing and timing, and a few ideas. - Infra is rolling out a 30 minute `transaction_timeout` (dbo-issue-tracker#790, production-engineering#25884). Connections that exceed it are terminated (FATAL 25P04). ### Production signal | Source | Signal | |---|---| | Epic 21116 (rescue `low_urgency_cpu_bound`) | `AutoMergeProcessWorker` had 61,796 jobs over 10 s in Aug 17-23 2026. That is 6.1% of the >10 s bucket and the third largest contributor. | | #592298 (open, no one from Code Review assigned) | Earlier sample: 9,583 jobs >10 s and 11,367 jobs >30 s, as quoted in that issue (not re-derived here). | | Production alerts | None found for `AutoMergeProcessWorker` long transactions. Searched gitlab-org/gitlab, gl-infra/production and production-engineering. The sibling `MergeTrains::RefreshWorker` has one (#627653). | | Related open issues | #602096 (concurrency safety of `AutoMergeService#process`, which assumes dedup works), #299306 (drop `worker_resource_boundary :cpu`), #592298. | ### gitlab-org/gitlab auto-merge MR counts Taken from the public API on 2026-09-30. | Count | Value | |---|---| | Open MRs targeting `master` | 1,705 | | Of those, with auto-merge enabled | 52 | | Breakdown of the 52 | 40 `unchecked`, 7 `discussions_not_resolved`, 2 `conflict`, 2 `ci_still_running`, 1 `ci_must_pass` | | Merges to `master` | 161-219 per day | These counts feed the fan-out estimate in Findings (c). </details> <details><summary>Method</summary> ### Abbreviations | Short form | Meaning | |---|---| | MWCP | `merge_when_checks_pass` auto-merge strategy | | AMTWCP | `add_to_merge_train_when_checks_pass` auto-merge strategy | | AMPW | `AutoMergeProcessWorker` | | MergeW | `MergeWorker` | | PAMFEW | `ProcessAutoMergeFromEventWorker` | | car | a merge train car (a row in `merge_trains`) | | RPC | Gitaly RPC | | txn | database transaction | ### Inline tracer (`am_trace.rb`) It runs `AutoMergeProcessWorker.new.perform(params)` inside `rails runner` on GDK. The call is wrapped in `Rails.application.executor.wrap` and `Gitlab::SafeRequestStore.ensure_request_store`, like a Sidekiq job. | Recorded | How | |---|---| | SQL statements | `sql.active_record` start and finish events. Cached queries are counted separately. | | Postgres transactions | Per connection, `BEGIN..COMMIT/ROLLBACK`, with the statements inside. Idle time is wall time minus statement time. Also any Gitaly, Redis, HTTP or enqueue that happened while the transaction was open. | | Gitaly RPCs | Prepend on `Gitlab::GitalyClient::Call#call`. Streamed responses are measured on consumption. | | Redis commands | Prepend on `RedisClient::Middlewares`. | | External HTTP | `request.external_http`. | | Sidekiq enqueues | Prepend on `Sidekiq::Client#push` and `push_bulk`. Jobs are captured, not pushed, so GDK's real Sidekiq never runs `MergeWorker` and similar jobs, and scenarios can be re-run. | | Phases | A phase stack (prepended markers on the worker, `AutoMergeService`, the strategies, `mergeable?`, each mergeability check, the `MergeabilityCheckService` steps and `merge_async`) attributes every event to a phase and prints an inclusive tree. | ### Scenario harness (`am_scenario.rb`) - Creates the branch and commits through Gitaly before the MR exists, so `PostReceive` refreshes nothing. - Creates the MR with `MergeRequests::CreateService`. - Sets the auto-merge columns directly (strategy, sha, merge user). - Creates or forces a head pipeline status. Branch pushes make GDK Sidekiq create pending pipelines, because gitlab-test has a `.gitlab-ci.yml`. - Resets the MR to `unchecked`, deletes `refs/merge-requests/:iid/merge` and destroys the `merge_head_diff`. This is the state `RefreshService` leaves after a target push. - Approves the MR. Adds approval rules, users and members, and an unresolved discussion. Enables merge trains and puts the MR on a train. - A warm-up run always comes first (development mode autoload), then the measured runs. ### GDK environment and caveats - Ultimate licence, praefect-fronted Gitaly, Postgres main and ci decomposed, external diffs disabled, no replicas. - Absolute numbers are GDK scale. The shares and counts are what transfer. - MWCP numbers are warm: the minimum of 2-3 runs. GDK Sidekiq was saturated by unrelated jobs, so wall time is noisy upwards. SQL and RPC counts were identical across repeats. - Merge train and EE numbers are second runs. Host load was 10-20 (another workstream was building 600-approver scenarios), so wall times are noisy. Counts are exact. - The baseline trace is run 1 (cold). The MWCP table has warm numbers for the baseline MR in that set. ### Four agents and their scope | Scope | Report | Source | |---|---|---| | Static audit of transactions and the call graph | `agents/static_audit/report.md` | Source code | | Trigger frequency and redundancy study | `agents/frequency/report.md` | GDK `sidekiq.log` analysis and the gitlab-org/gitlab public API | | MWCP dynamic scenarios | `agents/mwcp/report.md`, `agents/mwcp/tables.md` | Tracer and harness on GDK | | Merge train and complex EE scenarios, including specs | `agents/trains/report.md` | Tracer and harness on GDK, and RSpec for security policies | Per-scenario traces are in `out/*.txt|json`. ### Baseline scenario Project 2 `gitlab-org/gitlab-test` (330 commits), MR !3324 (id 5572): 3 commits x 2 files, MWCP, approved (the project has an any-approver rule), success head pipeline, `merge_status` `unchecked`, merge ref missing. ### Projects used | Project | Used for | |---|---| | 2 `gitlab-org/gitlab-test` | Baseline MR. MWCP states, early exits, approvals, diff discussions and fan-out scenarios (baseline MR in the MWCP set: 4 commits/6 files, any-approver rule). | | 4 (Wget2) | MWCP scale MRs. | | 7 `flightjs/Flight` (500 commits) | MR !8. Merge train, abort, cancel, status check and lock scenarios. | | 5 `Commit451/lab-coat` (988 commits) | MRs !9 (code owners) and !10 (group approvers). | </details> <details><summary>Results</summary> ### Baseline phase breakdown MWCP, `unchecked`, no merge ref. Run 1, cold GDK (`baseline_mwcp_unchecked_run1.txt`). - Wall: 3,238 ms. - SQL: 76 statements, 227 ms (13 writes, 7 cached). - Gitaly: 24 RPCs, 2,881 ms. - Redis: 17 calls. - Transactions: 2 (11 ms and 235 ms). - Enqueued: `AutoMergeProcessWorker` (itself, from the `after_transition => :can_be_merged` callback) and `MergeWorker`. | Phase | Time | Detail | |---|---|---| | `mergeable_state?` (16 checks, `use_cache` false) | 42 ms | 24 SQL, 0 Gitaly. `CheckApprovedService` is the largest at 17 ms / 11 SQL. | | `check_mergeability` (sync `MergeabilityCheckService`) | 3,150 ms (97%) | Sub-phases in the next table. | | `mergeable_git_state?` | 37 ms | `CheckConflictStatusService` does a `FindCommit` for `source_branch_sha` (30 ms). | | `merge_async` | 4 ms | `MergeWorker` enqueue captured, `update_merge_jid`, etag expiry. | Inside `check_mergeability`: | Sub-phase | Time | Detail | |---|---|---| | `merge_to_ref` | 810 ms | `UserMergeToRef` 795 ms plus `FindCommit`. | | `update_diff_discussion_positions!` | 7 ms | No notes. | | `can_git_merge?` | 758 ms | `FindBranch` 358 ms and `ListConflictFiles` 398 ms. Redundant, see Findings (d). | | `write_merge_status` (TXN 1) | 24 ms | Transaction 11 ms, 3 statements, 8 ms idle. Enqueues the self re-enqueue after commit. | | `reload_merge_head_diff` | 1,539 ms | 17 Gitaly RPCs and 33 SQL, including TXN 2 (235 ms, 18 statements, 39 ms idle, no Gitaly or HTTP inside). | | payload | 12 ms | Another `FindCommit`, for `merge_ref_head`. | Detail for the two transactions: - TXN 1 statements: `SELECT ... FOR UPDATE` on `merge_requests`, the assignee COUNT validation, `UPDATE` of `merge_status`. - `reload_merge_head_diff` Gitaly RPCs include 4x `FindCommit`, `ListCommits`, `FindChangedPaths`, `GetFileAttributes`, `CheckBlobsGenerated`, `FindMergeBase`, `CommitDiff`, `GetPatchID` (296 ms) and `GetBlobs`. - TXN 2 statements: `INSERT` into `merge_request_diffs`, `merge_request_commits_metadata` lookup (107 ms), `INSERT` into `merge_request_diff_commits` (82 ms), `INSERT` into `merge_request_diff_files`, count and `UPDATE` statements. Second run on the same MR (now `can_be_merged`, which is what the self re-enqueue does): 64 ms, 29 SQL, 1 Gitaly RPC. It enqueues `MergeWorker` again. In production `MergeWorker`'s own dedup drops it, but `merge_jid` is overwritten with nil. `pipeline_id` form: the same work plus loading the pipeline (`Ci::Pipeline.id_in` without `partition_id`: 251 ms cold on GDK), the fork network and the MR lookup. Cold baseline next to the warm run of the baseline MR in the MWCP set: | Item | Baseline run 1 (cold) | Warm (MWCP set) | |---|---|---| | Wall | 3,238 ms | 640 ms | | SQL | 76 statements, 227 ms (13 writes, 7 cached) | 76 statements (13 writes, 18 ms) | | Gitaly | 24 RPCs, 2,881 ms | 24 RPCs, 490 ms (77% of wall) | | `mergeable_state?` | 42 ms, 24 SQL | 34 ms, 24 SQL (`CheckApprovedService` 13 ms / 11 SQL) | | `merge_to_ref` | 810 ms | 247 ms (`UserMergeToRef` 241 ms) | | `update_diff_discussion_positions!` | 7 ms | 2 ms | | `can_git_merge?` | 758 ms | 72 ms, 3 RPCs | | `write_merge_status` | phase 24 ms, transaction 11 ms | phase 47 ms (includes `run_after_commit` and the self-enqueue), transaction 7 ms, 3 statements | | `reload_merge_head_diff` | 1,539 ms | 214 ms, 33 SQL, 17 RPCs (transaction 20 ms, 18 statements, 16 ms idle) | | payload `FindCommit` | 12 ms | 3-12 ms | | `mergeable_git_state?` | 37 ms | 9 ms | | `merge_async` | 4 ms | 4 ms | ### MWCP scenarios Warm GDK, minimum of 2-3 runs. Project 2 (baseline MR 4 commits/6 files, any-approver rule) and project 4 (Wget2, scale MRs). | Scenario | Wall ms | SQL (writes) | Gitaly RPCs (ms) | Txns: count, longest ms / statements / idle ms | Enqueued | Outcome | |---|---|---|---|---|---|---| | `unchecked`, no merge ref | 640 (median 1505) | 76 (13) | 24 (492) | 2: 20 / 18 / 16 | AMPW + MergeW | `merge_async` | | `can_be_merged` (the self re-enqueued second run) | 51 | 29 (1) | 1 (3) | 0 | MergeW | `merge_async` again, new `merge_jid` | | `checking`, lease free | 756 | 76 (13) | 24 (602) | 2 | AMPW + MergeW | Full recheck, same as `unchecked` (`recheck_merge_status?` covers all check states) | | `cannot_be_merged_recheck` | 608 | 76 (13) | 24 (473) | 2 | AMPW + MergeW | Full recheck | | `cannot_be_merged` (terminal) | 43 | 26 | 1 (3) | 0 | none | 16 state checks, then the conflict check fails | | Mergeability lease held elsewhere | 136-197 | 26 | 1 (3) | 0 | none | All 16 state checks run, then `in_write_lock` gives up in 0.4 ms (`retry_lease` false) | | Head pipeline running | 21 | 14 | 0 | 0 | none | Stops at `CheckCiStatusService` (12 checks) | | Unresolved discussion (setting on) | 19 | 10 | 0 | 0 | none | Stops at `CheckDiscussionsStatusService` (11 checks) | | Blocked by another MR | 13 | 5 | 0 | 0 | none | 6 checks | | Draft | 4.4 | 2 | 0 | 0 | none | Stops at the 2nd check | | 30 commits / 90 files | 536 | 69 (13) | 24 (403) | 2: 26 / 18 / 21 | AMPW + MergeW | `merge_async` | | 150 commits / 750 files | 719 | 69 (13) | 24 (542) | 2: 89-109 / 18 / 71-93 | AMPW + MergeW | `reload_merge_head_diff` is 63% of wall | | 300 commits / 300 files, 510 KB diff | 728 | 69 (13) | 24 (594) | 2: 63-81 / 18 / 51-54 | AMPW + MergeW | | | 5 rules x 100 approvers, approved | 473 | 78 (13) | 24 (352) | 2: 14 / 18 / 11 | AMPW + MergeW | `CheckApprovedService` 42 ms / 13 SQL | | Same, unapproved | 61 | 28 | 0 | 0 | none | 42 of 61 ms is `CheckApprovedService` | | 40 active diff discussions | 742 | 156 (53) | 46 (511) | 2: 16 / 18 / 12 | AMPW + MergeW | `update_diff_discussion_positions!` 290 ms, 22 RPCs, 81 SQL (N+1: ~7 ms + 0.5 RPC per discussion) | | `pipeline_id` fan-out, 5 MRs `unchecked` | 1777 | 295 (51) | 95 (1367) | 8: 37 max | AMPW x4 + MergeW x3 | 440-485 ms per MR. The dev-only Gitaly N+1 guard (30 RPCs/job) killed the job before the 5th MR. | | `pipeline_id` fan-out, 5 MRs `can_be_merged` | 660 | 193 (17) | 28 | 2 | MergeW x5 + AMPW | ~35 ms per already-checked MR | | Bare integer argument | 431 | 75 (13) | 24 (342) | 2 | AMPW + MergeW | Same as the hash form | AMPW = `AutoMergeProcessWorker`, MergeW = `MergeWorker`. A row with a single `Txns` number gives only the count. Empty cells were not recorded. ### Merge train and EE scenarios On GDK: project 7 (`flightjs/Flight`, 500 commits, MR !8) and project 5 (`Commit451/lab-coat`, 988 commits, MRs !9 and !10). Second runs. Host load was 10-20, so wall times are noisy. Counts are exact. S-numbers follow `trains_report.md`. | Scenario | Path | Wall ms | SQL (writes) | Gitaly RPCs / ms | Txns (longest ms) | Enqueued | Outcome | |---|---|---|---|---|---|---|---| | S1a AMTWCP, `unchecked`, no merge ref | `mergeable?`, `MergeabilityCheckService`, `MergeTrainService#available_for?`, `#execute` | 1091 | 114 (18) | 24 / 738 | 3 (165) | AMPW x2 | Car created, strategy becomes `merge_train`, system note | | S1b `merge_train` on train | `MergeTrainService#process` | 6 | 2 | 0 | 0 | `MergeTrains::RefreshWorker` | No-op | | S2 AMTWCP, trains disabled after opt-in | Full S1a work, then abort | 573 | 92 (17) | 23 / 415 | 3 (19) | AMPW, `CheckOrganizationIsolationStatusWorker` | Auto-merge cleared, abort note | | S2b same, with a car | `MergeTrainService#abort` | 619 | 90 (18) | 22 / 481 | 3 (16) | Same | Car deleted, note, GraphQL trigger | | S3 unknown strategy | `BaseService#cancel`, `execute_hooks` | 67 | 29 (2) | 1 / 3 | 1 (7.5) | `WebHookWorker`, `CheckOrganizationIsolationStatusWorker` | Cleared. Webhook enqueued, no sync HTTP. | | S4 MWCP + CODEOWNERS (3 sections x 10 owners) | Happy path | 660 | 92 (13) | 24 / 462 | 5 (20) | AMPW, MergeW | Merge enqueued. Code owner rule sync = 3 extra txns. | | S4 rerun on `can_be_merged` | Self re-enqueue path | 84 | 29 (1) | 3 / 24 | 0 | MergeW (again) | | | S5 MWCP + 3 rules x 50-member groups | Happy path | 489 | 74 (13) | 24 / 367 | 2 (29) | 2 jobs | `CheckApprovedService` 24 ms / 9 SQL, loads `users.*` for all 150 members | | S5b unapproved | Fails at `CheckApprovedService` | 52 | 26 | 0 | 0 | none | Stops before any Gitaly | | S6 MWCP + 2 external status checks passed | Happy path | 586 | 73 (13) | 24 / 454 | 2 (22) | 2 jobs | 1 `status_check_responses` query per check, no HTTP | | S6b one check pending | Fails at the 3rd check | 15 | 8 | 0 | 0 | none | | | S8a path lock by another user | Fails at `CheckPathLocksService` | 97 | 18 | 2 / 48 | 0 | none | Gitaly `FindChangedPaths` as soon as any lock row exists | | S8c LFS lock by another user | Fails at `CheckLfsFileLocksService` | 51 | 12 | 2 / 30 | 0 | none | | Reference: the plain MWCP happy path on project 7 is 70-73 SQL, 24-26 Gitaly RPCs and 2 txns. S1a phase tree (AMTWCP add to train): | Phase | Time | Detail | |---|---|---| | `mergeable?` | 985 ms | State checks 32 ms / 19 SQL. `check_mergeability` 953 ms: `merge_to_ref` 331 ms, `can_git_merge?` 224 ms (`FindBranch` 181 ms and `ListConflictFiles` 43 ms), `write_merge_status` transaction 7 ms / 3 statements, `reload_merge_head_diff` 377 ms / 33 SQL / 17 RPCs with a transaction of 15-165 ms / 18 statements. | | `MergeTrainService#available_for?` | 45 ms | A second full check pass: 19 SQL, 10 Redis. | | `MergeTrainService#execute` | 56 ms | Transaction 3: 20 ms, 9 statements (assignee COUNT, open-MR uniqueness `SELECT`, reviewer COUNT, `UPDATE merge_requests`, `INSERT merge_trains`, car index COUNT, `INSERT notes` with pre-rendered HTML, `INSERT system_note_metadata`, `DELETE merge_request_user_mentions`). The Markdown is rendered inside the transaction. | | `execute_hooks('update')` | 30 ms | A third full check pass for the webhook payload: 20 SQL, 1 Gitaly. See Findings (b). | | `AutoMergeProcessWorker` enqueue | | Both follow-up jobs take the 6 ms S1b path and only enqueue `MergeTrains::RefreshWorker`. | Abort transaction (`clear_auto_merge`): 7-8 statements, including a pending-todo `SELECT`, `INSERT` into `notes`, `DELETE` of mentions and (with a car) `DELETE` from `merge_trains`. The cold run took 189 ms, with a todos `INSERT`, a `feature_gates` read and two Redis writes inside. All are bounded (tens of ms), but they have the same shape `MergeWorker` was fixed for. ### Security policy spec variants RSpec with factories, test environment (`spec/workers/security_policy_trace_spec.rb`). | Variant | Wall ms | SQL | Gitaly RPCs | Enqueued | Trace in `out/` | |---|---|---|---|---|---| | Approved, no violations | 566 | 93 | 22 | `MergeWorker` | `spec_s7a_policy_synced_approved.txt` | | Failed violation | 91 | 28 | 0 | none | `spec_s7b_policy_violation_failed.txt` | | Desync | 23 | 12 | 0 | `ResyncMergeRequestRulesWorker` | `spec_s7c_policy_rules_desynced.txt` | With a failed violation, `CheckSecurityPolicyViolationsService` computes the whole `ApprovalState` itself (22 SQL). ### Transaction inventory | ID | Transaction | Statements | Observed on GDK | What grows | |---|---|---|---|---| | T1 | Merge status write (`MergeabilityCheckService#write_merge_status`) | `SELECT ... FOR UPDATE` on `merge_requests` (no `lock_timeout` or `NOWAIT`), MR validations including the assignee COUNT and `validate_fork` SQL, a possible `Banzai` re-render of title and description when `cached_markdown_version` is stale, `UPDATE`. | 3 statements, 5-11 ms (11 ms cold baseline, 7 ms warm), 8 ms idle in the baseline. | Nothing with MR size. The row-lock wait depends on other transactions. | | T2 | Merge head diff `DELETE` | `DELETE` of the old merge head diff, which cascades to diff files. | Not reported separately. The harness destroys the old merge head diff in its reset step. | The diff file rows removed by the cascade. | | T3 | Merge head diff `INSERT` (`merge_head_diff.save!`, `save_git_content`) | All merge-ref commits via `MergeRequestDiffCommit.create_bulk` in 1000-row batches with `DiffCommitUser` and `CommitsMetadata` find-or-create, all diff files in one `legacy_bulk_insert`, count `UPDATE`s. A `CarrierWave` external diff upload runs inside when `external_diffs` is `always`. | 18 statements. 235 ms cold baseline. Warm: 20 ms at 6 files, 26 ms at 90 files, 63-81 ms at 300 files (510 KB diff), 89-109 ms at 750 files. S1a: 15-165 ms. | Commits and files (size-proportional). Idle time grows about 0.1 ms per file: 16 ms at 6 files, 21 ms at 90, 51-54 ms at 300, 71-93 ms at 750. | | T4 | `MergeTrainService#execute` | MR `save!` with full validations, car `INSERT`, and `SystemNoteService.merge_train` inside: `Banzai` render of the note, `notes` and `system_note_metadata` `INSERT`, `touch_noteable` `UPDATE`, mentions upsert. | 9 statements, 20 ms (S1a). | The system note work done inside the transaction. | | T5 | `clear_auto_merge` on abort or cancel | MR `UPDATE`, car `DELETE`, system note, `TodoService.merge_train_removed` (pending-todo `SELECT`/`INSERT` per participant, todo count cache) and `GraphqlTriggers` (WAL queries and `ActionCable` Redis publish) inside the transaction, then `next_car.outdate_pipeline`. | 7-8 statements on abort. Longest transaction in the run: 19 ms (S2), 16 ms (S2b). Cold run 189 ms. Cancel (S3): 7.5 ms. | Participants (todos). Redis publish. | | T6 | EE code owner rule sync | Many short transactions, one set per CODEOWNERS entry. | S4: 3 extra transactions for 3 sections x 10 owners. Longest of the 5 transactions in the run: 20 ms. | Number of CODEOWNERS entries. | | T7 | Autocommit single statements | `update_merge_jid`, `DiffNotePosition` upserts, patch id. | Not transactions. | Not applicable. | No transaction this worker opens contains a Gitaly RPC, an HTTP call or a per-approver loop. </details> <details><summary>Findings</summary> ### (a) Transaction length - Every transaction this worker opens is short: 3 statements for the merge status write, 18 for the merge head diff insert, 9 for the add-to-train, 7-11 for abort and cancel. - None contains a Gitaly RPC, an HTTP call or a per-approver loop. No loop runs for minutes. - The merge head diff insert (T3) is the only size-dependent transaction. About 0.1 ms per diff file of Ruby row-building runs inside it. It took 89-109 ms at 750 files on GDK. It is unbounded in commits and files. - The only plausible way this worker shows up in a >10 min transaction alert is a wait on the `merge_requests` row lock. The `SELECT ... FOR UPDATE` in `write_merge_status` (T1) has no `lock_timeout` or `NOWAIT`. T4 and T5 can wait the same way. The holder would be another long transaction, for example the old `MergeWorker` post-merge transaction that #628868 already fixed. - The static audit also names an extreme-size merge head diff insert as a possible long transaction. Nothing on the MWCP path can plausibly reach minutes at the sizes we traced (up to 150 commits / 750 files, and 300 commits / 300 files). - Self-managed instances with `external_diffs.when: always` and object storage do an object storage PUT inside the merge head diff transaction (the `CarrierWave` external diff store runs inside it). GDK has external diffs disabled. The GitLab.com value is not checked (see Open questions). - Merge train add (T4) and abort or cancel (T5) have the same shape `MergeWorker` was fixed for: system note, todo and GraphQL trigger work inside the locked window. They are bounded at tens of ms (189 ms on one cold run). - No production alert was found for this worker. The sibling `MergeTrains::RefreshWorker` has one (#627653). ### (b) Per-run cost and redundant work - A successful run is Gitaly-bound: 24 RPCs are 77% of wall (490 ms of 640 ms, warm GDK). - 97% of the baseline run is the synchronous merge status recheck (3,150 of 3,238 ms, cold GDK). - Three of the four Gitaly-heavy steps are avoidable or deferrable: | Step | Cold (run 1) | Warm | Verdict | |---|---|---|---| | `merge_to_ref` | 810 ms | 247 ms | The merge itself. Not among the avoidable steps. | | `can_git_merge?` | 758 ms | 72 ms, 3 RPCs | Redundant after a successful `merge_to_ref` (Findings (d)). | | Repeated `FindCommit` for the same merge ref | 4-6 uncached calls | payload call 3-12 ms | Avoidable. | | `reload_merge_head_diff` | 1,539 ms | 214 ms, 17 RPCs | Deferrable. The merge itself never reads it. 33-63% of wall. | - Diff discussion position capture is N+1 per active discussion. 40 active diff discussions cost `update_diff_discussion_positions!` 290 ms, 22 RPCs and 81 SQL (about 7 ms and 0.5 RPC per discussion). It loads one diff file per discussion only to compute paths, although `position.paths` already exists. - AMTWCP runs the check suite 2 times per run. Pass 2 is `MergeTrainService#available_for?`. It pays the full Gitaly bill before it can abort: S2 took 573 ms with 23 RPCs (415 ms), S2b 619 ms with 22 RPCs (481 ms). - AMTWCP runs the suite 3 times when a hook payload is built (`detailed_merge_status`). The static audit expected this for projects with MR webhooks or external status checks. The S1a trace shows that an unrelated active integration is enough. `Integrations::Base::Integration#async_execute` calls `data.with_indifferent_access` before checking `supported_events`. The third pass took 30 ms, 20 SQL, 1 Gitaly. - Approvals (measured): `CheckApprovedService` is the largest state check in the baseline (17 ms / 11 SQL). With 5 rules x 100 approvers it takes 42 ms and 13 SQL, and an unapproved MR spends 42 of its 61 ms there. With 3 rules x 50-member groups it takes 24 ms and 9 SQL and loads `users.*` for all 150 members. - Approvals (in the code): every member of every approver group is loaded. Role-based code owners load all project members with the role. `temporarily_unapproved?` does a Redis `EXISTS` that is not memoised (2 times per pass). The bypass check does about 4 `EXISTS` per scan-result rule. - Locks: the path lock check calls Gitaly `FindChangedPaths` as soon as any lock row exists (S8a: 97 ms, 2 RPCs / 48 ms). The LFS lock check took 51 ms (S8c: 2 RPCs / 30 ms). - External status checks: 1 `status_check_responses` query per check, no HTTP (S6: 586 ms). - EE code owner sync after the merge head diff rebuild costs O(paths x patterns) CPU and 3 small transactions per entry (static audit). S4 shows 3 extra transactions. - Cheap exits are cheap: 4-43 ms in the MWCP table. Fail-fast tiering works. The CI, discussion, blocked and draft exits use no Gitaly. A CI-bound no-op costs 21 ms and 14 SQL after 12 checks. - A run that loses the mergeability lease takes 136-197 ms, because all 16 state checks run before the lease is tried. - `mergeable?` in `process` passes no skip params for MWCP. `skipped_auto_merge_checks` only feeds `available_for?`. `use_cache: false` changes behaviour for only 3 cacheable checks (LFS locks, path locks, approvals). The other 13 are never cached. - Check order (EE): open, draft, external status checks, jira, merge time, blocked by other MRs, requested changes, title regex, security policy violations, LFS locks, discussions, CI, commits, path locks, approved, security policy pipeline. Then `check_mergeability`, then rebase and conflict. - Fan-out by `pipeline_id`: 440-485 ms per unchecked MR (5 MRs: 1777 ms) and about 35 ms per already-checked MR. The dev-only Gitaly N+1 guard (30 RPCs/job) killed the job before the 5th MR. - Worker config: - `data_consistency :sticky`: after the first write, everything runs on the primary, including later MRs in the pipeline and project loops. - `retry: 3` on a non-idempotent job: one failing MR re-runs up to 30 in the project form. - `worker_resource_boundary :cpu`, although the time is mostly Gitaly and DB wait (#299306). - `PROJECT_MR_LIMIT = 30` with `take` and no ORDER BY. ### (c) Scheduling and frequency Trigger matrix (triggers A-H): | Trigger | What enqueues the worker | Notes | |---|---|---| | A | Pipeline completion | Every completed pipeline on the instance, including child pipelines and `success->success` loopbacks. GDK: 87% no-ops at ~5.6 queries. | | B | `after_transition => :can_be_merged` | Widget poll every 10 s, REST entities, show page, branch change, merged-results pipeline creation, and the worker's own sync check (the self re-enqueue). | | C | `RefreshService` on every branch push | Up to 20 auto-merge MRs on the pushed target branch (`first(20)` on an unordered relation), staggered 3 s. Source-branch MRs are enqueued and then aborted a few lines later. | | D | Enabling auto-merge | Includes AMTWCP converting itself to `merge_train`. | | E | Approval reset | Bare integer argument form. | | F | Title regex setting change | Project form, max 30 MRs. | | G | Scheduled-merge cron | Every minute for every past-schedule MR: 1,440 runs/day per blocked MR. Rows are never deleted. | | H | `ProcessAutoMergeFromEventWorker` on 11 events | Approved, draft toggle, discussions resolved, mergeable event on page load, external status check passed, and others. | - On gitlab-org/gitlab the `RefreshService` fan-out alone is about 20 runs per master push x 161-219 pushes/day = 3.2k-4.4k runs/day, nearly all no-ops. MRs blocked only by conflicts (2 of 52) pay a Gitaly merge-to-ref on every target push. - Self re-enqueue on GDK: for AMTWCP the duplicate finished before the original and added the train car. `merge_async` in the duplicate gets nil from the deduplicated `MergeWorker` push and stores nil in `merge_jid`. - The dedup does not help: every enqueue runs, and concurrent runs for the same MR are normal (see (d)). - Most triggers patch a known "nothing re-fired" hole, so trimming them needs proof that another trigger covers the case (see Constraints from history). Lifecycle estimate for a typical MWCP MR: | Item | Value | |---|---| | Assumed lifecycle | 3 source pushes, 2 approvals, 5 target pushes, threads resolved once | | Runs | 11 (8 AMPW + 3 PAMFEW) | | Runs with an open MR tab and 5 child pipelines per MR pipeline | ~20 | | Runs that can merge | exactly 1 | | Guaranteed duplicates | 1 (the self re-enqueue) | | All other runs | Die at `CheckCiStatusService` after up to 11 other checks | ### (d) Verified facts These were checked in the code, not only seen in traces. 1. **The dedup is inert.** The declaration `deduplicate :until_executed, if_deduplicated: :reschedule_once` does nothing in practice. `DeduplicatesWhenScheduling#deduplicate?` returns false unless `duplicate_job.idempotent?` (`lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/deduplicates_when_scheduling.rb:42`), and the worker never calls `idempotent!`. The signalling key is set and then immediately cleared by the client, so `reschedule_once` cannot fire outside a millisecond race. Every enqueue runs, and concurrent runs for the same MR are normal. Cost: 3-4 client and 2-3 server Redis round trips per job for nothing. 2. **`can_git_merge?` after a successful `merge_to_ref` is redundant.** `UserMergeToRef` raises on `:merge_conflict` (`lib/gitlab/gitaly_client/operation_service.rb:171-173`) and `MergeToRefService` turns that into an error result. So `merge_to_ref_success` already proves there are no conflicts. The check dates from d0ecb89774dc (2019). 3. **`merge_head_diff` consumers.** UI diff views (`diffs_controller.rb:193-194`, `rapid_diffs` compare versions, `diffs_batch_cache_key`), `diffable_merge_ref?`, and EE `Gitlab::CodeOwners` (`ee/lib/gitlab/code_owners.rb:82-90`), which prefers it when `can_be_merged?` for code owner rule sync. Approval patch ids come from `merge_request_diff.get_patch_id_sha`, not from the merge head diff. 4. **`MergeWorker` re-runs the full check suite** (`mergeable?` with `use_cache` false). So MWCP pays the suite twice end to end. 5. **Per-check timings are already logged in production.** `MergeRequests::Mergeability::Logger` writes `mergeability.<check>.duration_s.values` to `Gitlab::AppJsonLogger` with `mergeability_merge_request_id`. Filter Kibana application logs by `json.meta.caller_id: AutoMergeProcessWorker`. </details> <details><summary>What to fix or improve</summary> Ideas are ranked within each group, first is highest. File references are as found in the audit. Numbers are GDK unless stated. ### A. Transaction length **A1 - Bound the row-lock wait** - Change: `SET LOCAL lock_timeout` (1-5 s) or `FOR UPDATE NOWAIT`, and treat a lock failure as discarded, in `MergeabilityCheckService#write_merge_status` (`mergeability_check_service.rb:159-171`, `merge_request.rb:353-355`). Optionally also in `AutoMerge::BaseService#execute` and `#clear_auto_merge`. - Effect: removes this worker from the >10 min alert class entirely. - Risk: low to medium. Under contention the status may stay `unchecked` until the next trigger. - Feature flag: yes. **A2 - Move side effects out of the auto-merge transactions** - Change: run `SystemNoteService.*`, `TodoService.merge_train_removed` and `GraphqlTriggers` after commit, instead of inside the `AutoMerge::BaseService#execute` and `#clear_auto_merge` transactions. Same shape as the `MergeWorker` fixes. - Effect: cuts those transactions to 2-4 statements. Removes `Banzai`, mention, todo and Redis work from the locked window. - Risk: low. - Feature flag: decide at implementation time. **A3 - Skip the stale-markdown re-render on the merge status and auto-merge saves** - Change: use `skip_markdown_cache_validation` (precedent: `app/services/markdown_content_rewriter_service.rb:46`). - Effect: avoids the possible `Banzai` re-render inside T1. No number measured. - Risk: low. - Feature flag: decide at implementation time. **A4 - Confirm `external_diffs.when` on GitLab.com** - Change: if it is `always`, move the external diff store out of the `save_git_content` transaction. This affects all `merge_request_diff` creation, not only this worker. - Effect: removes an object storage PUT from the transaction on such instances. No number measured. - Risk: not rated. - Feature flag: decide at implementation time. ### B. Per-run speed **B1 - Skip `can_git_merge?` when `merge_to_ref` succeeded** - Change: `mergeability_check_service.rb:139-143`. - Effect: -3 RPCs, and -72 ms warm or -758 ms cold on GDK. It applies to every merge status recheck instance-wide (also `MergeRequestMergeabilityCheckWorker` and the widget). - Risk: low to medium. Semantics are identical unless the target moves between the two RPCs. The `FOR UPDATE` inputs check already covers the retarget case. - Feature flag: yes. **B2 - Take `reload_merge_head_diff` off the auto-merge critical path** - Change: run it (and the EE code owner rule sync it triggers) in a follow-up job, or skip it when the run is about to call `merge_async`. Alternative: rebuild only when the merge ref parents changed. - Effect: -214 to -463 ms and -17 RPCs per run on GDK. Removes the only size-dependent transaction from the worker. - Consumers that must keep working: merge-ref diff UI views, `diffable_merge_ref?`, EE `Gitlab::CodeOwners` (prefers `merge_head_diff` when `can_be_merged`, feeds approval rules). - Risk: medium to high in EE. Code owner rules that gate approvals must not be skipped. `MergeWorker` re-runs the checks itself. - Feature flag: yes. **B3 - Reuse the commit id `MergeToRefService` already returned** - Change: stop doing 4-6 uncached `FindCommit`-by-ref lookups for `merge_ref_head` in `CaptureDiffNotePositionService`, `ReloadMergeHeadDiffService`, `MergeRequestDiff#ensure_commit_shas` (x2) and the payload. - Effect: 4-6 fewer `FindCommit` RPCs per run (the payload one took 3-12 ms on GDK). - Risk: low. - Feature flag: decide at implementation time. **B4 - Cheaper diff discussion position capture** - Change: in `CaptureDiffNotePositions`, use `discussion.position.paths` instead of loading a diff file per discussion, preload `note_diff_file`, or move the capture to a follow-up job. - Effect: targets the cost of about 7 ms and 0.5 RPC per active discussion (40 discussions: `update_diff_discussion_positions!` 290 ms, 22 RPCs, 81 SQL on GDK). - Risk: low. - Feature flag: decide at implementation time. **B5 - AMTWCP: cheap availability facts first, no second suite run** - Change: evaluate the cheap `MergeTrainService` availability facts (trains enabled, head pipeline complete, `can_be_merged_by?`) before `mergeable?`. Do not re-run `execute_merge_checks` inside `available_for?` when the caller just ran the suite. - Effect: saves a full check pass per add (45 ms, 19 SQL, 10 Redis) and 400-700 ms of Gitaly plus 90 SQL per abort run. Also removes the abort-on-stale-cache window. - Risk: low. - Feature flag: decide at implementation time. **B6 - Check the event before building the webhook payload** - Change: `Integrations::Base::Integration#async_execute` forces the `Gitlab::Lazy` merge data (a third full check pass including a Gitaly conflict check) before checking `supported_events`. Check the event first. - Effect: benefits every MR update hook instance-wide. The third pass measured 30 ms, 20 SQL, 1 Gitaly in S1a. Related: thin webhooks (!242260). - Risk: low. - Feature flag: decide at implementation time. **B7 - Reorder inside `process`** - Change: run `CheckCiStatusService` first, and try the mergeability lease before the 16 state checks. - Effect: no-op runs drop from about 12 checks to about 3. Lease-collision runs drop from 136-197 ms to about 1 ms. - Risk: low. Reuse the check service (constraint C5). - Feature flag: yes. **B8 - Micro changes** - Change: lock checks use `merge_request_diff.modified_paths` instead of Gitaly `FindChangedPaths`. Batch `status_check_responses`. Memoise `temporarily_unapproved?`, `pipeline_creating?` and `approval_rule_bypassed?`. Preload `approval_policy_rule: :security_policy`. `pluck` ids instead of loading `users.*` for group approvers. Check discussions with `EXISTS`. - Effect: not quantified as a whole. Related measurements: S5 (`CheckApprovedService` 24 ms / 9 SQL, `users.*` for 150 members), S8a (97 ms), S6 (1 `status_check_responses` query per check). - Risk: not rated. - Feature flag: decide at implementation time. ### C. Fewer runs **C1 - Suppress the self re-enqueue** - Change: when the run itself made the `can_be_merged` transition, mark the MR id in `SafeRequestStore` around the sync check (`merge_request.rb:303-310` records instead of enqueueing). Enqueue only if the run did not reach `merge_async` or train execute. Also have `MergeTrainService#execute` enqueue `MergeTrains::RefreshWorker` directly instead of a second `AutoMergeProcessWorker` (`base_service.rb:16`). - Effect: -1 full run per merge. No duplicate `merge_async`. Fixes the `merge_jid` nil clobber and the AMTWCP double add. - Risk: low if scoped to the MR id and the process call. Transitions made elsewhere must keep enqueueing (#592733). - Feature flag: yes (`gitlab_com_derisk`). **C2 - Make the dedup real** - Change: declare `idempotent!` (gated by the `feature_flag:` option of `deduplicate`, `worker_attributes.rb:283-286`), keep `until_executed` with `reschedule_once`, and move `ResetApprovalsService` to the hash form. Do not add `including_scheduled` (constraint C7). - Effect: collapses queued duplicates under backlog. A mid-run duplicate becomes one rerun. - Risk: low to medium. Side effects already repeat today. - Feature flag: through the `feature_flag:` option of `deduplicate`. **C3 - Do not enqueue for child pipelines** - Change: `next if pipeline.parent_pipeline?` (`pipeline.rb:383-390`). Instance-wide. - Effect: no isolated number measured. See the lifecycle estimate: ~20 runs with an open MR tab and 5 child pipelines per MR pipeline, against 11 without. - Risk: low. - Feature flag: yes. **C4 - Narrow the `RefreshService` fan-out** - Change (`refresh_service.rb:217-225`): skip MRs it is about to abort (source-branch pushes), skip `merge_train`-strategy MRs, and use a deterministic order or drop the cap once C2 lands. - Effect: on gitlab-org/gitlab, about 20 runs per master push drop to a few (3-4k no-op runs/day today). - Risk: low for the skips. - Feature flag: yes. **C5 - Bulk-enqueue per-MR jobs from the fan-out forms** - Change: bulk-enqueue per-MR `merge_request_id` jobs from the `pipeline_id` and `project_id` forms, and hand PAMFEW off to AMPW. - Effect: jobs become bounded (the dev Gitaly N+1 guard trips at the 2nd unchecked MR of a fan-out) and dedup applies across forms. - Risk: medium. It adds a hop. PAMFEW is `:always` and AMPW is `:sticky`. - Feature flag: yes. **C6 - Limit the scheduled-merge cron** - Change: only enqueue MRs whose `merge_after` passed within a trailing window. Measure the GitLab.com count first. - Effect: not quantified. Today the cron runs every minute for every past-schedule MR (1,440 runs/day per blocked MR). - Risk: not rated. - Feature flag: decide at implementation time. **C7 - `worker_resource_boundary :cpu` mismatches the profile** - Change: see #299306 (drop `worker_resource_boundary :cpu`). The time is mostly Gitaly and DB wait. - Effect: not quantified. - Risk: not rated. - Feature flag: decide at implementation time. ### Suggested order 1. B1, B3, C1, A2, B5, B7: small, low risk, measurable. 2. C2, C3, C4: scheduling changes, need Kibana before and after numbers. 3. B2 and A1: larger, need care. 4. B4, B6, B8, C5, C6. A3, A4 and C7 have no position in this order. </details> <details><summary>Constraints from history</summary> These constraint IDs (C1-C7) are separate from the idea IDs C1-C7 in the fix list. Any change must respect them. | Constraint | What it says | |---|---| | C1 | Every trigger patches a known "nothing re-fired" hole: #592733 (`checking`), #594868 (sibling `unchecked`), #505829 (temporarily unapproved), #600832 (no pipeline), and rule or membership changes that are only covered by page views. Remove a trigger only with proof that another covers it. | | C2 | `process` bails silently on a transient `checking`. | | C3 | Primary reads did not fix #596177 (reverted in !239951). | | C4 | Never let the approval cache decide a merge (!231445). This is why `use_cache: false` exists. | | C5 | Do not re-implement CI gating outside `CheckCiStatusService` (dcaf9d0d156d). | | C6 | `head_pipeline_id` is written asynchronously. | | C7 | `including_scheduled` with `until_executed` drops scheduled duplicates without a reschedule. | If triggers are trimmed, add a structural safety net (a bounded reschedule-on-checking, or a sweep). </details> <details><summary>How to verify</summary> ### GDK re-run 1. Extract `am_trace_bundle.tar.gz`. The scripts use `require_relative 'lib/am_scenario'`, so keep `lib/` next to them. Traces are written to `out/` next to `lib/`, or to the directory in `AM_OUT`. 2. Run a script from the GDK checkout: `cd ~/work/gdk/gitlab && timeout 550 bin/rails runner <script>` 3. Do a warm-up run first (development mode autoload), then the measured runs. Never run two measured scenarios at the same time. 4. Compare wall, SQL, Gitaly RPC and transaction counts before and after a change. Counts are exact and repeatable. Wall time is noisy on a busy GDK. | Script | What it covers (from its header comment) | |---|---| | `run_baseline.rb` | Baseline: MWCP on project 2, `unchecked` MR with no merge ref, success pipeline, no rules. | | `mwcp_a_states.rb` | `merge_status` states, early exits and lease held, on the baseline MR. | | `mwcp_b1_setup_scale.rb` and `mwcp_b2_run_scale.rb` | MR size scale on project 4 (setup, then run). | | `mwcp_c_setup.rb` and `mwcp_c_run.rb` | Approval-heavy MR (5 rules x 100 approvers) approved, then unapproved (setup, then run). | | `mwcp_e_diffnotes.rb` | 40 active diff notes. | | `mwcp_f_fanout.rb` | One source branch, 5 MRs to 5 target branches: `pipeline_id` fan-out and the bare integer form. | | `run_trains_a.rb` | Project 7: merge train strategies, abort, legacy cancel and webhook. | | `run_approvals_b.rb` | Project 5: code owners (3 sections x 10 owners) and group approvers (3 groups x 50), MWCP `unchecked` flow. | | `run_groups_b3.rb` | Project 5, branch `am-groups`: group approvers, MWCP `unchecked` flow, approved and then unapproved (S5 and S5b traces). It has no header comment. | | `run_locks_c.rb` | Project 7 with trains disabled: external status checks, path locks, LFS locks under MWCP. | | `spec/workers/security_policy_trace_spec.rb` | Security policy variants. RSpec with factories, not a runner script: run it with `bundle exec rspec <path>`. Edit the two hard-coded scratchpad paths in it first (the `require` of `lib/am_trace` and `OUT`). | | `probe_dedup.rb` | Read-only probe (no DB or Redis writes). Prints `idempotent?`, the dedup strategy and options of `AutoMergeProcessWorker`, `ProcessAutoMergeFromEventWorker` and `MergeWorker`, the duplicate job state per argument form, and the auto-merge and merge schedule counts of the instance it runs on. | | `sidekiq_peek.rb` | Prints GDK Sidekiq stats, the largest queues, busy job classes and scheduled jobs. | | `summarize.rb` | Builds summary table rows from the `out/*.json` files a run writes: `ruby summarize.rb <out_dir> <label-regex>`. | ### Production (Kibana) | Query or field | What it tells you | |---|---| | Sidekiq logs, `json.class: AutoMergeProcessWorker`, split by `json.meta.caller_id` | Which trigger enqueued each run (mapping below). | | `json.duration_s`, `json.gitaly_calls`, `json.db_count` | Histograms of run duration, Gitaly calls and DB statements. | | `json.duplicate-of: *` | How often real dedup would act. | | Application log, `json.meta.caller_id: AutoMergeProcessWorker`, field `mergeability.<check>.duration_s` | Per-check timings, written by `MergeRequests::Mergeability::Logger`. | | `json.meta.caller_id` | Trigger | |---|---| | `PipelineProcessWorker`, `Ci::InitialPipelineProcessWorker` | Pipeline completion | | `UpdateMergeRequestsWorker` | `RefreshService` fan-out | | `AutoMergeProcessWorker` | Self re-enqueue, or train conversion | | `MergeRequestMergeabilityCheckWorker`, web and API | `can_be_merged` transition, or enabling auto-merge | | `ProcessScheduledMergeWorker` | Scheduled-merge cron | | `MergeRequestResetApprovalsWorker` | Approval reset | ### Success metrics - The >10 s count of `AutoMergeProcessWorker` jobs in epic 21116 (61,796 per week, Aug 17-23 2026), and the queue apdex. - The Patroni long-transaction alert stays silent for this worker. </details> <details><summary>Pick-up notes</summary> ### Attachments Tools and bundle: | File | What it is | |---|---| | [am_trace.rb](/uploads/fd75197213377fa17e50647b898cc1cd/am_trace.rb) | The inline tracer. Runs the worker in `rails runner` and records SQL, transactions, Gitaly RPCs, Redis, HTTP, enqueues and phases. | | [am_scenario.rb](/uploads/44623f793e18aeda6e9b2c8220c1b0e8/am_scenario.rb) | The scenario harness. Builds branches, commits, MRs, pipelines, approvals, discussions and trains on GDK, and runs the worker under the tracer. | | [run_baseline.rb](/uploads/f54069ef6cf421e25c2c17ee2672ca5e/run_baseline.rb) | Script for the baseline scenario. | | [am_trace_bundle.tar.gz](/uploads/3e9b393340fe06951f715c18465dadc4/am_trace_bundle.tar.gz) | Everything in one archive: `lib/` (`am_trace.rb`, `am_scenario.rb`, `am_patient.rb`), `agents/` (the four reports and the MWCP tables), `out/` (text traces and console logs of all scenarios), all scenario scripts and helper scripts, and the spec `spec/workers/security_policy_trace_spec.rb`. | Reports: | File | What it is | |---|---| | [static_audit_report.md](/uploads/2f30cc8a30b99778dac3d941b9b3e997/static_audit_report.md) | Static audit of transactions and the call graph (T1-T7). | | [frequency_report.md](/uploads/7e40b05e1ac56e5f00f7f9a07977d71d/frequency_report.md) | Trigger frequency and redundancy study. | | [mwcp_report.md](/uploads/4981b5e2ad61bacd5fcad030b2de1f3f/mwcp_report.md) | MWCP dynamic scenario report. | | [mwcp_tables.md](/uploads/2bc86747798bd2bf3dcc6523833c3f16/mwcp_tables.md) | One summary row per traced run: wall, SQL, Gitaly, Redis, transactions, enqueued jobs, last check. | | [trains_report.md](/uploads/e2ab8d5bbbc7c7eaf5b005100002b42d/trains_report.md) | Merge train and complex EE scenario report. | Traces (text, from the tracer): | File | What it is | |---|---| | [baseline_mwcp_unchecked_run1.txt](/uploads/27d2f890f6c3ef19c850de976a34188d/baseline_mwcp_unchecked_run1.txt) | Baseline scenario, run 1 (cold). Source of the baseline phase breakdown. | | [trace_s1a_amtwcp_add_to_train_run2.txt](/uploads/b211b6e525d52deb7711207a5bbb6134/trace_s1a_amtwcp_add_to_train_run2.txt) | AMTWCP add to merge train (S1a in the merge train table), run 2. | | [trace_s2b_unchecked_r2.txt](/uploads/d2fc4b9b4421e6364d024e765ea0bed4/trace_s2b_unchecked_r2.txt) | MWCP on the 150 commits / 750 files MR, run 2 (longest transaction 109 ms, the merge head diff insert). | | [trace_s3_approved_500_r2.txt](/uploads/9a5aa392fb957dc0a8b902f68f25b8b8/trace_s3_approved_500_r2.txt) | MWCP with 5 rules x 100 approvers, approved, run 2. | | [trace_s4_codeowners_mwcp_unchecked_run2.txt](/uploads/0d2516f8b84650c9e1056c1f8f02c0c5/trace_s4_codeowners_mwcp_unchecked_run2.txt) | MWCP with CODEOWNERS (3 sections x 10 owners), `unchecked`, run 2 (S4 in the merge train table). | | [trace_s5_diffnotes40_r2.txt](/uploads/376e12d31474d439deaf2ca2b50e27d2/trace_s5_diffnotes40_r2.txt) | MWCP with 40 active diff discussions, run 2. | Each trace has the totals, the inclusive phase tree, the transactions with their statements, every Gitaly RPC, HTTP, Redis by command, SQL grouped by statement and the 12 slowest statements, and the enqueued jobs. Trace names in `out/` are `<scenario>_<run>.txt`. The MWCP set and the merge train set each number their scenarios from `s1`, so the same prefix can be two different scenarios (`s1a_unchecked_r2` is the MWCP baseline MR, `s1a_amtwcp_add_to_train_run2` is the AMTWCP add to train). The bundle holds the `.txt` traces only. The matching `.json` files stay in the local scratchpad. ### Where things live - Local copies of everything are in the session scratchpad: `/private/tmp/claude-501/-Users-marcshaw-work-gdk-gitlab/ce8135c4-5cda-4223-abf0-350e4191dd9c/scratchpad/` - Claude memory pointer: `project_628868_auto_merge_process_worker.md`. ### GDK leftovers - Project 2: MR !3324 (baseline, MWCP enabled) and the fan-out MRs. `only_allow_merge_if_all_discussions_are_resolved` was restored. - Project 4: scale MRs (`am-trace` branches). - Project 5: MRs !9 (code owners, CODEOWNERS committed on master) and !10 (group approvers), open with MWCP enabled. - Project 7: MR !8. Settings restored. - Users `amtrace-*` and `amtrain-*` (`amtrain-1..180`), groups `amtrain-approvers-0..2`. ### Harness gotchas - Branch pushes make GDK Sidekiq create pending pipelines, so force the head pipeline status. - The any-approver rule on gitlab-test needs an approval. - `PostReceive` and `RefreshService` abort auto-merge on the test MR. The harness helper `reset_to_unchecked!` re-enables it. - Clear the 300 s `pipeline_creation` marker and the mergeability cache. - Warm up once before measuring. Never run measured scenarios concurrently. - Never use bare `python3` on this machine (the pyenv shim hangs). - GDK Sidekiq saturation inflates wall times. Counts stay exact. </details> <details><summary>Open questions</summary> - GitLab.com Sidekiq `statement_timeout` and `lock_timeout` values. Not checked. They frame A1, because the `SELECT ... FOR UPDATE` in `write_merge_status` has no `lock_timeout` or `NOWAIT` in the application code. - `external_diffs.when` on GitLab.com. Not checked. If it is `always`, A4 applies (object storage PUT inside the merge head diff transaction). - The GitLab.com count of past-schedule auto-merge MRs. Needed for C6: each one runs every minute (1,440 runs/day per blocked MR). - Whether this worker ever fired the Patroni long-transaction alert. No issue found mentions it. The sibling `MergeTrains::RefreshWorker` has one (#627653). - Kibana sizing has not been run. The production numbers above come from epic 21116, #592298 and the public API. The `json.meta.caller_id` split, the no-op share and the `json.duplicate-of` rate are still unmeasured. </details> ### Related - #628868 - MR workers executing long running transactions (parent work item). - #592298 - earlier sample of slow `AutoMergeProcessWorker` jobs. - #602096 - concurrency safety of `AutoMergeService#process`. - #299306 - drop `worker_resource_boundary :cpu`. - #627653 - long-transaction alert for the sibling `MergeTrains::RefreshWorker`. - Epic 21116 (rescue `low_urgency_cpu_bound`): https://gitlab.com/groups/gitlab-org/-/epics/21116 - `transaction_timeout` rollout: https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/issues/790