### Summary `MergeRequests::CreatePipelineWorker` experiences severe performance degradation, taking approximately 45 minutes to execute when handling a merge request with a large number of changed files and commits (~28k commits). This latency occurs both when triggering a pipeline manually on a large MR and during the initial MR creation. ### Steps to reproduce GitLab 18.9.1. 1. Create a merge request containing a large number of changed files and commits (~28k commits, 2k files). 2. Alternatively, trigger a new pipeline on an existing merge request of this size. 3. Navigate to the Sidekiq dashboard and wait for the `MergeRequests::CreatePipelineWorker` job to appear. 4. Observe the execution duration and resource utilization: the job runs for ~45 minutes with a single core at 100% cpu usage. ### What is the current *bug* behavior? Following the upgrade to GitLab 18.9.1, the `MergeRequests::CreatePipelineWorker` job takes significantly longer to execute than in previous versions. ### What is the expected *correct* behavior? Merge requests and pipelines should be created quickly, maintaining performance parity with earlier versions, where specific delays with large file counts were not experienced. ### Relevant logs and/or screenshots rbspy flame graph of a slow `MergeRequests::CreatePipelineWorker` job. {width=184 height=600} <details> <summary>Expand for sidekiq job log</summary> This worker had 2 of these jobs running, totaling 5.2k cpu seconds in 6k seconds total time. ```json { "severity": "INFO", "time": "2026-03-03T12:37:38.501Z", "retry": 3, "queue": "default", "version": 0, "store": null, "queue_namespace": "pipeline_creation", "args": ["71", "11", "52078", "[FILTERED]"], "class": "MergeRequests::CreatePipelineWorker", "jid": "3322bea29fabede86c41a89a", "created_at": "2026-03-03T09:08:19.477Z", "trace_propagation_headers": { "sentry-trace": "6ae3ce4c89e54a11b3f3f6c4936641a9-646c5bf991c146c1", "baggage": "sentry-trace_id=6ae3ce4c89e54a11b3f3f6c4936641a9,sentry-environment=,sentry-release=91256b9c8dc" }, "correlation_id": "01KJSF957FKJAWSBAY928NEDTB", "meta.caller_id": "POST /api/:version/projects/:id/merge_requests/:merge_request_iid/pipelines", "meta.remote_ip": "10.0.0.10", "meta.feature_category": "pipeline_composition", "meta.user": "ps", "meta.gl_user_id": 11, "meta.project": "...", "meta.root_namespace": "...", "meta.client_id": "user/11", "meta.root_caller_id": "POST /api/:version/projects/:id/merge_requests/:merge_request_iid/pipelines", "worker_data_consistency": "sticky", "worker_data_consistency_per_db": {"main": "sticky", "ci": "sticky"}, "wal_locations": {}, "wal_location_sources": {"main": "primary", "ci": "primary"}, "idempotency_key": "resque:gitlab:duplicate:default:1b49c74fd1c697cccd545a53d2932a3640cd4fb9a801aeee8f237fc8a7851794", "size_limiter": "validated", "enqueued_at": "2026-03-03T09:08:19.482Z", "interrupted_count": 2, "job_size_bytes": 164, "pid": 2750627, "sidekiq_tid": "1xv07", "sidekiq_thread_name": "sidekiq.default/processor", "message": "MergeRequests::CreatePipelineWorker JID-3322bea29fabede86c41a89a: done: 6061.314606 sec", "job_status": "done", "queue_duration_s": 6497.704963, "scheduling_latency_s": 6497.704981, "gitaly_calls": 13, "gitaly_duration_s": 30.618957, "redis_calls": 86, "redis_duration_s": 0.185521, "redis_read_bytes": 2239, "redis_write_bytes": 13688, "redis_action_cable_calls": 1, "redis_action_cable_duration_s": 0.000157, "redis_action_cable_read_bytes": 1, "redis_action_cable_write_bytes": 214, "redis_cache_calls": 4, "redis_cache_duration_s": 0.001805, "redis_cache_read_bytes": 48, "redis_cache_write_bytes": 365, "redis_feature_flag_calls": 15, "redis_feature_flag_duration_s": 0.150658, "redis_feature_flag_read_bytes": 1920, "redis_feature_flag_write_bytes": 1031, "redis_queues_calls": 10, "redis_queues_duration_s": 0.002155, "redis_queues_read_bytes": 10, "redis_queues_write_bytes": 8079, "redis_queues_metadata_calls": 10, "redis_queues_metadata_duration_s": 0.016625, "redis_queues_metadata_read_bytes": 14, "redis_queues_metadata_write_bytes": 1536, "redis_repository_cache_calls": 12, "redis_repository_cache_duration_s": 0.008319, "redis_repository_cache_read_bytes": 107, "redis_repository_cache_write_bytes": 561, "redis_shared_state_calls": 33, "redis_shared_state_duration_s": 0.005645, "redis_shared_state_read_bytes": 138, "redis_shared_state_write_bytes": 1688, "db_count": 99, "db_write_count": 11, "db_cached_count": 20, "db_txn_count": 1, "db_replica_txn_count": 0, "db_primary_txn_count": 0, "db_replica_count": 0, "db_primary_count": 99, "db_replica_write_count": 0, "db_primary_write_count": 11, "db_replica_cached_count": 0, "db_primary_cached_count": 20, "db_replica_wal_count": 0, "db_primary_wal_count": 0, "db_replica_wal_cached_count": 0, "db_primary_wal_cached_count": 0, "db_replica_txn_max_duration_s": 0.0, "db_primary_txn_max_duration_s": 0.0, "db_replica_txn_duration_s": 0.0, "db_primary_txn_duration_s": 0.0, "db_replica_duration_s": 0.0, "db_primary_duration_s": 6.548, "db_main_txn_count": 0, "db_ci_txn_count": 1, "db_main_replica_txn_count": 0, "db_ci_replica_txn_count": 0, "db_main_count": 79, "db_ci_count": 20, "db_main_replica_count": 0, "db_ci_replica_count": 0, "db_main_write_count": 3, "db_ci_write_count": 8, "db_main_replica_write_count": 0, "db_ci_replica_write_count": 0, "db_main_cached_count": 19, "db_ci_cached_count": 1, "db_main_replica_cached_count": 0, "db_ci_replica_cached_count": 0, "db_main_wal_count": 0, "db_ci_wal_count": 0, "db_main_replica_wal_count": 0, "db_ci_replica_wal_count": 0, "db_main_wal_cached_count": 0, "db_ci_wal_cached_count": 0, "db_main_replica_wal_cached_count": 0, "db_ci_replica_wal_cached_count": 0, "db_main_txn_max_duration_s": 0.0, "db_ci_txn_max_duration_s": 0.085, "db_main_replica_txn_max_duration_s": 0.0, "db_ci_replica_txn_max_duration_s": 0.0, "db_main_txn_duration_s": 0.0, "db_ci_txn_duration_s": 0.085, "db_main_replica_txn_duration_s": 0.0, "db_ci_replica_txn_duration_s": 0.0, "db_main_duration_s": 6.372, "db_ci_duration_s": 0.176, "db_main_replica_duration_s": 0.0, "db_ci_replica_duration_s": 0.0, "cpu_s": 2643.275649, "mem_objects": 7320509, "mem_bytes": 7774897551, "mem_mallocs": 621569, "mem_total_bytes": 8067717911, "worker_id": "sidekiq_4", "rate_limiting_gates": ["pipelines_create", "pipelines_created_per_user"], "exclusive_lock_requested_count": 1, "exclusive_lock_wait_duration_s": 0, "exclusive_lock_hold_duration_s": 0, "duration_s": 6061.314606, "completed_at": "2026-03-03T12:37:38.501Z", "load_balancing_strategy": "primary_no_wal", "db_duration_s": 6.780762, "urgency": "high", "target_duration_s": 10, "target_scheduling_latency_s": 10 } ``` </details> <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> ### Output of checks #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> gitlab-rake gitlab:env:info System information System: Debian 12 Proxy: no Current User: git Using RVM: no Ruby Version: 3.3.10 Gem Version: 3.7.1 Bundler Version:2.7.1 Rake Version: 13.0.6 Redis Version: 7.2.11 Sidekiq Version:7.3.9 Go Version: unknown GitLab information Version: 18.9.1-ee Revision: 91256b9c8dc Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 16.11 URL: https://gitlab...de HTTP Clone URL: https://gitlab...de/some-group/some-project.git SSH Clone URL: git@gitlab...de:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 14.45.6 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 18.9.1 - default Git Version: 2.52.gaea8cc3 </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.45.6 ? ... OK (14.45.6) Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/2 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/2 ... yes 2/3 ... yes 23/5 ... yes <snip> 1039/1024 ... yes 110/1025 ... yes 110/1026 ... yes Redis version >= 6.2.14? ... yes Ruby version >= 3.0.6 ? ... yes (3.3.10) Git user has default SSH configuration? ... yes Active users: ... 144 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-9.x or OpenSearch version 1.x-3.x ... skipped (advanced search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished </pre> </details> ### Possible fixes Not a fix but a workaround to get a working instance. ```patch --- /opt/gitlab/embedded/service/gitlab-rails/app/models/merge_request_diff.rb.ps.bak 2026-03-03 14:10:46.834644291 +0100 +++ /opt/gitlab/embedded/service/gitlab-rails/app/models/merge_request_diff.rb 2026-03-03 14:12:08.658621061 +0100 @@ -39,7 +39,8 @@ has_many :merge_request_diff_commits, -> { order(:merge_request_diff_id, :relative_order) }, inverse_of: :merge_request_diff do def with_users - associations_to_preload = [{ merge_request_commits_metadata: [:commit_author, :committer] }, :commit_author, :committer] + # associations_to_preload = [{ merge_request_commits_metadata: [:commit_author, :committer] }, :commit_author, :committer] + associations_to_preload = [:commit_author, :committer] ActiveRecord::Associations::Preloader.new(records: self, associations: associations_to_preload).call self end ``` This reverts part of [43380f988d](https://gitlab.com/gitlab-org/gitlab/-/commit/43380f988dfb40ec754234130a35ab08e7a3f234#line_290ecb739_A42). That way the job finishes in 2 minutes.