## Resolution _Use the [available AI skills](https://gitlab.com/gitlab-org/quality/analytics/ci-alerts/-/tree/main/.agents/skills) to investigate this incident._ <!-- RESOLUTION_START --> | Field | Value | |---|---| | Causing MR | [!252575](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252575) — commit `58fda4f695a8`, merged 2026-08-29 18:46 UTC. Sized the Node heap at 75% of detected memory (24081 MB of a 32109 MB machine). Aggravated by Vue 3 rollout bundle growth: `616227-vue3-migrate-clusters` (09-11 15:04 UTC), `616240-vue3-migrate-cluster-agents` (15:58 UTC), `vue3-migrate-broadcast-messages` (16:26 UTC). | | Fix MR | None yet. | | Why green in MRs | Most MR pipelines never really compile — they hit the assets-hash cache and finish in 129–196 s instead of 2000–3400 s. !252575's own last full compile ran 09-11 02:59, before the Vue 3 cluster rollouts landed, and !236153's head pipeline was a merge-train run with only 2 jobs. The failure is also probabilistic, so the same code can pass in an MR and be killed on `master`. | | Corrective actions | 1. Size the heap for the whole job, not just the main process — subtract a terser-worker budget (≈12–16 GB heap on a 32 GB box) or cap `AssetsHeapSizing` absolutely. 2. Bound terser explicitly (`parallel: 4`) instead of one worker per host CPU. 3. Pass `--max-old-space-size` only to the webpack invocation so worker children don't inherit a 24 GB allowance. 4. Set a real memory limit on the runner container so cgroup detection stops falling back to whole-VM `MemTotal`. 5. Address the 34–58 min compile times / bundle growth separately. | <!-- RESOLUTION_END --> ### Root cause `compile-production-assets` is SIGKILLed by the kernel because the Node heap is sized at 75% of the machine's RAM, leaving no room for the terser workers that run *outside* that heap limit. **The change.** [!252575](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252575) replaced a dead runner-tag check in `lib/tasks/gitlab/assets.rake` with `scripts/lib/assets_heap_sizing.rb`: | | Heap in CI | |---|---| | Before | `--max-old-space-size=8192` (the 16 GB branch tested `GITLAB_LARGE_RUNNER_OPTIONAL == "saas-linux-large-amd64"`, which no longer matches — the job runs on `gitlab-org-docker-linux-large-amd64`) | | After | 75% of detected memory | The executor is `docker+machine` with no container memory limit, so the cgroup read returns "unlimited" and it falls back to `/proc/meminfo` — the whole VM. Every failing trace prints: ``` Node heap limit: 24081 MB (detected memory: 32109 MB) ``` **Why 24 GB of 32 GB breaks.** `config/webpack.config.js:535` states it in its own comment: ```js // Terser spawns one worker per host CPU, outside the node heap limit minimize: IS_PRODUCTION && !NO_MINIFY, ``` Real consumers are the main webpack process (now permitted 24 GB) + one out-of-heap terser worker-farm child per CPU (each also inheriting `NODE_OPTIONS`, which is set for the whole `yarn build`) + production `devtool: 'source-map'` + the `docker:28.5.2-dind` service in the same container. Nothing is left, so the kernel kills the container — hence `Killed`, `exit code 137`, and the cascading `worker-farm` EPIPE stacks. **The MR traded failure modes rather than reducing memory use.** Before it (job `16164960051`, 08-28): `FATAL ERROR: Ineffective mark-compacts near heap limit`, exit **134**. After it: exit **137**. ### Onset `ci_metrics.failure_analysis_metrics`, `failure_category = 'oom_killed'`, `job_name = 'compile-production-assets'`: | Date | Count | |---|---| | 2026-09-09 | 1 (`rails-next`) | | 2026-09-10 | 6 | | 2026-09-11 | 29 | The 10-day gap after the merge closed as the Vue 3 rollout grew the bundle. Between the 09-11 11:35 pass and the 20:16 OOM the infection scanner went from `Promoted 154` → `171` pages and `Duplicated modules check: 387` → `419 page state(s)` — 17 new `vue3_migration.yml` files, each adding a duplicate Vue-3 entrypoint. This is **threshold behaviour, not a bad commit**: successful real compiles on 09-11 took 2032–3454 s (34–58 min) and pass/fail interleave on adjacent commits. ### Ruled out - **!236153** (`fix/webpack-overlay-vscode-languageserver-warnings`, merge `9659a2b5476`) is the first row in the activity table above and looks causal, but OOM kills already occurred at 16:33 (`6014be928a9`) and 17:20 (`4d7f0465256`). Its real effect: touching `config/webpack.config.js` invalidated the assets-hash cache, so every later pipeline ran a **real** compile instead of a ~150 s cache hit, converting intermittent OOMs into a continuous wall from 18:31. - The 18:14 failure (job `16454196700`, 382 s) was Gitaly `resource exhausted` on git fetch — unrelated. ### Ownership and routing There is no single owner, which is plausibly why the 08-28 instance was never actioned. | Path | CODEOWNERS | |---|---| | `.gitlab/ci/frontend.gitlab-ci.yml` (job definition) | `@gl-dx/pipeline-maintainers` + `@gitlab-org/maintainers/frontend`, section `^[Pipeline configuration]` | | `config/webpack.config.js` | `@gitlab-org/maintainers/frontend` (via `*.js`) | | `lib/tasks/gitlab/assets.rake` | **no specific entry** — catch-all `/*` list only | | `scripts/lib/assets_heap_sizing.rb` | **no specific entry** — catch-all `/scripts/` list only | The two files that contain the defect have no owning team in CODEOWNERS. These are also maintainer *approval* pools rather than accountable teams. No job→owner mapping exists in `.gitlab/ci/` or `tooling/`. The incident process did not attribute it either: [master-broken-incidents#28788](https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues/28788) (the instance !252575 was fixing) is `master-broken::undetermined`, unassigned, `missed::SLA`, and still open. Recent authorship of `lib/tasks/gitlab/assets.rake` is one commit each from five people, so there is no de facto owner either. People with the most relevant context: `@stanhu` (authored !252575), Stanislav Lashmanov (Rspack/assets CI; the only recent author of both `webpack.config.js` and `assets.rake`), and `@mrincon` (Vue 3 rollout driving the bundle growth). Suggested routing — **Developer Experience** as owner with **Frontend** as collaborator: corrective actions 1–3 are frontend build-tooling changes, action 4 is a runner/infrastructure change. This is inferred from path ownership, not a documented routing rule. ### Current status — updated 2026-09-14 Still unresolved; no commit has touched `assets_heap_sizing.rb` or `assets.rake` since the incident opened. **It has recurred on `master` today.** `oom_killed` counts for `compile-production-assets`: | Date | Count | Refs | |---|---|---| | 2026-09-09 | 1 | `rails-next` | | 2026-09-10 | 6 | `rails-next`, `master` | | 2026-09-11 | 30 | `rails-next`, `master`, `ruby-next` | | 2026-09-12 | 1 | `rails-next` | | 2026-09-13 | 1 | `ruby-next` | | 2026-09-14 | 12 | `master` (01:11–09:43 UTC) | The 09-12/09-13 lull was the weekend — only scheduled refs, and no frontend change invalidating the assets cache. It resumed as soon as `master` traffic did, which is consistent with threshold behaviour rather than anything having been fixed. **Forward risk:** `4e94d27d23ac` "Extend vue3_migration.yml to global bundles" (`@mrincon`) merged today at 12:27 UTC. By the mechanism above, extending the Vue 3 migration to global bundles adds further duplicate entrypoints and should be expected to make this worse. To be explicit, it is **not** the cause of today's burst — every 09-14 failure listed above predates it. --- ## At First Detection | Metric | Value | |---|---| | Projects | gitlab-org/gitlab | | Failure category | oom_killed | | Blocked pipelines | 21 (19 master, 2 MR) | | Impacted MRs | 1 | --- ## Recent Activity <!-- RECENT_ACTIVITY_START --> | Failed at (UTC) | Job | Pipeline | Ref | Signature | Error | |---|---|---|---|---|---| | 2026-09-11 20:44 UTC | [compile-production-assets](https://gitlab.com/gitlab-org/gitlab/-/jobs/16456028952) | [pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2841962972) | master | `oom_killed__b5ffc29704853f17` | Command failed with exit code 137 | | 2026-09-11 20:28 UTC | [compile-production-assets](https://gitlab.com/gitlab-org/gitlab/-/jobs/16455939065) | [pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2841949382) | master | `oom_killed__b5ffc29704853f17` | Command failed with exit code 137 | | 2026-09-11 19:59 UTC | [compile-production-assets](https://gitlab.com/gitlab-org/gitlab/-/jobs/16455495234) | [pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2841893291) | master | `oom_killed__b5ffc29704853f17` | Command failed with exit code 137 | | 2026-09-11 19:39 UTC | [compile-production-assets](https://gitlab.com/gitlab-org/gitlab/-/jobs/16455165301) | [pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2841849761) | ruby-next | `oom_killed__b5ffc29704853f17` | Command failed with exit code 137 | | 2026-09-11 19:38 UTC | [compile-production-assets](https://gitlab.com/gitlab-org/gitlab/-/jobs/16454471981) | [pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2841757539) | master | `oom_killed__b5ffc29704853f17` | Command failed with exit code 137 | <!-- RECENT_ACTIVITY_END --> ### Failure Signatures **[`oom_killed__b5ffc29704853f17`](https://dashboards.gitlab.net/d/dx-failure-analysis/dx3a-failure-analysis-dashboard?from=now-12h&to=now&var-failure_category_var=oom_killed&var-failure_signature_var=oom_killed__b5ffc29704853f17) — 21 pipelines** ``` Command failed with exit code <ID> ``` <!-- failure_signature: oom_killed --> <!-- failure_signature: oom_killed__b5ffc29704853f17 --> <!-- last_seen_at: 2026-09-11T20:44:00Z --> <!-- slack_ts: 1789159666.742819 --> <!-- slack_channel_id: C0AS5PF9WCU -->