## Summary On **2026-09-21 (Monday)**, 24 of 631 `DUO_WORKFLOW`-sourced `workload` jobs in `gitlab-org/gitlab` failed in the `prepare_script` step with: ``` exec /usr/bin/dumb-init: argument list too long ERROR: Job failed: prepare environment: exit code 1 ``` The job dies **before `get_sources`** — the build container never starts. These are deterministic, payload-driven failures: each was a distinct one-shot workload pipeline, never retried, spread across both Linux runner managers and across 09:26–21:09 CEST. ## Affected flows (confirmed) All 24 failing workloads were matched to their session via the project sessions API (`project.duoWorkflowWorkflows`), correlated by `createdAt` (Δ –2s to +41s): | Flow | Failing workloads | Monday total for flow | Failure rate | |---|---|---|---| | `recommend_reviewers/v1` | **15** | 186 (WEB) | **8.1%** | | `code_review/v1` | **9** | 313 (281 AMBIENT / 32 WEB) | **2.9%** | Both are MR-centric flows. The AMBIENT `code_review` failures are notable — ambient/`@duo` mention triggers are exactly the path that injects the unbounded `triggering_conversation` (full comment thread) into `DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT`. Monday flow volume for context: `code_review/v1` 313, `recommend_reviewers/v1` 186, `ai_catalog_agent` 69, `developer/v1` 58, `fix_pipeline/v1` 3, `sast_fp_detection/v1` 2. ## Root cause `execve()` enforces **`ARG_MAX` = 2 MiB total** (argv + envp, x86_64). The runner passes the whole serialized job environment to `dumb-init`; when it exceeds ~2 MiB the exec fails. Each workload job's environment is assembled in `Ai::DuoWorkflows::StartWorkflowService` (`git_clone_variables + base_variables + token_variables + git_environment_variables + sandbox.environment_variables`). Most variables are small/fixed (~3–6 KiB floor). The large, variable-sized ones: | Variable | Cap today | Content | |---|---|---| | `DUO_WORKFLOW_GOAL` | ✅ 127 KiB (per-var, `GOAL_MAX_BYTESIZE`, [work_item 616260](https://gitlab.com/gitlab-org/gitlab/-/work_items/616260)) | user goal text | | `DUO_WORKFLOW_METADATA` | ✅ 1 KiB | model metadata | | `DUO_WORKFLOW_FLOW_METADATA` | ✅ 1 KiB | | | `DUO_WORKFLOW_FLOW_CONFIG` | ❌ **none** | serialized flow config | | `DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT` | ❌ **none** | standard + trigger + resource context | | `DUO_WORKFLOW_AGENTS` | ❌ none | workspace agents JSON | | `DUO_WORKFLOW_SERVICE_TOKEN` | (~JWT, KBs) | | The gap: **`GOAL_MAX_BYTESIZE` guards only the per-variable 128 KiB `MAX_ARG_STRLEN`, not the aggregate 2 MiB `ARG_MAX`.** `DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT` is serialized via `Gitlab::Json.dump` from `BuildAdditionalContextService` and includes unbounded contributors: - **`triggering_conversation`** — for `@duo` mention triggers this is the full comment thread from `params[:input]`, passed straight through with no truncation (`FlowTriggers::RunService#mention_triggering_conversation` → `TriggerContextBuilder`). - **user-supplied `additional_context`** — forwarded verbatim (only reserved *categories* are stripped; no size limit). - **`resolve_additional_context_for`** resolvers — currently small, but the interface permits arbitrary content. A mention trigger on a long MR discussion thread, or a large flow config, can push the **aggregate** env past 2 MiB even though each individual string stays under its own cap. ## Evidence - All 24 failing jobs stopped at `prepare_script`; zero reached `step_script`, so the `$DUO_WORKFLOW_DEFINITION` echo in `workflow_echo_commands` never ran and the trace leaks nothing. (Contrast: the **2026-09-18** incident was a different, Gitaly "resource exhausted" failure in `get_sources`; that has been clean since Fri 21:39 CEST.) - Affected jobs all carry `tags: ['gitlab--duo']` (`Ai::DuoWorkflows::Workflow::WORKLOAD_TAG`). - Job env byte sizes are **not retrievable via REST or GraphQL** (`CiJob.inputs` is `[]` for these jobs; there is no job `variables` field), so the specific oversized variable can't be confirmed from the API — but the implicated flows are now known, and both inject large MR-thread/MR-content context. Failing pipeline IDs (24): `2867067675, 2867289953, 2867293272, 2867484450, 2867506932, 2867570434, 2867604429, 2867644744, 2867663307, 2867693475, 2867698384, 2867719880, 2867852883, 2867927841, 2867968099, 2868181070, 2868363951, 2868660816, 2868665333, 2868672345, 2868692703, 2868843523, 2869166340, 2869180656` Matched workflow (session) IDs, in failing-job time order: `8502553, 8502326, 8496425, 8493617, 8493153, 8492984, 8492904, 8486907, 8484824, 8481073, 8480313, 8478816, 8476522, 8476017, 8475921, 8475260, 8474861, 8474121, 8473498, 8472197, 8471708, 8467280, 8467239, 8461647` To confirm the specific oversized variable per failing workload, a Maintainer can inspect `ci_job_variables` for the workload's build, or `duo_workflows_workflows.goal` / the additional-context payload for the matched workflow IDs above. ## Suggested fix direction 1. **Guard the aggregate, not just per-var**: in `Ci::Workloads::RunWorkloadService` / `Ci::Workloads::WorkloadDefinition#to_job_hash`, sum the byte sizes of all job variables and reject (or offload) with a clear error when total approaches `ARG_MAX` — same spirit as work_item 616260, but for the whole environment. 2. **Move big payloads out of env vars**: `DUO_WORKFLOW_FLOW_CONFIG`, `DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT`, and `triggering_conversation` are candidates to pass via a file/artifact or a runtime-fetched reference (session URL + token) instead of the environment. 3. **Truncate `triggering_conversation`** to a bounded number of recent notes before serializing. 4. **Repro aid**: give `recommend_reviewers/v1` / `code_review/v1` an oversized MR discussion thread and confirm the workload fails in `prepare_script`. --- *Analysis performed with the Duo agent: enumerated all 631 Monday `DUO_WORKFLOW` `workload` jobs via GraphQL, classified all 42 failures by trace, matched the 24 `prepare_script` failures to their sessions via `project.duoWorkflowWorkflows`, and traced the variable-assembly code path in `StartWorkflowService` / `BuildAdditionalContextService`.*