Duo workflow workload jobs fail in prepare step with dumb-init: argument list too long — no aggregate cap on job environment size

Summary

On 2026-09-21 (Monday), 24 of 631 DUO_WORKFLOW-sourced workload jobs in gitlab-org/gitlab failed in the prepare_script step with:

exec /usr/bin/dumb-init: argument list too long
ERROR: Job failed: prepare environment: exit code 1

The job dies before get_sources — the build container never starts. These are deterministic, payload-driven failures: each was a distinct one-shot workload pipeline, never retried, spread across both Linux runner managers and across 09:26–21:09 CEST.

Affected flows (confirmed)

All 24 failing workloads were matched to their session via the project sessions API (project.duoWorkflowWorkflows), correlated by createdAt (Δ –2s to +41s):

Flow Failing workloads Monday total for flow Failure rate
recommend_reviewers/v1 15 186 (WEB) 8.1%
code_review/v1 9 313 (281 AMBIENT / 32 WEB) 2.9%

Both are MR-centric flows. The AMBIENT code_review failures are notable — ambient/@duo mention triggers are exactly the path that injects the unbounded triggering_conversation (full comment thread) into DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT.

Monday flow volume for context: code_review/v1 313, recommend_reviewers/v1 186, ai_catalog_agent 69, developer/v1 58, fix_pipeline/v1 3, sast_fp_detection/v1 2.

Root cause

execve() enforces ARG_MAX = 2 MiB total (argv + envp, x86_64). The runner passes the whole serialized job environment to dumb-init; when it exceeds ~2 MiB the exec fails.

Each workload job's environment is assembled in Ai::DuoWorkflows::StartWorkflowService (git_clone_variables + base_variables + token_variables + git_environment_variables + sandbox.environment_variables). Most variables are small/fixed (~3–6 KiB floor). The large, variable-sized ones:

Variable Cap today Content
DUO_WORKFLOW_GOAL ✅ 127 KiB (per-var, GOAL_MAX_BYTESIZE, work_item 616260) user goal text
DUO_WORKFLOW_METADATA ✅ 1 KiB model metadata
DUO_WORKFLOW_FLOW_METADATA ✅ 1 KiB
DUO_WORKFLOW_FLOW_CONFIG ❌ none serialized flow config
DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT ❌ none standard + trigger + resource context
DUO_WORKFLOW_AGENTS ❌ none workspace agents JSON
DUO_WORKFLOW_SERVICE_TOKEN (~JWT, KBs)

The gap: GOAL_MAX_BYTESIZE guards only the per-variable 128 KiB MAX_ARG_STRLEN, not the aggregate 2 MiB ARG_MAX. DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT is serialized via Gitlab::Json.dump from BuildAdditionalContextService and includes unbounded contributors:

  • triggering_conversation — for @duo mention triggers this is the full comment thread from params[:input], passed straight through with no truncation (FlowTriggers::RunService#mention_triggering_conversation → TriggerContextBuilder).
  • user-supplied additional_context — forwarded verbatim (only reserved categories are stripped; no size limit).
  • resolve_additional_context_for resolvers — currently small, but the interface permits arbitrary content.

A mention trigger on a long MR discussion thread, or a large flow config, can push the aggregate env past 2 MiB even though each individual string stays under its own cap.

Evidence

  • All 24 failing jobs stopped at prepare_script; zero reached step_script, so the $DUO_WORKFLOW_DEFINITION echo in workflow_echo_commands never ran and the trace leaks nothing. (Contrast: the 2026-09-18 incident was a different, Gitaly "resource exhausted" failure in get_sources; that has been clean since Fri 21:39 CEST.)
  • Affected jobs all carry tags: ['gitlab--duo'] (Ai::DuoWorkflows::Workflow::WORKLOAD_TAG).
  • Job env byte sizes are not retrievable via REST or GraphQL (CiJob.inputs is [] for these jobs; there is no job variables field), so the specific oversized variable can't be confirmed from the API — but the implicated flows are now known, and both inject large MR-thread/MR-content context.

Failing pipeline IDs (24): 2867067675, 2867289953, 2867293272, 2867484450, 2867506932, 2867570434, 2867604429, 2867644744, 2867663307, 2867693475, 2867698384, 2867719880, 2867852883, 2867927841, 2867968099, 2868181070, 2868363951, 2868660816, 2868665333, 2868672345, 2868692703, 2868843523, 2869166340, 2869180656

Matched workflow (session) IDs, in failing-job time order: 8502553, 8502326, 8496425, 8493617, 8493153, 8492984, 8492904, 8486907, 8484824, 8481073, 8480313, 8478816, 8476522, 8476017, 8475921, 8475260, 8474861, 8474121, 8473498, 8472197, 8471708, 8467280, 8467239, 8461647

To confirm the specific oversized variable per failing workload, a Maintainer can inspect ci_job_variables for the workload's build, or duo_workflows_workflows.goal / the additional-context payload for the matched workflow IDs above.

Suggested fix direction

  1. Guard the aggregate, not just per-var: in Ci::Workloads::RunWorkloadService / Ci::Workloads::WorkloadDefinition#to_job_hash, sum the byte sizes of all job variables and reject (or offload) with a clear error when total approaches ARG_MAX — same spirit as work_item 616260, but for the whole environment.
  2. Move big payloads out of env vars: DUO_WORKFLOW_FLOW_CONFIG, DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT, and triggering_conversation are candidates to pass via a file/artifact or a runtime-fetched reference (session URL + token) instead of the environment.
  3. Truncate triggering_conversation to a bounded number of recent notes before serializing.
  4. Repro aid: give recommend_reviewers/v1 / code_review/v1 an oversized MR discussion thread and confirm the workload fails in prepare_script.

Analysis performed with the Duo agent: enumerated all 631 Monday DUO_WORKFLOW workload jobs via GraphQL, classified all 42 failures by trace, matched the 24 prepare_script failures to their sessions via project.duoWorkflowWorkflows, and traced the variable-assembly code path in StartWorkflowService / BuildAdditionalContextService.

Edited by 🤖 GitLab Bot 🤖