### Problem to solve When the Kubernetes executor's helper container is killed by the OOM (Out of Memory) killer, the pipeline fails with the following cryptic error in the job log: ``` ERROR: Job failed (system failure): unable to upgrade connection: container not found ("helper") ``` This message gives the user no indication that the root cause is a memory limit being exceeded. Customers must independently diagnose the issue — often through `kubectl describe pod` or cluster-level tooling — before discovering that the helper container was OOM-killed and finding the [helper container memory sizing recommendations](https://docs.gitlab.com/runner/executors/kubernetes/#helper-container-memory-sizing-recommendations) in the documentation. This is a poor experience, particularly for users who do not have direct access to cluster-level observability. ### Proposal When the runner detects that the helper container is no longer present (the `container not found ("helper")` condition), it should query the Kubernetes API for the pod's container statuses and check whether the helper container's last termination reason was `OOMKilled`. If so, the error message surfaced in the job log should be enriched to clearly communicate the OOM cause and link directly to the relevant documentation. **Example of improved error output:** ``` ERROR: Job failed (system failure): helper container was OOM-killed (memory limit exceeded). Consider increasing `helper_memory_limit` or setting the KUBERNETES_HELPER_MEMORY_LIMIT CI/CD variable. See: https://docs.gitlab.com/runner/executors/kubernetes/#helper-container-memory-sizing-recommendations ``` This follows the pattern established by https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/31052 (merged in !4211), which added Kubernetes warning event surfacing for image pull failures. ### Current behaviour ``` ERROR: Job failed (system failure): unable to upgrade connection: container not found ("helper") ``` No indication of the root cause. The user must independently investigate cluster events. ### Expected behaviour When the helper container was OOM-killed, the error message clearly states: - That the container was terminated due to memory exhaustion - What configuration knobs are available to fix it (`helper_memory_limit`, `KUBERNETES_HELPER_MEMORY_LIMIT`) - A direct link to the documentation ### Implementation notes - The relevant code path is in the Kubernetes executor, likely in the connection upgrade / attach logic under `executors/kubernetes/` - The fix should check `pod.Status.ContainerStatuses` for the `helper` container and inspect `LastTerminationState.Terminated.Reason == "OOMKilled"` before constructing the error string - This should be a targeted change (1–2 files, the executor and possibly a test file) consistent with the scope guidelines - No documentation changes are needed — the existing docs page is the target of the new link ### Who benefits - **Developers** running jobs on Kubernetes who hit memory limits and see an opaque failure - **Platform/infra operators** who need to diagnose runner failures quickly - **Support teams** who currently have to walk customers through cluster-level diagnosis ### Related - https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/29026 — original OOM memory usage report (closed) - https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/31052 — precedent: K8s pod failure event surfacing (closed/merged) - https://gitlab.com/groups/gitlab-org/-/work_items/10891 — "Job failures should be distinguishable between infrastructure and build problems" (open) - [Helper container memory sizing docs](https://docs.gitlab.com/runner/executors/kubernetes/#helper-container-memory-sizing-recommendations) ### Nice to have Allow customers to supply a **custom error message** that is appended to (or replaces) the runner-generated OOM error output. This is useful in environments where security policies mandate specific variable naming conventions that differ from the defaults suggested in the error message (e.g. a policy that requires `RUNNER_K8S_HELPER_MEM_LIMIT` instead of `KUBERNETES_HELPER_MEMORY_LIMIT`). **Proposed mechanism:** a new optional CI/CD variable (e.g. `KUBERNETES_HELPER_OOM_MESSAGE`) whose value, when set, is appended to or replaces the standard OOM error text in the job log. For example: ``` ERROR: Job failed (system failure): helper container was OOM-killed (memory limit exceeded). Consider increasing `helper_memory_limit` or setting the RUNNER_K8S_HELPER_MEM_LIMIT CI/CD variable. See: https://docs.gitlab.com/runner/executors/kubernetes/#helper-container-memory-sizing-recommendations ``` **Behaviour options (to be decided during implementation):** - **Append mode (default):** the custom message is printed after the standard runner message. - **Replace mode:** if the variable is set to a non-empty value, it fully replaces the runner-generated text, giving operators complete control over the wording. **Why this matters:** organisations with strict security policies often standardise on internal variable names or internal documentation links. Hardcoding the runner's own variable names in the error output can conflict with those policies or cause confusion when the customer-facing variable name differs from the runner config key.