You need to sign in or sign up before continuing.
Job failed" errors are logged in the main gitlab-runner pod instead of the job-specific pod
Problem Description
When running GitLab CI/CD pipelines in Kubernetes, errors like the following appear:
WARNING: Job failed: command terminated with exit code 1
WARNING: Job failed: command terminated with exit code 3
These messages are logged in the main GitLab Runner pod (gitlab-runner-7bb977c778-fs2d8
) instead of the temporary job-specific pod (e.g., runner-sijwob5yi-project-69309276-concurrent-0-15yuo30m
). This makes debugging harder since the actual job execution logs should logically appear in the job’s own pod.
Environment
-
GitLab Runner:
- Version:
17.11.0
- Installed via Helm chart:
gitlab-runner-0.76.0
- Version:
-
Kubernetes:
- Server:
v1.30.1
- Client:
v1.30.0
- Server:
Steps to Reproduce
- Run a pipeline with a job that fails (e.g., due to a script error).
- Check logs using
stern
:stern -n gitlab-runner -l app=gitlab-runner . | grep "Job failed: command terminated with exit code"
- Observe that errors are logged in the main
gitlab-runner-*
pod rather than the ephemeral job pod.
Expected Behavior
Job failure messages (e.g., non-zero exit codes) should appear in the logs of the temporary job pod where the script executes (e.g., runner-*-concurrent-0-*
).
Current Behavior
Errors are logged in the main GitLab Runner pod, forcing users to manually trace which job failed.