Skip to content

Job failed" errors are logged in the main gitlab-runner pod instead of the job-specific pod

Problem Description

When running GitLab CI/CD pipelines in Kubernetes, errors like the following appear:

WARNING: Job failed: command terminated with exit code 1  
WARNING: Job failed: command terminated with exit code 3  

These messages are logged in the main GitLab Runner pod (gitlab-runner-7bb977c778-fs2d8) instead of the temporary job-specific pod (e.g., runner-sijwob5yi-project-69309276-concurrent-0-15yuo30m). This makes debugging harder since the actual job execution logs should logically appear in the job’s own pod.

Environment

  • GitLab Runner:
    • Version: 17.11.0
    • Installed via Helm chart: gitlab-runner-0.76.0
  • Kubernetes:
    • Server: v1.30.1
    • Client: v1.30.0

Steps to Reproduce

  1. Run a pipeline with a job that fails (e.g., due to a script error).
  2. Check logs using stern:
    stern -n gitlab-runner -l app=gitlab-runner . | grep "Job failed: command terminated with exit code"
  3. Observe that errors are logged in the main gitlab-runner-* pod rather than the ephemeral job pod.

Expected Behavior

Job failure messages (e.g., non-zero exit codes) should appear in the logs of the temporary job pod where the script executes (e.g., runner-*-concurrent-0-*).

Current Behavior

Errors are logged in the main GitLab Runner pod, forcing users to manually trace which job failed.