Skip to content

Service logs are not captured if the service container fails too quickly

Summary

When a service fails before all containers are started, it's logs are not collected even if CI_DEBUG_SERVICES is true. This happens with the k8s executor.

In a setup where runners are managed by a centralized devops team and the teams using the runners only have access to the pipeline UI and not the actual logs in k8s, this can make debugging services difficult.

Steps to reproduce

Have services for a job and one those fails, e.g. for a port being already in use since multiple services cannot run on the same port. Below is a simple example with two services with one always failing and the other successfully having it's logs collected.

.gitlab-ci.yml
variables:
  CI_DEBUG_SERVICES: "true"

example-job:
  image: alpine:3
  services:
    - name: alpine
      alias: failing-service
      command: ["sh", "-c", "echo 'Service will fail'; exit 1"]
    - name: debian
      alias: running-servive
      command: ["sh", "-c", "echo 'Service will not fail'; sleep 2; echo 'Success!'"]
  script:
    - echo "Job running"
    - sleep 10

Actual behavior

Service container logs are not visible in the pipeline UI.

Expected behavior

Service container logs should always be visible in the pipeline UI when CI_DEBUG_SERVICES is true.

Relevant logs and/or screenshots

job log
Subscribing to Kubernetes Pod events...
Type     Reason      Message
Normal   Scheduled   Successfully assigned xxxxxxxx to xxxxxxxxx
Normal   Pulled   Container image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v18.2.1" already present on machine
Normal   Created   Created container: init-permissions
Normal   Started   Started container init-permissions
Normal   Pulling   Pulling image "alpine:3"
Normal   Pulled   Successfully pulled image "alpine:3" in 472ms (472ms including waiting). Image size: 3810510 bytes.
Normal   Created   Created container: build
Normal   Started   Started container build
Normal   Pulled   Container image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v18.2.1" already present on machine
Normal   Created   Created container: helper
Normal   Started   Started container helper
Normal   Pulling   Pulling image "alpine"
Normal   Pulled   Successfully pulled image "alpine" in 419ms (419ms including waiting). Image size: 3810510 bytes.
Normal   Created   Created container: failing-service
Normal   Started   Started container failing-service
Normal   Pulling   Pulling image "debian"
Normal   Pulled   Successfully pulled image "debian" in 3.274s (3.274s including waiting). Image size: 49289936 bytes.
Normal   Created   Created container: running-servive
Normal   Started   Started container running-servive
WARNING: failed to open log stream for container failing-service: container "failing-service" terminated with non-zero status: 1
[service:debian-running-servive] 2025-09-24T13:54:18.362290512Z Service will not fail
[service:debian-running-servive] 2025-09-24T13:54:20.366291643Z Success!
...
Executing "step_script" stage of the job script 00:10
$ echo "Job running"
Job running
$ sleep 10
Cleaning up project directory and file based variables 00:01
Job succeeded

Environment description

Self-managed runner with the k8s executor. Self-managed Gitlab instance.

Used GitLab Runner version

18.2.1

Possible fixes

Collecting the container logs here if that is possible.