Skip to content

Capture helper service logs into job/tasks main trace log [kubernetes]

Axel von Bertoldi requested to merge avonbertoldi/2119/capture-service-logs-k8s into avonbertoldi/2119/capture-service-logs

Status

What does this MR do?

This MR captures and streams logs from helper service containers to the CI task/job's main trace logs for the kubernetes executor. This functionality is enabled when the variable CI_DEBUG_SERVICES = trace. Currently no other values for this variable are supported, but more could be added in the future. While the logs are currently written to the jobs trace logs, they could easily be written elsewhere (file, log aggregator service, syslog...). This approach relies on the kubernetes.Clientset().CoreV1().Pods().GetLogs() API. Despite the knows problems with that API, we're using it here as a first iteration, which may turn out to be good enough (see discussion #2119 (comment 1072311180)).

This is the kubernetes counterpart of !3551 (closed), and in fact builds upon that MR, so start reviewing at commit Move service log capturing code and tests from docker to common.

Why was this MR needed?

!2119 (merged) asks for service logs to be captured somehow. This is to help debug failing jobs when the failure is (at least in part) caused by behaviour in one of the service container services' (though not necessarily a failure to start said service container). A few possible approaches are mentioned in the issue; this MR takes the most "iteration friendly" (i.e. simplest) approach of copying the service logs inline into the main trace logs, but leaves room for the logs to be written elsewhere (e.g. a file) in the future.

What's the best way to test this MR?

  1. BASELINE: Don't specify CI_DEBUG_SERVICES and run a CI job with a service container. The output of the main log trace should be unchanged from main.

  2. Set CI_DEBUG_SERVICES to a bogus value. The error message invalid value '<xxx>' for CI_DEBUG_SERVICES variable should appear in the main trace logs.

Example https://gitlab.com/avonbertoldi/test-project/-/jobs/2890895755

  1. Set CI_DEBUG_SERVICES = trace in the CI configuration
    1. register a runner with a docker executor
    2. create a job that includes a service which writes logs (example below)
    3. run the job (using a runner built from this branch)
    4. the service container's logs should appear in the job's main log trace in grey colour, with the container name prefixed to the log lines.

gitlab-ci.yaml

stages:
  - test

variables:
  POSTGRES_PASSWORD: password
  CI_DEBUG_SERVICES: trace

format:
  stage: test
  image:
    name: golang:latest
  services:
    - postgres:latest
    - redis:latest
  script:
    - go fmt $(go list ./... | grep -v /vendor/)
    - go vet $(go list ./... | grep -v /vendor/)
    - go test -race $(go list ./... | grep -v /vendor/)

Example https://gitlab.com/avonbertoldi/test-project/-/jobs/2850804762

What are the relevant issue numbers?

Notes:

  • Best reviewed commit-at-a-time.
  • Integration tests are still in the works and will be added to THIS MR when complete.
  • See !3551 (closed) for the docker executor version of this MR.
Edited by Axel von Bertoldi

Merge request reports